Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,5 @@ By default, alerts are not active. To enable them, go to **Setup → Alert Rules

## Related

- [Performance Alerts](/features/performance-monitoring/performance-alerts) — setup guide and API reference for creating `pipeline_task_performance` monitors
- [Automated monitors overview](/features/anomaly-detection/automated-monitors) — how automated monitors work across freshness, volume, and performance
- [Performance Alerts](/cloud/features/performance-monitoring/performance-alerts) — setup guide and API reference for creating `pipeline_task_performance` monitors
- [Automated monitors overview](/cloud/features/anomaly-detection/automated-monitors) — how automated monitors work across freshness, volume, and performance
38 changes: 38 additions & 0 deletions docs/snippets/guides/reduce-on-run-end-time.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,44 @@ vars:
Disabling these hooks means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development). This will reduce the amount of data processed and uploaded at the end of each run, but will impact the completeness of reports and monitoring for those runs.
</Warning>

## Disable test table cleanup during on-run-end

Elementary creates temporary tables during test execution and cleans them up at the end of each run. For large projects with many tests, this cleanup phase can become a significant portion of the `on_run_end` time.

If you observe in the logs that the slow phase is the test table cleanup, you can disable it during `on_run_end` and offload it to a dedicated scheduled job instead.

### Disable automatic cleanup

Add the following to your `dbt_project.yml`:

```yaml dbt_project.yml
vars:
clean_elementary_temp_tables: false
```

### Run cleanup in a dedicated job

After disabling automatic cleanup, schedule a recurring dbt operation to clean up stale test tables:

```bash
dbt run-operation elementary.cleanup_stale_test_tables
```

You can pass optional arguments to control the cleanup behavior:

```bash
dbt run-operation elementary.cleanup_stale_test_tables --args '{"hours": 48, "limit": 1000}'
```

**Parameters:**

- **`hours`** (default: `24`): Only delete test tables that are older than this many hours. Increase this value if you want to retain temporary tables for longer before cleanup.
- **`limit`** (default: `2000`): Maximum number of tables to delete per run. Use this to cap the cleanup operation and avoid long-running jobs when many stale tables have accumulated.

<Info>
We recommend scheduling this job to run daily. If a large number of stale tables have built up, the `limit` parameter prevents the first cleanup from running indefinitely — you can run the job multiple times until all stale tables are removed.
</Info>

## Limit Hooks to Production or Specific Targets

If you only need full observability in production, configure Elementary to run hooks only for specific targets. This is especially useful when you want to:
Expand Down
Loading