From a0ea514db7b22d4057cc1c42592ee71714fe6ffd Mon Sep 17 00:00:00 2001 From: GuyEshdat Date: Tue, 9 Jun 2026 11:09:24 +0300 Subject: [PATCH] docs: add test table cleanup section to reduce-on-run-end-time guide Documents the new clean_elementary_temp_tables flag and cleanup_stale_test_tables macro introduced in dbt-data-reliability#1022, explaining the hours and limit params. Also fixes pre-existing broken relative links in cloud-pipeline-task-performance.mdx. Co-Authored-By: Claude Sonnet 4.6 (1M context) --- .../cloud-pipeline-task-performance.mdx | 4 +- .../guides/reduce-on-run-end-time.mdx | 38 +++++++++++++++++++ 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/docs/cloud/features/anomaly-detection/cloud-pipeline-task-performance.mdx b/docs/cloud/features/anomaly-detection/cloud-pipeline-task-performance.mdx index 3b3cf3e33..99ed47f45 100644 --- a/docs/cloud/features/anomaly-detection/cloud-pipeline-task-performance.mdx +++ b/docs/cloud/features/anomaly-detection/cloud-pipeline-task-performance.mdx @@ -54,5 +54,5 @@ By default, alerts are not active. To enable them, go to **Setup → Alert Rules ## Related -- [Performance Alerts](/features/performance-monitoring/performance-alerts) — setup guide and API reference for creating `pipeline_task_performance` monitors -- [Automated monitors overview](/features/anomaly-detection/automated-monitors) — how automated monitors work across freshness, volume, and performance +- [Performance Alerts](/cloud/features/performance-monitoring/performance-alerts) — setup guide and API reference for creating `pipeline_task_performance` monitors +- [Automated monitors overview](/cloud/features/anomaly-detection/automated-monitors) — how automated monitors work across freshness, volume, and performance diff --git a/docs/snippets/guides/reduce-on-run-end-time.mdx b/docs/snippets/guides/reduce-on-run-end-time.mdx index bce2d0a13..73f98b7b3 100644 --- a/docs/snippets/guides/reduce-on-run-end-time.mdx +++ b/docs/snippets/guides/reduce-on-run-end-time.mdx @@ -62,6 +62,44 @@ vars: Disabling these hooks means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development). This will reduce the amount of data processed and uploaded at the end of each run, but will impact the completeness of reports and monitoring for those runs. +## Disable test table cleanup during on-run-end + +Elementary creates temporary tables during test execution and cleans them up at the end of each run. For large projects with many tests, this cleanup phase can become a significant portion of the `on_run_end` time. + +If you observe in the logs that the slow phase is the test table cleanup, you can disable it during `on_run_end` and offload it to a dedicated scheduled job instead. + +### Disable automatic cleanup + +Add the following to your `dbt_project.yml`: + +```yaml dbt_project.yml +vars: + clean_elementary_temp_tables: false +``` + +### Run cleanup in a dedicated job + +After disabling automatic cleanup, schedule a recurring dbt operation to clean up stale test tables: + +```bash +dbt run-operation elementary.cleanup_stale_test_tables +``` + +You can pass optional arguments to control the cleanup behavior: + +```bash +dbt run-operation elementary.cleanup_stale_test_tables --args '{"hours": 48, "limit": 1000}' +``` + +**Parameters:** + +- **`hours`** (default: `24`): Only delete test tables that are older than this many hours. Increase this value if you want to retain temporary tables for longer before cleanup. +- **`limit`** (default: `2000`): Maximum number of tables to delete per run. Use this to cap the cleanup operation and avoid long-running jobs when many stale tables have accumulated. + + +We recommend scheduling this job to run daily. If a large number of stale tables have built up, the `limit` parameter prevents the first cleanup from running indefinitely — you can run the job multiple times until all stale tables are removed. + + ## Limit Hooks to Production or Specific Targets If you only need full observability in production, configure Elementary to run hooks only for specific targets. This is especially useful when you want to: