steveahnahn opened a new pull request, #70923: URL: https://github.com/apache/airflow/pull/70923
`airflow db clean` has not purged the `callback` table since Airflow 3.2.0. Migration `b87d2135fa50` renamed `callback_request` to `callback`, but the cleanup configuration in `db_cleanup.py` kept the old name. A configured table that does not exist is skipped with only a warning, so the entry became a no-op and nothing purges `callback` any more. The table does accumulate in practice. Executor callbacks are set to `success`/`failed` by the scheduler and never deleted (only Dag-processor callbacks are removed, as they are dispatched). And because `deadline.callback_id` references `callback.id` with `ON DELETE CASCADE`, `callback` is the parent, so purging `deadline` leaves its callback rows orphaned permanently. There was also no workaround available: `--tables callback` was rejected as an invalid choice, so operators could not purge the table through `airflow db clean` at all. ### What this changes * Register the table under its current name, with `created_at` as the recency column. * Purge only callbacks that can no longer run. A callback still awaiting execution owns its `deadline` row through the `ON DELETE CASCADE` foreign key, so deleting one would silently drop a deadline that has not fired yet. Dag-processor callbacks carry no state and reference no deadline, so rows outliving the retention window are purged too. The existing `TERMINAL_STATES` constant is reused rather than re-hardcoding the state literals, since that kind of drift is what caused this bug. * Add `dependent_tables=["deadline"]` so deadline rows are cleaned and archived ahead of their parent, matching how `dag` and `dag_run` already handle the same cascade. * Drop the `sla_miss` entry, whose table no longer exists in Airflow 3. ### Verification Verified end to end against a real Postgres metadata database, seeding rows and running the real `run_cleanup`, with a `log` row as a control to prove the run did actual work: | Case | Result | | --- | --- | | old + `success` / `failed` | purged | | old + NULL state (Dag-processor) | purged | | old + `scheduled` / `queued` / `running` | kept | | recent + `success` | kept | | unfired deadline owned by a live callback | survives | | control `log` row | purged | The full `test_db_cleanup.py` suite passes on Postgres, MySQL and SQLite. All ten new tests fail without the source change. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
