kaxil opened a new pull request, #73173: URL: https://github.com/apache/airflow/pull/73173
Follow-up to #70923, which restored the coverage check for `db clean`. That check compares table *names* against the config, not the columns the config names, so it cannot catch this. `airflow db clean --dag-ids` or `--exclude-dag-ids` silently purges nothing from three tables. Each names a `dag_id` column it does not have: `task_reschedule.dag_id` was dropped in 3.0.0, `deadline.dag_id` in 3.1.0, and `asset_event` has only ever had `source_dag_id`. The query fails on a missing column, `run_cleanup` suppresses a failing table into a warning, and the command exits 0. An operator who always scopes by Dag has been growing those three tables forever while cleanup reported success. `deadline` and `task_reschedule` now reach their Dag the way the schema does, through `dag_run` and `task_instance`. **Why scope them rather than just drop the filter**, which would be the smaller diff: `deadline` is cleaned as a dependent of `dag_run` precisely so its rows are archived before the `ON DELETE CASCADE` removes them. An unscoped pass would purge deadlines belonging to the very Dags whose runs `--exclude-dag-ids` is preserving, turning a silent no-op into silent data loss. **`--exclude-dag-ids` now also treats a NULL dag id as eligible.** A row attributed to no Dag is not one of the excluded Dags' rows, but `NOT IN` alone yields NULL for it and retains it forever. That is every `job` row, since core never sets `Job.dag_id`, and it becomes reachable for `asset_event` and `log` too. This changes behaviour for those tables, in the direction the flag already documents. The guard test is the part that stops this recurring. It only ever called `run_cleanup` without Dag filters, and the Dag filter is the only thing that dereferences these column names, so all three configs drifted unnoticed across two releases. It is now parametrized over the filter combinations and fails on each of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
