arose26 opened a new pull request, #71732: URL: https://github.com/apache/airflow/pull/71732
--- closes: #69030 `airflow db clean` fails on `asset_event`, `deadline` and `task_reschedule` whenever `--dag-ids` or `--exclude-dag-ids` is passed, while the same command without a dag filter succeeds: ``` RuntimeError: airflow db clean encountered errors on the following tables and did not clean them: ['asset_event', 'deadline', 'task_reschedule'] ``` ### Cause All three are configured with `dag_id_column_name="dag_id"`, and none of them has a `dag_id` column: | table | actual column | |---|---| | `asset_event` | `source_dag_id` | | `task_reschedule` | `ti_id` → `task_instance.id` | | `deadline` | `dagrun_id` → `dag_run.id` | `_build_query` only reads `dag_id_column` when a dag filter is requested, which is why the failure appears solely with those two flags. I checked the remaining entries in `config_list` against the table metadata rather than assuming the report was exhaustive — `dag`, `dag_run`, `dag_version`, `job`, `log`, `task_instance`, `task_instance_history`, `task_state_store` and `xcom` all genuinely have `dag_id`. These three are the only broken ones. ### Change `asset_event` filters on `source_dag_id`. The other two have no dag id of their own, so `_TableConfig` gains `dag_id_via=(fk_column, parent_table, parent_pk_column)` and filters through the parent's `dag_id`. It is built as a correlated `EXISTS`, matching the `skip_if_referenced` guard just above it in the same function. `EXISTS` rather than `IN` / `NOT IN` because `deadline.dagrun_id` is nullable. A deadline attached to no dag run belongs to no dag, so it is kept under `--dag-ids` and removed under `--exclude-dag-ids`; with `NOT IN`, the NULL would swallow the comparison and the row would survive a filter that does not name it. Setting both `dag_id_column_name` and `dag_id_via` raises rather than silently ANDing two filters. Dry-run output prints the resolved path (`ti_id -> task_instance.dag_id`) instead of `None`, which would otherwise read as "this flag does not apply to this table". Credit where it is due: #69177 was an earlier attempt at this issue and identified the same three tables before being closed by its author. This PR reaches the filter differently — a correlated `EXISTS` reusing the existing `skip_if_referenced` idiom rather than added join metadata — and audits the rest of `config_list` for the same class of mistake. The review comment on that PR, asking for the parent-cascade path to be covered as well as the child's own filter, is answered by the parametrized test below. ### Tests - cleanup of the three tables under `--dag-ids` and under `--exclude-dag-ids`. This needs no fixture rows — the statement fails against the schema — and reproduces the reported `RuntimeError` verbatim, naming the same three tables. - rows for two dags in `task_reschedule` and `asset_event`, cleaned with `--dag-ids dag1`: only `dag2`'s rows survive. Parametrized to also run with `task_instance` and `dag_run` in the table list, so the path where the parents are cleaned first and cascade-delete the children is covered as well as the child's own filter. Archiving is on (the default), so the archive-and-delete path runs against the new subquery too. - the `_TableConfig` guard rejecting both filter styles at once. All four cleanup cases fail before this change and pass after. `test_db_cleanup.py` and `test_db_command.py` pass (177 passed, 6 skipped). Eight errors in `test_db_cleanup.py` are a local environment gap — `ModuleNotFoundError: No module named 'airflow_shared'` raised in the `cap_structlog` fixture — and occur identically on an unmodified checkout. `mypy` and `ruff` are clean on both files. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
