hkc-8010 opened a new pull request, #65239:
URL: https://github.com/apache/airflow/pull/65239
## What do the changes do?
`airflow db clean` (the CLI command wrapping
`airflow.utils.db_cleanup.run_cleanup`) currently suppresses all per-table
cleanup errors via `_suppress_with_logging()` and always exits 0, even when one
or more tables could not be cleaned. This makes it **impossible for operators
to detect that their database is not actually being cleaned** without manually
inspecting task logs and grepping for specific warning strings.
This PR adds:
1. An opt-in `--error-on-cleanup-failure` flag that causes the command to
exit 1 (raise `AirflowException` from `run_cleanup()`) if any table cleanup
encountered an error. Default behaviour is **unchanged** (backward-compatible).
2. A warning summary that lists all tables that were not cleaned due to
errors. This summary is **always** emitted when failures occur, without
requiring any opt-in, making the silent failure visible in task logs.
## Why?
A large deployment ran `airflow db clean` daily for 14+ months. Each run
silently failed on the `log` table because the CTAS archival step exceeded
`statement_timeout` (300 s) and was rolled back. The DAG task showed green on
every run. The `log` table grew to 337 M rows / 151 GB. When the deployment was
later upgraded to Airflow 3.x, a migration adding a new column + index to the
`log` table could not complete within `statement_timeout`, leaving the
deployment in a migration loop for ~15 hours.
Root cause: `_suppress_with_logging()` catches `OperationalError` /
`ProgrammingError`, logs a WARNING, and swallows the exception. `run_cleanup()`
has no way to know any table failed.
## Changes
| File | Change |
|------|--------|
| `airflow/utils/db_cleanup.py` | `_suppress_with_logging` yields a
`SimpleNamespace(failed=False)` context so callers can detect suppression;
`run_cleanup` collects failed table names, emits a warning summary, and
optionally raises `AirflowException` |
| `airflow/cli/cli_config.py` | Add `ARG_DB_ERROR_ON_CLEANUP_FAILURE`; wire
into `db clean` ActionCommand |
| `airflow/cli/commands/db_command.py` | Forward `error_on_cleanup_failure`
arg from CLI to `run_cleanup()` |
| `tests/utils/test_db_cleanup.py` | Unit tests for the new flag and warning
summary behaviour |
## Usage
```bash
# Default: errors are suppressed, exits 0 (unchanged behaviour)
airflow db clean --clean-before-timestamp 2024-01-01 --yes
# Opt-in: exit 1 if any table cleanup failed
airflow db clean --clean-before-timestamp 2024-01-01 --yes
--error-on-cleanup-failure
```
In a DAG-based cleanup workflow:
```python
BashOperator(
task_id="clean_db",
bash_command=(
"airflow db clean --yes "
"--clean-before-timestamp '{{ macros.ds_add(ds, -21) }}' "
"--error-on-cleanup-failure"
),
)
```
## Note on `--skip-archive`
When the CTAS archival step is itself the source of the timeout (as in the
motivating incident), combining `--error-on-cleanup-failure` with
`--skip-archive` is recommended: `--skip-archive` deletes rows directly without
the costly `CREATE TABLE … AS SELECT`, making the cleanup both faster and less
likely to time out.
## Checklist
- [x] My PR is targeted at the `main` branch
- [x] My changes are backward-compatible (new flag defaults to `False`)
- [x] I have added unit tests
- [ ] I have updated the documentation (follow-up:
`docs/apache-airflow/howto/use-airflow-db-cleanup.rst`)
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
