vatsrahul1001 opened a new issue, #63534: URL: https://github.com/apache/airflow/issues/63534
### Apache Airflow version main (development) ### If "Other Airflow 3 version" selected, which one? _No response_ ### What happened? Migration 0094_3_2_0_replace_deadline_inline_callback_with_fkey downgrade deletes rows from the callback table after restoring inline callback data to the deadline table. The deadline table has a foreign key on callback_id referencing callback(id) with ON DELETE CASCADE, but there is no index on deadline.callback_id. The downgrade processes in batches of 1000. Each batch updates 1000 deadline rows (restoring inline callback JSON, setting callback_id = NULL), then deletes 1000 callback rows: `conn.execute(callback_table.delete().where(callback_table.c.id.in_(callback_ids_to_delete)))` Even though callback_id is already NULLed out by the preceding UPDATE, PostgreSQL still performs the cascade check scan because the FK constraint is evaluated at statement level, not row level. Observed impact: With 10M deadline rows the first batch of 1000 deletes ran for over 5 minutes without completing a single deletion. Extrapolated to all 10,000 batches, the migration would take roughly 38 days. In practice it would never finish — PostgreSQL would eventually hit lock table limits or OOM before completing. Indexes present on deadline at time of failure: ``` deadline_pkey deadline_missed_deadline_time_idx (no index on callback_id) ``` ### What you think should happen instead? _No response_ ### How to reproduce - Start Airflow on 3.1.8 with PostgreSQL backend - Insert a meaningful number of deadline data (100K+ is enough to see significant slowdown, 1M+ makes it impractical) - Upgrade to 3.2.0 (airflow db migrate) — this creates the callback table and populates it from deadline data - Run airflow db downgrade -n 3.1.8 - Observe that migration 0094 downgrade hangs on DELETE FROM callback WHERE id IN (...) ### Operating System MAC ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
