ColtenOuO opened a new pull request, #72003: URL: https://github.com/apache/airflow/pull/72003
### Summary A manually triggered Dag run's `logical_date` can coincide with a later, unrelated scheduled run's `logical_date` -- they represent different data intervals despite sharing that value (a manual trigger's data interval is derived from when it fires, not the period it's labeled with). The `dag_run` table's `(dag_id, logical_date)` uniqueness constraint didn't distinguish between run types, so whichever run landed first silently and permanently claimed that logical_date: the scheduler's dedup check would find the manual run, assume the scheduled run for that period already existed, and skip creating it -- dropping that scheduling period for good, with only an easily-missed `WARNING` log as a trace. This isn't only an in-memory dedup-check bug. Verified directly against the metadata database: even bypassing the scheduler's pre-check, inserting the correctly-computed scheduled run fails with the same unique-constraint violation as long as a manual run already occupies that `(dag_id, logical_date)` pair. The fix has to widen what the database considers a duplicate, not just how the scheduler checks for one. ### Change - Added a migration widening the `dag_run` unique constraint from `(dag_id, logical_date)` to `(dag_id, logical_date, run_type)`.w only matches `SCHEDULED` runs, so a manual (or backfill/asset-triggered) run sharing a `logical_date` is no longer mistaken for the scheduled run itself. - `SchedulerJobRunner._create_dag_runs`'s dedup lookup now only matches `SCHEDULED` runs, so a manual (or backfill/asset-triggered) run sharing a `logical_date` is no longer mistaken for the scheduled run itself. - `Backfill._create_backfill_dag_run_non_partitioned` and `_handle_clear_run` now lock and update the specific `DagRun` row by `id` instead of by `(dag_id, logical_date)`, since that pair can now match more than one row. - Added `test_scheduler_create_dag_runs_manual_run_does_not_block_scheduled_run` (`test_scheduler_job.py`) and `test_handle_clear_run_only_updates_the_targeted_run` (`test_backfill.py`) as regression tests; both confirmed to fail without their corresponding fix. ### **Downgrade handling** After this constraint has been in effect, data can legitimately exist that violates the narrower, pre-migration constraint (a manual and a scheduled run sharing a `logical_date`), so downgrading straight to the old constraint would fail outright. The downgrade first finds every `(dag_id, logical_date)` group with more than one row, keeps the lowest-id row of each group in `dag_run`, and moves the rest into an `_airflow_moved__`-prefixed table before re-adding the narrower constraint -- data is preserved for review, not discarded. This mirrors the existing `0032_..._rename_execution_date_to_logical_date_and_nullable` migration, which handles the equivalent case for NULL `logical_date` rows the same way. closes: #41208 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes -- Claude Code (Sonnet 5) for writing test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
