hkc-8010 commented on issue #68721: URL: https://github.com/apache/airflow/issues/68721#issuecomment-6116926675
We hit this in production and traced it. The trigger is long backfill creation, not fast tasks finishing between scheduler loops. Queued runs already block completion through the `~exists(...)` clause. `_create_backfill` commits the `Backfill` row first. It then creates every `DagRun` and `BackfillDagRun` in a single transaction, which only commits when `create_session()` exits. While that transaction is open, `_mark_backfills_complete` (every 30s) sees a backfill with no visible queued/running runs and sets `completed_at`. What we saw, Airflow 3.2.1, Postgres, schedule of 10 runs/day, 3.75 years = 13,760 runs: ``` 18:53:48Z backfill created (POST /api/v2/backfills) 18:53:51Z scheduler: "marking 1 backfills as complete" 19:01:57Z POST returns 200 after 488.9s, all 13,760 runs become visible ``` Creation ran at about 28 runs/s. The backfill then executed all its runs with `completed_at` already set. Side effects: - The UI banner (`completed_at === null` filter) never shows, so Pause/Cancel aren't available. - `cancel_backfill` returns `409 Backfill is already completed.`, so thousands of queued runs can't be cancelled. - A resubmit in the few seconds before the scheduler closes the previous attempt gets `409 There is already a running backfill`. - Resubmits made while the first transaction is still open block on its uncommitted `dag_run` unique keys until `statement_timeout`, which surfaces as `QueryCanceled` and HTTP 500. #62561 (in 3.2.2) narrows this but doesn't close it. The guard only holds while `Backfill.created_at >= now - 2 minutes` (`initializing_cutoff`). Its `BackfillDagRun` evidence is written in the same uncommitted transaction, so it isn't visible either. Any backfill that takes more than about 2 minutes to create is still closed early on 3.2.2 and on current `main`. The repro in this issue (1,096 runs, SQLite) is consistent with that. I haven't confirmed its creation time. Possible directions: - Commit runs in batches, so `BackfillDagRun` rows become visible while creation is still going. - Have the scheduler skip a backfill until creation is complete, using an explicit "initializing" marker on `Backfill` instead of a wall-clock cutoff. - Create the `Backfill` row and its runs in one transaction, and enforce one active backfill per Dag with a lock or partial unique index instead of the early commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
