safaehar opened a new issue, #77201: URL: https://github.com/apache/airflow/issues/77201
### Description Could the scheduler reduce how long it holds locks on RUNNING DagRuns, so clearing tasks in one run is less dependent on the scheduling cost of other runs in the same batch? On Airflow 3.3.2 with PostgreSQL, task-clear API requests have returned HTTP 500 when an update to `dag_run` reached our configured 10-second `statement_timeout`. PostgreSQL showed the update waiting on another transaction's lock. This was a statement timeout during a lock wait, not a confirmed deadlock. The blocking backend was not conclusively mapped to a scheduler in that capture; the scheduler code below identifies a concrete contention path to investigate. Our scheduler batch limit was `scheduler.max_dagruns_per_loop_to_schedule = 120` (above the default of 20). This request concerns the lifetime of those locks, including smaller batches with expensive scheduling decisions. ### Current mechanism In 3.3.2: 1. [`DagRun.get_running_dag_runs_to_examine`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/models/dagrun.py#L654) selects and locks the batch with `SKIP LOCKED`. 2. [`SchedulerJobRunner._schedule_all_dag_runs`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2811) checks the selected runs sequentially and commits after the loop. 3. [`clear_task_instances`](https://github.com/apache/airflow/blob/3.3.2/airflow-core/src/airflow/models/taskinstance.py#L357) updates DagRun fields such as `clear_number` and `queued_at`, including for runs that are already RUNNING. Consequently, a clear targeting run A can wait for the remainder of the batch even after the scheduler has finished checking A. `SKIP LOCKED` lets other schedulers skip those rows; it does not prevent a clear's update from waiting. Source inspection of `main` at `3b35135afe9549e99088c4d557c62e5deafb125b` still shows the [batch lock acquisition](https://github.com/apache/airflow/blob/3b35135afe9549e99088c4d557c62e5deafb125b/airflow-core/src/airflow/models/dagrun.py#L815) and [commit after the scheduling loop](https://github.com/apache/airflow/blob/3b35135afe9549e99088c4d557c62e5deafb125b/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L3368). We have not reproduced the API failure on that revision. ### Use case/motivation Keep interactive task clears responsive while the scheduler processes expensive DagRuns, without compromising HA scheduler coordination or transactional consistency. Would smaller independently acquired/committed batches, or another approach to bounding lock lifetime, be viable? Simply committing after each run while retaining a preselected batch would release locks on all remaining rows, so that alone would not preserve the current safety guarantees. Reducing the configured batch limit is a possible mitigation, but not a bound on transaction duration. We do not yet have a controlled before/after result proving that it resolves these timeouts. ### Suggested regression test Proposed PostgreSQL concurrency test, not an executed reproduction: 1. Have the scheduler acquire two RUNNING runs A and B through its normal selection path. 2. Allow its scheduling decision for A to finish, then pause processing B at a test barrier. 3. From a separate session, clear a failed task in A using the normal clear path and a short statement timeout. 4. Check whether the clear waits until B is released and the scheduler batch commits. An improved mechanism should bound this wait while ensuring another scheduler cannot process unowned or stale preselected runs. ### Related issues - #72730 fixes starvation when selecting RUNNING backfills; it does not change batch-level lock lifetime. - #66817 concerns stale session objects and cross-scheduler deadlocks; this request concerns lock duration and interactive clear latency. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
