GayathriSrividya opened a new pull request, #69973: URL: https://github.com/apache/airflow/pull/69973
closes #66794 ## What this PR changes - split executor event processing into two phases in `SchedulerJobRunner.process_executor_events` - process `QUEUED` task events first without `FOR UPDATE` row locks, updating only `external_executor_id` - keep `FOR UPDATE SKIP LOCKED` processing for the remaining state transitions (`FAILED`, `SUCCESS`, `RUNNING`, `RESTARTING`) - use per-phase try-number maps so queued and non-queued events for the same TI primary key do not interfere - add a regression test ensuring QUEUED-only event handling does not invoke `with_row_locks` ## Why `QUEUED` executor events only update metadata (`external_executor_id`) and do not require lock-protected state transitions. Removing them from the locked batch reduces lock contention in multi-scheduler MySQL deployments and avoids heartbeat deadlock pressure while preserving existing behavior for terminal transitions. ## Tests - /opt/homebrew/bin/ruff format airflow-core/src/airflow/jobs/scheduler_job_runner.py airflow-core/tests/unit/jobs/test_scheduler_job.py - /opt/homebrew/bin/ruff check --fix airflow-core/src/airflow/jobs/scheduler_job_runner.py airflow-core/tests/unit/jobs/test_scheduler_job.py - uv run --project airflow-core pytest airflow-core/tests/unit/jobs/test_scheduler_job.py -k "queued_event_does_not_take_row_lock or process_executor_events_multiple_try_numbers_warns" -xvs *(fails during collection in this environment: ModuleNotFoundError: tests_common)* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
