ashb opened a new pull request, #65711: URL: https://github.com/apache/airflow/pull/65711
When a scheduler crashes between dispatching a task to Celery and processing the QUEUED event that persists `external_executor_id`, the replacement scheduler cannot adopt the in-flight task. Without the Celery task ID in the database, `try_adopt_task_instances` has no `AsyncResult` to look up, so the task is reset and re-queued — causing duplicate execution of an already-running task. Fix this by generating `external_executor_id` via a DB-side UUID function (`gen_random_uuid` on PostgreSQL, `UUID()` on MySQL, a Python `uuid4` registered on SQLite) in the same bulk UPDATE that sets state=QUEUED. The ID is committed atomically with the state transition — no second write, no race window. RETURNING is used on PostgreSQL and SQLite to read back the generated UUIDs without a second round-trip; MySQL falls back to a SELECT. The CeleryExecutor passes the pre-assigned ID to `apply_async()` as the Celery `task_id`, making it deterministic from DB state. Other executors ignore it and overwrite with their own ID (e.g. ECS task ARN) during event processing. This also fixes the separate race in #55004 where `external_executor_id` is lost when the task instance row is locked during event processing. `process_executor_events` uses `skip_locked=True`, and `get_event_buffer()` flushes the executor's in-memory buffer into a local variable. If a TI is locked and skipped, its QUEUED event is consumed from the buffer but never processed — the event and its task ID are silently dropped. With the ID now written to the database before the task is even sent to Celery, adoption no longer depends on the event being processed. Closes: #55004 Closes: #58570 Closes: #64971 (cherry picked from commit 3b188b904fbb9525e17173f6ef1f5e5d388a75fc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
