KidAmnesiac1 opened a new pull request, #74446: URL: https://github.com/apache/airflow/pull/74446
### Description Makes the scheduler mark an executor callback as `FAILED` when it can't be queued, instead of crashing the scheduler loop. ### Reason for the change If queuing an executor callback raises, for example because the executor's database write is rejected at flush, the error escapes `_enqueue_executor_callbacks` and the scheduler loop rolls back. The callback stays `PENDING`, so every restarted scheduler retries it and crashes again. Scheduling stops for the whole deployment until someone deletes the callback's Dag run. We hit this in production with a `SyncCallback` deadline on the `EdgeExecutor`, whose `edge_job.command` column rejected the oversized workload (fixed separately in #74445). Any error at this point causes the same crash loop. ### Fix - Each callback is queued inside its own savepoint (`session.begin_nested()`). If queuing fails, only that savepoint rolls back, the callback is marked `FAILED`, and the scheduler carries on with the rest. - The full error goes to the scheduler log. `callback.output` stores only the exception type, because a database error's message includes the workload JSON and its token. - A new test makes the database reject one of two callbacks at flush time. Without this change the `IntegrityError` escapes; with it, that callback is `FAILED` and the other is `QUEUED`. Executors that track queued work in memory may keep the failed callback's key until their own reconciliation runs. For Edge, `_purge_jobs()` already drops keys that have no `edge_job` row. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
