ccoliu commented on issue #71923:
URL: https://github.com/apache/airflow/issues/71923#issuecomment-5367419024
Great analysis and impressive investigation on the timeline!
Regarding the fix direction for your PR, here are a few ideas that might
help:
1. **Re-issue `TaskReschedule` cleanup before flushing**:
In `prepare_db_for_next_try()` / `save_to_db()`, executing a second
`DELETE FROM task_reschedule WHERE ti_id = <old_id>` right before flushing the
`ti.id` UPDATE can guarantee that any concurrent inserts from the Execution API
during the listener stall are cleaned up before the FK constraint is checked.
2. **Isolate Listener execution from the DB transaction**:
Longer-term, listener hooks (like `on_task_instance_failed`) shouldn't
block an open database transaction in the scheduler's critical path. Moving
listener invocations outside the transaction scope (or executing them
post-flush/commit) would eliminate the window amplifier altogether.
3. **Scheduler exception handling (Defense in depth)**:
Wrapping the DB flush in failure-handling paths with `try...except
IntegrityError` (or catching `ForeignKeyViolation`) to gracefully handle/retry
the single TI update, rather than letting the exception bubble up and crash the
main scheduler loop.
Wish you all the best on solving issue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
