seanghaeli opened a new issue, #77278: URL: https://github.com/apache/airflow/issues/77278
When many deferred tasks finish at once with remote logging on, the triggerer uploads each finished trigger's log inline in its main loop, one after the other (`TriggerRunnerSupervisor._handle_request`, `TriggerStateChanges`: `upload_to_remote()` then `close()` per finished trigger). While that runs it doesn't heartbeat or handle events. After a long batch it logs that its runner looks deadlocked and skips its heartbeat, and with more than one triggerer its other triggers are reassigned and run twice. This happens on ordinary workloads: many deferrable sensors waiting on one external event, or a restart after which many timers are already due. Suggestion: upload finished triggers' logs from a small thread pool instead of inline, so the main loop keeps heartbeating. The pool size also bounds how hard the remote store is hit. Done when: with remote logging on, a few thousand triggers finishing together don't stop the triggerer's heartbeat, their logs are all uploaded, and no trigger is reassigned, with a test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
