dabla commented on issue #66842: URL: https://github.com/apache/airflow/issues/66842#issuecomment-4441303527
> async tasks (AIP-98, in 3.2) run in the triggerer event loop but the current task state and asset state accessors are synchronous and calling them blocks the event loop. Currently, all context accessors — variables, connections, XCom, and task state, go through `SUPERVISOR_COMMS.send()` which is synchronous and blocks the event loop on every call. No async client path exists in the SDK yet. > > The server-side async methods (`aget/aset/adelete/aclear`) on `MetastoreStateBackend` already exist. The gap is entirely client-side. > > One potential approach would be to: > > * Add async methods to the relevant operations in task sdk client > * Add `async def get/set/delete/clear` to `TaskStateAccessor` (and other accessors) that bypass supervisor comms and hit the execution API directly via async HTTP > * Wire the async context at task startup when running in triggerer mode What do you actually mean by async tasks run in the triggerer event loop? To my knowledge async tasks run on the worker event loop, not the triggerer but maybe I’m missing something? Good point regarding the TaskStateAccessor, will have a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
