gopidesupavan commented on issue #50185: URL: https://github.com/apache/airflow/issues/50185#issuecomment-2925180245
> > [@x42005e1f](https://github.com/x42005e1f) sorry was having hard time last two days at my day job, back to this now, Could you please elaborate with some example i couldn't think of what you referring sorry? > > Collisions occur due to simultaneous reading of a single descriptor - usually the one referenced by `sys.stdin`. In `CommsDecoder.get_message()`, it is read synchronously via `sys.stdin` directly. In `TriggerRunner.sync_state_to_supervisor()`, it is read asynchronously via `asyncio.StreamReader`. The idea is that to resolve collisions and avoid deadlocks, it is sufficient to allow the file descriptor to be read in mixed mode in the same thread, but associate the read with the one who sent the request. > > When using the lock described above (which can be implemented via GLock or independently), there is only one such mixed read situation - async -> sync. In this case we can, for example, do three things: > > 1. Replace `asyncio.StreamReader` with `concurrent.futures.ThreadPoolExecutor(1)`. > 2. Call `executor.submit()` in `TriggerRunner.sync_state_to_supervisor()`, store a reference to the future in `SUPERVISOR_COMMS`, wait for the future asynchronously with `asyncio.wrap_future()`. > 3. In `CommsDecoder.get_message()`, wait for the future synchronously, if any, and only then read the `sys.stdin` content itself. > > That way we can eliminate this type of collision with minimal pain. At least hypothetically. Looks promising, looking at now. :) thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
