renat-sagut commented on code in PR #65622:
URL: https://github.com/apache/airflow/pull/65622#discussion_r3159976108
##########
task-sdk/src/airflow/sdk/execution_time/comms.py:
##########
@@ -193,14 +194,36 @@ class CommsDecoder(Generic[ReceiveMsgType, SendMsgType]):
# Async lock for async operations
_async_lock: asyncio.Lock = attrs.field(factory=asyncio.Lock, repr=False)
+ @contextmanager
+ def _lock_sync(self):
+ """Acquire the thread lock for synchronous operations."""
+ with self._thread_lock:
+ yield
+
+ @asynccontextmanager
+ async def _lock_async(self):
+ """
+ Acquire both the async lock and the thread lock for asynchronous
operations.
+
+ The thread lock is acquired via a thread executor so the event loop
+ is not blocked while waiting for other holders.
+ """
+ async with self._async_lock:
+ loop = asyncio.get_running_loop()
+ await loop.run_in_executor(None, self._thread_lock.acquire)
Review Comment:
@dabla is this explanation correct?
##########
task-sdk/src/airflow/sdk/execution_time/comms.py:
##########
@@ -193,14 +194,36 @@ class CommsDecoder(Generic[ReceiveMsgType, SendMsgType]):
# Async lock for async operations
_async_lock: asyncio.Lock = attrs.field(factory=asyncio.Lock, repr=False)
+ @contextmanager
+ def _lock_sync(self):
+ """Acquire the thread lock for synchronous operations."""
+ with self._thread_lock:
+ yield
+
+ @asynccontextmanager
+ async def _lock_async(self):
+ """
+ Acquire both the async lock and the thread lock for asynchronous
operations.
+
+ The thread lock is acquired via a thread executor so the event loop
+ is not blocked while waiting for other holders.
+ """
+ async with self._async_lock:
+ loop = asyncio.get_running_loop()
+ await loop.run_in_executor(None, self._thread_lock.acquire)
Review Comment:
The comms socket can be accessed at the same time by async tasks (in the
async thread) and sync tasks (in other threads). For example, sync_to_async
runs sync functions in a shared thread. That's why we need the sync lock as
well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
