dabla commented on PR #44240:
URL: https://github.com/apache/airflow/pull/44240#issuecomment-2506400146

   > > > What was the stactrace of such issue @dabla -> I have a hard time to 
figure out what race/parallelism could have caused it.
   > > 
   > > 
   > > Or maybe this is a bug in the Python 3.9 implementation of threadlocal?
   > 
   > Yeah. that why I wanted to see stacktrace, because I cannot find a path 
where you would have missing attribute - when I compare "before" and "after" 
the change i cannot see anything that could have fixed that problem, other than 
slightly bigger overhead of executing the method, which could have mitigitated 
some race condition, having such a stack trace would definitely help us to see 
the root cause. I am pretty reluctant to approve and merge change that I do not 
understand how it works :).
   
   Yes, I completely understand, me personally I also don't understand why the 
"'fix" fixes this issue as I wouldn't have expected this behaviour.  Here you, 
I've re-executed the DAG without the patched ExecutorSafeguard:
   
   ```
   [2024-11-28, 15:43:59 UTC] {taskinstance.py:3311} ERROR - Task failed with 
exception
   Traceback (most recent call last):
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py", 
line 767, in _execute_task
       result = _execute_callable(context=context, **execute_callable_kwargs)
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py", 
line 733, in _execute_callable
       return ExecutionCallableRunner(
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/utils/operator_helpers.py", 
line 252, in run
       return self.func(*args, **kwargs)
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", 
line 417, in wrapper
       return func(self, *args, **kwargs)
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 424, 
in execute
       self.max_active_tis_per_dag,
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 346, 
in _run_tasks
       for task, future in futures:
     File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 771, in get
       raise self._value
     File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 125, in 
worker
       result = (True, func(*args, **kwds))
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 325, 
in _run_task
       )  # Always open new event loop as this is executed in multithreaded
     File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in 
run_until_complete
       return future.result()
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 392, 
in _run_operator
       operator: BaseOperator = cast(BaseOperator, task_instance.task)
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 219, 
in run
       return await wait_for(
     File "/usr/local/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
       return fut.result()
     File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 169, 
in _run_callable
       return callable_runner.run(*args, **kwargs)
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/utils/operator_helpers.py", 
line 252, in run
       return self.func(*args, **kwargs)
     File 
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", 
line 410, in wrapper
       sentinel = 
cls._sentinel.callers.pop(f"{func.__qualname__.split('.')[0]}__sentinel", None)
   AttributeError: '_thread._local' object has no attribute 'callers'
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to