dabla commented on PR #44240:
URL: https://github.com/apache/airflow/pull/44240#issuecomment-2506400146
> > > What was the stactrace of such issue @dabla -> I have a hard time to
figure out what race/parallelism could have caused it.
> >
> >
> > Or maybe this is a bug in the Python 3.9 implementation of threadlocal?
>
> Yeah. that why I wanted to see stacktrace, because I cannot find a path
where you would have missing attribute - when I compare "before" and "after"
the change i cannot see anything that could have fixed that problem, other than
slightly bigger overhead of executing the method, which could have mitigitated
some race condition, having such a stack trace would definitely help us to see
the root cause. I am pretty reluctant to approve and merge change that I do not
understand how it works :).
Yes, I completely understand, me personally I also don't understand why the
"'fix" fixes this issue as I wouldn't have expected this behaviour. Here you,
I've re-executed the DAG without the patched ExecutorSafeguard:
```
[2024-11-28, 15:43:59 UTC] {taskinstance.py:3311} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py",
line 767, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py",
line 733, in _execute_callable
return ExecutionCallableRunner(
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/baseoperator.py",
line 417, in wrapper
return func(self, *args, **kwargs)
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 424,
in execute
self.max_active_tis_per_dag,
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 346,
in _run_tasks
for task, future in futures:
File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 125, in
worker
result = (True, func(*args, **kwds))
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 325,
in _run_task
) # Always open new event loop as this is executed in multithreaded
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in
run_until_complete
return future.result()
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 392,
in _run_operator
operator: BaseOperator = cast(BaseOperator, task_instance.task)
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 219,
in run
return await wait_for(
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
return fut.result()
File "/airflow/plugins/infrabel/operators/streamedoperator.py", line 169,
in _run_callable
return callable_runner.run(*args, **kwargs)
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
File
"/airflow/.venv/lib/python3.9/site-packages/airflow/models/baseoperator.py",
line 410, in wrapper
sentinel =
cls._sentinel.callers.pop(f"{func.__qualname__.split('.')[0]}__sentinel", None)
AttributeError: '_thread._local' object has no attribute 'callers'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]