GitHub user csm10495 added a comment to the discussion: Can `with DAG(..)` be
made thread local?
I agree with you completely. There shouldn't be something time consuming in
there.. but alas, that's how they wrote it. Even if it wasn't time consuming,
there is still a chance for them to hit an issue here (though a smaller chance
I guess). We're on older Python, so the GIL is still locking, though I'm
guessing network or other IO calls within the DAG building process allow
context switching.
They build the DAG inside a function inside a service outside of Airflow. The
dag building has slow pieces and network calls inside (yes i agree this is
bad). After they build the DAG, they parse it to figure out the flow the DAG
would have then dump that info into another service.
Sort of like:
```
os.environ["_AIRFLOW__AS_LIBRARY"] = "1"
from airflow import DAG
def load_dag_info_into_service(...):
with DAG(...) as dag:
...
# go through tasks in the dag, load them into the other service
```
The problem they recently hit was because the DAG-building-service uses a
thread pool executor to distribute jobs (to that function ^) and they
implicitly use the current dag, they sometimes get parsing issues because the
current dag changes for 'the other thread' while 'this thread' is building.
GitHub link:
https://github.com/apache/airflow/discussions/53691#discussioncomment-13880130
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
