GitHub user pedro-cf added a comment to the discussion: DAG Parsing Performance
with Large Number of Dynamically Generated DAGs
Thank you @tanelk
I have added this to my dag parsing:
```python
CURRENT_DAG_ID = get_parsing_context().dag_id
def create_dynamic_dags(dynamic_dags):
"""Create DAG objects from configurations."""
for dynamic_dag in dynamic_dags:
if CURRENT_DAG_ID is not None and CURRENT_DAG_ID !=
dynamic_dag.dynamic_dag_id:
continue
globals()[dynamic_dag.dynamic_dag_id] = create_dag(dynamic_dag)
```
Have there been any new updates regarding this topic?
> Have several standalone dag processors and splitting the processing by
> --subdir flags - you can split your DAGs by subdirectories and parse each
> subdirectory by one or more separate dag file processors.
I would like to avoid generating dag files during my parsing if possible.
GitHub link:
https://github.com/apache/airflow/discussions/44727#discussioncomment-13194200
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
