respecht opened a new issue, #34978:
URL: https://github.com/apache/airflow/issues/34978
### Apache Airflow version
2.7.2
### What happened
If a DAG is dynamically generated, with the code for that generation in a
separate file, the Dag File Processor will not pick up the file where the
DAG-generating function is called. See reproduction for an example.
A workaround for this is to import airflow.decorators.dag at the top of the
file where the DAG is called. Although this is an unused import, the Dag File
Processor will process the file and recognize the DAG.
### What you think should happen instead
Airflow should pick up valid DAGs in the dags directory regardless of how
they are instantiated.
### How to reproduce
DAG factory at dags/dag_factory.py:
```
import time
from airflow.decorators import dag, task
from airflow.utils.dates import days_ago
def generate_dag(name):
@dag(name, start_date=days_ago(10), tags=['test'])
def job_dag():
@task
def add_ids_to_queue():
time.sleep(5)
print('Task complete!')
add_ids_to_queue()
return job_dag()
```
dags/test_no_import_dag.py (does not show up in DAG list)
```
from dag_factory import generate_dag
generate_dag('test_no_import')
```
dags/test_with_import_dag.py (shows up in DAG list)
```
from dag_factory import generate_dag
from airflow.decorators import dag
generate_dag('test_has_import')
```
Although the import `from airflow.decorators import dag` is unused, it is a
workaround for having Airflow pick up these dynamically generated DAGs.
### Operating System
Mac OSX Ventura 13.5.1
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-sqlite==3.4.3
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I created a clean Airflow deployment following the instructions at
https://airflow.apache.org/docs/apache-airflow/stable/start.html
### Anything else
Latest logs from logs/dag_processor_manager/dag_processor_manager.log:
```
[2023-10-16T15:03:48.829-0400] {manager.py:887} INFO -
================================================================================
DAG File Processing Stats
File Path
PID Runtime # DAGs #
Errors Last Runtime Last Run
--------------------------------------------------------------------------------------------------------------------------
----- --------- -------- ---------- -------------- -------------------
/Users/RSpecht/airflow/dags/dag_factory.py
0
0 0.57s 2023-10-16T19:03:23
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py
0 0 0.58s
2023-10-16T19:02:57
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api_virtualenv.py
0 0 0.58s
2023-10-16T19:03:30
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_kubernetes_executor.py
0 0 0.57s
2023-10-16T19:03:01
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/workday.py
0
0 0.58s 2023-10-16T19:03:26
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/event_listener.py
0 0
0.58s 2023-10-16T19:03:29
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/plugins/listener_plugin.py
0 0 0.63s
2023-10-16T19:03:28
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/subdags/subdag.py
0
0 0.56s 2023-10-16T19:03:25
/Users/RSpecht/airflow/dags/test_with_import_dag.py
1
0 0.62s 2023-10-16T19:03:22
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_latest_only.py
1 0
0.63s 2023-10-16T19:03:00
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_task_group_decorator.py
1 0 0.68s
2023-10-16T19:03:39
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_complex.py
1
0 0.64s 2023-10-16T19:03:10
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_python_operator.py
1 0 0.65s
2023-10-16T19:02:34
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_params_trigger_ui.py
1 0 0.65s
2023-10-16T19:02:39
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_bash_operator.py
1 0
0.61s 2023-10-16T19:03:21
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_short_circuit_operator.py
1 0 0.64s
2023-10-16T19:03:47
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_setup_teardown.py
1 0
0.63s 2023-10-16T19:02:30
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_operator.py
1 0 0.62s
2023-10-16T19:03:14
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_labels.py
1 0
0.62s 2023-10-16T19:03:16
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py
1 0
0.63s 2023-10-16T19:03:32
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_task_group.py
1 0
0.69s 2023-10-16T19:03:41
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial.py
1
0 0.64s 2023-10-16T19:03:34
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sla_dag.py
1
0 0.67s 2023-10-16T19:03:44
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_passing_params_via_test_command.py
1 0 0.64s
2023-10-16T19:02:36
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_latest_only_with_trigger.py
1 0 0.62s
2023-10-16T19:02:59
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_time_delta_sensor_async.py
1 0 0.64s
2023-10-16T19:03:38
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dag_decorator.py
1 0
0.82s 2023-10-16T19:03:08
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_xcom.py
1
0 0.62s 2023-10-16T19:02:54
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_operator_decorator.py
1 0 0.65s
2023-10-16T19:03:13
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_day_of_week_operator.py
1 0 0.63s
2023-10-16T19:03:17
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_skip_dag.py
1 0
0.63s 2023-10-16T19:03:45
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_short_circuit_decorator.py
1 0 0.72s
2023-10-16T19:03:48
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/tutorial_dag.py
1
0 0.61s 2023-10-16T19:03:33
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_trigger_target_dag.py
1 0 0.64s
2023-10-16T19:02:55
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sensor_decorator.py
1 0 0.63s
2023-10-16T19:02:32
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_python_dop_operator_3.py
1 0 0.63s
2023-10-16T19:03:12
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_sensors.py
1
0 0.64s 2023-10-16T19:02:31
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_setup_teardown_taskflow.py
1 0 0.63s
2023-10-16T19:02:28
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_nested_branch_dag.py
1 0 0.68s
2023-10-16T19:02:41
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
1 0 0.62s 2023-10-16T19:03:04
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_trigger_controller_dag.py
1 0 0.63s
2023-10-16T19:03:36
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_params_ui_tutorial.py
1 0 0.67s
2023-10-16T19:02:37
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_dynamic_task_mapping.py
1 0 0.61s
2023-10-16T19:03:05
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_external_task_marker_dag.py
2 0 0.59s
2023-10-16T19:03:02
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_xcomargs.py
2 0
0.63s 2023-10-16T19:02:52
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_branch_datetime_operator.py
3 0 0.61s
2023-10-16T19:03:19
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_subdag_operator.py
3 0 0.66s
2023-10-16T19:03:43
/opt/homebrew/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py
6 0
0.62s 2023-10-16T19:03:06
================================================================================
```
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
