GitHub user gbonazzoli created a discussion: Airflow 3.x Huge Versioning
Problem In Dynamically Created DAGS
### Apache Airflow version
3.0.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a Python Source that generate `PythonOperator` Tasks on a DAG.
It is the load of the Staging Area in a Datawarehouse, so during this phase
there are hundreds of tables that we need to load from the transactional
database to the analytical one, Airflow limits the load through
`max_active_tasks=32` DAG parameter.
The code is:
```python
with TaskGroup(group_id="MOVEX100") as DLTHUB_MOVEX100:
for source_table_list in [SOURCE_MOVEX100_TABLES]:
for source_table_obj in source_table_list:
table_name = source_table_obj["table_name"]
dlt_query = source_table_obj["dlt_query"]
dlt_load_movex100 = PythonOperator(
task_id="dlt_movex100_" + table_name,
python_callable=execute_dlt_movex100,
retries=3, # Retry up to 3 times (original + 3 retries)
retry_delay=timedelta(minutes=randrange(5) + 1), # Wait
from 1 to 5 minutes between retries
templates_dict={
"table_name": f"{table_name}",
"dlt_query": f"{dlt_query}",
},
)
Beg_Load_Staging >> dlt_load_movex100 >> End_Load_Staging
```
The problem with this code is that every days Airflow 3.x registers hundreds of
different versions for the DAG.
The reality is that is dynamically generated and so it is obvious that changes
dynamically.
Is there anyway to limit the problem ?
### What you think should happen instead?
_No response_
### How to reproduce
I think that is a general problem afflicting Airflow 3.x with DAG that are
dinamically generated through code.
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.6.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.4
apache-airflow-providers-fab==2.3.1
apache-airflow-providers-git==0.0.5
apache-airflow-providers-postgres==6.2.2
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-smtp==2.1.2
apache-airflow-providers-ssh==4.1.2
apache-airflow-providers-standard==1.5.0
```
### Deployment
Virtualenv installation
### Deployment details
An ubuntu vanilla installation in python 3.12 virtual env.
### Anything else?
no
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/54348
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
