gbonazzoli opened a new issue, #54337:
URL: https://github.com/apache/airflow/issues/54337
### Apache Airflow version
3.0.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a Python Source that generate `PythonOperator` Tasks on a DAG.
It is the load of the Staging Area in a Datawarehouse, so during this phase
there are hundreds of tables that we need to load from the transactional
database to the analytical one, Airflow limits the load through
`max_active_tasks=32` DAG parameter.
The code is:
```python
with TaskGroup(group_id="MOVEX100") as DLTHUB_MOVEX100:
for source_table_list in [SOURCE_MOVEX100_TABLES]:
for source_table_obj in source_table_list:
table_name = source_table_obj["table_name"]
dlt_query = source_table_obj["dlt_query"]
dlt_load_movex100 = PythonOperator(
task_id="dlt_movex100_" + table_name,
python_callable=execute_dlt_movex100,
retries=3, # Retry up to 3 times (original + 3 retries)
retry_delay=timedelta(minutes=randrange(5) + 1), # Wait
from 1 to 5 minutes between retries
templates_dict={
"table_name": f"{table_name}",
"dlt_query": f"{dlt_query}",
},
)
Beg_Load_Staging >> dlt_load_movex100 >> End_Load_Staging
```
The problem with this code is that every days Airflow 3.x registers hundreds
of different versions for the DAG.
The reality is that is dynamically generated and so it is obvious that
changes dynamically.
Is there anyway to limit the problem ?
### What you think should happen instead?
_No response_
### How to reproduce
I think that is a general problem afflicting Airflow 3.x with DAG that are
dinamically generated through code.
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.6.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.4
apache-airflow-providers-fab==2.3.1
apache-airflow-providers-git==0.0.5
apache-airflow-providers-postgres==6.2.2
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-smtp==2.1.2
apache-airflow-providers-ssh==4.1.2
apache-airflow-providers-standard==1.5.0
```
### Deployment
Virtualenv installation
### Deployment details
An ubuntu vanilla installation in python 3.12 virtual env.
### Anything else?
no
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
