GitHub user mateuscarestiato added a comment to the discussion: How do disable the DAG version ?
Hi @trongchata! The rapid version increase in your multi-datacenter setup is almost certainly caused by two DAG processors in different instances processing the same DAGs and producing slightly different serializations — leading to each instance detecting "changes" made by the other. Root cause in multi-datacenter setups: When DAG processors run in two separate Airflow instances sharing the same metadata database, each processor independently serializes and stores DAG versions. If there's any non-determinism in serialization (timezone handling, float precision, object ordering), each instance will see the other's write as a "change" and create a new version — causing a rapid version increment loop. Options to address this: 1. Designate a single DAG processor (recommended for 3.x): In Airflow 3.x, DAG processing is handled by the dag-processor component. Run it in only one datacenter and have the other datacenter's scheduler consume DAG metadata from the shared DB: yaml # In the second DC's airflow.cfg — disable local DAG processing [dag_processor] standalone = false 2. Increase the DAG file processing interval to reduce parse frequency: ini # airflow.cfg [scheduler] dag_dir_list_interval = 300 # default is 300s, increase if lower min_file_process_interval = 60 # seconds between re-processing same file 3. If using DB-stored DAGs (your case), ensure a single writer: Since your DAGs are saved in DB and generated dynamically, ensure only one instance writes DAG definitions to the database. The second datacenter's DAG processor should be configured as read-only or disabled. 4. Check for non-determinism in your dynamic DAG generation (see related discussion #66103 for diagnostics). Note: There is currently no config flag to fully disable DAG versioning in Airflow 3.x — it's a core feature of the new architecture. The correct solution is controlling which instances run the DAG processor. Hope this helps! Are both instances using the same shared metadata database or separate DBs that are synced? GitHub link: https://github.com/apache/airflow/discussions/66514#discussioncomment-16886826 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
