LeonardoSanBenitez commented on issue #58310: URL: https://github.com/apache/airflow/issues/58310#issuecomment-5203808826
Airflow already has a check for this and it has been on by default since 3.2, which I think is the fastest way for anyone here to find out which DAG is doing it. [`[dag_processor] dag_version_inflation_check_level`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-version-inflation-check-level) defaults to `warning`, and the DAG processor runs a static check on every parse ([`dag_version_inflation_checker.py`](https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/utils/dag_version_inflation_checker.py), called from [`processor.py`](https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/dag_processing/processor.py#L236)). When it fires you get a DAG warning in the UI naming the line: ``` This Dag uses runtime-variable values in Dag construction. It causes the Dag version to increase as values change on every Dag parse. Line 7, Col 4 Code: BashOperator(task_id='t', bash_command=f'echo test{uuid.uuid4()}') Issue: Don't use runtime-varying values as arguments of task within with Dag block ``` That is the `uuid4()` example from earlier in this thread, run through 3.3.0. It also catches `datetime.now()` in the DAG constructor or inside `default_args`, the `from uuid import uuid4` form, tasks built in a loop, and a value passed through an intermediate variable (it prints `Line 6: 'run_marker' related 'uuid.uuid4()'`). Setting the level to `error` turns it into a DAG import error instead of a warning. One caveat, since silence from it is not proof: it follows `m = uuid.uuid4()` but not `m = uuid.uuid4().hex` — wrapping the runtime call in an attribute access loses the trace. So it rules out the obvious patterns, not every one. If the check is quiet and versions still climb, the cause is probably not in the DAG file at all, and #66103 is worth reading. There the reporter compared `dag_code.source_code_hash` between consecutive versions, found it identical while `serialized_dag.data` differed, and traced it to two DAG processors disagreeing — one was running an older build of an in-house provider, so `template_fields` differed by one entry. Another report in that thread sees 20-70 new versions from a single rolling redeploy, which fits the same shape. Code hash unchanged plus serialized data changed means the parsing environment is varying rather than your DAG. On the question further up about "parse, parse, parse" not bumping the version while "trigger a run, parse" does: that is [`SerializedDagModel.write_dag`](https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/models/serialized_dag.py#L726). When the serialization changes, a new `DagVersion` is only created if the current version already has task instances attached; otherwise the existing row is overwritten in place, because rewriting a version that a run has already executed against would rewrite history. On 3.3.0, five consecutive parses of a DAG with a `uuid4()` in it left `version_number` at 1 while the stored hash changed every time; after attaching a task instance, each further parse added exactly one version. So the counter tracks how often the DAG changed *while the previous version was already in use*, which is why a high run rate turns a small non-determinism into tens of thousands of rows. On cleanup: `dag_version` is covered by `airflow db clean`, and on 3.3 it does drain. Six versions each pinned by an aged run went to a single row in one pass, keeping the newest version per DAG ([`db_cleanup.py`](https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/utils/db_cleanup.py)). The constraint worth knowing is that a version still referenced by a task instance is skipped, since `task_instance.dag_version_id` is `ON DELETE RESTRICT` — so what actually governs how fast the table drains is your `task_instance` retention window, not anything about versions. If the table grew while cleanup was already configured, it is worth checking the old logs before assuming the retention window was the whole story. On 3.2.2 the same case does this: ``` Checking table dag_version Found 5 rows meeting deletion criteria. [warning ] Encountered error when attempting to clean table 'dag_version'. [warning ] The following tables were not cleaned due to errors: ['dag_version']. exit code: 0 ``` Exit code 0, so a nightly job would have looked green while nothing was cleaned. On 3.3.0 the same case prints `Found 0 rows meeting deletion criteria` and exits cleanly. I did not reproduce the MySQL variant, where that delete is reported to hang on metadata locks (#66177). On the api-server running out of memory: I have not looked into that properly and cannot say it has the same root cause. The grid structure endpoint on 3.3 already limits historical versions to the runs on the current page and streams them, so it is not loading all 54k versions. Two open threads look adjacent: #69832 is a PR against that same endpoint, about it holding a database connection across deserialization for DAGs with many versions, and #64877 is the api-server hanging when you open a task that no longer exists in the latest DAG code — which is close to what going back to an older run of a heavily versioned DAG would do. If neither matches, it is probably worth its own issue with the version counts and pod memory limits in it. --- Drafted-by: Claude Code (Opus 5); reviewed by @LeonardoSanBenitez before posting. If you believe this contribution was not useful to the airflow community, please let me know and I will stop posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
