litzius opened a new issue, #65696: URL: https://github.com/apache/airflow/issues/65696
**Category:** Airflow Core **Apache Airflow version:** 3.2.0 --- ### What happened and how to reproduce it? When a DAG is updated but its current `DagVersion` has no associated task instances, `SerializedDagModel.write_dag` updates the serialized DAG **in-place** — same `dag_version_id` UUID, new content in the database. The scheduler holds a long-lived `DBDagBag` instance (`self.scheduler_dag_bag`) whose `_dags` dict caches deserialized `SerializedDAG` objects keyed by `dag_version_id`. Because the UUID does not change, subsequent calls to `_get_dag` return the **stale cached object** — the scheduler never re-reads from the database. **Steps to reproduce:** 1. Deploy a DAG with tasks A and B. Let it be parsed and serialized. 2. Trigger a run and let it complete. Wait until all task instances for that `DagVersion` are gone (e.g. deleted via retention or manual cleanup), so the version has zero associated TIs. 3. Update the DAG code — add task C or remove task B. 4. Wait for the DAG processor to re-serialize. Confirm `created_at != updated_at` on the `dag_version` row and that the `dag_version_id` UUID is unchanged (in-place update path taken). 5. The API server / UI reflects the new task structure immediately. 6. Trigger a new DAG run. 7. **Observed:** the new run uses the stale `SerializedDAG` from the scheduler's in-memory cache. Tasks added in the update appear as removed; tasks removed in the update are still scheduled. ### What you think should happen instead? The scheduler should use the current serialized DAG content from the database. When a `dag_version_id` maps to content that has been updated in-place, the cached entry is stale and must not be returned. ### Root cause (code pointers) - `airflow/models/serialized_dag.py:708–746` — in-place update: same `dag_version_id`, new `_data` written to the row when `not has_task_instances` - `airflow/models/dagbag.py:99–107` — `_get_dag` returns the cached `SerializedDAG` on UUID match with no staleness check - `airflow/jobs/scheduler_job_runner.py:308` — the `DBDagBag` instance is created once and lives for the lifetime of the scheduler process ### Workaround Rollout-restart the scheduler. This clears the in-memory `_dags` dict and forces a fresh DB read on the next loop. ### Possible fixes (looking for input on the right approach) - **A** — Scheduler never caches (always queries DB). Simplest; removes the stale-cache class of bugs entirely. May affect performance if many DAG runs share the same DAG within one scheduling loop. - **B** — Store the `dag_hash` alongside the cached entry; re-query if the hash fetched from DB differs. Surgical but adds a DB round-trip on every cache hit to compare hashes. - **C** — Use a short TTL on the scheduler's `_dags` dict so stale entries expire automatically. ### Operating System Linux (Kubernetes node) ### Deployment Official Apache Airflow Helm Chart ### Are you willing to submit a PR? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
