hkc-8010 opened a new issue, #66301:
URL: https://github.com/apache/airflow/issues/66301
### Apache Airflow version
3.2.0
### What happened?
We hit a case where a DAG was being actively parsed from the current bundle
path, but the serialized DAG metadata still pointed to an older bundle path.
In our case, DAG-level `on_failure_callback` behavior was failing before a
manual serialized-DAG refresh, then started working again after we deleted the
serialized row and let Airflow recreate it.
There is still some uncertainty about the exact internal callback execution
path, but the stale serialized metadata was definitely present, and refreshing
it correlated directly with the fix.
We also saw repeated API/UI-side errors for the same DAG version while this
was happening:
```text
2026-05-01T17:58:24.234995Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=ece1848c-2ea4-4355-9757-440b8eb725fd
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=ece1848c-2ea4-4355-9757-440b8eb725fd
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found
[airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113
request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09
version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
```
### What you think should happen instead?
If a DAG's `fileloc` changes, the serialized DAG row should be updated to
reflect the new path even when the serialized DAG content hash is unchanged.
At minimum, `fileloc` should not stay stale indefinitely when the DAG is
being actively reparsed from a new location.
### How to reproduce
We do not yet have a minimal standalone repro, but this is the observed
pattern:
1. Run Airflow with versioned or changing DAG bundle paths.
2. Keep the DAG definition effectively unchanged so the serialized DAG hash
remains the same.
3. Move the same DAG to a new bundle path.
4. Observe that `DagModel.fileloc` and current parsing reflect the new path,
but `SerializedDagModel` may still point to the old path.
5. DAG-level callback behavior and some UI/version-history lookups may then
behave incorrectly.
### Operating System
Linux
### Versions of Apache Airflow Providers
Not central to the issue
### Deployment
Custom / Astro Hosted runtime based on Airflow 3.2.0
### Deployment details
Observed on Airflow 3.2.0 with versioned DAG bundle paths.
### Anything else?
A few concrete observations from the incident:
- `DagModel.fileloc` was current and pointed at the active bundle path.
- `DagModel.last_parsed_time` was current.
- `SerializedDagModel.last_updated` was still old.
- `SerializedDagModel` still pointed at the old bundle path.
- The old bundle path no longer existed on the current pod.
- DAG-level failure callbacks did not appear to fire for failed runs before
refresh.
- After deleting the serialized DAG row and letting Airflow recreate it, the
serialized `fileloc` updated to the current path and the next customer test
worked.
The serialization logic appears relevant here. In
`SerializedDagModel.write_dag`, the row is skipped when `dag_hash` and
`processor_subdir` are unchanged:
```python
if (
serialized_dag_db is not None
and serialized_dag_db.dag_hash == new_serialized_dag.dag_hash
and serialized_dag_db.processor_subdir ==
new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB",
dag.dag_id)
return False
```
But `fileloc` is set separately from `dag.fileloc` when building the
serialized row:
```python
self.fileloc = dag.fileloc
self.fileloc_hash = DagCode.dag_fileloc_hash(self.fileloc)
```
So if only `fileloc` changes while the serialized DAG content hash stays the
same, the row may never be rewritten.
### Are you willing to submit PR?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
