ahujaanmol1289 opened a new pull request, #70715:
URL: https://github.com/apache/airflow/pull/70715
## What this does
`SerializedDagModel.__init__()` and `hash()` each independently called
`_sort_serialized_dag_dict()` followed by `json.dumps(sort_keys=True)`.
The two JSON outputs differ only by whether `fileloc` and `bundle_name`
are present — every DAG write paid for two full recursive sorts and two
full JSON serializations when one of each suffices.
This PR adds `_compute_hash_and_storage_json()` that:
- Calls `_sort_serialized_dag_dict()` **once**
- Pops `fileloc`/`bundle_name`, generates hash JSON, computes MD5
- Restores the popped fields, generates storage JSON
- Returns `(dag_hash, storage_json_bytes, sorted_data)` in one pass
`__init__()` and `hash()` both delegate to it instead of doing their
own independent sort + serialize.
## Why
The duplicate work was unnecessary — the hash computation and storage
JSON generation operate on the same sorted dict and differ only by
two fields. Merging them removes one full recursive deep-sort and one
full `json.dumps()` call per DAG per parse cycle.
## Benchmark (50-task DAG, 100 iterations)
| Metric | Before | After |
|--------|--------|-------|
| Mean | 1.83 ms | 1.61 ms |
| Median | 1.79 ms | 1.56 ms |
~12% improvement on `_compute_hash_and_storage_json` in isolation.
This runs on every DAG on every parse cycle, so the improvement
compounds linearly with DAG count.
## What does NOT change
- The hash value produced is **bit-identical** to before (verified across
PYTHONHASHSEED=42, 123, 999)
- The stored JSON is **identical**
- Compression behavior (`compress_serialized_dags`) is unchanged
- `write_dag()` logic and control flow is untouched
- `hash()` remains a public classmethod with the same signature and
return value (backward compatible)
- All 86 existing `test_serialized_dag` tests pass unmodified
## Testing
8 new tests in `TestComputeHashAndStorageJson`:
- Hash equivalence with the legacy implementation
- `fileloc` and `bundle_name` excluded from hash, present in storage JSON
- Hash changes when actual DAG content changes
- Input dict is not mutated by the computation
- `__data_cache` and `_data`/`_data_compressed` behavior preserved
(parametrized: compressed + uncompressed)
Validated across: sqlite, postgres, mysql. Hash-stable across 3
PYTHONHASHSEED values. mypy clean. Full Core suite (3321 tests) passes.
Related: #56471, #64929
---
##### Was generative AI tooling used to co-author this PR?
- [] Yes (please specify the tool below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
