sabz90 opened a new pull request, #74182:
URL: https://github.com/apache/airflow/pull/74182
## Summary
This draft explores preserving `DagRun.conf` JSON semantics while avoiding
Python object-graph materialization in the Execution API server and task
supervisor.
It changes the `DagRun.conf` task-start path as follows:
- use PostgreSQL `JSON` rather than canonicalizing `JSONB`;
- defer ORM loading of `DagRun.conf` in the task-start endpoint;
- select the value as text for `TIRunContext`;
- transport compact JSON through the existing `DagRun.conf` field;
- decode once in the task process immediately before params/template-context
construction;
- convert the string back to a dictionary for older Execution API and foreign
SDK protocol versions.
Normal ORM and public API callers continue to see `DagRun.conf` as a
dictionary. MySQL and SQLite retain their existing JSON storage.
## Motivation
This combines the `DagRun.conf` portions of:
- #74025: avoid API-server/supervisor object-graph amplification;
- #74023: preserve exponent-form finite floats through PostgreSQL;
- #73712: avoid MessagePack overflow for out-of-range integers inside
`DagRun.conf`.
It does not replace the general large-integer transport work in #73783
because
oversized integers can originate outside `DagRun.conf`.
## Difference from #74048
#74048 serialized an already-materialized dictionary in the supervisor. The
task-start API and supervisor had already paid the object-graph cost.
This draft projects serialized text directly from the database, so neither
the
Execution API server nor supervisor creates the nested dictionary. The task
still materializes it once because existing params, templates, and user code
require dictionary behavior.
## Evidence
PostgreSQL 16:
```text
JSON: {"float":1.7E308,"big":18446744073709551616}
JSONB: exponent expanded into a 309-digit integer token
```
Synthetic isolated-phase benchmark with an 11,461,903-byte nested payload:
| Phase | Current peak | Opaque peak | Current time | Opaque time |
|---|---:|---:|---:|---:|
| API | 26.74 MB | 17.19 MB | 119.27 ms | 9.77 ms |
| Supervisor | 26.66 MB | 17.19 MB | 229.62 ms | 6.17 ms |
| Task | 40.79 MB | 42.78 MB | 190.00 ms | 133.27 ms |
The task-side cost remains. Before marking this ready, this needs concurrent,
end-to-end pod RSS and startup-latency validation.
## Validation
Passed in the Apache Airflow Linux CI image:
- `DagRun.conf` persistence and number-type regression;
- Execution API serialized transport;
- null-conf compatibility;
- public `order_by=conf`;
- Task SDK out-of-range integer string transport;
- task-runner materialization;
- foreign SDK downgrade conversion;
- Ruff for every changed Python file;
- Execution API and supervisor schema-version checks;
- migration anti-pattern check;
- `git diff --check`.
## Compatibility notes
- Existing JSONB rows keep their already-normalized values during migration;
lost exponent semantics cannot be reconstructed.
- Older Task SDK clients receive a dictionary through Cadwyn and retain the
old memory/MessagePack behavior.
- Public ORM/API consumers still receive a dictionary.
- PostgreSQL JSON query/index implications need maintainer review.
Related: #74025, #74023, #73712, #74048, #73783, #74093
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — GitHub Copilot (implementation support, test generation, and PR
description drafting). The design and changes were reviewed and validated
by
the contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
