nvidyasagar0912 opened a new pull request, #75935:
URL: https://github.com/apache/airflow/pull/75935
## Summary
Reduce unnecessary Python object materialization when tasks start with large,
nested `DagRun.conf` values.
The Execution API reads persisted configuration as JSON text. Python
supervisors
forward that text without constructing the nested dictionary; the task
process
restores `context["dag_run"].conf` before parameters and template context
are built.
The task-visible dictionary/null contract is retained.
**This is a draft for review.** Local measurements show lower API-server and
worker peaks, but **do not establish a lower full-stack peak**. The
common-case
database/transport costs and remaining validation are explicit below.
Quick historical findings: API own peak **24.1% lower**, worker own peak
**9.8% lower**, **no demonstrated aggregate RAM reduction**, and one extra
asset-collection query per task start. These are local synthetic
observations,
not production guarantees.
Related: #74025. Related prior draft: #74392. This draft provides a separate
implementation and an openly inspectable validation record.
## Why this change
Large JSON configuration is much larger as a Python object graph than as
stored
JSON. In the existing task-start path, the API server materializes it,
serializes
it for HTTP, and the supervisor materializes it again before encoding startup
details for the child process.
The API server and Python supervisor do not need to examine those nested
values
to authorize/start the task. The task itself still needs the dictionary.
```text
Before:
JSONB -> API dictionary -> HTTP JSON -> supervisor dictionary
-> MessagePack objects -> task dictionary
With this change, for Python task startup:
JSON text -> HTTP string -> supervisor string -> MessagePack string
-> one task-side JSON decode -> task dictionary
```
## Implementation and compatibility
- Add optional `TIRunContext.serialized_dag_run_conf`, without changing
ordinary
DagRun detail endpoints.
- Retrieve JSON text with a route-local projection and defer the ORM `conf`
column. Do not mutate the attached ORM object's configuration.
- Load consumed asset events separately to avoid repeating the large text for
every joined event row. Preserve team and metric-tag eager loading.
- Validate the JSON root in the database before committing the task-start
transaction. Invalid persisted roots return 422 instead of the previous
in-route 500; rollback is preserved, including for older negotiated
versions.
- Restore and cache dictionary/null configuration in the Python task runtime,
clear the serialized field, and preserve mutable dictionary identity.
- Cache JSON object keys rather than unique manifest values in the task
decoder.
- Preserve the compact legacy environment envelope for `run_as_user` re-exec.
- Convert older Execution API responses back to dictionary configuration.
- Restore the dictionary and strip the new field for pinned foreign-runtime
startup messages, including the in-progress supervisor version.
- Accumulate changes in the existing in-progress API/IPC versions and
regenerate
Task SDK models and the supervisor schema with the official tools.
- Log task identifiers rather than complete configuration-bearing startup
models.
No scheduler optimization, database migration, compression codec, claim-check
storage or direct worker database access is introduced.
## Validation on the publication candidate
Source baseline: `6056deb572e1e8fd5a51654ad0a22c963e5a6a69`.
The publication branch was subsequently rebased onto
`ddb89d5e006ebabea298dd22396b73ee0a0e463c`. All candidate files and
application
production code were unchanged by that rebase; upstream test infrastructure
and unrelated tests changed.
| Test layer | Result |
| --- | ---: |
| Task SDK conf/IPC/migration/context/impersonation regression selections |
442 passed |
| SQLite Execution API startup and transport | 96 passed |
| PostgreSQL startup, transport and older API versions | 121 passed |
| MySQL startup, transport and older API versions | 121 passed |
| **Total** | **780 passed; 0 failures, 0 errors** |
Coverage includes explicit SQL NULL separately from JSON null, empty/nested
configuration, four negotiated API versions, invalid-root rollback,
parameter-override settings, mutable identity, Python/foreign startup wire
shapes,
and impersonation envelope round trips.
Also executed:
- Official SDK-model and supervisor-schema generation using the locked
`datamodel-code-generator==0.83.0`.
- Ruff lint, formatting and Git whitespace checks.
- Targeted mypy on seven changed production modules.
**Not claimed passed:** the complete pre-commit/manual/CI suite.
`prek install` succeeded in the Linux validation checkout, but
hook-environment
installation was blocked by PyPI dependency-download timeouts. The draft is
published with that blocker disclosed; full hook execution remains pending.
## Real local end-to-end comparison
The performance data below is historical and has an exact revision boundary:
- Baseline: `750f849cc2372ee42dc6910541d4b6a866102ac9`.
- Candidate: that baseline plus the archived, exact tested patch.
- The performance comparison has **not** been rerun on the rebased
publication
head. Later changes are review cleanups and explicit SQL-NULL test
coverage.
- Same pinned dependency image, resources and workload in all arms.
The private Docker stack ran real API, scheduler, Dag processor, Celery
worker
and child tasks, triggerer, PostgreSQL and Redis. Runs were triggered
through the
public API, dispatched through Redis/Celery and started through the
authenticated
Execution API. Task output was checked through XCom.
| Scenario, per arm | DagRuns | Records per configuration |
| --- | ---: | ---: |
| Small | 4 | 20 |
| Medium | 8 | 2,000 |
| Large | 20 | 23,000, approximately 11.9 MB |
| Contract cases | 5 | Empty conf, false parameter, planned retry,
deferral/resume, planned negative hash |
Worker concurrency was **four**, not twenty executing child tasks.
Across baseline / candidate / baseline-again arms, **111 declared run
outcomes
were verified**: 108 successes and three intentionally failed negative cases.
Performance runs succeeded on the first attempt; complete payload hashes,
record counts, parameters and mutable dictionary behavior were checked.
### Large-workload results
The group peak is measured from the sum of all seven component working sets
at each sampling point, approximately every 200 ms. It is **not** a sum of
independently observed component maxima.
| Metric | Baseline A1 | Candidate | Baseline A2 |
| --- | ---: | ---: | ---: |
| Simultaneous group working-set peak, MiB | 5,091.7 | 5,031.3 | 4,795.5 |
| Completion time for 20 runs, seconds | 126.6 | 105.2 | 124.3 |
**No total-stack peak-memory reduction was demonstrated:** the candidate was
1.8% above the mean of the two baselines, inside their 6.0% spread.
Completion was 16.1% shorter than the baseline mean in this single candidate
observation, not a statistically established production result.
Individual component maxima, which can occur at different times:
| Component | Baseline mean own peak, MiB | Candidate own peak, MiB |
Observed change |
| --- | ---: | ---: | ---: |
| API server | 915.2 | 694.4 | 24.1% lower |
| Worker, including supervisors and task processes | 1,610.9 | 1,453.2 |
9.8% lower |
| Scheduler | 2,247.1 | 2,248.7 | Essentially unchanged |
| Dag processor | 250.9 | 250.6 | Essentially unchanged |
| Triggerer | 451.8 | 450.7 | Essentially unchanged |
| PostgreSQL | 513.8 | 695.7 | 35.4% higher; cause not established |
| Redis | 7.5 | 6.9 | Small absolute difference |
### Startup-readiness qualification
Each main arm had 16 recovered task-start HTTP 400 responses after
intentional
API restarts. These are retained in the logs, not presented as clean
requests.
An additional unchanged-baseline control completed four cold and four
execution-health-warmed runs: cold startup reproduced four HTTP 400s; warming
`/execution/health` produced zero. This supports a readiness/cold-start
effect,
but does not establish the precise transport failure mechanism. The full
large
comparison was not rerun with the warm-up.
The API was deliberately restarted after public DagRun admission and before
execution sampling in every scenario. Admission time was recorded separately;
the measurement isolates scheduling/startup/execution, not the public trigger
endpoint's large-body allocations.
## Reproduction and raw data
Supporting material is on a **separate, fork-only evidence branch**, not in
this upstream source diff:
- [Readable report, test protocol, raw results and reproduction
harness](https://github.com/nvidyasagar0912/airflow/tree/8748c0475df7ad649013b9b0bda13b216f7ac7d9)
- [Current-candidate JUnit reports and exact validation
selectors](https://github.com/nvidyasagar0912/airflow/tree/8748c0475df7ad649013b9b0bda13b216f7ac7d9/current-head-validation)
- [Time-aligned memory chart and machine-readable
comparison](https://github.com/nvidyasagar0912/airflow/tree/8748c0475df7ad649013b9b0bda13b216f7ac7d9/e2e-20261028)
The evidence records the source/patch hashes, image digest, resource limits,
synthetic payload definition, raw memory/CPU samples, state transitions,
service logs, expected failures and cleanup. The historical tests are local
synthetic Airflow tests, not cloud-cluster or external ingestion/search
tests.
## Trade-offs and work remaining before ready-for-review
- [ ] Complete the required three alternating baseline and three candidate
runs,
with an unaffected control, health warm-up and median comparisons.
- [ ] Measure empty/small, legacy-client and foreign-runtime costs, including
HTTP body/compressed-transfer bytes and MessagePack frame sizes.
- [ ] Quantify the extra asset-collection query and native JSON-type/cast
work.
The PostgreSQL peak increase is an observation, not an attributed
cause.
- [ ] Complete boundary numeric parity through database/API/IPC,
minimum-supported
SQLite non-finite handling and the remaining converter/log-canary
checks.
- [ ] Complete full pre-commit/manual checks and the commands selected by
`breeze verify`; fix CI findings before marking this ready.
- [ ] Confirm full in-process/real re-exec/old-language-runtime
interoperability
beyond the tested context and migration selections.
Same-epoch mixed development snapshots are not a supported compatibility
boundary; use matching core/SDK development sources. Released older API
responses retain dictionary configuration through conversion.
The intended claim is reduced **API/Python-supervisor materialization and
component pressure**, not a proven overall RAM reduction, universal speedup
or
regression-free production rollout.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — GitHub Copilot SDK in VS Code (implementation, tests, analysis
and drafting).
Generated-by: GitHub Copilot SDK in VS Code following
[the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions).
---
Drafted-by: GitHub Copilot SDK in VS Code (no human line-by-line review
before posting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
