github-actions[bot] opened a new pull request, #71626:
URL: https://github.com/apache/airflow/pull/71626
* Release DB connection before deserializing DAGs in grid structure endpoint
`get_dag_structure` (`GET /ui/grid/structure/{dag_id}`) streamed historical
`SerializedDagModel` rows with a `yield_per` server-side cursor while running
CPU-bound `serdag.dag` deserialization and task-group merging between
fetches.
That keeps the read transaction — and, under PgBouncer transaction pooling,
the
pooled server connection — pinned for the entire render of a large DAG.
At scale this holds connections open for minutes, exhausting the PgBouncer
pool
and starving task-instance heartbeats, contributing to the contention
tracked in
apache/airflow#65712.
Materialize the (page-bounded) historical serialized DAGs, detach them, and
commit the read transaction so the connection returns to the pool *before*
the
deserialization/merge. `SerializedDagModel.dag` only reads the already-loaded
`data` column, so it works on detached instances.
This mirrors the per-unit session-release pattern already applied to the
streaming `ti_summaries` endpoint in apache/airflow#65010.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Add newsfragment for #69832
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Batch historical serdag loading to keep memory bounded
Address review: instead of materializing all historical SerializedDagModel
rows at once (which regressed the yield_per=5 memory profile), fetch their
ids
and process them in batches of 5, each batch loaded and detached in its own
short-lived session that is closed before the batch is deserialized.
This keeps peak memory bounded to one batch (matching the previous behaviour)
while still releasing the DB connection during the CPU-bound deserialization.
Also drop the newsfragment per review.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Address review: trim comment, drop unnecessary order_by, use
session.close()
- Comment above serdag_id_query was too verbose for a simple query; the
rationale belongs in the commit message. Trimmed per dstandish's
suggestion.
- Dropped .order_by(SerializedDagModel.id): the original code had no explicit
ordering, and _merge_node_dicts is sensitive to processing order for nodes
that differ across historical Dag versions, so this restores exact prior
behavior instead of introducing an unreviewed ordering guarantee.
- Replaced session.commit() with session.close(): this session has done no
writes (pure read), so close() is the more accurate call and, per review,
reads clearer while releasing the connection the same way.
---------
(cherry picked from commit b278b6227369a37763970aa2eb6b7364d719d0bd)
Co-authored-by: Sean Muth <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
