seanmuth opened a new pull request, #69832:
URL: https://github.com/apache/airflow/pull/69832
## Overview
`get_dag_structure` (`GET /ui/grid/structure/{dag_id}`) streams historical
`SerializedDagModel` rows with a `yield_per` server-side cursor while
performing CPU-bound `serdag.dag` deserialization and task-group merging
*between* fetches. Because the cursor stays open across that work, the read
transaction — and, under PgBouncer transaction pooling, the pooled server
connection — stays pinned for the entire render of a large DAG.
For DAGs with many tasks and many historical versions this holds a
connection open for a long time (multiple minutes in the field). At scale that
exhausts the PgBouncer pool and starves task-instance heartbeat updates,
contributing to the metadata-DB contention tracked in #65712.
## Change
Materialize the (page-bounded) set of historical serialized DAGs, detach
them from the session, and commit the read transaction so the connection
returns to the pool **before** the deserialization/merge loop runs.
`SerializedDagModel.dag` only reads the already-loaded `data` column, so
deserialization works on detached instances with no further DB access. The
result set is bounded by this endpoint's page of runs (a single DAG's
versions), so materializing it is a safe trade for not holding the connection
open across CPU-bound work.
This mirrors the per-unit session-release pattern already applied to the
streaming `ti_summaries` endpoint in #65010.
## Related
- Contributes to #65712
## Testing
- Existing grid `structure` endpoint tests pass, including
`test_structure_includes_historical_removed_task_with_proper_shape`, which
exercises the historical-versions loop that was changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code) — Claude Opus
4.8 (1M context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
