The GitHub Actions job "npm_and_yarn in /ts-sdk - Update #1525727542" on airflow.git/main has failed. Run started by GitHub user dependabot[bot] (triggered by dependabot[bot]).
Head commit for run: b278b6227369a37763970aa2eb6b7364d719d0bd / Sean Muth <[email protected]> Release DB connection before deserializing DAGs in grid structure endpoint (#69832) * Release DB connection before deserializing DAGs in grid structure endpoint `get_dag_structure` (`GET /ui/grid/structure/{dag_id}`) streamed historical `SerializedDagModel` rows with a `yield_per` server-side cursor while running CPU-bound `serdag.dag` deserialization and task-group merging between fetches. That keeps the read transaction — and, under PgBouncer transaction pooling, the pooled server connection — pinned for the entire render of a large DAG. At scale this holds connections open for minutes, exhausting the PgBouncer pool and starving task-instance heartbeats, contributing to the contention tracked in apache/airflow#65712. Materialize the (page-bounded) historical serialized DAGs, detach them, and commit the read transaction so the connection returns to the pool *before* the deserialization/merge. `SerializedDagModel.dag` only reads the already-loaded `data` column, so it works on detached instances. This mirrors the per-unit session-release pattern already applied to the streaming `ti_summaries` endpoint in apache/airflow#65010. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Add newsfragment for #69832 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Batch historical serdag loading to keep memory bounded Address review: instead of materializing all historical SerializedDagModel rows at once (which regressed the yield_per=5 memory profile), fetch their ids and process them in batches of 5, each batch loaded and detached in its own short-lived session that is closed before the batch is deserialized. This keeps peak memory bounded to one batch (matching the previous behaviour) while still releasing the DB connection during the CPU-bound deserialization. Also drop the newsfragment per review. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Address review: trim comment, drop unnecessary order_by, use session.close() - Comment above serdag_id_query was too verbose for a simple query; the rationale belongs in the commit message. Trimmed per dstandish's suggestion. - Dropped .order_by(SerializedDagModel.id): the original code had no explicit ordering, and _merge_node_dicts is sensitive to processing order for nodes that differ across historical Dag versions, so this restores exact prior behavior instead of introducing an unreviewed ordering guarantee. - Replaced session.commit() with session.close(): this session has done no writes (pure read), so close() is the more accurate call and, per review, reads clearer while releasing the connection the same way. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Report URL: https://github.com/apache/airflow/actions/runs/31821451014 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
