fat-catTW opened a new pull request, #71331:
URL: https://github.com/apache/airflow/pull/71331
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Why
`SchedulerJobRunner._create_dag_runs()` had a TODO asking whether the
scheduler should call `session.flush()` or `session.expunge_all()` while
creating scheduled DagRuns.
The concern behind the TODO is that the scheduler may iterate over multiple
Dags, create scheduled DagRuns, update `DagModel` scheduling fields, and create
task instance rows in the same SQLAlchemy session. For larger Dags or larger
batches of Dags, leaving ORM state pending until the outer transaction boundary
can make it harder to reason about session growth and memory behavior.
This PR resolves that TODO by flushing after each successful scheduled
DagRun creation and related `DagModel` scheduling-state update.
## Why `flush()` Instead of `expunge_all()`
The narrower operation is enough for the state observed in this path.
`session.flush()` writes pending ORM changes to the database transaction
while keeping objects attached to the session. That directly addresses the
observed pending/dirty scheduler state after creating a scheduled DagRun and
updating the corresponding `DagModel`.
`session.expunge_all()` does something broader: it detaches every ORM object
from the session. That can reduce identity-map retention, but it also carries
more behavioral risk because later scheduler code may still expect ORM objects
to be attached, refreshable, or tracked by the session.
To decide whether `expunge_all()` was needed, this path was checked for
retained ORM objects after DagRun creation. If creating larger Dags left one
tracked `DagRun` or `TaskInstance` object per created row, `flush()` would not
address that memory growth because it does not clear the session identity map.
A local pressure check with 20 Dags and 500 tasks per Dag created 10,000
TaskInstance rows. After `_create_dag_runs()` returned, the observed session
state was:
```text
session.new: 0
session.dirty: 0
identity_map: Counter({'DagModel': 20, 'DagVersion': 1,
'SerializedDagModel': 1, 'DagCode': 1})
```
<img width="728" height="122" alt="螢幕擷取畫面 2026-08-08 204423"
src="https://github.com/user-attachments/assets/d154a3f9-27fd-441c-9072-ee57cf6a32d9";
/>
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- `[X]` Yes (please specify the tool below)
Generated-by: [Codex] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
