ephraimbuddy opened a new pull request, #71621:
URL: https://github.com/apache/airflow/pull/71621
The Dag processor's manager persists parse results one file at a time, and
on small deployments that process is the CPU bottleneck. Two costs were paid
there for every parsed file.
Import errors were read by loading the whole import_error table, even for
files that had none. The lookup is now scoped to the keys being written. The
table has no index on (bundle_name, filename), so this saves materialising
every row rather than the scan itself.
The serialized Dag was hashed twice per write, once for change detection and
again in the model constructor, and hashing re-sorts and re-encodes the whole
structure.
The compressed-storage test parametrization never ran, because the module
binds that flag at import. It is made real here, since this change moves code
into that branch.
From @seanmuth's investigation of the Airflow 2-to-3 dag-processor CPU
regression: github.com/seanmuth/af2-af3-scheduler-cpu-repro. This is two of the
smaller levers; the per-sweep result batching is left for follow-up.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes (please specify the tool below)
Generated-by: Claude Opus 5 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
