kjh0623 opened a new pull request, #70058:
URL: https://github.com/apache/airflow/pull/70058
**What this adds**
An opt-in `refresh_on_initialize` kwarg (default `True`, no behavior change
for existing users) on `GitDagBundle`, for deployments where many ephemeral
processes initialize the same bundle over shared storage:
```
GitDagBundle(tracking_ref="main", refresh_on_initialize=False)
```
Today, unversioned `initialize()` always ends in a remote fetch. That is the
right default for a dag processor keeping the bundle fresh, but with
KubernetesExecutor every worker pod initializing the bundle pays that fetch on
startup — for our repository this adds ~15s of cold-start to every single task,
even though the shared volume already holds a tracking repo the dag processor
refreshes continuously. Shallow/sparse clones shave the cost down but every pod
still pays a network round-trip.
With `refresh_on_initialize=False`, `initialize()` reuses the tracking repo
already on disk as-is and skips all network operations. Scope guards:
- the repository is still cloned (and fetched) when nothing is on disk yet,
- explicit `refresh()` calls still fetch as usual (both repo handles are
assigned in the reuse path),
- version-pinned initialization is unaffected (it has its own reuse paths
already),
- any failure reusing the on-disk repo falls back to the full initialization
path.
This generalizes what we currently run in production as a small
`GitDagBundle` subclass (workers reuse the PVC-resident tracking repo, only the
dag processor fetches). Related context on the worker cold-start motivation was
discussed in #70003.
**Included**
- `refresh_on_initialize` kwarg + docstring
- Unit tests: reuse path serves the on-disk state and skips the fetch while
explicit `refresh()` still fetches; default behavior unchanged (second
initialize fetches latest)
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
