kjh0623 opened a new pull request, #70003: URL: https://github.com/apache/airflow/pull/70003
`GitDagBundle` fetches the repository itself: every refresh runs `git fetch` and resolves the bundle's Airflow Connection inside the dag processor. For large repositories that are parsed frequently, this puts git traffic, latency and credentials in the parsing path — and pairing Airflow 3 with the Kubernetes [git-sync](https://github.com/kubernetes/git-sync) sidecar (the standard pattern many Airflow 2 helm-chart deployments migrate from) currently requires writing a custom bundle class, since `LocalDagBundle` loses the symlink-swap consistency and git-sync awareness. **What this adds** A new read-only `GitSyncDagBundle` in the git provider for checkouts maintained by an external synchronization process: - No `git` command and no Connection resolution ever happens in Airflow — the external process (e.g. the git-sync sidecar) owns pulling; Airflow only reads files. - git-sync publishes each synced commit as a separate worktree and atomically swaps a symlink. The bundle resolves that symlink on each `refresh()` and pins the resolved worktree until the next refresh, so a sync happening mid-parse cannot mix files from two different commits into one parsing round (docs recommend git-sync's `--stale-worktree-timeout` so the previous worktree stays readable while a parse that pinned it is in flight). - `supports_versioning = False` (the checkout only ever contains the current state); a `version` kwarg is rejected with a clear error. - Plain directories (no symlink indirection) also work, for other sync mechanisms. **Production context** We run this pattern in production over a ~3,600-DAG repository after `GitDagBundle`'s built-in fetch contended with parsing load: git-sync is pull-only, the dag processor reads the synced worktree, and DAG changes are picked up in seconds without any clone/fetch overhead in Airflow. This PR upstreams that (previously internal) bundle class. **Included** - `airflow.providers.git.bundles.gitsync.GitSyncDagBundle` + provider.yaml registration - Unit tests covering symlink pinning across mid-parse swaps, refresh repointing, subdir, plain-directory fallback, missing-path and version-kwarg errors - Documentation section in the git provider bundles page with a config example and operational notes (git-sync flags, worker/triggerer access to the synced path) <!-- SPDX-License-Identifier: Apache-2.0 https://www.apache.org/licenses/LICENSE-2.0 --> --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes (please specify the tool below) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
