ephraimbuddy opened a new pull request, #75805: URL: https://github.com/apache/airflow/pull/75805
Depends on https://github.com/apache/airflow/pull/75502, which depends on https://github.com/apache/airflow/pull/75498. This branch builds on that stack and should merge after both PRs; the current diff includes their changes. Allow `ManifestLocalDagBundle` to publish a shared local source directory automatically when `source_path` is configured. Deployments can update that directory while Dag runs continue to use immutable, content-addressed releases. Publication waits until source metadata remains unchanged for `source_stability_seconds`. Publishers coordinate the stability period through shared state and recheck it under the publication lock. Once a process confirms the current source, unchanged refreshes compare metadata without rehashing file contents. Empty sources are rejected by default, and an existing release remains available while the source is changing or publication fails. Before the first release is ready, the Dag processor retries initialization without blocking other bundles or consuming pending priority requests. Pinned versions continue to load their recorded snapshots. Documentation covers configuration, shared-filesystem and permission requirements, clock synchronization, and the limits of the stability check. Validation: - The focused bundle and Dag processor tests passed in Breeze on the final rebased branch: 494 passed, 6 skipped. All four new regression cases failed against the original implementation and passed with the fixes. - Regular and manual prek checks passed on the final rebased branch. - Both core suites selected by `breeze verify` passed across the API, Always, CLI, Core, Other, and Serialization groups: database tests had 12,045 passed, 77 skipped, 1 xfailed; non-database tests had 5,797 passed, 12 skipped, 2 xfailed. The non-database run used five xdist workers. - `breeze build-docs apache-airflow` passed. The broad suites and docs build ran before the final rebase over unrelated UI and provider commits. That rebase preserved all eleven patches; the changed Dag processing source, tests, and documentation are byte-identical to the tested version. Performance was measured in Breeze with five alternating runs per revision, comparing the dependency version with this change using 500 files of 4 KiB each on container-local storage. Medians: | Operation | Dependency version | This change | | --- | ---: | ---: | | Unchanged manual refresh, warm cache | 56.349 ms | 57.383 ms | | Explicit publication of a new snapshot | 1571.968 ms | 1107.471 ms | | Unchanged metadata-walk control | 11.924 ms | 11.147 ms | The control moved by 6.5%; the manual refresh difference was 1.8%. Automatic unchanged-source refresh measured 68.036 ms, adding 10.653 ms over manual mode for the opt-in source checks. These measurements cover local storage and warm refreshes; they do not establish performance on network filesystems or with concurrent publishers. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) and OpenAI Codex (GPT-6) Generated-by: Claude Code (Opus 5.5) and OpenAI Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
