potiuk opened a new pull request, #75996: URL: https://github.com/apache/airflow/pull/75996
Several workflows read, modify and write back state that other runs share. When two of them overlap, one run's result is silently lost. - **Provider registry.** An incremental registry build downloads the bucket's `providers.json` and `modules.json`, merges in its own providers and syncs the files back. Docs publishes trigger it, release managers fire subset and fix-up publishes, and the backfill writes to the same bucket. Two overlapping runs drop whichever one finished first from the index. The registry publish job and the backfill's `publish-versions` job now share `registry-publish-<destination>`. The group is set on the job because `registry-build.yml` is also called from `publish-docs-to-s3.yml`. - **Docs publishes.** Provider docs and Airflow docs land in different paths, so their syncs keep running side by side, with one group per bucket and kind (`publish-docs-<destination>-<providers|airflow>`). `manifest/packages-metadata.json` is different: it is rebuilt from a scan of the whole bucket, so both kinds share it. Rebuilding it now runs in a separate job, `update-packages-metadata`, with one group per bucket. That job runs only when the sync step succeeded, so a schema-publish failure after a complete sync still updates the metadata, and a partial sync is never advertised. The new `publish-docs-to-s3` flags `--skip-packages-metadata` and `--packages-metadata-only` make this split possible. - **Constraints.** `release-constraints.yml`, `update-constraints-on-push.yml` and `update-constraints-on-push-stable.yml` push to the same `constraints-X-Y` branch. Their publish jobs now share `constraints-publish-<branch>`, so a release tag push and a refresh no longer race into a non-fast-forward. The stable workflow runs from the `v*-stable` branches, so it takes effect there only after a backport. - **Docker Hub release.** The group was the bare version string, so any other workflow that used a version as its group would collide with it. It now has a `release-dockerhub-image-` prefix. All new groups use `cancel-in-progress: false`. ### Known limitations - **GitHub keeps only one pending run per group.** A third run arriving while one is running and one is waiting cancels the waiting one. That run shows as cancelled and has to be re-run, which is still better than the silent loss it replaces. The metadata job does not have this problem, because the surviving run rescans the whole bucket. - **A mixed docs publish is grouped as `airflow`.** For example, `include-docs` set to `all` or to `apache-airflow amazon` gets the `airflow` group, but it also writes provider paths. Such a run can still overlap a provider-only publish of the same provider. Covering this would mean splitting the publish into separately grouped provider and non-provider jobs. Mixed publishes are rare manual runs, so this PR does not do that. ### Checks run - `uv run --project dev/breeze pytest dev/breeze/tests/test_publish_docs_to_s3.py dev/breeze/tests/test_release_management_commands.py -k publish_docs_to_s3`: passed. The new tests cover skipping the metadata during a sync, the metadata-only mode, and option validation. - `breeze setup regenerate-command-images --check-only` and `breeze setup check-all-params-in-groups`: passed. - `prek run --from-ref main --stage pre-commit`: passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
