This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 115eb43aba8 Build docs and registry CI images separately so docs stay
on default Python (#70959)
115eb43aba8 is described below
commit 115eb43aba8b7989be377e682f6cdb2aedb631a0
Author: Jarek Potiuk <[email protected]>
AuthorDate: Mon Aug 3 21:17:44 2026 +0200
Build docs and registry CI images separately so docs stay on default Python
(#70959)
* Build docs and registry CI images separately so docs stay on default
Python
The docs build and the provider registry shared a single CI image, so they
also
shared its Python. That image is built at 3.12 because the registry has
always
extracted on 3.12, which silently moved the docs build there too and broke
publishing on 2026-07-28: Sphinx mocks third-party modules while building
provider docs, and from 3.12 functools copies __type_params__, for which a
mock
hands back another mock rather than a tuple. Any provider decorating methods
with functools.wraps over a mocked callable then fails to import, and the
build
also began reporting cross-reference ambiguities it had not reported before.
The image job already matrixes over the Python versions it is given, and
every
cache it touches - registry buildx tag, image stash, commit marker and mount
cache - is keyed by Python version. Asking it for both versions therefore
builds
them in parallel, each seeding and storing its own cache, and needs no
second
job. Docs go back to the default Python and the registry keeps 3.12.
The stashes are also named for what they are rather than for the ref alone,
so
release images stop sharing a key with anything else that stashes an image
for
the same ref.
* Build each CI image in the job that uses it instead of a shared job
Docs and the registry now want different Pythons, and a job that builds an
image
for another job has to hand it over as a multi-gigabyte tarball - exported,
stashed, restored and imported - for an image whose only consumer is a
single
downstream job. Building in place skips that round trip entirely, and the
two
builds still run in parallel because both jobs now wait only on build-info.
What is worth keeping between runs is the BuildKit mount cache rather than
the
image, so the docs job restores and stashes it under this ref and its own
Python. The registry side already knew how to build its own image when the
caller had not stashed one; it just could not be told which ref to build,
so it
would have documented main rather than the tag being published.
* Pin the docs build to the default Python again
`breeze build-docs` gained a --python option, and because that option reads
PYTHON_MAJOR_MINOR_VERSION, which every job of the docs publishing workflow
sets, the docs quietly started building on whatever the workflow happened to
export rather than on the Python they are meant for. That is how they ended
up
on 3.12, where Sphinx's mocking of third-party modules makes functools.wraps
raise and providers fail to import.
Which interpreter documents Airflow is not a per-invocation choice, so the
option is gone and the build pins itself to the default again. The workflow
reads the same constant instead of naming a version of its own, so the
image it
prepares cannot drift away from the one the docs build asks for.
* Name the docs publishing stashes after the workflow that writes them
The prefix exists to keep these images from sharing a key with anything else
stashing an image for the same ref, so it should say which workflow put them
there rather than describe them as release artifacts.
* Build the CI image in the job that uses it, cache and all
Each image now has a single consumer, so a job that only builds one can do
no
more than export it and have the consumer import it straight back. Both
docs and
the registry build in place instead, which leaves the registry workflow
with one
job as well.
What that job hands to the next run is unchanged: the image, the commit it
was
built from and the mount cache are stashed under the ref, so a later run
for the
same ref seeds its build from them and skips the build entirely when the
sources
have not moved. That is the whole reason the stash exists - publishing an
RC and
then the final docs is the same ref twice - so it moves into the jobs with
the
build rather than being dropped along with the job that used to do it.
The logic lives in a composite action so both jobs share one copy;
ci-image-build.yml stays for callers that do want an image job of their own.
* Balance the registry job condition after dropping the image dependency
Removing the build-ci-image guard left the parenthesis that opened its
group, so
GitHub refused to parse the workflow and the dispatch failed before any job
ran.
* Wait for the registry before reporting the docs publish done
The registry still runs alongside the docs build - nothing it does depends
on
them - but the publish now waits for it, so a green run means both halves
of a
publish are actually out rather than only the S3 upload. Its result is
checked
explicitly because the registry is skipped for non-provider distributions,
and a
skipped dependency would otherwise skip the publish with it.
* Read the image-build action from the workflow's own version
A local action is resolved against the workspace, not against the ref the
workflow definition came from, so publishing a tag cut before the action
existed
found nothing to run and both image builds failed immediately. The
workflow's
own version is now checked out alongside the sources being published and the
action is taken from there, which is what lets a workflow branch be tested
against an older tag at all. The image itself is still built from the
checked
out sources.
---
.../actions/build_ci_image_with_cache/action.yml | 221 +++++++++++++++++++++
.github/workflows/publish-docs-to-s3.yml | 143 ++++++-------
.github/workflows/registry-build.yml | 96 ++++-----
dev/breeze/doc/images/output_build-docs.svg | 44 ++--
dev/breeze/doc/images/output_build-docs.txt | 2 +-
.../airflow_breeze/commands/developer_commands.py | 13 +-
.../commands/developer_commands_config.py | 1 -
dev/breeze/tests/test_developer_commands.py | 33 +--
8 files changed, 380 insertions(+), 173 deletions(-)
diff --git a/.github/actions/build_ci_image_with_cache/action.yml
b/.github/actions/build_ci_image_with_cache/action.yml
new file mode 100644
index 00000000000..3309891832d
--- /dev/null
+++ b/.github/actions/build_ci_image_with_cache/action.yml
@@ -0,0 +1,221 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Build CI image with cache'
+description: >
+ Builds the CI image in the job that uses it, seeding the build from what a
previous run stashed
+ for the same ref and stashing the result for the next one. Exists so a job
needing an image does
+ not have to take it from another job as a multi-gigabyte artifact;
ci-image-build.yml remains the
+ reusable-workflow form for callers that do want a job of their own.
+inputs:
+ python:
+ description: 'Python version to build the image for'
+ required: true
+ platform:
+ description: 'Platform to build for'
+ default: 'linux/amd64'
+ image-stash-ref:
+ description: >
+ Discriminator the image, its commit marker and the mount cache are
stashed under. Scoped to
+ the ref rather than the branch: these images are built from a ref that
is not the branch tip,
+ so the shared per-branch stash would hand the next reader sources it
never asked for.
+ required: true
+ github-token:
+ description: 'Token used to log in to ghcr.io and read the registry build
cache'
+ required: true
+ constraints-github-repository:
+ description: 'Repository the constraints are taken from'
+ default: 'apache/airflow'
+runs:
+ using: "composite"
+ steps:
+ # A stashed image is only an answer when the sources have not moved since;
otherwise it is
+ # cache. The commit it was built from tells the two apart and is stashed
on its own, so
+ # deciding costs a few bytes rather than the image the decision may make
unnecessary.
+ - name: "Restore the commit the stashed CI image was built from"
+ uses:
apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-image-commit-v3-${{ inputs.platform }}-${{ inputs.python }}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/mnt/"
+ only-current-branch: 'true'
+ id: restore-commit
+ - name: "Check whether the stashed CI image was built from this commit"
+ id: stashed-image
+ env:
+ COMMIT_FILE: "/mnt/ci-image-commit-${{ inputs.python }}.txt"
+ shell: bash
+ run: |
+ head_sha="$(git rev-parse HEAD)"
+ stashed_sha="$(cat "${COMMIT_FILE}" 2>/dev/null || true)"
+ if [[ "${stashed_sha}" == "${head_sha}" ]]; then
+ echo "The stashed image was built from ${head_sha} - reusing it,
without a build."
+ echo "reusable=true" >> "${GITHUB_OUTPUT}"
+ else
+ echo "The stashed image was built from '${stashed_sha:-unknown}',
not ${head_sha}."
+ echo "It can only seed the build cache."
+ fi
+ if: steps.restore-commit.outputs.stash-hit == 'true'
+ # Restored ahead of the caches that feed the build, so a build made
unnecessary skips them too.
+ - name: "Restore the CI image stashed for this ref"
+ uses:
apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-image-save-v3-${{ inputs.platform }}-${{ inputs.python }}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/mnt/"
+ only-current-branch: 'true'
+ id: restore-image
+ - name: "Load the stashed CI image as the image to use"
+ shell: bash
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ run: breeze ci-image load --platform "${PLATFORM}" --python "${PYTHON}"
--image-file-dir "/mnt"
+ if: steps.stashed-image.outputs.reusable == 'true'
+ # BuildKit reads `--cache-from` from a registry, never from the local
engine, so the restored
+ # image has to be served from one to contribute anything.
+ - name: "Serve the stashed CI image as build cache"
+ id: serve-cache
+ shell: bash
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ CACHE_IMAGE: "localhost:5000/ci-image-cache:${{ inputs.python }}"
+ run: |
+ docker run -d --name cache-registry -p 5000:5000 registry:2
+ breeze ci-image load --platform "${PLATFORM}" --python "${PYTHON}" \
+ --image-file-dir "/mnt" --tag-as "${CACHE_IMAGE}"
+ docker push "${CACHE_IMAGE}"
+ echo "cache-from-image=${CACHE_IMAGE}" >> "${GITHUB_OUTPUT}"
+ if: >
+ steps.stashed-image.outputs.reusable != 'true' &&
+ steps.restore-image.outputs.stash-hit == 'true'
+ # Scoped to the ref for the same reason the image is: the mount cache
holds the dependency set
+ # the sources resolve to, and a ref's and the branch tip's are exactly
what differ.
+ - name: "Restore the mount cache stashed for this ref"
+ uses:
apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-cache-mount-save-v3-${{ inputs.platform }}-${{ inputs.python
}}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/tmp/"
+ id: restore-mount-cache
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Import the mount cache"
+ shell: bash
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ inputs.python }}
+ CACHE_FILE: "/tmp/ci-cache-mount-save-v3-${{ inputs.python }}.tar.gz"
+ run: |
+ if [[ ! -f "${CACHE_FILE}" ]]; then
+ echo "${CACHE_FILE} is missing - the stash restore may have timed
out. Building without it."
+ exit 0
+ fi
+ breeze ci-image import-mount-cache --cache-file "${CACHE_FILE}"
+ if: >
+ steps.stashed-image.outputs.reusable != 'true' &&
+ steps.restore-mount-cache.outputs.stash-hit == 'true'
+ - name: "Login to ghcr.io"
+ shell: bash
+ env:
+ GITHUB_TOKEN: ${{ inputs.github-token }}
+ ACTOR: ${{ github.actor }}
+ run: echo "${GITHUB_TOKEN}" | docker login ghcr.io -u "${ACTOR}"
--password-stdin
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Build the CI image"
+ shell: bash
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ inputs.python }}
+ DOCKER_CACHE: "registry"
+ DISABLE_AIRFLOW_REPO_CACHE: "false"
+ UPGRADE_TO_NEWER_DEPENDENCIES: "false"
+ CONSTRAINTS_GITHUB_REPOSITORY: ${{
inputs.constraints-github-repository }}
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ inputs.github-token }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ PUSH: "false"
+ VERBOSE: "true"
+ # Empty when nothing was stashed for this ref; breeze then builds off
the registry cache
+ # alone. BuildKit reads --cache-from from a registry, never from the
local engine.
+ CACHE_FROM_IMAGE: ${{ steps.serve-cache.outputs.cache-from-image }}
+ run: |
+ if breeze ci-image build --platform "${PLATFORM}"; then
+ exit 0
+ fi
+ # Reached when this ref's pyproject.toml has drifted far enough from
the branch the
+ # registry cache was built for that the cached layers cannot satisfy
it. Slow, but correct
+ # - and the case that matters here, since these refs are cut days or
weeks before main.
+ echo "Build with the registry cache failed - retrying with the cache
disabled."
+ DOCKER_CACHE="disabled" breeze ci-image build --platform "${PLATFORM}"
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Stop serving the stashed image as cache"
+ # The registry holds a second copy of a multi-gigabyte image and the
export below needs room.
+ shell: bash
+ env:
+ CACHE_FROM_IMAGE: ${{ steps.serve-cache.outputs.cache-from-image }}
+ run: |
+ docker rm -f cache-registry
+ docker rmi "${CACHE_FROM_IMAGE}"
+ if: always() && steps.serve-cache.outputs.cache-from-image != ''
+ - name: "Export the CI image for the next run"
+ shell: bash
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ COMMIT_FILE: "/mnt/ci-image-commit-${{ inputs.python }}.txt"
+ run: |
+ breeze ci-image save --platform "${PLATFORM}" --image-file-dir "/mnt"
+ git rev-parse HEAD > "${COMMIT_FILE}"
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Stash the CI image"
+ uses:
apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-image-save-v3-${{ inputs.platform }}-${{ inputs.python }}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/mnt/ci-image-save-*-${{ inputs.python }}.tar"
+ if-no-files-found: 'error'
+ # Read by the next run for this same ref - an RC and then the final,
days apart.
+ retention-days: '6'
+ if: steps.stashed-image.outputs.reusable != 'true'
+ # Saved last and with the image's retention, so that finding this commit
is enough to know the
+ # image it describes is there to be restored.
+ - name: "Stash the commit the CI image was built from"
+ uses:
apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-image-commit-v3-${{ inputs.platform }}-${{ inputs.python }}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/mnt/ci-image-commit-${{ inputs.python }}.txt"
+ if-no-files-found: 'error'
+ retention-days: '6'
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Export the mount cache for the next run"
+ shell: bash
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ inputs.python }}
+ run: >
+ breeze ci-image export-mount-cache
+ --cache-file
/tmp/ci-cache-mount-save-v3-${PYTHON_MAJOR_MINOR_VERSION}.tar.gz
+ if: steps.stashed-image.outputs.reusable != 'true'
+ - name: "Stash the mount cache"
+ uses:
apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ with:
+ key: "ci-cache-mount-save-v3-${{ inputs.platform }}-${{ inputs.python
}}\
+ -${{ inputs.image-stash-ref }}"
+ path: "/tmp/ci-cache-mount-save-v3-${{ inputs.python }}.tar.gz"
+ if-no-files-found: 'error'
+ retention-days: '6'
+ if: steps.stashed-image.outputs.reusable != 'true'
diff --git a/.github/workflows/publish-docs-to-s3.yml
b/.github/workflows/publish-docs-to-s3.yml
index f94760b64dd..1c613209794 100644
--- a/.github/workflows/publish-docs-to-s3.yml
+++ b/.github/workflows/publish-docs-to-s3.yml
@@ -98,9 +98,18 @@ jobs:
publish-supervisor-schema: ${{
steps.parameters.outputs.publish-supervisor-schema }}
# yamllint disable rule:line-length
skip-write-to-stable-folder: ${{ inputs.skip-write-to-stable-folder &&
'--skip-write-to-stable-folder' || '' }}
- # The docs build and the registry extraction share one CI image, so they
share its
- # Python too. 3.12 is what the registry has always extracted on.
- default-python-version: "3.12"
+ # Docs and the registry each build the image they need rather than
sharing one. Sharing
+ # meant sharing the Python too, which put the docs on 3.12 and broke
publishing: Sphinx
+ # mocks third-party modules, and from 3.12 functools copies
__type_params__, so a mocked
+ # decorator raises TypeError and providers fail to import. `breeze
build-docs` pins itself
+ # to the default Python, so this follows the same constant rather than
restating a version.
+ docs-python-version: ${{
steps.default_python.outputs.default-python-version }}
+ registry-python-version: "3.12"
+ # Discriminator for every stash this workflow writes. A bare ref would
collide with any
+ # other workflow stashing an image for the same ref, and the images here
are special: built
+ # from a release tag rather than a branch tip, and kept for days so the
next publish of that
+ # tag reuses them. Naming them after this workflow keeps them
recognisably ours.
+ image-stash-ref: "publish-docs-${{ inputs.ref }}"
registry-providers: ${{
steps.derive_registry_inputs.outputs.registry-providers }}
registry-full-build: ${{
steps.derive_registry_inputs.outputs.registry-full-build }}
if: contains(fromJSON('[
@@ -165,6 +174,21 @@ jobs:
exit 0
fi
python3 dev/registry/derive_wave_providers.py
+ - name: "Derive the default Python version"
+ id: default_python
+ shell: bash
+ # `breeze build-docs` always documents on breeze's default Python, so
the image this
+ # workflow builds for it has to be that same one. Reading the constant
keeps the two from
+ # drifting apart and silently making the docs job build an image
nothing then uses.
+ run: |
+ DEFAULT_PYTHON=$(python3 -c "
+ import re, pathlib
+ text =
pathlib.Path('dev/breeze/src/airflow_breeze/global_constants.py').read_text()
+ versions =
re.search(r'^ALL_PYTHON_MAJOR_MINOR_VERSIONS\s*=\s*\[(.*?)\]', text, re.M |
re.S)
+ print(re.findall(r'\"([0-9]+\.[0-9]+)\"', versions.group(1))[0])
+ ")
+ echo "Default Python version: '${DEFAULT_PYTHON}'"
+ echo "default-python-version=${DEFAULT_PYTHON}" >> "${GITHUB_OUTPUT}"
- name: "Input parameters summary"
shell: bash
id: parameters
@@ -228,46 +252,8 @@ jobs:
echo "publish-execution-api-schema=${PUBLISH_EXEC}" >>
${GITHUB_OUTPUT}
echo "publish-supervisor-schema=${PUBLISH_SUP}" >> ${GITHUB_OUTPUT}
- build-ci-image:
- name: "Build CI image"
- needs: [build-info]
- uses: ./.github/workflows/ci-image-build.yml
- permissions:
- contents: read
- packages: write
- with:
- runners: '["ubuntu-22.04"]'
- platform: "linux/amd64"
- # Built from the docs ref so the image matches the sources being
documented, and
- # cached against main's registry cache, which the regular Test workflow
keeps warm.
- checkout-ref: ${{ inputs.ref }}
- push-image: "false"
- upload-image-artifact: "true"
- # Leaves the BuildKit mount cache behind for the next publish of this
same ref;
- # ci-image-build.yml always restores it, so consecutive publishes stop
re-downloading
- # the whole dependency set.
- upload-mount-cache-artifact: "true"
- # Docs are published from refs cut days or weeks before main, and main's
registry cache
- # stops matching at the first Dockerfile.ci change made since the cut -
#70285 alone
- # costs a from-scratch Python build. The image the previous publish
stashed was built
- # from these very sources, so it is the cache main's cannot be.
Publishing the same ref
- # twice - an RC and then the final docs - skips the build outright.
- seed-cache-from-stashed-image: "true"
- image-stash-ref: ${{ inputs.ref }}
- python-versions: ${{ format('["{0}"]',
needs.build-info.outputs.default-python-version) }}
- branch: "main"
- constraints-branch: "constraints-main"
- use-uv: "true"
- upgrade-to-newer-dependencies: "false"
- docker-cache: "registry"
- disable-airflow-repo-cache: "false"
- # Building against main's cache can fail outright when the ref's
pyproject.toml has
- # diverged from main — the case the docs job's inline `breeze ci-image
build || docker
- # buildx build` fallback used to cover.
- retry-without-cache: "true"
-
build-docs:
- needs: [build-info, build-ci-image]
+ needs: [build-info]
timeout-minutes: 150
name: "Build documentation"
runs-on: ubuntu-latest
@@ -282,7 +268,7 @@ jobs:
VERBOSE: "true"
EXTRA_BUILD_OPTIONS: ${{ needs.build-info.outputs.extra-build-options }}
APPLY_COMMITS: ${{ inputs.apply-commits || '' }}
- PYTHON_MAJOR_MINOR_VERSION: "${{
needs.build-info.outputs.default-python-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{
needs.build-info.outputs.docs-python-version }}"
DOCKER_CACHE: "registry"
steps:
- name: "Cleanup repo"
@@ -313,6 +299,16 @@ jobs:
ref: ${{ inputs.ref }}
fetch-tags: true
fetch-depth: 0
+ # A local action is read from the workspace, not from the ref the
workflow itself runs from,
+ # so an action added after this tag was cut is simply absent. Checking
the workflow's own
+ # version out alongside supplies the definition; the image is still
built from the sources
+ # above. It has to come after the checkout that populates the root,
which cleans untracked
+ # siblings away.
+ - name: "Checkout the workflow version for its actions"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+ path: workflow-version
- name: "Apply patch commits if provided"
run: |
if [[ "${APPLY_COMMITS}" != "" ]]; then
@@ -332,26 +328,21 @@ jobs:
- name: "Install Breeze from the ${{ inputs.ref }} reference"
uses: ./.github/actions/breeze
with:
- python-version: "${{ needs.build-info.outputs.default-python-version
}}"
- # The image comes from the `build-ci-image` job, which builds it from
this same ref and
- # stashes it under the ref's own key - the shared per-branch stash holds
main's image,
- # not this ref's. The registry build restores the very same stash, so a
run builds it
- # once, and a run that had nothing to build restores what the previous
publish left.
- - name: >
- Restore CI docker image built for ref ${{ inputs.ref }}
- linux/amd64:${{ needs.build-info.outputs.default-python-version }}
- uses:
apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355
+ python-version: "${{ needs.build-info.outputs.docs-python-version }}"
+ # Built here rather than in a job of its own: a separate job could only
hand the image over
+ # as a multi-gigabyte artifact, for a single consumer. The image, the
commit it was built
+ # from and the mount cache are still stashed under this ref, so the next
publish of the same
+ # ref seeds its build from them - or skips the build outright when the
sources have not moved.
+ - name: "Build CI image linux/amd64:${{
needs.build-info.outputs.docs-python-version }}"
+ uses: ./workflow-version/.github/actions/build_ci_image_with_cache
with:
- key: "ci-image-save-v3-linux/amd64-\
- ${{ needs.build-info.outputs.default-python-version }}-${{
inputs.ref }}"
- path: "/mnt/"
- only-current-branch: 'true'
- fail-on-download: 'true'
- - name: "Load CI image linux/amd64:${{
needs.build-info.outputs.default-python-version }}"
- env:
- PYTHON: ${{ needs.build-info.outputs.default-python-version }}
- run: >
- breeze ci-image load --platform "linux/amd64" --python "${PYTHON}"
--image-file-dir "/mnt"
+ python: "${{ needs.build-info.outputs.docs-python-version }}"
+ platform: "linux/amd64"
+ image-stash-ref: "${{ needs.build-info.outputs.image-stash-ref }}"
+ github-token: "${{ secrets.GITHUB_TOKEN }}"
+ constraints-github-repository: >-
+ ${{ secrets.CONSTRAINTS_GITHUB_REPOSITORY != '' &&
+ secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
- name: "Restore docs inventory cache"
uses:
apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355
with:
@@ -499,7 +490,14 @@ jobs:
overwrite: 'true'
publish-docs-to-s3:
- needs: [build-docs, build-java-sdk-docs, build-info]
+ needs: [build-docs, build-java-sdk-docs, build-info, update-registry]
+ # `update-registry` is skipped for non-provider distributions, and a job
whose dependency was
+ # skipped is skipped in turn, so its result is checked rather than implied
- which means the
+ # two that are not conditional have to be asserted here as well.
+ if: >
+ !cancelled() && needs.build-docs.result == 'success' &&
+ needs.build-java-sdk-docs.result == 'success' &&
+ needs.update-registry.result != 'failure'
name: "Publish documentation to S3"
permissions:
id-token: write
@@ -511,7 +509,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
INCLUDE_SUCCESS_OUTPUTS: false
- PYTHON_MAJOR_MINOR_VERSION: "${{
needs.build-info.outputs.default-python-version }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{
needs.build-info.outputs.docs-python-version }}"
VERBOSE: "true"
steps:
- name: "Cleanup repo"
@@ -553,7 +551,7 @@ jobs:
env:
AIRFLOW_VERSION: ${{ needs.build-info.outputs.airflow-version }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- PYTHON_VERSION: "${{ needs.build-info.outputs.default-python-version
}}"
+ PYTHON_VERSION: "${{ needs.build-info.outputs.docs-python-version }}"
FORCE: "true"
run: >
breeze sbom update-sbom-information
@@ -643,8 +641,9 @@ jobs:
update-registry:
# Runs alongside the docs build rather than after it: the registry reads
nothing the docs
- # publish produces, so waiting only added its whole duration to the
release manager's wait.
- needs: [build-info, build-ci-image]
+ # build produces, so ordering the two only added its whole duration to the
wait. The publish
+ # does wait for it, so a run reports success once both halves are actually
out.
+ needs: [build-info]
if: needs.build-info.outputs.registry-providers != '' ||
needs.build-info.outputs.registry-full-build == 'true'
name: "Update Provider Registry"
permissions:
@@ -654,10 +653,12 @@ jobs:
with:
destination: ${{ needs.build-info.outputs.destination }}
provider: ${{ needs.build-info.outputs.registry-providers }}
- python-version: ${{ needs.build-info.outputs.default-python-version }}
- # `build-ci-image` already built and stashed the image this run - under
the ref's key.
- ci-image-already-built: true
- image-stash-ref: ${{ inputs.ref }}
+ # The registry extracts on 3.12 and the docs build on the default
Python, so each builds the
+ # image it needs in the job that uses it, keyed by this ref so
consecutive publishes reuse it.
+ python-version: ${{ needs.build-info.outputs.registry-python-version }}
+ # Without this the registry would read main rather than the ref being
published.
+ checkout-ref: ${{ inputs.ref }}
+ image-stash-ref: ${{ needs.build-info.outputs.image-stash-ref }}
secrets:
DOCS_AWS_ACCESS_KEY_ID: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
DOCS_AWS_SECRET_ACCESS_KEY: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
diff --git a/.github/workflows/registry-build.yml
b/.github/workflows/registry-build.yml
index c5bae276172..da239882629 100644
--- a/.github/workflows/registry-build.yml
+++ b/.github/workflows/registry-build.yml
@@ -55,11 +55,13 @@ on: # yamllint disable-line rule:truthy
required: false
type: string
default: "3.12"
- ci-image-already-built:
- description: "Caller has already built and stashed the CI image for
this run"
+ checkout-ref:
+ description: >
+ Ref the registry is built from. Defaults to the branch tip; a
release publish passes the
+ tag it is publishing so the extraction reads those sources rather
than main's.
required: false
- type: boolean
- default: false
+ type: string
+ default: ""
# `github.event_name` inside a called workflow reports the *caller's*
event, never
# `workflow_call`, so it cannot distinguish the two entry points. An
input declared
# only here can: it defaults to true when called and is undefined
(falsy) on a
@@ -88,57 +90,13 @@ permissions:
packages: read
jobs:
- build-ci-image:
- name: "Build CI image"
- uses: ./.github/workflows/ci-image-build.yml
- permissions:
- contents: read
- packages: write
- # Skipped when the caller stashed an image for this run; a standalone
dispatch has no
- # such image and still builds its own.
- if: >
- (inputs.ci-image-already-built != true) && (
- inputs.is-workflow-call ||
- contains(fromJSON('[
- "ashb",
- "bugraoz93",
- "eladkal",
- "ephraimbuddy",
- "jedcunningham",
- "jscheffl",
- "kaxil",
- "pierrejeambrun",
- "shahar1",
- "potiuk",
- "uranusjr",
- "utkarsharma2",
- "vincbeck"
- ]'), github.event.sender.login))
- with:
- runners: '["ubuntu-22.04"]'
- platform: "linux/amd64"
- push-image: "false"
- upload-image-artifact: "true"
- # Kept so a standalone registry dispatch reuses the mount cache its last
run left behind.
- upload-mount-cache-artifact: "true"
- python-versions: ${{ format('["{0}"]', inputs.python-version) }}
- branch: "main"
- constraints-branch: "constraints-main"
- use-uv: "true"
- upgrade-to-newer-dependencies: "false"
- docker-cache: "registry"
- disable-airflow-repo-cache: "false"
-
build-and-publish-registry:
timeout-minutes: 45
name: "Build & Publish Registry"
- needs: [build-ci-image]
- # `build-ci-image` is skipped when the caller stashed the image, so this
cannot simply
- # inherit its result. On a dispatch it enforces the committer allowlist
below; on the
- # `workflow_call` path the caller has already gated on its own allowlist,
so this must
- # not re-check it against a second, separately-maintained copy.
+ # On a dispatch this enforces the committer allowlist below; on the
`workflow_call` path the
+ # caller has already gated on its own allowlist, so this must not re-check
it against a
+ # second, separately-maintained copy.
if: >
- !cancelled() && needs.build-ci-image.result != 'failure' && (
inputs.is-workflow-call ||
contains(fromJSON('[
"ashb",
@@ -154,7 +112,7 @@ jobs:
"uranusjr",
"utkarsharma2",
"vincbeck"
- ]'), github.event.sender.login))
+ ]'), github.event.sender.login)
runs-on: ubuntu-latest
env:
SCARF_ANALYTICS: "false"
@@ -169,25 +127,51 @@ jobs:
REGISTRY_CACHE_CONTROL: public, max-age=300
permissions:
contents: read
+ # The image is built here now, so this job reads the ghcr registry build
cache itself.
+ packages: read
steps:
- name: "Checkout repository"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
with:
persist-credentials: false
+ # Empty means the branch tip; a release publish passes the tag it is
publishing, so the
+ # extraction reads the sources being released rather than whatever
main holds.
+ ref: ${{ inputs.checkout-ref }}
# Tags drive the phantom-version filter in extract_metadata.py
# (only versions with a real `providers-<id>/<ver>` tag are
# treated as released). Without this, the filter silently
# falls back to `versions[0]` and ships phantom versions.
fetch-tags: true
- - name: "Prepare breeze & CI image"
- uses: ./.github/actions/prepare_breeze_and_image
+ # A local action is read from the workspace, not from the ref the
workflow itself runs from,
+ # so an action added after the checked-out ref was cut is simply absent.
Checking the
+ # workflow's own version out alongside supplies the definition; the
image is still built
+ # from the sources checked out above.
+ - name: "Checkout the workflow version for its actions"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+ path: workflow-version
+
+ - name: "Make /mnt writeable and cleanup"
+ run: ./scripts/ci/make_mnt_writeable.sh
+ - name: "Free up disk space"
+ run: ./scripts/tools/free_up_disk_space.sh
+ - name: "Install Breeze"
+ uses: ./.github/actions/breeze
+ with:
+ python-version: "${{ inputs.python-version }}"
+
+ # Built in this job rather than handed over by one of its own: the image
has a single
+ # consumer, so a separate job could only export it and import it
straight back. It is still
+ # stashed under this ref, so the next run for the same ref seeds its
build from it.
+ - name: "Build CI image linux/amd64:${{ inputs.python-version }}"
+ uses: ./workflow-version/.github/actions/build_ci_image_with_cache
with:
python: "${{ inputs.python-version }}"
platform: "linux/amd64"
- use-uv: "true"
- make-mnt-writeable-and-cleanup: "true"
image-stash-ref: "${{ inputs.image-stash-ref }}"
+ github-token: "${{ secrets.GITHUB_TOKEN }}"
- name: "Install AWS CLI v2"
run: |
diff --git a/dev/breeze/doc/images/output_build-docs.svg
b/dev/breeze/doc/images/output_build-docs.svg
index 78ab4748143..99bc5dbc0de 100644
--- a/dev/breeze/doc/images/output_build-docs.svg
+++ b/dev/breeze/doc/images/output_build-docs.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 1587.1999999999998"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 1538.3999999999999"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-build-docs-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="1536.1999999999998" />
+ <rect x="0" y="0" width="1463.0" height="1487.3999999999999" />
</clipPath>
<clipPath id="breeze-build-docs-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -225,15 +225,9 @@
<clipPath id="breeze-build-docs-line-59">
<rect x="0" y="1441.1" width="1464" height="24.65"/>
</clipPath>
-<clipPath id="breeze-build-docs-line-60">
- <rect x="0" y="1465.5" width="1464" height="24.65"/>
- </clipPath>
-<clipPath id="breeze-build-docs-line-61">
- <rect x="0" y="1489.9" width="1464" height="24.65"/>
- </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1585.2" rx="8"/><text
class="breeze-build-docs-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: build-docs</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1536.4" rx="8"/><text
class="breeze-build-docs-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: build-docs</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -289,23 +283,21 @@
</text><text class="breeze-build-docs-r5" x="0" y="1069.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-43)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1069.2" textLength="353.8"
clip-path="url(#breeze-build-docs-line-43)">--include-removed-providers  </text><text
class="breeze-build-docs-r1" x="402.6" y="1069.2" textLength="561.2"
clip-path="url(#breeze-build-docs-line-43)">Whether to include providers that are removed.</te
[...]
</text><text class="breeze-build-docs-r5" x="0" y="1093.6" textLength="1464"
clip-path="url(#breeze-build-docs-line-44)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-build-docs-r1" x="1464" y="1093.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-44)">
</text><text class="breeze-build-docs-r5" x="0" y="1118" textLength="24.4"
clip-path="url(#breeze-build-docs-line-45)">╭─</text><text
class="breeze-build-docs-r5" x="24.4" y="1118" textLength="170.8"
clip-path="url(#breeze-build-docs-line-45)"> Misc options </text><text
class="breeze-build-docs-r5" x="195.2" y="1118" textLength="1244.4"
clip-path="url(#breeze-build-docs-line-45)">──────────────────────────────────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-build-docs-r5" x="0" y="1142.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-46)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1142.4" textLength="244"
clip-path="url(#breeze-build-docs-line-46)">--python            </text><text
class="breeze-build-docs-r6" x="292.8" y="1142.4" textLength="24.4"
clip-path="url(#breeze-build-docs-line-46)">-p</text><text
class="breeze-build-docs-r1 [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1166.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-47)">│</text><text
class="breeze-build-docs-r7" x="341.6" y="1166.8" textLength="317.2"
clip-path="url(#breeze-build-docs-line-47)">3.11 | 3.12 | 3.13 | 3.14)</text><text
class="breeze-build-docs-r5" x="1451.8" y="1166.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-47)">│</text><text
class="breeze-build-docs-r1" x="1464" y="1166.8" [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1191.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-48)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1191.2" textLength="244"
clip-path="url(#breeze-build-docs-line-48)">--include-commits   </text><text
class="breeze-build-docs-r1" x="341.6" y="1191.2" textLength="451.4"
clip-path="url(#breeze-build-docs-line-48)">Include commits in the documentation.</text><text
class="breeze-bu [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1215.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-49)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1215.6" textLength="244"
clip-path="url(#breeze-build-docs-line-49)">--github-repository </text><text
class="breeze-build-docs-r6" x="292.8" y="1215.6" textLength="24.4"
clip-path="url(#breeze-build-docs-line-49)">-g</text><text
class="breeze-build-docs-r1" x="341.6" y="1215.6" textLength="597.8"
clip-path="ur [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1240" textLength="12.2"
clip-path="url(#breeze-build-docs-line-50)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1240" textLength="244"
clip-path="url(#breeze-build-docs-line-50)">--builder           </text><text
class="breeze-build-docs-r1" x="341.6" y="1240" textLength="768.6"
clip-path="url(#breeze-build-docs-line-50)">Buildx builder used to perform&#
[...]
-</text><text class="breeze-build-docs-r5" x="0" y="1264.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-51)">│</text><text
class="breeze-build-docs-r7" x="341.6" y="1264.4" textLength="73.2"
clip-path="url(#breeze-build-docs-line-51)">(TEXT)</text><text
class="breeze-build-docs-r5" x="1451.8" y="1264.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-51)">│</text><text
class="breeze-build-docs-r1" x="1464" y="1264.4" textLength="12.2"
clip-path="url(#breeze-build-doc [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1288.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-52)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1288.8" textLength="244"
clip-path="url(#breeze-build-docs-line-52)">--distributions-list</text><text
class="breeze-build-docs-r1" x="341.6" y="1288.8" textLength="1098"
clip-path="url(#breeze-build-docs-line-52)">Optional, contains space separated list of package ids that
[...]
-</text><text class="breeze-build-docs-r5" x="0" y="1313.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-53)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1313.2" textLength="1098"
clip-path="url(#breeze-build-docs-line-53)">documentation building, and document publishing. It is an easier alternative to adding    </text><text
class="breeze-build-docs-r5" x="1451.8" y="1313.2" textLength="12.2 [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1337.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-54)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1337.6" textLength="1098"
clip-path="url(#breeze-build-docs-line-54)">individual packages as arguments to every command. This overrides the packages passed as  </text><text
class="breeze-build-docs-r5" x="1451.8" y="1337.6" textLength="12.2" cli [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1362" textLength="12.2"
clip-path="url(#breeze-build-docs-line-55)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1362" textLength="134.2"
clip-path="url(#breeze-build-docs-line-55)">arguments. </text><text
class="breeze-build-docs-r7" x="475.8" y="1362" textLength="73.2"
clip-path="url(#breeze-build-docs-line-55)">(TEXT)</text><text
class="breeze-build-docs-r5" x="1451.8" y="1362" textLength="12.2"
clip-path="url(#breeze- [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1386.4" textLength="1464"
clip-path="url(#breeze-build-docs-line-56)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-build-docs-r1" x="1464" y="1386.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-56)">
-</text><text class="breeze-build-docs-r5" x="0" y="1410.8" textLength="24.4"
clip-path="url(#breeze-build-docs-line-57)">╭─</text><text
class="breeze-build-docs-r5" x="24.4" y="1410.8" textLength="195.2"
clip-path="url(#breeze-build-docs-line-57)"> Common options </text><text
class="breeze-build-docs-r5" x="219.6" y="1410.8" textLength="1220"
clip-path="url(#breeze-build-docs-line-57)">────────────────────────────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-build-docs-r5" x="0" y="1435.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-58)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1435.2" textLength="109.8"
clip-path="url(#breeze-build-docs-line-58)">--dry-run</text><text
class="breeze-build-docs-r6" x="158.6" y="1435.2" textLength="24.4"
clip-path="url(#breeze-build-docs-line-58)">-D</text><text
class="breeze-build-docs-r1" x="207.4" y="1435.2" textLength="719.8"
clip-path="url(#breeze-buil [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1459.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-59)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1459.6" textLength="109.8"
clip-path="url(#breeze-build-docs-line-59)">--verbose</text><text
class="breeze-build-docs-r6" x="158.6" y="1459.6" textLength="24.4"
clip-path="url(#breeze-build-docs-line-59)">-v</text><text
class="breeze-build-docs-r1" x="207.4" y="1459.6" textLength="585.6"
clip-path="url(#breeze-buil [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1484" textLength="12.2"
clip-path="url(#breeze-build-docs-line-60)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1484" textLength="109.8"
clip-path="url(#breeze-build-docs-line-60)">--answer </text><text
class="breeze-build-docs-r6" x="158.6" y="1484" textLength="24.4"
clip-path="url(#breeze-build-docs-line-60)">-a</text><text
class="breeze-build-docs-r1" x="207.4" y="1484" textLength="329.4"
clip-path="url(#breeze-build-d [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1508.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-61)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1508.4" textLength="109.8"
clip-path="url(#breeze-build-docs-line-61)">--help   </text><text
class="breeze-build-docs-r6" x="158.6" y="1508.4" textLength="24.4"
clip-path="url(#breeze-build-docs-line-61)">-h</text><text
class="breeze-build-docs-r1" x="207.4" y="1508.4" textLength="329.4"
clip-path="u [...]
-</text><text class="breeze-build-docs-r5" x="0" y="1532.8" textLength="1464"
clip-path="url(#breeze-build-docs-line-62)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-build-docs-r1" x="1464" y="1532.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-62)">
+</text><text class="breeze-build-docs-r5" x="0" y="1142.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-46)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1142.4" textLength="244"
clip-path="url(#breeze-build-docs-line-46)">--include-commits   </text><text
class="breeze-build-docs-r1" x="341.6" y="1142.4" textLength="451.4"
clip-path="url(#breeze-build-docs-line-46)">Include commits in the documentation.</text><text
class="breeze-bu [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1166.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-47)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1166.8" textLength="244"
clip-path="url(#breeze-build-docs-line-47)">--github-repository </text><text
class="breeze-build-docs-r6" x="292.8" y="1166.8" textLength="24.4"
clip-path="url(#breeze-build-docs-line-47)">-g</text><text
class="breeze-build-docs-r1" x="341.6" y="1166.8" textLength="597.8"
clip-path="ur [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1191.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-48)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1191.2" textLength="244"
clip-path="url(#breeze-build-docs-line-48)">--builder           </text><text
class="breeze-build-docs-r1" x="341.6" y="1191.2" textLength="768.6"
clip-path="url(#breeze-build-docs-line-48)">Buildx builder used to per
[...]
+</text><text class="breeze-build-docs-r5" x="0" y="1215.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-49)">│</text><text
class="breeze-build-docs-r7" x="341.6" y="1215.6" textLength="73.2"
clip-path="url(#breeze-build-docs-line-49)">(TEXT)</text><text
class="breeze-build-docs-r5" x="1451.8" y="1215.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-49)">│</text><text
class="breeze-build-docs-r1" x="1464" y="1215.6" textLength="12.2"
clip-path="url(#breeze-build-doc [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1240" textLength="12.2"
clip-path="url(#breeze-build-docs-line-50)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1240" textLength="244"
clip-path="url(#breeze-build-docs-line-50)">--distributions-list</text><text
class="breeze-build-docs-r1" x="341.6" y="1240" textLength="1098"
clip-path="url(#breeze-build-docs-line-50)">Optional, contains space separated list of package ids that are
[...]
+</text><text class="breeze-build-docs-r5" x="0" y="1264.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-51)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1264.4" textLength="1098"
clip-path="url(#breeze-build-docs-line-51)">documentation building, and document publishing. It is an easier alternative to adding    </text><text
class="breeze-build-docs-r5" x="1451.8" y="1264.4" textLength="12.2 [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1288.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-52)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1288.8" textLength="1098"
clip-path="url(#breeze-build-docs-line-52)">individual packages as arguments to every command. This overrides the packages passed as  </text><text
class="breeze-build-docs-r5" x="1451.8" y="1288.8" textLength="12.2" cli [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1313.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-53)">│</text><text
class="breeze-build-docs-r1" x="341.6" y="1313.2" textLength="134.2"
clip-path="url(#breeze-build-docs-line-53)">arguments. </text><text
class="breeze-build-docs-r7" x="475.8" y="1313.2" textLength="73.2"
clip-path="url(#breeze-build-docs-line-53)">(TEXT)</text><text
class="breeze-build-docs-r5" x="1451.8" y="1313.2" textLength="12.2"
clip-path="url( [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1337.6" textLength="1464"
clip-path="url(#breeze-build-docs-line-54)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-build-docs-r1" x="1464" y="1337.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-54)">
+</text><text class="breeze-build-docs-r5" x="0" y="1362" textLength="24.4"
clip-path="url(#breeze-build-docs-line-55)">╭─</text><text
class="breeze-build-docs-r5" x="24.4" y="1362" textLength="195.2"
clip-path="url(#breeze-build-docs-line-55)"> Common options </text><text
class="breeze-build-docs-r5" x="219.6" y="1362" textLength="1220"
clip-path="url(#breeze-build-docs-line-55)">──────────────────────────────────────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-build-docs-r5" x="0" y="1386.4" textLength="12.2"
clip-path="url(#breeze-build-docs-line-56)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1386.4" textLength="109.8"
clip-path="url(#breeze-build-docs-line-56)">--dry-run</text><text
class="breeze-build-docs-r6" x="158.6" y="1386.4" textLength="24.4"
clip-path="url(#breeze-build-docs-line-56)">-D</text><text
class="breeze-build-docs-r1" x="207.4" y="1386.4" textLength="719.8"
clip-path="url(#breeze-buil [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1410.8" textLength="12.2"
clip-path="url(#breeze-build-docs-line-57)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1410.8" textLength="109.8"
clip-path="url(#breeze-build-docs-line-57)">--verbose</text><text
class="breeze-build-docs-r6" x="158.6" y="1410.8" textLength="24.4"
clip-path="url(#breeze-build-docs-line-57)">-v</text><text
class="breeze-build-docs-r1" x="207.4" y="1410.8" textLength="585.6"
clip-path="url(#breeze-buil [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1435.2" textLength="12.2"
clip-path="url(#breeze-build-docs-line-58)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1435.2" textLength="109.8"
clip-path="url(#breeze-build-docs-line-58)">--answer </text><text
class="breeze-build-docs-r6" x="158.6" y="1435.2" textLength="24.4"
clip-path="url(#breeze-build-docs-line-58)">-a</text><text
class="breeze-build-docs-r1" x="207.4" y="1435.2" textLength="329.4"
clip-path="url(#breeze [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1459.6" textLength="12.2"
clip-path="url(#breeze-build-docs-line-59)">│</text><text
class="breeze-build-docs-r4" x="24.4" y="1459.6" textLength="109.8"
clip-path="url(#breeze-build-docs-line-59)">--help   </text><text
class="breeze-build-docs-r6" x="158.6" y="1459.6" textLength="24.4"
clip-path="url(#breeze-build-docs-line-59)">-h</text><text
class="breeze-build-docs-r1" x="207.4" y="1459.6" textLength="329.4"
clip-path="u [...]
+</text><text class="breeze-build-docs-r5" x="0" y="1484" textLength="1464"
clip-path="url(#breeze-build-docs-line-60)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-build-docs-r1" x="1464" y="1484" textLength="12.2"
clip-path="url(#breeze-build-docs-line-60)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_build-docs.txt
b/dev/breeze/doc/images/output_build-docs.txt
index cc79708d4cd..75dd9355bcf 100644
--- a/dev/breeze/doc/images/output_build-docs.txt
+++ b/dev/breeze/doc/images/output_build-docs.txt
@@ -1 +1 @@
-c6881341938aadc0176ded803153998d
+e22969fb5e92a3efafb744881fea5ef8
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index 0233a6919b0..8a41b104136 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -814,7 +814,6 @@ def _build_python_docs(
*,
generated_path: Path,
builder: str,
- python: str,
clean_build: bool,
clean_inventory_cache: bool,
refresh_airflow_inventories: bool,
@@ -830,9 +829,14 @@ def _build_python_docs(
spellcheck_only: bool,
doc_packages: tuple[str, ...],
):
+ # Docs are always built on the default Python. The Sphinx configuration
mocks third-party
+ # modules, and what that mocking does depends on the interpreter - on 3.12
functools copies
+ # __type_params__, for which a mock returns another mock rather than a
tuple, so providers
+ # decorating methods with functools.wraps over a mocked callable fail to
import. Letting a
+ # caller pick the interpreter here silently changes what the docs build
can document.
build_params = BuildCiParams(
github_repository=github_repository,
- python=python,
+ python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
builder=builder,
)
rebuild_or_pull_ci_image_if_needed(command_params=build_params)
@@ -887,7 +891,7 @@ def _build_python_docs(
)
shell_params = ShellParams(
github_repository=github_repository,
- python=python,
+ python=DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
mount_sources=MOUNT_ALL,
)
result = execute_command_in_shell(shell_params,
project_name="breeze-docs", command=cmd)
@@ -934,7 +938,6 @@ def _build_python_docs(
@option_github_repository
@option_include_not_ready_providers
@option_include_removed_providers
-@option_python
@click.option(
"--one-pass-only",
help="Builds documentation in one pass only. This is useful for debugging
sphinx errors.",
@@ -979,7 +982,6 @@ def build_docs(
include_commits: bool,
one_pass_only: bool,
package_filter: tuple[str, ...],
- python: str,
distributions_list: str,
spellcheck_only: bool,
sdk: tuple[str, ...],
@@ -1010,7 +1012,6 @@ def build_docs(
include_commits=include_commits,
one_pass_only=one_pass_only,
package_filter=package_filter,
- python=python,
distributions_list=distributions_list,
spellcheck_only=spellcheck_only,
doc_packages=doc_packages,
diff --git
a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
index f3816139abe..17d32146110 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
@@ -385,7 +385,6 @@ DEVELOPER_PARAMETERS: dict[str, list[dict[str, str |
list[str]]]] = {
{
"name": "Misc options",
"options": [
- "--python",
"--include-commits",
"--github-repository",
"--builder",
diff --git a/dev/breeze/tests/test_developer_commands.py
b/dev/breeze/tests/test_developer_commands.py
index d5b0ca1a070..2336b0585ba 100644
--- a/dev/breeze/tests/test_developer_commands.py
+++ b/dev/breeze/tests/test_developer_commands.py
@@ -23,6 +23,7 @@ import pytest
from click.testing import CliRunner
from airflow_breeze.commands.developer_commands import build_docs
+from airflow_breeze.global_constants import DEFAULT_PYTHON_MAJOR_MINOR_VERSION
@pytest.fixture
@@ -31,7 +32,12 @@ def runner():
class TestBuildDocsPythonVersion:
- """`breeze build-docs` must document with the Python version it was asked
for."""
+ """`breeze build-docs` always documents on the default Python.
+
+ The Sphinx config mocks third-party modules and what that mocking does
depends on the
+ interpreter, so the docs build must not follow whatever Python the caller
happens to have
+ selected - see the comment in ``_build_python_docs``.
+ """
@pytest.fixture(autouse=True)
def _no_docker(self, monkeypatch):
@@ -54,15 +60,18 @@ class TestBuildDocsPythonVersion:
runner.invoke(build_docs, args, env=env, catch_exceptions=False)
return mock_rebuild, mock_shell
- @pytest.mark.parametrize(
- ("args", "env"),
- [
- pytest.param(["--python", "3.12"], None, id="python-option"),
- pytest.param([], {"PYTHON_MAJOR_MINOR_VERSION": "3.12"},
id="python-env-var"),
- ],
- )
- def test_selected_python_is_used_for_image_and_shell(self, runner, args,
env):
- mock_rebuild, mock_shell = self._invoke(runner, [*args,
"--docs-only"], env=env)
+ def test_environment_python_does_not_change_the_docs_build(self, runner):
+ # PYTHON_MAJOR_MINOR_VERSION is set on every job of the docs
publishing workflow, so an
+ # option reading it silently decided what the docs were built with.
+ mock_rebuild, mock_shell = self._invoke(
+ runner, ["--docs-only"], env={"PYTHON_MAJOR_MINOR_VERSION": "3.12"}
+ )
+
+ assert mock_rebuild.call_args.kwargs["command_params"].python ==
DEFAULT_PYTHON_MAJOR_MINOR_VERSION
+ assert mock_shell.call_args.args[0].python ==
DEFAULT_PYTHON_MAJOR_MINOR_VERSION
+
+ def test_python_option_is_rejected(self, runner):
+ result = runner.invoke(build_docs, ["--python", "3.12", "--docs-only"])
- assert mock_rebuild.call_args.kwargs["command_params"].python == "3.12"
- assert mock_shell.call_args.args[0].python == "3.12"
+ assert result.exit_code != 0
+ assert "no such option" in result.output.lower()
