This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-4-test by this push:
new 5941209499e [v3-4-test] Accelerate CI image materialization with
same-run snapshots (#74173) (#76183)
5941209499e is described below
commit 5941209499e7a8e04e95e4d64f35689df8891467
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Oct 31 22:02:42 2026 +0100
[v3-4-test] Accelerate CI image materialization with same-run snapshots
(#74173) (#76183)
* Restore the CI image from a snapshot of Docker's data directory
Every job that prepares the CI image runs `docker image load` on the stash,
which unpacks and checksums every layer of an 8 GB image again, about two
minutes per job. Extracting a copy of the image store into a stopped daemon
gives the same image in about half a minute on the same runners. The image
stash stays as it is, for other consumers and as the fallback when a
snapshot does not fit the daemon a job runs on.
* Recover Docker safely when CI snapshot materialization fails
Snapshot failures must not prevent the authoritative stash from loading,
and an older branch snapshot must not substitute an image for the current
checkout.
* Scope CI snapshots to the current workflow run
Raw daemon snapshots are job handoffs, so publishing them into branch
caches unnecessarily exposes later runs to arbitrary checkout refs.
* Publish authoritative caches before optional CI snapshot execution
The snapshot script runs checkout code and prunes Docker state. Existing
cache publications must finish before that optional execution.
* Isolate CI snapshot creation from branch cache publishers
The optional snapshot producer executes checkout code without publishing
branch caches; its artifacts are available only to consumers in the same
workflow run.
* Bind optional snapshot jobs to the workflow commit
Snapshot producers must not accept arbitrary checkout refs in privileged
workflow contexts, and a partially failed Docker stop must still trigger
restart cleanup.
* Avoid reloading the CI image before snapshot creation
Preparing the authoritative image in a second runner adds nearly three
minutes before consumers can start. All branch publications must finish before
the optional snapshot code prunes Docker state.
* Execute the snapshot script from the immutable workflow commit
The builder can check out a configurable ref before publishing caches.
Snapshot execution must read the workflow commit itself rather than trust files
left in that working tree.
* Keep optional snapshots in an isolated read-only job
Executing snapshot code inside the configurable-ref builder introduces a
cache-poisoning finding. A separate job preserves the security boundary; its
latency cost must remain explicit when measuring the runner-time benefit.
* Preserve Docker state when snapshot inventory checks fail
An unavailable daemon must not be mistaken for an empty image store before
destructive snapshot materialization.
* Retrigger CI after provider compatibility timeout
* Create CI snapshots in the builder after publishing caches
Reloading the freshly built image in a second job delays every consumer by
nearly four minutes. Trusted workflow commands reuse the builder daemon without
executing code from the configurable checkout.
* Write CI snapshots off the Docker data disk
The builder bind-mounts Docker from /mnt. Reading its image store and
writing the compressed archive to that same disk competes for I/O; runner temp
keeps the output on the root disk.
* Cache CI dependencies separately from workspace source
Ordinary source edits invalidate the dependency installation layer because
it follows the full checkout copy. Reuse locked third-party dependencies while
retaining the complete source installation and its resolution fallback.
* Make dependency manifest extraction portable
Use portable path-preserving shell commands for the BuildKit manifest stage
and cover nested paths containing spaces.\n\nCo-authored-by: Copilot
<[email protected]>
* Run dependency manifest extraction with Bash
Use the shell required by the manifest extraction stage so the BuildKit RUN
command works with Docker's default shell.\n\nCo-authored-by: Copilot
<[email protected]>
* Harden dependency manifest extraction
Restore the current UV and prek versions, propagate manifest discovery and
copy failures, and cover failure injection in the extraction
tests.\n\nCo-authored-by: Copilot <[email protected]>
* Address CI snapshot review and separate dependency caching
* Name the CI image snapshot artifact v1
There has never been an earlier version of this artifact on main, so
the -v2 suffix left over from earlier branch revisions would puzzle the
next reader. Snapshot artifacts are scoped to the workflow run that
uploads them and kept for two days, so no consumer can see an artifact
under the old name and no compatibility shim is needed.
A test now checks that the upload and restore steps use the same
artifact name, so the two cannot drift apart silently.
* Keep CI image snapshots for one day
A snapshot is only downloaded by consumer jobs of the run that uploaded
it. Once it expires, a re-run restores the image stash instead, which
keeps its two-day retention. One day halves the storage the snapshot
adds on top of the stash.
* Drop snapshot-ci-images assertion per review
* Keep CI image snapshots faithful across reruns and check them before
download
- Archive and extract xattrs in every namespace: GNU tar restores only
user.* by default, which dropped overlay2's trusted.overlay.opaque and
file capabilities, so a restored image could differ from the built one.
- Upload the snapshot with overwrite, plus a small metadata artifact that
is uploaded even when the export fails; it reads "unavailable" until the
archive is complete, so a rerun never restores an earlier attempt.
- Check the metadata against the checkout and daemon before downloading
the snapshot, and bound extraction with a timeout that falls back to
the stash after cleaning partial state.
- Load images for prepare_breeze_and_image and prepare_single_ci_image
through one shared load_image action.
* Verify restored CI image snapshots and route them through breeze
- Record the image ID and a hash of its merged file tree when creating the
snapshot and compare both after restoring: a restore that lost an opaque
directory marker keeps the image ID but changes the tree, so it now falls
back to the stash instead of passing `bash -c true`.
- Restore through `breeze ci-image load --snapshot-file`, keeping breeze's
environment checks, image check, diagnostics and rebuild marker.
- Bound the daemon stop and start as well as the extraction.
- Download the snapshot metadata by pattern, which adds no error annotation
when the builder made no snapshot.
- Limit snapshots to GitHub-hosted runners, where self-hosted runners
already restore the stash from their own storage.
* Check that an unsupported image store skips pruning and publishes only
the unavailable marker
* Sort the snapshot file-tree listing outside the container
Sorting the CI image's 265k-entry listing inside the container spilled
to temporary files in /tmp while find was still walking it, so every run
listed different files and every consumer rejected the snapshot.
* Create CI image snapshots through breeze ci-image save
The builder now calls `breeze ci-image save --snapshot-file`, mirroring the
`breeze ci-image load --snapshot-file` that consumers restore with, so both
sides of the snapshot go through breeze and its environment checks.
* Retire the snapshot metadata when its archive upload fails and
fingerprint ownership and size
The export writes metadata for a complete archive before the upload; if the
upload then fails, that metadata would point consumers at an earlier attempt's
archive. The tree fingerprint now also covers owner, group and size, and the
artifact actions use the v7.0.2/v8.0.2 pins used elsewhere in the repository.
Generated-by: Claude Opus 5
---------
(cherry picked from commit bf177e2948514f0807bf33c6462cde81ac43238e)
Co-authored-by: Yossi Eliaz <[email protected]>
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Shahar Epstein <[email protected]>
Co-authored-by: Jarek Potiuk <[email protected]>
---
.github/actions/load_image/action.yml | 116 ++++++
.../actions/prepare_breeze_and_image/action.yml | 25 +-
.github/actions/prepare_single_ci_image/action.yml | 19 +-
.github/workflows/ci-image-build.yml | 59 +++
dev/breeze/doc/images/output_ci-image_load.svg | 32 +-
dev/breeze/doc/images/output_ci-image_load.txt | 2 +-
dev/breeze/doc/images/output_ci-image_save.svg | 26 +-
dev/breeze/doc/images/output_ci-image_save.txt | 2 +-
.../airflow_breeze/commands/ci_image_commands.py | 53 ++-
.../commands/ci_image_commands_config.py | 2 +
.../commands/common_image_options.py | 12 +
dev/breeze/tests/test_ci_image_commands.py | 103 +++++
scripts/ci/docker_data_root_snapshot.sh | 207 ++++++++++
scripts/tests/ci/test_docker_data_root_snapshot.py | 459 +++++++++++++++++++++
14 files changed, 1058 insertions(+), 59 deletions(-)
diff --git a/.github/actions/load_image/action.yml
b/.github/actions/load_image/action.yml
new file mode 100644
index 00000000000..bb90c98ef76
--- /dev/null
+++ b/.github/actions/load_image/action.yml
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Load image (CI or PROD)'
+description: >
+ Loads the current image for one Python version: from the CI image snapshot
when it fits this
+ runner's daemon, otherwise from the image stash.
+inputs:
+ python:
+ description: 'Python version of the image to load'
+ required: true
+ platform:
+ description: 'Platform of the image - linux/amd64 or linux/arm64'
+ required: true
+ image-type:
+ description: 'Which image type to load (ci/prod)'
+ default: "ci"
+ image-stash-ref:
+ description: 'Ref whose image stash to restore (empty = the per-branch
one)'
+ required: false
+ default: ""
+runs:
+ using: "composite"
+ steps:
+ # The metadata is a few bytes, so a snapshot that cannot fit is rejected
before its download.
+ # A pattern, unlike a name, downloads nothing without an error when the
builder made no snapshot.
+ # Self-hosted runners restore the stash from their own storage, which a
run artifact cannot beat.
+ - name: "Download CI image snapshot metadata ${{ inputs.platform }}:${{
inputs.python }}"
+ id: snapshot-meta
+ continue-on-error: true
+ uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333
# v8.0.2
+ with:
+ pattern: >-
+ ci-image-snapshot-v1-${{ inputs.python }}-${{
+ inputs.platform == 'linux/amd64' && 'amd64' || 'arm64' }}-meta
+ merge-multiple: true
+ path: "/mnt/"
+ if: >
+ github.event_name == 'pull_request' && runner.environment ==
'github-hosted' &&
+ inputs.image-type == 'ci' && inputs.image-stash-ref == ''
+ - name: "Check CI image snapshot ${{ inputs.platform }}:${{ inputs.python
}}"
+ id: snapshot-check
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ run: |
+ if ./scripts/ci/docker_data_root_snapshot.sh check \
+ "/mnt/ci-image-snapshot-${PLATFORM//\//_}-${PYTHON}.tar.zst"; then
+ echo "fits=true" >> "${GITHUB_OUTPUT}"
+ else
+ echo "Falling back to loading the image stash"
+ fi
+ shell: bash
+ if: steps.snapshot-meta.outcome == 'success'
+ - name: "Download CI image snapshot ${{ inputs.platform }}:${{
inputs.python }}"
+ id: snapshot-download
+ continue-on-error: true
+ uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333
# v8.0.2
+ with:
+ name: >-
+ ci-image-snapshot-v1-${{ inputs.python }}-${{
+ inputs.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
+ path: "/mnt/"
+ if: steps.snapshot-check.outputs.fits == 'true'
+ # Falls back to the image stash below whenever the snapshot cannot be
restored.
+ - name: "Unpack CI image snapshot ${{ inputs.platform }}:${{ inputs.python
}}"
+ id: snapshot
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ run: |
+ if breeze ci-image load --platform "${PLATFORM}" --python "${PYTHON}" \
+ --snapshot-file
"/mnt/ci-image-snapshot-${PLATFORM//\//_}-${PYTHON}.tar.zst"; then
+ echo "restored=true" >> "${GITHUB_OUTPUT}"
+ else
+ echo "Falling back to loading the image stash"
+ fi
+ shell: bash
+ if: steps.snapshot-download.outcome == 'success'
+ - name: >
+ Restore ${{ inputs.image-type }} docker image ${{ inputs.platform
}}:${{ inputs.python }}
+ ${{ inputs.image-stash-ref != '' && format('built for ref {0}',
inputs.image-stash-ref) || '' }}
+ uses: ./.github/actions/stash-restore
+ with:
+ key: "${{ inputs.image-type }}-image-save-v3-${{ inputs.platform
}}-${{ inputs.python }}\
+ ${{ inputs.image-stash-ref != '' && format('-{0}',
inputs.image-stash-ref) || '' }}"
+ path: "/mnt/"
+ # Must stay true: runs on RunsOn (canaries and release-branch pushes)
save these image
+ # stashes as S3 digests only.
+ only-current-branch: 'true'
+ fail-on-download: 'true'
+ if: steps.snapshot.outputs.restored != 'true'
+ - name: "Load ${{ inputs.image-type }} image ${{ inputs.platform }}:${{
inputs.python }}"
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ PYTHON: ${{ inputs.python }}
+ IMAGE_TYPE: ${{ inputs.image-type }}
+ run: >
+ breeze ${IMAGE_TYPE}-image load --platform "${PLATFORM}" --python
"${PYTHON}" --image-file-dir "/mnt"
+ shell: bash
+ if: steps.snapshot.outputs.restored != 'true'
diff --git a/.github/actions/prepare_breeze_and_image/action.yml
b/.github/actions/prepare_breeze_and_image/action.yml
index 7cad3d17033..141d65e1624 100644
--- a/.github/actions/prepare_breeze_and_image/action.yml
+++ b/.github/actions/prepare_breeze_and_image/action.yml
@@ -84,23 +84,10 @@ runs:
run: |
echo "Checking free space!"
df -H
- - name: >
- Restore ${{ inputs.image-type }} docker image ${{ inputs.platform
}}:${{ inputs.python }}
- ${{ inputs.image-stash-ref != '' && format('built for ref {0}',
inputs.image-stash-ref) || '' }}
- uses: ./.github/actions/stash-restore
- with:
- key: "${{ inputs.image-type }}-image-save-v3-${{ inputs.platform
}}-${{ inputs.python }}\
- ${{ inputs.image-stash-ref != '' && format('-{0}',
inputs.image-stash-ref) || '' }}"
- path: "/mnt/"
- # Must stay true: runs on RunsOn (canaries and release-branch pushes)
save these image
- # stashes as S3 digests only.
- only-current-branch: 'true'
- fail-on-download: 'true'
- name: "Load ${{ inputs.image-type }} image ${{ inputs.platform }}:${{
inputs.python }}"
- env:
- PLATFORM: ${{ inputs.platform }}
- PYTHON: ${{ inputs.python }}
- IMAGE_TYPE: ${{ inputs.image-type }}
- run: >
- breeze ${IMAGE_TYPE}-image load --platform "${PLATFORM}" --python
"${PYTHON}" --image-file-dir "/mnt"
- shell: bash
+ uses: ./.github/actions/load_image
+ with:
+ python: ${{ inputs.python }}
+ platform: ${{ inputs.platform }}
+ image-type: ${{ inputs.image-type }}
+ image-stash-ref: ${{ inputs.image-stash-ref }}
diff --git a/.github/actions/prepare_single_ci_image/action.yml
b/.github/actions/prepare_single_ci_image/action.yml
index 8a734ef1102..df23eb1c1f2 100644
--- a/.github/actions/prepare_single_ci_image/action.yml
+++ b/.github/actions/prepare_single_ci_image/action.yml
@@ -35,20 +35,9 @@ inputs:
runs:
using: "composite"
steps:
- - name: "Restore CI docker images ${{ inputs.platform }}:${{ inputs.python
}}"
- uses: ./.github/actions/stash-restore
- with:
- key: ci-image-save-v3-${{ inputs.platform }}-${{ inputs.python }}
- path: "/mnt/"
- # Must stay true: runs on RunsOn (canaries and release-branch pushes)
save these image
- # stashes as S3 digests only.
- only-current-branch: 'true'
- fail-on-download: 'true'
- if: contains(inputs.python-versions-list-as-string, inputs.python)
- name: "Load CI image ${{ inputs.platform }}:${{ inputs.python }}"
- env:
- PLATFORM: ${{ inputs.platform }}
- PYTHON: ${{ inputs.python }}
- run: breeze ci-image load --platform "${PLATFORM}" --python "${PYTHON}"
--image-file-dir "/mnt/"
- shell: bash
+ uses: ./.github/actions/load_image
+ with:
+ python: ${{ inputs.python }}
+ platform: ${{ inputs.platform }}
if: contains(inputs.python-versions-list-as-string, inputs.python)
diff --git a/.github/workflows/ci-image-build.yml
b/.github/workflows/ci-image-build.yml
index b16296c8a14..4c81cce0c0e 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -412,3 +412,62 @@ jobs:
steps.stashed-image.outputs.reusable != 'true'
- name: "Check disk space after build"
run: df -H
+
+ # Snapshot only after cache publication: pruning removes the mount-cache
image.
+ - name: "Snapshot CI image ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ id: snapshot-export
+ continue-on-error: true
+ timeout-minutes: 5
+ env:
+ PLATFORM: ${{ inputs.platform }}
+ # Docker is bind-mounted from /mnt on AMD; write the archive on the
root disk.
+ SNAPSHOT_FILE: >-
+ ${{ runner.temp }}/ci-image-snapshot-${{
+ inputs.platform == 'linux/amd64' && 'linux_amd64' || 'linux_arm64'
+ }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar.zst
+ run: >
+ breeze ci-image save --platform "${PLATFORM}" --python
"${PYTHON_MAJOR_MINOR_VERSION}"
+ --snapshot-file "${SNAPSHOT_FILE}"
+ shell: bash
+ if: >
+ github.event_name == 'pull_request' && runner.environment ==
'github-hosted' &&
+ inputs.upload-image-artifact == 'true' && inputs.image-stash-ref ==
'' &&
+ (inputs.checkout-ref == '' || inputs.checkout-ref == github.sha)
+ - name: "Upload CI image snapshot ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ id: snapshot-upload
+ continue-on-error: true
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
# v7.0.2
+ with:
+ name: >-
+ ci-image-snapshot-v1-${{ env.PYTHON_MAJOR_MINOR_VERSION }}-${{
+ inputs.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
+ path: "${{ runner.temp }}/ci-image-snapshot-*-${{
env.PYTHON_MAJOR_MINOR_VERSION }}.tar.zst"
+ if-no-files-found: 'error'
+ compression-level: '0'
+ # Read only within this run; once it expires, consumers fall back to
the stash.
+ retention-days: '1'
+ # A rerun replaces the snapshot of its earlier attempt instead of
failing to upload.
+ overwrite: true
+ if: steps.snapshot-export.outcome == 'success'
+ # The export wrote metadata for a complete archive; without the uploaded
archive it would point
+ # consumers at the snapshot of an earlier attempt.
+ - name: "Retire CI image snapshot metadata ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ run: >
+ for meta in
"${RUNNER_TEMP}"/ci-image-snapshot-*-"${PYTHON_MAJOR_MINOR_VERSION}".tar.zst.meta;
+ do echo "unavailable" > "${meta}"; done
+ shell: bash
+ if: steps.snapshot-export.outcome == 'success' &&
steps.snapshot-upload.outcome != 'success'
+ # Uploaded even when the export fails, so its "unavailable" marker
retires the snapshot of an
+ # earlier attempt; consumers read it before downloading the snapshot.
+ - name: "Upload CI image snapshot metadata ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ continue-on-error: true
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
# v7.0.2
+ with:
+ name: >-
+ ci-image-snapshot-v1-${{ env.PYTHON_MAJOR_MINOR_VERSION }}-${{
+ inputs.platform == 'linux/amd64' && 'amd64' || 'arm64' }}-meta
+ path: "${{ runner.temp }}/ci-image-snapshot-*-${{
env.PYTHON_MAJOR_MINOR_VERSION }}.tar.zst.meta"
+ if-no-files-found: 'error'
+ retention-days: '1'
+ overwrite: true
+ if: steps.snapshot-export.outcome != 'skipped'
diff --git a/dev/breeze/doc/images/output_ci-image_load.svg
b/dev/breeze/doc/images/output_ci-image_load.svg
index d02615077d4..4ed12ccbef8 100644
--- a/dev/breeze/doc/images/output_ci-image_load.svg
+++ b/dev/breeze/doc/images/output_ci-image_load.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 830.8"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 904.0"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-load-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="779.8" />
+ <rect x="0" y="0" width="1463.0" height="853.0" />
</clipPath>
<clipPath id="breeze-ci-image-load-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -138,9 +138,18 @@
<clipPath id="breeze-ci-image-load-line-30">
<rect x="0" y="733.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-image-load-line-31">
+ <rect x="0" y="757.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-load-line-32">
+ <rect x="0" y="782.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-load-line-33">
+ <rect x="0" y="806.7" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="828.8" rx="8"/><text
class="breeze-ci-image-load-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci-image load</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="902" rx="8"/><text
class="breeze-ci-image-load-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci-image load</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -175,13 +184,16 @@
</text><text class="breeze-ci-image-load-r5" x="0" y="556.8" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-22)">│</text><text
class="breeze-ci-image-load-r1" x="414.8" y="556.8" textLength="256.2"
clip-path="url(#breeze-ci-image-load-line-22)">from the latest job. </text><text
class="breeze-ci-image-load-r7" x="671" y="556.8" textLength="73.2"
clip-path="url(#breeze-ci-image-load-line-22)">(TEXT)</text><text
class="breeze-ci-image-load-r5" x="1451.8" y=" [...]
</text><text class="breeze-ci-image-load-r5" x="0" y="581.2" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-23)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="581.2" textLength="317.2"
clip-path="url(#breeze-ci-image-load-line-23)">--from-run                </text><text
class="breeze-ci-image-load-r1" x="414.8" y="581.2" textLength="793"
clip-path="url(#breeze-ci-image-load-line-23)"> [...]
</text><text class="breeze-ci-image-load-r5" x="0" y="605.6" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-24)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="605.6" textLength="317.2"
clip-path="url(#breeze-ci-image-load-line-24)">--skip-image-file-deletion</text><text
class="breeze-ci-image-load-r1" x="414.8" y="605.6" textLength="414.8"
clip-path="url(#breeze-ci-image-load-line-24)">Skip image deletion after loading.</text><text
class="bree [...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="630" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-25)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="630" textLength="317.2"
clip-path="url(#breeze-ci-image-load-line-25)">--tag-as                  </text><text
class="breeze-ci-image-load-r1" x="414.8" y="630" textLength="671"
clip-path="url(#breeze-ci-image-load-line-2 [...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="654.4" textLength="1464"
clip-path="url(#breeze-ci-image-load-line-26)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-load-r1" x="1464" y="654.4" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-26)">
-</text><text class="breeze-ci-image-load-r5" x="0" y="678.8" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-27)">╭─</text><text
class="breeze-ci-image-load-r5" x="24.4" y="678.8" textLength="195.2"
clip-path="url(#breeze-ci-image-load-line-27)"> Common options </text><text
class="breeze-ci-image-load-r5" x="219.6" y="678.8" textLength="1220"
clip-path="url(#breeze-ci-image-load-line-27)">─────────────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-28)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="703.2" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-28)">--dry-run</text><text
class="breeze-ci-image-load-r6" x="158.6" y="703.2" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-28)">-D</text><text
class="breeze-ci-image-load-r1" x="207.4" y="703.2" textLength="719.8"
clip-path= [...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-29)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="727.6" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-29)">--verbose</text><text
class="breeze-ci-image-load-r6" x="158.6" y="727.6" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-29)">-v</text><text
class="breeze-ci-image-load-r1" x="207.4" y="727.6" textLength="585.6"
clip-path= [...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="752" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-30)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="752" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-30)">--help   </text><text
class="breeze-ci-image-load-r6" x="158.6" y="752" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-30)">-h</text><text
class="breeze-ci-image-load-r1" x="207.4" y="752" textLength="329.4" cli [...]
-</text><text class="breeze-ci-image-load-r5" x="0" y="776.4" textLength="1464"
clip-path="url(#breeze-ci-image-load-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-load-r1" x="1464" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-31)">
+</text><text class="breeze-ci-image-load-r5" x="0" y="630" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-25)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="630" textLength="317.2"
clip-path="url(#breeze-ci-image-load-line-25)">--snapshot-file           </text><text
class="breeze-ci-image-load-r1" x="414.8" y="630" textLength="1024.8"
clip-path="url(#breeze-ci-image-load-line-25)">Restore the image&
[...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="654.4" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-26)">│</text><text
class="breeze-ci-image-load-r1" x="414.8" y="654.4" textLength="1024.8"
clip-path="url(#breeze-ci-image-load-line-26)">Exits with the snapshot script's code when the snapshot does not fit this daemon.   </text><text
class="breeze-ci-image-load-r5" x="1451.8" y="654.4 [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-27)">│</text><text
class="breeze-ci-image-load-r7" x="414.8" y="678.8" textLength="73.2"
clip-path="url(#breeze-ci-image-load-line-27)">(FILE)</text><text
class="breeze-ci-image-load-r5" x="1451.8" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-27)">│</text><text
class="breeze-ci-image-load-r1" x="1464" y="678.8" textLength="12.2"
clip-path="url( [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-28)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="703.2" textLength="317.2"
clip-path="url(#breeze-ci-image-load-line-28)">--tag-as                  </text><text
class="breeze-ci-image-load-r1" x="414.8" y="703.2" textLength="671"
clip-path="url(#breeze-ci-image-load- [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="727.6" textLength="1464"
clip-path="url(#breeze-ci-image-load-line-29)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-load-r1" x="1464" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-29)">
+</text><text class="breeze-ci-image-load-r5" x="0" y="752" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-30)">╭─</text><text
class="breeze-ci-image-load-r5" x="24.4" y="752" textLength="195.2"
clip-path="url(#breeze-ci-image-load-line-30)"> Common options </text><text
class="breeze-ci-image-load-r5" x="219.6" y="752" textLength="1220"
clip-path="url(#breeze-ci-image-load-line-30)">───────────────────────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-31)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="776.4" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-31)">--dry-run</text><text
class="breeze-ci-image-load-r6" x="158.6" y="776.4" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-31)">-D</text><text
class="breeze-ci-image-load-r1" x="207.4" y="776.4" textLength="719.8"
clip-path= [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-32)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="800.8" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-32)">--verbose</text><text
class="breeze-ci-image-load-r6" x="158.6" y="800.8" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-32)">-v</text><text
class="breeze-ci-image-load-r1" x="207.4" y="800.8" textLength="585.6"
clip-path= [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="825.2" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-33)">│</text><text
class="breeze-ci-image-load-r4" x="24.4" y="825.2" textLength="109.8"
clip-path="url(#breeze-ci-image-load-line-33)">--help   </text><text
class="breeze-ci-image-load-r6" x="158.6" y="825.2" textLength="24.4"
clip-path="url(#breeze-ci-image-load-line-33)">-h</text><text
class="breeze-ci-image-load-r1" x="207.4" y="825.2" textLength="32 [...]
+</text><text class="breeze-ci-image-load-r5" x="0" y="849.6" textLength="1464"
clip-path="url(#breeze-ci-image-load-line-34)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-load-r1" x="1464" y="849.6" textLength="12.2"
clip-path="url(#breeze-ci-image-load-line-34)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_ci-image_load.txt
b/dev/breeze/doc/images/output_ci-image_load.txt
index 5fb965482c8..3240c4afd5d 100644
--- a/dev/breeze/doc/images/output_ci-image_load.txt
+++ b/dev/breeze/doc/images/output_ci-image_load.txt
@@ -1 +1 @@
-39e243d6fa50474dd1b3e83c6c388fd0
+bc1033ea3ef8562140765ee300ba64b4
diff --git a/dev/breeze/doc/images/output_ci-image_save.svg
b/dev/breeze/doc/images/output_ci-image_save.svg
index 5524e6ba1c7..05859507161 100644
--- a/dev/breeze/doc/images/output_ci-image_save.svg
+++ b/dev/breeze/doc/images/output_ci-image_save.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 513.5999999999999"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 562.4"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-image-save-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="462.59999999999997" />
+ <rect x="0" y="0" width="1463.0" height="511.4" />
</clipPath>
<clipPath id="breeze-ci-image-save-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -99,9 +99,15 @@
<clipPath id="breeze-ci-image-save-line-17">
<rect x="0" y="416.3" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-image-save-line-18">
+ <rect x="0" y="440.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-image-save-line-19">
+ <rect x="0" y="465.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="511.6" rx="8"/><text
class="breeze-ci-image-save-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci-image save</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="560.4" rx="8"/><text
class="breeze-ci-image-save-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci-image save</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -124,12 +130,14 @@
</text><text class="breeze-ci-image-save-r5" x="0" y="264" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-10)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="264" textLength="231.8"
clip-path="url(#breeze-ci-image-save-line-10)">--image-file       </text><text
class="breeze-ci-image-save-r1" x="329.4" y="264" textLength="439.2"
clip-path="url(#breeze-ci-image-save-line-10)">Optional file to save the image
[...]
</text><text class="breeze-ci-image-save-r5" x="0" y="288.4" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-11)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="288.4" textLength="231.8"
clip-path="url(#breeze-ci-image-save-line-11)">--image-file-dir   </text><text
class="breeze-ci-image-save-r1" x="329.4" y="288.4" textLength="866.2"
clip-path="url(#breeze-ci-image-save-line-11)">The path to the directory where the 
[...]
</text><text class="breeze-ci-image-save-r5" x="0" y="312.8" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-12)">│</text><text
class="breeze-ci-image-save-r7" x="329.4" y="312.8" textLength="134.2"
clip-path="url(#breeze-ci-image-save-line-12)">(DIRECTORY)</text><text
class="breeze-ci-image-save-r5" x="1451.8" y="312.8" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-12)">│</text><text
class="breeze-ci-image-save-r1" x="1464" y="312.8" textLength="12.2" clip-path
[...]
-</text><text class="breeze-ci-image-save-r5" x="0" y="337.2" textLength="1464"
clip-path="url(#breeze-ci-image-save-line-13)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-save-r1" x="1464" y="337.2" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-13)">
-</text><text class="breeze-ci-image-save-r5" x="0" y="361.6" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-14)">╭─</text><text
class="breeze-ci-image-save-r5" x="24.4" y="361.6" textLength="195.2"
clip-path="url(#breeze-ci-image-save-line-14)"> Common options </text><text
class="breeze-ci-image-save-r5" x="219.6" y="361.6" textLength="1220"
clip-path="url(#breeze-ci-image-save-line-14)">─────────────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-ci-image-save-r5" x="0" y="386" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-15)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="386" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-15)">--verbose</text><text
class="breeze-ci-image-save-r6" x="158.6" y="386" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-15)">-v</text><text
class="breeze-ci-image-save-r1" x="207.4" y="386" textLength="585.6"
clip-path="url(#br [...]
-</text><text class="breeze-ci-image-save-r5" x="0" y="410.4" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-16)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="410.4" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-16)">--dry-run</text><text
class="breeze-ci-image-save-r6" x="158.6" y="410.4" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-16)">-D</text><text
class="breeze-ci-image-save-r1" x="207.4" y="410.4" textLength="719.8"
clip-path= [...]
-</text><text class="breeze-ci-image-save-r5" x="0" y="434.8" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-17)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="434.8" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-17)">--help   </text><text
class="breeze-ci-image-save-r6" x="158.6" y="434.8" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-17)">-h</text><text
class="breeze-ci-image-save-r1" x="207.4" y="434.8" textLength="32 [...]
-</text><text class="breeze-ci-image-save-r5" x="0" y="459.2" textLength="1464"
clip-path="url(#breeze-ci-image-save-line-18)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-save-r1" x="1464" y="459.2" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-18)">
+</text><text class="breeze-ci-image-save-r5" x="0" y="337.2" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-13)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="337.2" textLength="231.8"
clip-path="url(#breeze-ci-image-save-line-13)">--snapshot-file    </text><text
class="breeze-ci-image-save-r1" x="329.4" y="337.2" textLength="1110.2"
clip-path="url(#breeze-ci-image-save-line-13)">Save a Docker data-root snapshot holdin
[...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="361.6" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-14)">│</text><text
class="breeze-ci-image-save-r1" x="329.4" y="361.6" textLength="756.4"
clip-path="url(#breeze-ci-image-save-line-14)">file. It removes every other image and the build cache first. </text><text
class="breeze-ci-image-save-r7" x="1085.8" y="361.6" textLength="73.2"
clip-path="url(#breeze-ci-image-s [...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="386" textLength="1464"
clip-path="url(#breeze-ci-image-save-line-15)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-save-r1" x="1464" y="386" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-15)">
+</text><text class="breeze-ci-image-save-r5" x="0" y="410.4" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-16)">╭─</text><text
class="breeze-ci-image-save-r5" x="24.4" y="410.4" textLength="195.2"
clip-path="url(#breeze-ci-image-save-line-16)"> Common options </text><text
class="breeze-ci-image-save-r5" x="219.6" y="410.4" textLength="1220"
clip-path="url(#breeze-ci-image-save-line-16)">─────────────────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="434.8" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-17)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="434.8" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-17)">--verbose</text><text
class="breeze-ci-image-save-r6" x="158.6" y="434.8" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-17)">-v</text><text
class="breeze-ci-image-save-r1" x="207.4" y="434.8" textLength="585.6"
clip-path= [...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="459.2" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-18)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="459.2" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-18)">--dry-run</text><text
class="breeze-ci-image-save-r6" x="158.6" y="459.2" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-18)">-D</text><text
class="breeze-ci-image-save-r1" x="207.4" y="459.2" textLength="719.8"
clip-path= [...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="483.6" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-19)">│</text><text
class="breeze-ci-image-save-r4" x="24.4" y="483.6" textLength="109.8"
clip-path="url(#breeze-ci-image-save-line-19)">--help   </text><text
class="breeze-ci-image-save-r6" x="158.6" y="483.6" textLength="24.4"
clip-path="url(#breeze-ci-image-save-line-19)">-h</text><text
class="breeze-ci-image-save-r1" x="207.4" y="483.6" textLength="32 [...]
+</text><text class="breeze-ci-image-save-r5" x="0" y="508" textLength="1464"
clip-path="url(#breeze-ci-image-save-line-20)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-image-save-r1" x="1464" y="508" textLength="12.2"
clip-path="url(#breeze-ci-image-save-line-20)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_ci-image_save.txt
b/dev/breeze/doc/images/output_ci-image_save.txt
index 273b299da9b..f04f007d717 100644
--- a/dev/breeze/doc/images/output_ci-image_save.txt
+++ b/dev/breeze/doc/images/output_ci-image_save.txt
@@ -1 +1 @@
-0a356161ee78afd4199f889d6f421f4e
+45653b55a46f2c1e43bc520ff5fe93e3
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index ee3e0d342b0..f39bd1d63a3 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -53,6 +53,8 @@ from airflow_breeze.commands.common_image_options import (
option_push,
option_python_image,
option_skip_image_file_deletion,
+ option_snapshot_file_to_load,
+ option_snapshot_file_to_save,
option_tag_as,
option_verify,
option_wait_for_image,
@@ -107,7 +109,7 @@ from airflow_breeze.utils.parallel import (
check_async_run_results,
run_with_pool,
)
-from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH, BUILD_CACHE_PATH
+from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH,
BUILD_CACHE_PATH, SCRIPTS_CI_PATH
from airflow_breeze.utils.python_versions import get_python_version_list
from airflow_breeze.utils.recording import generating_command_images
from airflow_breeze.utils.run_tests import verify_an_image
@@ -516,6 +518,7 @@ def run_verify_in_parallel(
@option_image_file_dir
@option_platform_single
@option_python
+@option_snapshot_file_to_save
@option_verbose
@option_dry_run
def save(
@@ -524,9 +527,13 @@ def save(
github_repository: str,
image_file: Path | None,
image_file_dir: Path,
+ snapshot_file: Path | None,
):
"""Save CI image to a file."""
perform_environment_checks()
+ if snapshot_file:
+ save_ci_image_snapshot(snapshot_file)
+ return
image_name = BuildCiParams(
python=python,
github_repository=github_repository,
@@ -560,6 +567,7 @@ def save(
@option_platform_single
@option_python
@option_skip_image_file_deletion
+@option_snapshot_file_to_load
@option_tag_as
@option_verbose
def load(
@@ -572,6 +580,7 @@ def load(
platform: str,
python: str,
skip_image_file_deletion: bool,
+ snapshot_file: Path | None,
tag_as: str | None,
):
"""Load CI image from a file."""
@@ -580,6 +589,10 @@ def load(
python=python,
github_repository=github_repository,
)
+ if snapshot_file:
+ restore_ci_image_snapshot(snapshot_file,
build_ci_params.airflow_image_name)
+ tag_and_mark_loaded_ci_image(build_ci_params, tag_as)
+ return
escaped_platform = platform.replace("/", "_")
if not image_file:
@@ -622,6 +635,41 @@ def load(
if result.returncode != 0:
console_print(f"[error]Error when loading image: {result.stdout}[/]")
sys.exit(result.returncode)
+ if not skip_image_file_deletion:
+ console_print(f"[info]Deleting image file {image_file_to_load}[/]")
+ image_file_to_load.unlink()
+ tag_and_mark_loaded_ci_image(build_ci_params, tag_as)
+
+
+def save_ci_image_snapshot(snapshot_file: Path) -> None:
+ """Save a Docker data-root snapshot holding only the CI image, exiting
when it cannot be made."""
+ console_print(f"[info]Saving Python CI image snapshot to
{snapshot_file}[/]")
+ result = run_command(
+ [os.fspath(SCRIPTS_CI_PATH / "docker_data_root_snapshot.sh"),
"create", os.fspath(snapshot_file)],
+ check=False,
+ )
+ if result.returncode != 0:
+ console_print(f"[warning]Snapshot {snapshot_file} not created[/]")
+ sys.exit(result.returncode)
+
+
+def restore_ci_image_snapshot(snapshot_file: Path, image_name: str) -> None:
+ """Restore the CI image from a Docker data-root snapshot, exiting when it
does not fit."""
+ console_print(f"[info]Restoring Python CI image from snapshot
{snapshot_file}[/]")
+ result = run_command(
+ [os.fspath(SCRIPTS_CI_PATH / "docker_data_root_snapshot.sh"),
"restore", os.fspath(snapshot_file)],
+ check=False,
+ )
+ if result.returncode != 0:
+ console_print(f"[warning]Snapshot {snapshot_file} not restored, load
the image file instead[/]")
+ sys.exit(result.returncode)
+ result = run_command(["docker", "image", "inspect", "--format", "{{.Id}}",
image_name], check=False)
+ if result.returncode != 0:
+ console_print(f"[error]The restored snapshot does not hold
{image_name}[/]")
+ sys.exit(result.returncode)
+
+
+def tag_and_mark_loaded_ci_image(build_ci_params: BuildCiParams, tag_as: str |
None) -> None:
if tag_as:
console_print(f"[info]Tagging {build_ci_params.airflow_image_name} as
{tag_as}[/]")
result = run_command(
@@ -631,9 +679,6 @@ def load(
if result.returncode != 0:
console_print(f"[error]Error when tagging image:
{result.stdout}[/]")
sys.exit(result.returncode)
- if not skip_image_file_deletion:
- console_print(f"[info]Deleting image file {image_file_to_load}[/]")
- image_file_to_load.unlink()
if get_verbose():
run_command(["docker", "images", "-a"])
mark_image_as_rebuilt(ci_image_params=build_ci_params)
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
index d8fb175538d..8ad22347f32 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py
@@ -167,6 +167,7 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str |
list[str]]]] = {
"--github-repository",
"--image-file",
"--image-file-dir",
+ "--snapshot-file",
],
},
],
@@ -183,6 +184,7 @@ CI_IMAGE_TOOLS_PARAMETERS: dict[str, list[dict[str, str |
list[str]]]] = {
"--from-pr",
"--from-run",
"--skip-image-file-deletion",
+ "--snapshot-file",
"--tag-as",
],
},
diff --git a/dev/breeze/src/airflow_breeze/commands/common_image_options.py
b/dev/breeze/src/airflow_breeze/commands/common_image_options.py
index 04a2c8d6a04..6142eb37a6f 100644
--- a/dev/breeze/src/airflow_breeze/commands/common_image_options.py
+++ b/dev/breeze/src/airflow_breeze/commands/common_image_options.py
@@ -216,6 +216,18 @@ option_skip_image_file_deletion = click.option(
is_flag=True,
envvar="SKIP_IMAGE_FILE_DELETION",
)
+option_snapshot_file_to_load = click.option(
+ "--snapshot-file",
+ help="Restore the image from this Docker data-root snapshot instead of an
image file. Exits with "
+ "the snapshot script's code when the snapshot does not fit this daemon.",
+ type=click.Path(dir_okay=False, path_type=Path),
+)
+option_snapshot_file_to_save = click.option(
+ "--snapshot-file",
+ help="Save a Docker data-root snapshot holding only the CI image to this
file instead of an image "
+ "file. It removes every other image and the build cache first.",
+ type=click.Path(dir_okay=False, path_type=Path),
+)
option_tag_as = click.option(
"--tag-as",
help="Additionally tag the loaded image with this reference.",
diff --git a/dev/breeze/tests/test_ci_image_commands.py
b/dev/breeze/tests/test_ci_image_commands.py
index 43c33330194..f85b279c798 100644
--- a/dev/breeze/tests/test_ci_image_commands.py
+++ b/dev/breeze/tests/test_ci_image_commands.py
@@ -28,6 +28,8 @@ from airflow_breeze.commands.ci_image_commands import (
get_ci_image_sources_hash_label,
import_mount_cache,
is_ci_image_built_from_current_sources,
+ load,
+ save,
)
from airflow_breeze.global_constants import CI_IMAGE_SOURCES_HASH_LABEL
from airflow_breeze.params.build_ci_params import BuildCiParams
@@ -206,3 +208,104 @@ def
test_import_mount_cache_does_not_prune_the_imported_cache_mount(
["docker", "rmi", "airflow-import-cache"],
["docker", "image", "prune", "-f"],
]
+
+
+def save_snapshot(snapshot_file, image_file_dir):
+ save.callback(
+ python="3.11",
+ platform="linux/amd64",
+ github_repository="apache/airflow",
+ image_file=None,
+ image_file_dir=image_file_dir,
+ snapshot_file=snapshot_file,
+ )
+
+
+def load_snapshot(snapshot_file, image_file_dir):
+ load.callback(
+ from_run=None,
+ from_pr=None,
+ github_repository="apache/airflow",
+ github_token=None,
+ image_file=None,
+ image_file_dir=image_file_dir,
+ platform="linux/amd64",
+ python="3.11",
+ skip_image_file_deletion=False,
+ snapshot_file=snapshot_file,
+ tag_as=None,
+ )
+
+
[email protected]("airflow_breeze.commands.ci_image_commands.mark_image_as_rebuilt",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.run_command",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.perform_environment_checks",
autospec=True)
+def test_load_restores_snapshot_with_the_checks_of_an_image_file_load(
+ mock_perform_environment_checks, mock_run_command,
mock_mark_image_as_rebuilt, tmp_path
+):
+ mock_run_command.return_value = mock.MagicMock(returncode=0)
+ snapshot_file = tmp_path / "ci-image-snapshot-linux_amd64-3.11.tar.zst"
+
+ load_snapshot(snapshot_file, tmp_path)
+
+ commands = [call.args[0] for call in mock_run_command.call_args_list]
+ assert commands[0][1:] == ["restore", str(snapshot_file)]
+ assert commands[0][0].endswith("scripts/ci/docker_data_root_snapshot.sh")
+ assert commands[1] == ["docker", "image", "inspect", "--format",
"{{.Id}}", CI_IMAGE]
+ assert not [command for command in commands if command[:3] == ["docker",
"image", "load"]]
+ mock_perform_environment_checks.assert_called_once()
+ mock_mark_image_as_rebuilt.assert_called_once()
+
+
[email protected](
+ ("returncodes", "expected_exit"),
+ [pytest.param([3], 3, id="snapshot-does-not-fit"), pytest.param([0, 1], 1,
id="image-missing")],
+)
[email protected]("airflow_breeze.commands.ci_image_commands.mark_image_as_rebuilt",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.run_command",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.perform_environment_checks",
autospec=True)
+def test_load_exits_when_snapshot_is_not_restored(
+ mock_perform_environment_checks,
+ mock_run_command,
+ mock_mark_image_as_rebuilt,
+ tmp_path,
+ returncodes,
+ expected_exit,
+):
+ mock_run_command.side_effect = [mock.MagicMock(returncode=code) for code
in returncodes]
+
+ with pytest.raises(SystemExit) as exit_info:
+ load_snapshot(tmp_path / "ci-image-snapshot-linux_amd64-3.11.tar.zst",
tmp_path)
+
+ assert exit_info.value.code == expected_exit
+ mock_mark_image_as_rebuilt.assert_not_called()
+
+
[email protected]("airflow_breeze.commands.ci_image_commands.run_command",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.perform_environment_checks",
autospec=True)
+def test_save_creates_snapshot_with_the_snapshot_script(
+ mock_perform_environment_checks, mock_run_command, tmp_path
+):
+ mock_run_command.return_value = mock.MagicMock(returncode=0)
+ snapshot_file = tmp_path / "ci-image-snapshot-linux_amd64-3.11.tar.zst"
+
+ save_snapshot(snapshot_file, tmp_path)
+
+ commands = [call.args[0] for call in mock_run_command.call_args_list]
+ assert len(commands) == 1
+ assert commands[0][0].endswith("scripts/ci/docker_data_root_snapshot.sh")
+ assert commands[0][1:] == ["create", str(snapshot_file)]
+ mock_perform_environment_checks.assert_called_once()
+
+
[email protected]("airflow_breeze.commands.ci_image_commands.run_command",
autospec=True)
[email protected]("airflow_breeze.commands.ci_image_commands.perform_environment_checks",
autospec=True)
+def test_save_exits_with_the_snapshot_script_code(
+ mock_perform_environment_checks, mock_run_command, tmp_path
+):
+ mock_run_command.return_value = mock.MagicMock(returncode=3)
+
+ with pytest.raises(SystemExit) as exit_info:
+ save_snapshot(tmp_path / "ci-image-snapshot-linux_amd64-3.11.tar.zst",
tmp_path)
+
+ assert exit_info.value.code == 3
diff --git a/scripts/ci/docker_data_root_snapshot.sh
b/scripts/ci/docker_data_root_snapshot.sh
new file mode 100755
index 00000000000..350d89f774a
--- /dev/null
+++ b/scripts/ci/docker_data_root_snapshot.sh
@@ -0,0 +1,207 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Moves the CI image between jobs as a copy of the Docker data directory that
holds it, so a job
+# restores it with one extraction instead of `docker image load` unpacking and
checksumming every
+# layer again.
+#
+# docker_data_root_snapshot.sh create SNAPSHOT_FILE
+# Creates the snapshot after the builder publishes its caches.
SNAPSHOT_FILE.meta reads
+# "unavailable" until the archive is complete, so a failed rerun retires
an earlier snapshot.
+# docker_data_root_snapshot.sh check SNAPSHOT_FILE
+# Checks SNAPSHOT_FILE.meta against this checkout and daemon before the
archive is downloaded.
+# docker_data_root_snapshot.sh restore SNAPSHOT_FILE
+# Restores the image. Incompatible snapshots leave the daemon untouched.
Failed or overdue
+# materialization cleans partial image state and restarts Docker for the
normal stash path.
+set -euo pipefail
+
+DATA_ROOT="/var/lib/docker"
+# overlay2 records opaque directories in trusted.* attributes and file
capabilities in security.*;
+# GNU tar extracts only user.* attributes unless told otherwise.
+TAR_ATTRIBUTES=(--xattrs "--xattrs-include=*" --acls --numeric-owner)
+# Every blocking step is bounded, so a stalled restore fails fast instead of
holding the job.
+RESTORE_TIMEOUT_SECONDS="${SNAPSHOT_RESTORE_TIMEOUT_SECONDS:-300}"
+DAEMON_TIMEOUT_SECONDS=60
+DAEMON_STOPPED=false
+RESTORE_IN_PROGRESS=false
+RESTORE_FILE=""
+IMAGE_ID=""
+EXPECTED_IMAGE=""
+
+function cleanup_daemon() {
+ local result=$?
+ trap - EXIT
+ if [[ -n "${RESTORE_FILE}" ]]; then
+ rm -f "${RESTORE_FILE}" "${RESTORE_FILE}.meta"
+ fi
+ if [[ "${RESTORE_IN_PROGRESS}" == true ]]; then
+ stop_daemon
+ remove_image_store
+ start_daemon
+ elif [[ "${DAEMON_STOPPED}" == true ]]; then
+ start_daemon
+ fi
+ exit "${result}"
+}
+trap cleanup_daemon EXIT
+
+function check_supported_daemon() {
+ local daemon=$1
+ local _ driver root
+ read -r _ driver _ root <<< "${daemon}"
+ if [[ "${driver}" != overlay2 || "${root}" != "${DATA_ROOT}" ]]; then
+ echo "::warning::Unsupported Docker image store; skipping snapshot:
${daemon}"
+ exit 3
+ fi
+}
+
+# A data directory is only readable by the daemon version and storage driver
that wrote it.
+function daemon_fingerprint() {
+ docker info --format '{{.ServerVersion}} {{.Driver}} {{.Architecture}}
{{.DockerRootDir}}'
+}
+
+function stop_daemon() {
+ DAEMON_STOPPED=true
+ sudo timeout "${DAEMON_TIMEOUT_SECONDS}" systemctl stop docker.socket
docker
+}
+
+function start_daemon() {
+ sudo timeout "${DAEMON_TIMEOUT_SECONDS}" systemctl start docker
+ DAEMON_STOPPED=false
+}
+
+# Identifies the image by its ID and by the file tree its layers combine into:
a restore that loses
+# a whiteout or an opaque-directory marker changes the tree while the ID stays
the same. The listing
+# is sorted on the host, since sorting it in the container would add sort's
temporary files to it.
+function image_fingerprint() {
+ local image=$1
+ local image_id tree
+ image_id="$(docker image inspect --format '{{.Id}}' "${image}")" || return
1
+ tree="$(timeout "${DAEMON_TIMEOUT_SECONDS}" docker run --rm --pull never
--network none \
+ --entrypoint find "${image}" / -xdev -printf '%p %y %m %U %G %s\n' |
LC_ALL=C sort | sha256sum)" \
+ || return 1
+ echo "${image_id} ${tree%% *}"
+}
+
+function remove_image_store() {
+ sudo rm -rf "${DATA_ROOT}/image" "${DATA_ROOT}/overlay2"
+}
+
+function create_snapshot() {
+ local snapshot_file="${1}"
+ local daemon image_id revision fingerprint image_fingerprint_id image_tree
+ rm -f "${snapshot_file}"
+ echo "unavailable" > "${snapshot_file}.meta"
+ daemon="$(daemon_fingerprint)"
+ check_supported_daemon "${daemon}"
+ image_id="$(docker images --quiet --filter
'label=org.apache.airflow.image=airflow-ci' | sort -u)"
+ if [[ -z "${image_id}" || "${image_id}" == *$'\n'* ]]; then
+ echo "Expected exactly one CI image in the daemon, found:
'${image_id}'" >&2
+ exit 1
+ fi
+ revision="$(git rev-parse HEAD)"
+ fingerprint="$(image_fingerprint "${image_id}")"
+ read -r image_fingerprint_id image_tree <<< "${fingerprint}"
+ docker ps --all --quiet | xargs --no-run-if-empty docker rm --force
>/dev/null
+ docker images --quiet | sort -u | {
+ grep --invert-match --fixed-strings --line-regexp "${image_id}" || true
+ } | xargs --no-run-if-empty docker rmi --force >/dev/null
+ docker builder prune --all --force >/dev/null
+ stop_daemon
+ sudo tar --directory "${DATA_ROOT}" "${TAR_ATTRIBUTES[@]}" --create --file
- image overlay2 \
+ | zstd -3 -T0 --quiet --force -o "${snapshot_file}"
+ printf '%s %s %s %s\n' "${daemon}" "${image_fingerprint_id}" "${revision}"
"${image_tree}" \
+ > "${snapshot_file}.meta"
+ start_daemon
+}
+
+# Exits without touching the daemon unless the snapshot was made from this
checkout by a daemon
+# like this one, and this daemon holds nothing the restore would drop.
+function check_snapshot() {
+ local snapshot_file="${1}"
+ local -a meta
+ local snapshot_fingerprint daemon existing_images existing_containers
+ if [[ ! -f "${snapshot_file}.meta" ]]; then
+ echo "No snapshot at ${snapshot_file}"
+ exit 2
+ fi
+ read -r -a meta < "${snapshot_file}.meta"
+ if [[ ${#meta[@]} != 7 ]]; then
+ echo "Snapshot unavailable or its metadata is invalid"
+ exit 3
+ fi
+ if [[ "${meta[5]}" != "$(git rev-parse HEAD)" ]]; then
+ echo "Snapshot belongs to a different checkout revision"
+ exit 3
+ fi
+ snapshot_fingerprint="${meta[*]:0:4}"
+ daemon="$(daemon_fingerprint)"
+ check_supported_daemon "${daemon}"
+ if [[ "${snapshot_fingerprint}" != "${daemon}" ]]; then
+ echo "::warning::Snapshot daemon fingerprint mismatch:
'${snapshot_fingerprint}' != '${daemon}'"
+ exit 3
+ fi
+ existing_images="$(docker images --all --quiet)"
+ existing_containers="$(docker ps --all --quiet)"
+ if [[ -n "${existing_images}" || -n "${existing_containers}" ]]; then
+ echo "The daemon already holds images, which restoring the snapshot
would drop"
+ exit 3
+ fi
+ IMAGE_ID="${meta[4]}"
+ EXPECTED_IMAGE="${meta[4]} ${meta[6]}"
+}
+
+function restore_snapshot() {
+ local snapshot_file="${1}"
+ local restored_image
+ RESTORE_FILE="${snapshot_file}"
+ if [[ ! -f "${snapshot_file}" ]]; then
+ echo "No snapshot at ${snapshot_file}"
+ exit 2
+ fi
+ check_snapshot "${snapshot_file}"
+ stop_daemon
+ RESTORE_IN_PROGRESS=true
+ remove_image_store
+ timeout "${RESTORE_TIMEOUT_SECONDS}" zstd -d -T0 --quiet --stdout
"${snapshot_file}" \
+ | sudo timeout "${RESTORE_TIMEOUT_SECONDS}" \
+ tar --directory "${DATA_ROOT}" "${TAR_ATTRIBUTES[@]}" --extract
--file -
+ start_daemon
+ if ! restored_image="$(image_fingerprint "${IMAGE_ID}")" \
+ || [[ "${restored_image}" != "${EXPECTED_IMAGE}" ]]; then
+ echo "The restored image does not match the snapshot:
'${restored_image:-}' != '${EXPECTED_IMAGE}'"
+ exit 4
+ fi
+ RESTORE_IN_PROGRESS=false
+}
+
+case "${1:-}" in
+ create)
+ create_snapshot "${2}"
+ ;;
+ check)
+ check_snapshot "${2}"
+ ;;
+ restore)
+ restore_snapshot "${2}"
+ ;;
+ *)
+ echo "Usage: ${0} {create|check|restore} SNAPSHOT_FILE" >&2
+ exit 1
+ ;;
+esac
diff --git a/scripts/tests/ci/test_docker_data_root_snapshot.py
b/scripts/tests/ci/test_docker_data_root_snapshot.py
new file mode 100644
index 00000000000..3a7b394cc30
--- /dev/null
+++ b/scripts/tests/ci/test_docker_data_root_snapshot.py
@@ -0,0 +1,459 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import hashlib
+import os
+import shlex
+import subprocess
+from pathlib import Path
+
+import pytest
+import yaml
+
+SCRIPT = Path(__file__).resolve().parents[2] / "ci" /
"docker_data_root_snapshot.sh"
+WORKFLOW = SCRIPT.parents[2] / ".github" / "workflows" / "ci-image-build.yml"
+ACTIONS = SCRIPT.parents[2] / ".github" / "actions"
+LOADER = ACTIONS / "load_image" / "action.yml"
+FINGERPRINT = "28.5.2 overlay2 x86_64 /var/lib/docker"
+STOP = ["sudo", "timeout", "60", "systemctl", "stop", "docker.socket",
"docker"]
+START = ["sudo", "timeout", "60", "systemctl", "start", "docker"]
+REMOVE_STORE = ["sudo", "rm", "-rf", "/var/lib/docker/image",
"/var/lib/docker/overlay2"]
+TREE = hashlib.sha256(b"/ d 755\n/tmp d 1777\n").hexdigest()
+
+
[email protected]
+def fake_tools(tmp_path):
+ """Shadow ``docker``, ``sudo``, ``zstd`` and ``git`` with a logger
answering the script's queries."""
+ tools = tmp_path / "bin"
+ tools.mkdir()
+ command = tools / "command"
+ command.write_text(
+ r"""#!/usr/bin/env bash
+name="${0##*/}"
+printf '%s\n' "${name} $*" >> "${COMMAND_LOG}"
+if [[ "${name}" == "git" && "$*" == "rev-parse HEAD" ]]; then
+ echo "${CHECKOUT_SHA:-revision1}"
+fi
+if [[ "${name}" == "zstd" ]]; then
+ if [[ -n "${ZSTD_SLEEP:-}" ]]; then
+ exec sleep "${ZSTD_SLEEP}"
+ fi
+ exit "${ZSTD_EXIT:-0}"
+fi
+if [[ "${name}" == "sudo" && ("$*" == "tar "* || "$*" == "timeout "*" tar "*)
]]; then
+ exit "${TAR_EXIT:-0}"
+fi
+if [[ "${name}" == "sudo" && "$*" == *"systemctl stop docker.socket docker" &&
"${STOP_EXIT:-0}" != 0 ]]; then
+ exit "${STOP_EXIT}"
+fi
+if [[ "${name}" == "sudo" && "$*" == *"systemctl start docker" &&
"${START_FAIL_ONCE:-0}" == 1 ]]; then
+ if [[ $(grep -c 'systemctl start docker' "${COMMAND_LOG}") == 1 ]]; then
+ exit 1
+ fi
+fi
+if [[ "${name}" == "docker" ]]; then
+ case "$*" in
+ "info --format "*) echo "${DAEMON_FINGERPRINT}" ;;
+ "images --quiet --filter label=org.apache.airflow.image=airflow-ci")
printf '%b' "${CI_IMAGES:-}" ;;
+ "images --quiet") printf '%b' "${ALL_IMAGES:-}" ;;
+ "images --all --quiet") printf '%b' "${ALL_IMAGES:-}"; exit
"${IMAGES_EXIT:-0}" ;;
+ "ps --all --quiet") printf '%b' "${CONTAINERS:-}"; exit
"${CONTAINERS_EXIT:-0}" ;;
+ "image inspect --format {{.Id}} "*) echo
"${IMAGE_DIGEST:-sha256:abc123}" ;;
+ "run "*)
+ if [[ "${RUN_EXIT:-0}" != 0 ]]; then
+ exit "${RUN_EXIT}"
+ fi
+ printf '%b' "${TREE_LISTING:-/tmp d 1777\\n/ d 755\\n}"
+ ;;
+ esac
+fi
+"""
+ )
+ command.chmod(0o755)
+ for name in ("docker", "sudo", "zstd", "git"):
+ (tools / name).symlink_to(command)
+ return {
+ "PATH": f"{tools}:{os.environ['PATH']}",
+ "COMMAND_LOG": str(tmp_path / "commands.log"),
+ "DAEMON_FINGERPRINT": FINGERPRINT,
+ }
+
+
[email protected]
+def snapshot(tmp_path):
+ snapshot_file = tmp_path / "ci-image-snapshot-linux_amd64-3.10.tar.zst"
+ snapshot_file.write_bytes(b"")
+ Path(f"{snapshot_file}.meta").write_text(f"{FINGERPRINT} sha256:abc123
revision1 {TREE}\n")
+ return snapshot_file
+
+
+def run_script(env, *args):
+ command = ["bash", "--noprofile", "--norc", str(SCRIPT), *args]
+ result = subprocess.run(
+ command,
+ env={**os.environ, **env},
+ capture_output=True,
+ text=True,
+ timeout=15,
+ check=False,
+ )
+ if args[0] == "restore":
+ assert not Path(args[1]).exists()
+ assert not Path(f"{args[1]}.meta").exists()
+ return result
+
+
+def read_commands(env):
+ log = Path(env["COMMAND_LOG"])
+ return [shlex.split(line) for line in log.read_text().splitlines()] if
log.exists() else []
+
+
+def is_tar(command, mode):
+ return command[0] == "sudo" and "tar" in command and mode in command
+
+
+def steps(path, *keys):
+ document = yaml.safe_load(path.read_text())
+ for key in keys:
+ document = document[key]
+ return document
+
+
+def test_restore_unpacks_snapshot_into_stopped_daemon(fake_tools, snapshot):
+ result = run_script(fake_tools, "restore", str(snapshot))
+
+ assert result.returncode == 0, result.stdout + result.stderr
+ commands = read_commands(fake_tools)
+ stop = commands.index(STOP)
+ remove = commands.index(REMOVE_STORE)
+ extract = next(i for i, c in enumerate(commands) if is_tar(c, "--extract"))
+ start = commands.index(START)
+ inspect = commands.index(["docker", "image", "inspect", "--format",
"{{.Id}}", "sha256:abc123"])
+ run = next(i for i, c in enumerate(commands) if c[:2] == ["docker", "run"])
+ assert stop < remove < extract < start < inspect < run
+ assert commands[run][:9] == [
+ "docker",
+ "run",
+ "--rm",
+ "--pull",
+ "never",
+ "--network",
+ "none",
+ "--entrypoint",
+ "find",
+ ]
+ assert commands[run][9:12] == ["sha256:abc123", "/", "-xdev"]
+ assert commands[extract][:3] == ["sudo", "timeout", "300"]
+ assert "--xattrs-include=*" in commands[extract]
+ assert not snapshot.exists()
+ assert not Path(f"{snapshot}.meta").exists()
+
+
[email protected](
+ ("env", "remove_meta", "expected_exit"),
+ [
+ pytest.param({}, True, 2, id="no-snapshot"),
+ pytest.param(
+ {"DAEMON_FINGERPRINT": "29.0.0 overlay2 x86_64 /var/lib/docker"},
False, 3, id="other-daemon"
+ ),
+ pytest.param({"ALL_IMAGES": "def456\\n"}, False, 3,
id="daemon-holds-images"),
+ ],
+)
+def test_restore_leaves_daemon_alone_when_snapshot_does_not_fit(
+ fake_tools, snapshot, env, remove_meta, expected_exit
+):
+ if remove_meta:
+ Path(f"{snapshot}.meta").unlink()
+
+ result = run_script({**fake_tools, **env}, "restore", str(snapshot))
+
+ assert result.returncode == expected_exit
+ if "DAEMON_FINGERPRINT" in env:
+ assert "::warning::" in result.stdout
+ assert not [c for c in read_commands(fake_tools) if "systemctl" in c]
+
+
[email protected](
+ "env",
+ [{"RUN_EXIT": "1"}, {"TREE_LISTING": "/ d 755\\n/data d 755\\n"},
{"IMAGE_DIGEST": "sha256:other"}],
+ ids=["does-not-run", "other-file-tree", "other-image"],
+)
+def test_restore_empties_store_again_when_image_does_not_match(fake_tools,
snapshot, env):
+ result = run_script({**fake_tools, **env}, "restore", str(snapshot))
+
+ assert result.returncode == 4
+ commands = read_commands(fake_tools)
+ start = commands.index(START)
+ after_start = commands[start + 1 :]
+ assert REMOVE_STORE in after_start
+ assert after_start[-1] == START
+
+
+def test_create_keeps_only_the_ci_image(fake_tools, tmp_path):
+ snapshot_file = tmp_path / "out.tar.zst"
+ env = {**fake_tools, "CI_IMAGES": "abc123\\nabc123\\n", "ALL_IMAGES":
"abc123\\nother1\\nabc123\\n"}
+
+ result = run_script(env, "create", str(snapshot_file))
+
+ assert result.returncode == 0, result.stdout + result.stderr
+ commands = read_commands(env)
+ assert ["docker", "rmi", "--force", "other1"] in commands
+ assert not [c for c in commands if c[:2] == ["docker", "rmi"] and "abc123"
in c]
+ assert ["docker", "builder", "prune", "--all", "--force"] in commands
+ assert Path(f"{snapshot_file}.meta").read_text() == f"{FINGERPRINT}
sha256:abc123 revision1 {TREE}\n"
+ fingerprint = next(i for i, c in enumerate(commands) if c[:2] ==
["docker", "run"])
+ assert fingerprint < commands.index(STOP)
+ archive = next(c for c in commands if is_tar(c, "--create"))
+ assert archive[-2:] == ["image", "overlay2"]
+ assert "--xattrs-include=*" in archive
+
+
+def test_create_refuses_ambiguous_ci_image(fake_tools, tmp_path):
+ env = {**fake_tools, "CI_IMAGES": "abc123\\ndef456\\n"}
+
+ result = run_script(env, "create", str(tmp_path / "out.tar.zst"))
+
+ assert result.returncode == 1
+ assert not [c for c in read_commands(env) if "systemctl" in c]
+
+
[email protected](
+ "failure",
+ [
+ {"ZSTD_EXIT": "1"},
+ {"TAR_EXIT": "1"},
+ {"START_FAIL_ONCE": "1"},
+ {"ZSTD_SLEEP": "10", "SNAPSHOT_RESTORE_TIMEOUT_SECONDS": "1"},
+ ],
+ ids=["zstd", "tar", "start", "stalled"],
+)
+def
test_restore_recovers_empty_running_daemon_after_materialization_failure(fake_tools,
snapshot, failure):
+ result = run_script({**fake_tools, **failure}, "restore", str(snapshot))
+ assert result.returncode != 0
+ commands = read_commands(fake_tools)
+ assert commands[-2:] == [
+ REMOVE_STORE,
+ START,
+ ]
+ assert not any(command[:2] == ["docker", "run"] for command in commands)
+
+
[email protected]("failure", [{"ZSTD_EXIT": "1"}, {"TAR_EXIT": "1"}])
+def test_create_restarts_daemon_after_archive_failure(fake_tools, tmp_path,
failure):
+ snapshot_file = tmp_path / "snapshot.tar.zst"
+ result = run_script({**fake_tools, **failure, "CI_IMAGES": "abc123"},
"create", str(snapshot_file))
+ assert result.returncode != 0
+ assert read_commands(fake_tools)[-1] == START
+ assert Path(f"{snapshot_file}.meta").read_text() == "unavailable\n"
+
+
+def test_failed_create_retires_earlier_snapshot(fake_tools, tmp_path):
+ snapshot_file = tmp_path / "snapshot.tar.zst"
+ snapshot_file.write_bytes(b"earlier attempt")
+ Path(f"{snapshot_file}.meta").write_text(f"{FINGERPRINT} sha256:abc123
revision1 {TREE}\n")
+
+ result = run_script({**fake_tools, "CI_IMAGES": ""}, "create",
str(snapshot_file))
+
+ assert result.returncode == 1
+ assert not snapshot_file.exists()
+ assert Path(f"{snapshot_file}.meta").read_text() == "unavailable\n"
+
+
+def test_check_accepts_fitting_snapshot_before_download(fake_tools, snapshot):
+ snapshot.unlink()
+
+ result = run_script(fake_tools, "check", str(snapshot))
+
+ assert result.returncode == 0, result.stdout + result.stderr
+ assert Path(f"{snapshot}.meta").exists()
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
[email protected](
+ "env",
+ [
+ {"DAEMON_FINGERPRINT": "29.0.0 overlay2 x86_64 /var/lib/docker"},
+ {"ALL_IMAGES": "def456\\n"},
+ {"CHECKOUT_SHA": "revision2"},
+ ],
+ ids=["other-daemon", "daemon-holds-images", "other-revision"],
+)
+def test_check_rejects_snapshot_that_does_not_fit(fake_tools, snapshot, env):
+ result = run_script({**fake_tools, **env}, "check", str(snapshot))
+
+ assert result.returncode == 3
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
+def test_check_rejects_unavailable_marker(fake_tools, snapshot):
+ Path(f"{snapshot}.meta").write_text("unavailable\n")
+
+ result = run_script(fake_tools, "check", str(snapshot))
+
+ assert result.returncode == 3
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
[email protected]("mode", ["create", "restore"])
[email protected]("daemon", ["28.5.2 btrfs x86_64 /var/lib/docker",
"28.5.2 overlay2 x86_64 /other"])
+def test_unsupported_storage_does_not_modify_daemon(fake_tools, snapshot,
mode, daemon):
+ result = run_script({**fake_tools, "DAEMON_FINGERPRINT": daemon}, mode,
str(snapshot))
+ assert result.returncode == 3
+ assert "::warning::" in result.stdout
+ commands = read_commands(fake_tools)
+ assert not any(command[0] == "sudo" for command in commands)
+ assert not any(command[:2] in (["docker", "rmi"], ["docker", "builder"])
for command in commands)
+ if mode == "create":
+ assert Path(f"{snapshot}.meta").read_text() == "unavailable\n"
+
+
+def test_existing_container_prevents_restore(fake_tools, snapshot):
+ result = run_script({**fake_tools, "CONTAINERS": "container1"}, "restore",
str(snapshot))
+ assert result.returncode == 3
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
[email protected]("failure", [{"IMAGES_EXIT": "1"}, {"CONTAINERS_EXIT":
"1"}])
+def test_failed_daemon_inventory_does_not_modify_store(fake_tools, snapshot,
failure):
+ result = run_script({**fake_tools, **failure}, "restore", str(snapshot))
+
+ assert result.returncode != 0
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
+def test_malformed_metadata_prevents_restore(fake_tools, snapshot):
+ Path(f"{snapshot}.meta").write_text("invalid\n")
+ result = run_script(fake_tools, "restore", str(snapshot))
+ assert result.returncode == 3
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
+def test_stale_snapshot_does_not_modify_daemon(fake_tools, snapshot):
+ result = run_script({**fake_tools, "CHECKOUT_SHA": "revision2"},
"restore", str(snapshot))
+ assert result.returncode == 3
+ assert not any(command[0] == "sudo" for command in
read_commands(fake_tools))
+
+
[email protected]("mode", ["create", "restore"])
+def test_partial_stop_failure_restarts_daemon(fake_tools, snapshot, mode):
+ result = run_script({**fake_tools, "STOP_EXIT": "1", "CI_IMAGES":
"abc123"}, mode, str(snapshot))
+ assert result.returncode != 0
+ assert read_commands(fake_tools)[-1] == START
+ assert not any(command[:2] == ["sudo", "rm"] for command in
read_commands(fake_tools))
+
+
+def test_snapshot_creation_uses_builder_after_cache_publication():
+ workflow = yaml.safe_load(WORKFLOW.read_text())
+ steps = workflow["jobs"]["build-ci-images"]["steps"]
+ snapshot = next(i for i, step in enumerate(steps) if step.get("id") ==
"snapshot-export")
+ cache_publishers = [
+ i
+ for i, step in enumerate(steps)
+ if "/stash/save@" in step.get("uses", "") or step.get("uses",
"").endswith("/stash-save")
+ ]
+ assert cache_publishers
+ assert all(publisher < snapshot for publisher in cache_publishers)
+
+
+def test_snapshot_artifacts_replace_earlier_attempts():
+ uploads = [
+ step
+ for step in steps(WORKFLOW, "jobs", "build-ci-images", "steps")
+ if "/upload-artifact@" in step.get("uses", "") and
step["name"].startswith("Upload CI image snapshot")
+ ]
+ assert len(uploads) == 2
+ assert all(upload["with"]["overwrite"] is True for upload in uploads)
+ archive, metadata = uploads
+ assert archive["if"] == "steps.snapshot-export.outcome == 'success'"
+ assert metadata["if"] == "steps.snapshot-export.outcome != 'skipped'"
+ assert metadata["with"]["path"].endswith(".tar.zst.meta")
+
+
+def test_failed_archive_upload_retires_metadata_before_its_upload():
+ workflow_steps = steps(WORKFLOW, "jobs", "build-ci-images", "steps")
+ by_name = {step["name"].split(" ${{")[0]: i for i, step in
enumerate(workflow_steps)}
+ retire = workflow_steps[by_name["Retire CI image snapshot metadata"]]
+ assert by_name["Upload CI image snapshot"] < by_name["Retire CI image
snapshot metadata"]
+ assert by_name["Retire CI image snapshot metadata"] < by_name["Upload CI
image snapshot metadata"]
+ assert workflow_steps[by_name["Upload CI image snapshot"]]["id"] ==
"snapshot-upload"
+ assert retire["if"] == (
+ "steps.snapshot-export.outcome == 'success' &&
steps.snapshot-upload.outcome != 'success'"
+ )
+ assert 'echo "unavailable" > "${meta}"' in retire["run"]
+
+
+def test_snapshot_artifact_names_match_between_upload_and_restore():
+ uploads = [
+ step["with"]["name"].replace("env.PYTHON_MAJOR_MINOR_VERSION",
"inputs.python")
+ for step in steps(WORKFLOW, "jobs", "build-ci-images", "steps")
+ if "/upload-artifact@" in step.get("uses", "") and
step["name"].startswith("Upload CI image snapshot")
+ ]
+ loader = {step.get("id"): step for step in steps(LOADER, "runs", "steps")}
+ assert uploads == [
+ loader["snapshot-download"]["with"]["name"],
+ loader["snapshot-meta"]["with"]["pattern"],
+ ]
+ assert uploads[0].startswith("ci-image-snapshot-v1-")
+
+
+def test_loader_checks_snapshot_before_downloading_it():
+ loader = steps(LOADER, "runs", "steps")
+ order = [step.get("id") for step in loader]
+ assert order[:4] == ["snapshot-meta", "snapshot-check",
"snapshot-download", "snapshot"]
+ by_id = {step.get("id"): step for step in loader}
+ assert by_id["snapshot-download"]["if"] ==
"steps.snapshot-check.outputs.fits == 'true'"
+ assert " check " in by_id["snapshot-check"]["run"]
+ assert "name" not in by_id["snapshot-meta"]["with"]
+ assert by_id["snapshot-meta"]["with"]["merge-multiple"] is True
+ assert "breeze ci-image load" in by_id["snapshot"]["run"]
+ stash = [step for step in loader if "stash" in step.get("uses", "")]
+ assert [step["uses"] for step in stash] ==
["./.github/actions/stash-restore"]
+ assert stash[0]["if"] == "steps.snapshot.outputs.restored != 'true'"
+ assert "--snapshot-file" in by_id["snapshot"]["run"]
+
+
+def test_snapshot_is_created_and_restored_through_breeze():
+ export = next(
+ step
+ for step in steps(WORKFLOW, "jobs", "build-ci-images", "steps")
+ if step.get("id") == "snapshot-export"
+ )
+ restore = next(step for step in steps(LOADER, "runs", "steps") if
step.get("id") == "snapshot")
+ assert "breeze ci-image save" in export["run"] and "--snapshot-file" in
export["run"]
+ assert "breeze ci-image load" in restore["run"] and "--snapshot-file" in
restore["run"]
+
+
+def test_snapshot_is_limited_to_github_hosted_runners():
+ export = next(
+ step
+ for step in steps(WORKFLOW, "jobs", "build-ci-images", "steps")
+ if step.get("id") == "snapshot-export"
+ )
+ metadata = next(step for step in steps(LOADER, "runs", "steps") if
step.get("id") == "snapshot-meta")
+ assert "runner.environment == 'github-hosted'" in export["if"]
+ assert "runner.environment == 'github-hosted'" in metadata["if"]
+
+
[email protected]("action", ["prepare_breeze_and_image",
"prepare_single_ci_image"])
+def test_image_preparing_actions_share_the_loader(action):
+ action_steps = steps(ACTIONS / action / "action.yml", "runs", "steps")
+ assert [step["uses"] for step in action_steps if "load_image" in
step.get("uses", "")] == [
+ "./.github/actions/load_image"
+ ]
+ assert not [step for step in action_steps if "image load" in
step.get("run", "")]
+ assert not [step for step in action_steps if "stash" in step.get("uses",
"")]
