shahar1 commented on code in PR #74173:
URL: https://github.com/apache/airflow/pull/74173#discussion_r4178551814
##########
.github/workflows/ci-image-build.yml:
##########
@@ -401,3 +401,60 @@ jobs:
steps.stashed-image.outputs.reusable != 'true'
- name: "Check disk space after build"
run: df -H
+
+ # Run checkout code in a separate job with no branch-cache publications.
This optional
+ # producer hands the snapshot only to consumers of the same workflow run.
+ snapshot-ci-images:
+ name: "Snapshot CI ${{ inputs.platform }} image ${{ matrix.python-version
}}"
+ needs: build-ci-images
+ if: >
+ github.event_name == 'pull_request' &&
+ inputs.upload-image-artifact == 'true' && inputs.image-stash-ref == '' &&
+ (inputs.checkout-ref == '' || inputs.checkout-ref == github.sha)
+ continue-on-error: true
+ timeout-minutes: 20
+ runs-on: ${{ fromJSON(inputs.runners) }}
+ permissions:
+ contents: read
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ${{ fromJSON(inputs.python-versions) }}
+ env:
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ steps:
+ - name: "Checkout sources"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ ref: ${{ github.sha }}
+ persist-credentials: false
+ - name: "Prepare authoritative CI image"
Review Comment:
This step took 178 s of the job's 229 s in run 37206844037: it restores and
`docker image load`s the stash that `build-ci-images` exported seconds earlier.
Creating the snapshot inside `build-ci-images` (after "Stash cache mount",
since `create` runs `docker builder prune --all`) removes this reload entirely;
the daemon there still reports `DockerRootDir=/var/lib/docker` because
`move_docker_to_mnt.sh` bind-mounts rather than reconfigures.
##########
.github/workflows/ci-image-build.yml:
##########
@@ -401,3 +401,60 @@ jobs:
steps.stashed-image.outputs.reusable != 'true'
- name: "Check disk space after build"
run: df -H
+
+ # Run checkout code in a separate job with no branch-cache publications.
This optional
+ # producer hands the snapshot only to consumers of the same workflow run.
+ snapshot-ci-images:
Review Comment:
Every `ci-amd.yml` job with `needs: build-ci-images` waits for the whole
called workflow, and this job is part of it (`continue-on-error` changes the
conclusion, not the wait). In this PR's run the first consumer started at
13:59:55, 3 min 56 s after the build job finished at 13:55:59; in a baseline
run from the same afternoon the gap was 0 s. The ~87 s saved per consumer is
paid with ~236 s added before any consumer starts, so end-to-end PR time gets
longer. See the review body for the measured numbers and the suggestion to
create the snapshot as the last step of `build-ci-images` instead.
##########
scripts/ci/docker_data_root_snapshot.sh:
##########
@@ -0,0 +1,154 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Moves the CI image between jobs as a copy of the Docker data directory that
holds it, so a job
+# restores it with one extraction instead of `docker image load` unpacking and
checksumming every
+# layer again.
+#
+# docker_data_root_snapshot.sh create SNAPSHOT_FILE
+# Keeps only the CI image in the daemon and writes its image store to
SNAPSHOT_FILE, plus
+# SNAPSHOT_FILE.meta naming the daemon that wrote it and the image.
+# docker_data_root_snapshot.sh restore SNAPSHOT_FILE
+# Restores the image. Incompatible snapshots leave the daemon untouched.
Failed
+# materialization cleans partial image state and restarts Docker for the
normal stash path.
+set -euo pipefail
+
+DATA_ROOT="/var/lib/docker"
+DAEMON_STOPPED=false
+RESTORE_IN_PROGRESS=false
+
+function cleanup_daemon() {
+ local result=$?
+ trap - EXIT
+ if [[ "${RESTORE_IN_PROGRESS}" == true ]]; then
+ stop_daemon
+ remove_image_store
+ start_daemon
+ elif [[ "${DAEMON_STOPPED}" == true ]]; then
+ start_daemon
+ fi
+ exit "${result}"
+}
+trap cleanup_daemon EXIT
+
+function check_supported_daemon() {
+ local daemon=$1
+ local _ driver root
+ read -r _ driver _ root <<< "${daemon}"
+ if [[ "${driver}" != overlay2 || "${root}" != "${DATA_ROOT}" ]]; then
Review Comment:
Minor: when runner images move to the containerd image store this check
rejects every snapshot, the producer keeps spending its minutes, and nothing
surfaces it because the create step is `continue-on-error`. An `echo
"::warning::..."` here and on the fingerprint-mismatch exit would make the
silent no-op visible in the run summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
