On Wed, Jun 26, 2024 at 4:04 PM Tom Lane <t...@sss.pgh.pa.us> wrote: > Thomas Munro <thomas.mu...@gmail.com> writes: > > But I thought of an easier way: instead of trying to do my own cache > > invalidation with shell script and duct tape, I can include the > > current OS major version in the cache key used to carry the > > macports directory between CI runs. Hopefully Cirrus's cache machinery > > is smart enough to age out the old stuff eventually. > > Sounds reasonable.
cfbot didn't like v2. It seems that github accounts using "instance:OSXCommunityInstance" are forced to use ghcr.io/cirruslabs/macos-runner:sonoma no matter what you ask for (example: [1]), while accounts configured to use user-supplied runners like the Mac Minis that cfbot is using *can't* use ghcr.io/cirruslabs/macos-runner:sonoma, and fail (example: [2]). I don't know why. So I think we should request ghcr.io/cirruslabs/macos-sonoma-base:latest. Personal github accounts will use macos-runner:sonoma instead, but at least it's the same OS release. Here's a new version like that, to see if cfbot likes it. Given that the OS release affects the macports_url we have to specify, I think this either means that we'll have to stay in sync with whatever macOS version is being forced for "instance:OSXCommunityInstance" users, or construct the macports_url automatically. Here is an attempt at the latter, as a second patch. Seems to work OK. For example, the setup_additional_packages step currently prints out: [06:23:08.584] macOS major version: 14 [06:23:09.672] MacPorts package URL: https://github.com/macports/macports-base/releases/download/v2.9.3/MacPorts-2.9.3-14-Sonoma.pkg As for the difference between the two types of image, they're described at [3]. The -runner images seem to be part of a project for faster starting VMs[4], which sounds like a pretty good reason to want to standardise on images to make pre-started instances fungible but there is perhaps also potential for selecting different xcode versions. > > FTR there is a newer macOS release that recently came out, Sequoia aka > > macOS 15, but the image available to us for CI is marked beta so I > > figured we can wait a bit longer for that. > > Indeed not; that's only beta and will be so till September-ish. > We don't really want to touch it yet because of this issue: > > https://www.postgresql.org/message-id/flat/CAMBWrQnEwEJtgOv7EUNsXmFw2Ub4p5P%2B5QTBEgYwiyjy7rAsEQ%40mail.gmail.com > > I'm not sure what the resolution of that will be, but we surely > don't want to gate CI improvement on that. Urgh. Also we have to wait for MacPorts to make a release for Sequoia, which might involve lots of maintainers hunting stuff like that. (If Cirrus starts forcing people to use Sequoia before then, that'd be a problem.) [1] https://cirrus-ci.com/task/4747151899623424 [2] https://cirrus-ci.com/task/6601239016767488 [3] https://github.com/cirruslabs/macos-image-templates [4] https://cirrus-runners.app/blog/2024/04/11/optimizing-startup-time-of-cirrus-runners/
From 07ca3cbefe03a220319485462ec8778b12d8a8a6 Mon Sep 17 00:00:00 2001 From: Thomas Munro <thomas.mu...@gmail.com> Date: Wed, 26 Jun 2024 10:43:50 +1200 Subject: [PATCH v3 1/2] ci: Upgrade macOS version from 13 to 14. We also need to change the macports_url to an installer for 14, and change the cache key we use to cache the whole macports installation to include the Darwin major version, since otherwise it would break. Back-patch to 15 where CI began. Discussion: https://postgr.es/m/CA%2BhUKGLqJdv6RcwyZ_0H7khxtLTNJyuK%2BvDFzv3uwYbn8hKH6A%40mail.gmail.com --- .cirrus.tasks.yml | 10 ++++++++-- src/tools/ci/ci_macports_packages.sh | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 33646faeadf..7cdc6bd26e3 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -411,7 +411,7 @@ task: task: - name: macOS - Ventura - Meson + name: macOS - Sonoma - Meson env: CPUS: 4 # always get that much for cirrusci macOS instances @@ -420,7 +420,9 @@ task: # work OK. See # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de TEST_JOBS: 8 - IMAGE: ghcr.io/cirruslabs/macos-ventura-base:latest + # If changing the macOS release, you probably also need to change + # macports_url in src/tools/ci/ci_macports_packages.sh + IMAGE: ghcr.io/cirruslabs/macos-sonoma-base:latest CIRRUS_WORKING_DIR: ${HOME}/pgsql/ CCACHE_DIR: ${HOME}/ccache @@ -460,6 +462,10 @@ task: # updates macports every time. macports_cache: folder: ${MACPORTS_CACHE} + fingerprint_script: | + # Include the darwin version in the cache key. If the OS image changes + # to a different major version, we need to reinstall. + uname -r | sed 's/\..*//' setup_additional_packages_script: | sh src/tools/ci/ci_macports_packages.sh \ ccache \ diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index f87256e0908..db811a28c8e 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -13,7 +13,7 @@ set -e packages="$@" -macports_url="https://github.com/macports/macports-base/releases/download/v2.8.1/MacPorts-2.8.1-13-Ventura.pkg"; +macports_url="https://github.com/macports/macports-base/releases/download/v2.9.3/MacPorts-2.9.3-14-Sonoma.pkg"; cache_dmg="macports.hfs.dmg" if [ "$CIRRUS_CI" != "true" ]; then -- 2.39.2
From 12959500616da20ff5a012cda83003951cb7de0a Mon Sep 17 00:00:00 2001 From: Thomas Munro <thomas.mu...@gmail.com> Date: Thu, 27 Jun 2024 16:21:13 +1200 Subject: [PATCH v3 2/2] ci: Find appropriate MacPorts package automatically. Instead of hard-coding a MacPorts installation URL, deduce it from the running macOS version and the available releases. This removes the need to keep the ci_macports_packages.sh in sync with .cirrus.task.yml, and to advance the MacPorts version from time to time. It also helps smooth over the transition if Cirrus decides to force an incompatible MacOS image on some users. Discussion: https://postgr.es/m/CA%2BhUKGLqJdv6RcwyZ_0H7khxtLTNJyuK%2BvDFzv3uwYbn8hKH6A%40mail.gmail.com --- .cirrus.tasks.yml | 2 -- src/tools/ci/ci_macports_packages.sh | 14 +++++++++++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 7cdc6bd26e3..a95c04ef1f3 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -420,8 +420,6 @@ task: # work OK. See # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de TEST_JOBS: 8 - # If changing the macOS release, you probably also need to change - # macports_url in src/tools/ci/ci_macports_packages.sh IMAGE: ghcr.io/cirruslabs/macos-sonoma-base:latest CIRRUS_WORKING_DIR: ${HOME}/pgsql/ diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index db811a28c8e..a002eca7e30 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -13,7 +13,19 @@ set -e packages="$@" -macports_url="https://github.com/macports/macports-base/releases/download/v2.9.3/MacPorts-2.9.3-14-Sonoma.pkg"; +macos_major_version="` sw_vers | grep ProductVersion | sed 's/[^0-9]*//;s/\..*//' `" +echo "macOS major version: $macos_major_version" + +# Scan the avialable MacPorts releases to find the latest one for the +# running macOS release. By default we assume the first match is the most +# recent MacPorts version but that can be changed below if it turns out to be +# problematic or a particular MacPorts release turns out to be broken. +macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; +macports_version_pattern=".*" +#macports_version_pattern="2\.9\.3" +macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$macports_version_pattern-$macos_major_version-[A-Za-z]*\.pkg\""; | sed 's/.*: "//;s/".*//' | head -1 )" +echo "MacPorts package URL: $macports_url" + cache_dmg="macports.hfs.dmg" if [ "$CIRRUS_CI" != "true" ]; then -- 2.39.2
