On Fri, Nov 04, 2022 at 06:59:46PM -0700, Andres Freund wrote:
> Hi,
>
> On 2022-11-04 18:54:12 -0500, Justin Pryzby wrote:
> > Subject: [PATCH 1/8] meson: PROVE is not required
> > Subject: [PATCH 3/8] meson: rename 'main' tasks to 'regress' and 'isolation'
>
> Pushed, thanks for the patches.
Thanks.
> > diff --git a/.cirrus.yml b/.cirrus.yml
> > index 9f2282471a9..99ac09dc679 100644
> > --- a/.cirrus.yml
> > +++ b/.cirrus.yml
> > @@ -451,12 +451,20 @@ task:
> >
> > build_script: |
> > vcvarsall x64
> > - ninja -C build
> > + ninja -C build |tee build/meson-logs/build.txt
> > + REM Since pipes lose exit status of the preceding command, rerun
> > compilation,
> > + REM without the pipe exiting now if it fails, rather than trying to
> > run checks
> > + ninja -C build > nul
>
> This seems mighty grotty :(. but I guess it's quick enough not worry about,
> and I can't come up with a better plan.
>
> It doesn't seem quite right to redirect into meson-logs/ to me, my
> interpretation is that that's "meson's namespace". Why not just store it in
> build/?
I put it there so it'd be included with the build artifacts.
Maybe it's worth adding a separate line to artifacts for stuff like
this, and ccache log ?
> > From e85fe83fd8a4b4c79a96d2bf66cd6a5e1bdfcd1e Mon Sep 17 00:00:00 2001
> > From: Justin Pryzby <pryz...@telsasoft.com>
> > Date: Sat, 26 Feb 2022 19:34:35 -0600
> > Subject: [PATCH 5/8] cirrus: build docs as a separate task..
> > + # Exercise HTML and other docs:
> > + ninja_docs_build_script: |
> > + mkdir build.ninja
> > + cd build.ninja
>
> Perhaps build-ninja instead? build.ninja is the filename for ninja's build
> instructions, so it seems a bit confusing.
Sure.
Do you think building docs with both autoconf and meson is what's
desirable here ?
I'm not sure if this ought to be combined with/before/after your "move
compilerwarnings task to meson" patch? (Regarding that patch: I
mentioned that it shouldn't use ccache -C, and it should use
meson_log_artifacts.)
> > From: Justin Pryzby <pryz...@telsasoft.com>
> > Date: Tue, 26 Jul 2022 20:30:02 -0500
> > Subject: [PATCH 6/8] cirrus/ccache: add explicit cache keys..
> >
> > Since otherwise, building with ci-os-only will probably fail to use the
> > normal cache, since the cache key is computed using both the task name
> > and its *index* in the list of caches (internal/executor/cache.go:184).
>
> Seems like this would potentially better addressed by reporting a bug to the
> cirrus folks?
You said that before, but I don't think so - since they wrote code to do
that, it's odd to file a bug that says that the behavior is wrong. I am
curious why, but it seems delibrate.
https://www.postgresql.org/message-id/20220828171029.GO2342%40telsasoft.com
> There's enough copies of this to make it worth deduplicating. If we
> use something like fingerprint_script: echo
> ccache/$CIRRUS_BRANCH/$CIRRUS_OS we can use a yaml ref?
>
>
>
I'll look into it...
> I think you experimented with creating a 'base' ccache dir (e.g. on the master
> branch) and then using branch specific secondar caches?
I have to revisit that sometime.
That's a new feaure in ccache 4.4, which is currently only in macos.
This is another thing that it'd be easier to test by having cfbot
clobber the cirrus.yml rather than committing to postgres repo.
(Technically, it should probably only do use the in-testing cirrus.yml
if the patch it's testing doesn't itself modify .cirrus.yml)
> How did that turn out? I think cfbot's caches constantly get removed
> due to overrunning the global space.
For cfbot, I don't know if there's much hope that any patch-specific
build artifacts will be cached from the previous run, typically ~24h
prior.
One idea I have, for the "Warnings" task (and maybe linux too), is to
defer pruning until after all the compilations. To avoid LRU pruning
during early tasks causing bad hit ratios of later tasks.
Another thing that probably happens is that task1 starts compiling
patch1, and then another instance of task1 starts compiling patch2. A
bit later, the first instance will upload its ccache result for patch1,
which will be summarily overwritten by the second instance's compilation
result, which doesn't include anything from the first instance.
Also, whenever ccache hits its MAXSIZE threshold, it prunes the cache
down to 80% of the configured size, which probably wipes away everything
from all but the most recent ~20 builds.
I also thought about having separate caches for each compilation in the
warnings task - but that requires too much repeated yaml just for that..
> > From: Justin Pryzby <pryz...@telsasoft.com>
> > Date: Sun, 3 Apr 2022 00:10:20 -0500
> > Subject: [PATCH 7/8] cirrus/ccache: disable compression and show stats
> >
> > linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte
> > 109616
> > macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
> > freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
> > XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte 51179
>
> I'm still somewhat doubtful this is a good idea. The mingw cache is huge, for
> example, and all that additional IO and memory usage is bound to show up.
I think you're referring to the proposed mingw task which runs under
windows, and not the existing cross-compilation ?
And you're right - I remember this now (I think it's due to PCH?)
In my local copy I'd "unset CCACHE_NOCOMPRESS". But I view that as an
oddity of windows headers, rather than an argument against disabling
compression elsewhere. BTW, freebsd ccache is too old to use
compression.
What about using CCACHE_HARDLINK (which implies no compression) ?
> > Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
> > The log should be written *outside* the ccache folder - it shouldn't be
> > preserved across cirrusci task invocations.
>
> I assume we don't have a new enough ccache everywhere yet?
No - see above.
I've added patches to update macos.
--
Justin
>From 12d90810ad86afbad4d692cf27ac0b363433a382 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Wed, 25 May 2022 21:53:22 -0500
Subject: [PATCH 01/11] cirrus/windows: add compiler_warnings_script
I'm not sure how to write this test in windows shell; it's also not easy to
write it in posix sh, since windows shell is somehow interpretting && and ||...
https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de
See also:
8a1ce5e54f6d144e4f8e19af7c767b026ee0c956
https://cirrus-ci.com/task/6241060062494720
https://cirrus-ci.com/task/6496366607204352
ci-os-only: windows
---
.cirrus.yml | 10 +++++++++-
src/tools/ci/windows-compiler-warnings | 24 ++++++++++++++++++++++++
2 files changed, 33 insertions(+), 1 deletion(-)
create mode 100755 src/tools/ci/windows-compiler-warnings
diff --git a/.cirrus.yml b/.cirrus.yml
index 9f2282471a9..7596b2058c6 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -451,12 +451,20 @@ task:
build_script: |
vcvarsall x64
- ninja -C build
+ ninja -C build |tee build/meson-logs/build.txt
+ REM Since pipes lose the exit status of the preceding command, rerun the compilation
+ REM without the pipe, exiting now if it fails, to avoid trying to run checks
+ ninja -C build > nul
check_world_script: |
vcvarsall x64
meson test %MTEST_ARGS% --num-processes %TEST_JOBS%
+ # This should be last, so check_world is always run
+ always:
+ compiler_warnings_script:
+ - sh src\tools\ci\windows-compiler-warnings build/meson-logs/build.txt build/meson-logs/build-warnings.txt
+
on_failure:
<<: *on_failure_meson
crashlog_artifacts:
diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings
new file mode 100755
index 00000000000..685aa995d5c
--- /dev/null
+++ b/src/tools/ci/windows-compiler-warnings
@@ -0,0 +1,24 @@
+#! /bin/sh
+# Success if the given exists and is empty, else fail
+# This is a separate file to avoid dealing with windows shell quoting and escaping,
+# or giving the impression that one-liner scripts will work trivially.
+set -e
+
+infile=$1
+outfile=$2
+
+# Looks like:
+# [19:39:27.130] c:\cirrus\src\backend\backup\basebackup_zstd.c(80) : warning C4715: 'bbsink_zstd_new': not all control paths return a value
+# should include linker warnings?
+grep ": warning " "$infile" > $outfile ||
+ [ $? -eq 1 ]
+
+if [ -s "$outfile" ]
+then
+ # Display the file's content, then exit indicating failure
+ cat "$outfile"
+ exit 1
+else
+ # Success
+ exit 0
+fi
--
2.25.1
>From 1df20b9bd9883a2f77618ac9e7611752128ddad1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sat, 26 Feb 2022 19:34:35 -0600
Subject: [PATCH 02/11] cirrus: build docs as a separate task..
This will run the doc build if any docs have changed, even if Linux
fails, to allow catch doc build failures.
XXX: it'd be nice to have cfbot inject this cirrus.yml for a couple days
to check that changesInclude() is working as intended.
This automatically shows up as a separate "column" on cfbot.
Also, in the future, this will hopefully upload each patch's changed HTML docs
as an artifact, for easy review.
ci-os-only: html
---
.cirrus.yml | 49 +++++++++++++++++++++++++++++++++++--------------
1 file changed, 35 insertions(+), 14 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 7596b2058c6..a3b84d3b1c0 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -580,20 +580,6 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- ###
- # Verify docs can be built
- ###
- # XXX: Only do this if there have been changes in doc/ since last build
- always:
- docs_build_script: |
- time ./configure \
- --cache gcc.cache \
- CC="ccache gcc" \
- CXX="ccache g++" \
- CLANG="ccache clang"
- make -s -j${BUILD_JOBS} clean
- time make -s -j${BUILD_JOBS} -C doc
-
###
# Verify headerscheck / cpluspluscheck succeed
#
@@ -617,3 +603,38 @@ task:
always:
upload_caches: ccache
+
+
+###
+# Verify docs can be built
+# changesInclude() will skip this task if none of the commits since
+# CIRRUS_LAST_GREEN_CHANGE touched any relevant files. The comparison appears
+# to be like "git log a..b -- ./file", not "git diff a..b -- ./file"
+###
+
+task:
+ name: Documentation
+
+ env:
+ CPUS: 1
+ BUILD_JOBS: 1
+
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
+ skip: "!changesInclude('.cirrus.yml', 'doc/**')"
+
+ container:
+ image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
+ cpu: $CPUS
+ memory: 2G
+
+ # Exercise HTML and other docs:
+ ninja_docs_build_script: |
+ mkdir build-ninja
+ cd build-ninja
+ time meson setup
+ time ninja docs
+
+ # Again, with autoconf
+ docs_build_script: |
+ time ./configure
+ make -s -j${BUILD_JOBS} -C doc
--
2.25.1
>From 8f3dd74efa522b0db0ca27456d1b9717b2bd8447 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Tue, 26 Jul 2022 20:30:02 -0500
Subject: [PATCH 03/11] cirrus/ccache: add explicit cache keys..
Since otherwise, building with ci-os-only will probably fail to use the
normal cache, since the cache key is computed using both the task name
and its *index* in the list of caches (internal/executor/cache.go:184).
---
.cirrus.yml | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index a3b84d3b1c0..061c8e1c7df 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -87,6 +87,9 @@ task:
ccache_cache:
folder: $CCACHE_DIR
+ fingerprint_key: ccache/freebsd
+ reupload_on_changes: true
+
# Workaround around performance issues due to 32KB block size
repartition_script: src/tools/ci/gcp_freebsd_repartition.sh
create_user_script: |
@@ -181,6 +184,8 @@ task:
ccache_cache:
folder: ${CCACHE_DIR}
+ fingerprint_key: ccache/linux
+ reupload_on_changes: true
sysinfo_script: |
id
@@ -359,6 +364,9 @@ task:
ccache_cache:
folder: $CCACHE_DIR
+ fingerprint_key: ccache/macos
+ reupload_on_changes: true
+
configure_script: |
brewpath="/usr/local"
PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
@@ -510,6 +518,8 @@ task:
ccache_cache:
folder: $CCACHE_DIR
+ fingerprint_key: ccache/warnings
+ reupload_on_changes: true
setup_additional_packages_script: |
#apt-get update
--
2.25.1
>From 54338b5a26c7eb8dcd0586bf93d67a1cb5c06803 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Wed, 2 Nov 2022 14:14:38 -0500
Subject: [PATCH 04/11] cirrus/ccache: avoid clearing cache until after all
compilations
Local storage use is of no concern, but it's bad if we have lower cache
hit ratio due to pruning which occurs during earlier compilations.
---
.cirrus.yml | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 061c8e1c7df..47c4caf51ac 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -236,7 +236,7 @@ task:
- name: Linux - Debian Bullseye - Meson
env:
- CCACHE_MAXSIZE: "400M" # tests two different builds
+ CCACHE_MAXSIZE: "800M" # tests two different builds
configure_script: |
su postgres <<-EOF
@@ -287,6 +287,10 @@ task:
PYTHONCOERCECLOCALE=0 LANG=C meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS}
EOF
+ ccache_stats_end_script:
+ ccache -s
+ CCACHE_MAXSIZE=400M ccache --cleanup
+
on_failure:
<<: *on_failure_meson
@@ -492,8 +496,10 @@ task:
BUILD_JOBS: 4
# Use larger ccache cache, as this task compiles with multiple compilers /
- # flag combinations
- CCACHE_MAXSIZE: "1GB"
+ # flag combinations. The cache size is set high now, and manually pruned later,
+ # to avoid cache cleaning cycles during the early compilations pruning
+ # compilation results which could've been used during later compilations.
+ CCACHE_MAXSIZE: "2G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -612,6 +618,10 @@ task:
time make -s cpluspluscheck EXTRAFLAGS='-fmax-errors=10'
always:
+ ccache_stats_end_script:
+ ccache -s
+ CCACHE_MAXSIZE=1G ccache --cleanup
+
upload_caches: ccache
--
2.25.1
>From 1914ac033f7fe416c691961c27f2b08cefb701b0 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 3 Apr 2022 00:10:20 -0500
Subject: [PATCH 05/11] cirrus/ccache: disable compression and show stats
Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x. But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should be shown and/or logged.
ccache --show-stats shows the *cumulative* stats (including prior
compilations)
ccache --zero-stats clears out not only the global stats, but the
per-file cache stats (from which the global stats are derived) - which
obviously makes the cache work poorly.
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
The log should be written *outside* the ccache folder - it shouldn't be
preserved across cirrusci task invocations.
freebsd, linux, macos
ci-os-only: linux
---
.cirrus.yml | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 47c4caf51ac..d9660149a73 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,11 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "750M"
+ CCACHE_HARDLINK: 1
+ #CCACHE_NOCOMPRESS: 1
+ # The statslog should be stored somewhere *outside* the persistent ccache dir
+ CCACHE_STATSLOG: ${CIRRUS_WORKING_DIR}/ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -236,7 +240,7 @@ task:
- name: Linux - Debian Bullseye - Meson
env:
- CCACHE_MAXSIZE: "800M" # tests two different builds
+ CCACHE_MAXSIZE: "2400M" # tests two different builds
configure_script: |
su postgres <<-EOF
@@ -289,7 +293,7 @@ task:
ccache_stats_end_script:
ccache -s
- CCACHE_MAXSIZE=400M ccache --cleanup
+ CCACHE_MAXSIZE=1200M ccache --cleanup
on_failure:
<<: *on_failure_meson
@@ -392,7 +396,7 @@ task:
-DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
build
- build_script: ninja -C build -j${BUILD_JOBS}
+ build_script: ninja -C build -j${BUILD_JOBS} && ccache --show-log-stats
upload_caches: ccache
test_world_script: |
@@ -499,7 +503,7 @@ task:
# flag combinations. The cache size is set high now, and manually pruned later,
# to avoid cache cleaning cycles during the early compilations pruning
# compilation results which could've been used during later compilations.
- CCACHE_MAXSIZE: "2G"
+ CCACHE_MAXSIZE: "6G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -620,7 +624,7 @@ task:
always:
ccache_stats_end_script:
ccache -s
- CCACHE_MAXSIZE=1G ccache --cleanup
+ CCACHE_MAXSIZE=3G ccache --cleanup
upload_caches: ccache
--
2.25.1
>From 61e27060dd249f38ce2fd96a6031101e95c492e5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Tue, 19 Jul 2022 12:38:45 -0500
Subject: [PATCH 06/11] cirrus/warnings: use a single/common 'always' block
ci-os-only: warnings
---
.cirrus.yml | 44 +++++++++++++++++++-------------------------
1 file changed, 19 insertions(+), 25 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index d9660149a73..6fb8ed0cded 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -526,11 +526,6 @@ task:
clang -v
export
- ccache_cache:
- folder: $CCACHE_DIR
- fingerprint_key: ccache/warnings
- reupload_on_changes: true
-
setup_additional_packages_script: |
#apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y install ...
@@ -545,8 +540,13 @@ task:
# different compilers to build with different combinations of dtrace on/off
# and cassert on/off.
- # gcc, cassert off, dtrace on
always:
+ ccache_cache:
+ folder: $CCACHE_DIR
+ fingerprint_key: ccache/warnings
+ reupload_on_changes: true
+
+ # gcc, cassert off, dtrace on
gcc_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -556,8 +556,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # gcc, cassert on, dtrace off
- always:
+ # gcc, cassert on, dtrace off
gcc_a_warning_script: |
time ./configure \
--cache gcc.cache \
@@ -567,8 +566,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert off, dtrace off
- always:
+ # clang, cassert off, dtrace off
clang_warning_script: |
time ./configure \
--cache clang.cache \
@@ -577,8 +575,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # clang, cassert on, dtrace on
- always:
+ # clang, cassert on, dtrace on
clang_a_warning_script: |
time ./configure \
--cache clang.cache \
@@ -589,8 +586,7 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- # cross-compile to windows
- always:
+ # cross-compile to windows
mingw_cross_warning_script: |
time ./configure \
--host=x86_64-w64-mingw32 \
@@ -600,16 +596,15 @@ task:
make -s -j${BUILD_JOBS} clean
time make -s -j${BUILD_JOBS} world-bin
- ###
- # Verify headerscheck / cpluspluscheck succeed
- #
- # - Don't use ccache, the files are uncacheable, polluting ccache's
- # cache
- # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose
- # - XXX have to disable ICU to avoid errors:
- # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de
- ###
- always:
+ ###
+ # Verify headerscheck / cpluspluscheck succeed
+ #
+ # - Don't use ccache, the files are uncacheable, polluting ccache's
+ # cache
+ # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose
+ # - XXX have to disable ICU to avoid errors:
+ # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de
+ ###
headers_headerscheck_script: |
time ./configure \
${LINUX_CONFIGURE_FEATURES} \
@@ -621,7 +616,6 @@ task:
headers_cpluspluscheck_script: |
time make -s cpluspluscheck EXTRAFLAGS='-fmax-errors=10'
- always:
ccache_stats_end_script:
ccache -s
CCACHE_MAXSIZE=3G ccache --cleanup
--
2.25.1
>From b76dd329bcf6af63de96ed26d8bb1945149467c7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 13 Nov 2022 14:03:27 -0600
Subject: [PATCH 07/11] cirrus: clean up windows task
- remove CIRRUS_ESCAPING_PROCESSES, which wasn't needed since the task
was converted to meson in e6b6ea025;
- name the task the same as the other tasks using meson;
- use the otherwise-unused TAR variable;
---
.cirrus.yml | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 6fb8ed0cded..3045278f4b1 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -410,7 +410,7 @@ task:
task:
- name: Windows - Server 2019, VS 2019 - Meson & ninja
+ name: Windows - Server 2019, VS 2019 - Meson
env:
# Half the allowed per-user CPU cores
@@ -428,13 +428,6 @@ task:
PG_TEST_USE_UNIX_SOCKETS: 1
PG_REGRESS_SOCK_DIR: "c:/cirrus/"
- # startcreate_script starts a postgres instance that we don't want to get
- # killed at the end of that script (it's stopped in stop_script). Can't
- # trivially use background_scripts because a) need pg_ctl's dropping of
- # permissions b) need to wait for startup to have finished, and we don't
- # currently have a tool for that...
- CIRRUS_ESCAPING_PROCESSES: 1
-
# Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That
# prevents crash reporting from working unless binaries do SetErrorMode()
# themselves. Furthermore, it appears that either python or, more likely,
@@ -463,7 +456,7 @@ task:
# Use /DEBUG:FASTLINK to avoid high memory usage during linking
configure_script: |
vcvarsall x64
- meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dssl=openssl -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=c:/windows/system32/tar.exe -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
+ meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dssl=openssl -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build
build_script: |
vcvarsall x64
--
2.25.1
>From 1ce768f023526313b86df24a32c8584a2fb578e1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 11 Nov 2022 21:14:39 -0600
Subject: [PATCH 08/11] cirrus: switch to "macos_instance"..
This uses an M1 ARM processor rather than intel.
See:
https://cirrus-ci.org/blog/2022/11/08/sunsetting-intel-macos-instances/
https://github.com/justinpryzby/postgres/runs/9446478561
| warning We are sunsetting support for Intel-based macOS instances! Please migrate before January 1st 2023.
ci-os-only: macos
---
.cirrus.yml | 10 ++++++----
src/test/kerberos/t/001_auth.pl | 14 ++++++++++++--
2 files changed, 18 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 3045278f4b1..ca49b2973ac 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -306,7 +306,7 @@ task:
name: macOS - Monterey - Meson
env:
- CPUS: 12 # always get that much for cirrusci macOS instances
+ CPUS: 4 # always get that much for cirrusci macOS instances
BUILD_JOBS: $CPUS
# Test performance regresses noticably when using all cores. 8 seems to
# work OK. See
@@ -326,8 +326,10 @@ task:
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
- osx_instance:
- image: monterey-base
+ macos_instance:
+ image: ghcr.io/cirruslabs/macos-monterey-base:latest
+ #cpus: 4
+ #memory: 8G
sysinfo_script: |
id
@@ -376,7 +378,7 @@ task:
reupload_on_changes: true
configure_script: |
- brewpath="/usr/local"
+ brewpath="/opt/homebrew"
PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
for pkg in icu4c krb5 openldap openssl zstd ; do
diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl
index a2bc8a5351e..1f6e45b372e 100644
--- a/src/test/kerberos/t/001_auth.pl
+++ b/src/test/kerberos/t/001_auth.pl
@@ -34,8 +34,18 @@ my ($krb5_bin_dir, $krb5_sbin_dir);
if ($^O eq 'darwin')
{
- $krb5_bin_dir = '/usr/local/opt/krb5/bin';
- $krb5_sbin_dir = '/usr/local/opt/krb5/sbin';
+ if (-d "/usr/local/opt/krb5")
+ {
+ # old
+ $krb5_bin_dir = '/usr/local/opt/krb5/bin';
+ $krb5_sbin_dir = '/usr/local/opt/krb5/sbin';
+ }
+ elsif (-d "/opt/homebrew/opt/krb5")
+ {
+ # new
+ $krb5_bin_dir = '/opt/homebrew/opt/krb5/bin';
+ $krb5_sbin_dir = '/opt/homebrew/opt/krb5/sbin';
+ }
}
elsif ($^O eq 'freebsd')
{
--
2.25.1
>From 0f4e47d0f6157ab8ee6ecd17d7933155d0bdc91a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 4 Nov 2022 15:43:56 -0500
Subject: [PATCH 09/11] cirrus: update to macos ventura
ci-os-only: macos
---
.cirrus.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ca49b2973ac..aa381432050 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -303,7 +303,7 @@ task:
task:
- name: macOS - Monterey - Meson
+ name: macOS - Ventura - Meson
env:
CPUS: 4 # always get that much for cirrusci macOS instances
@@ -327,7 +327,7 @@ task:
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
macos_instance:
- image: ghcr.io/cirruslabs/macos-monterey-base:latest
+ image: ghcr.io/cirruslabs/macos-ventura-base:latest
#cpus: 4
#memory: 8G
--
2.25.1
>From 045de16bf365302805f33557c0824011300545e8 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Fri, 24 Jun 2022 00:09:12 -0500
Subject: [PATCH 10/11] cirrus/freebsd: run with more CPUs+RAM and do not
repartition
There was some historic problem where tests under freebsd took 8+ minutes (and
before 4a288a37f took 15 minutes).
This reduces test time from 10min to 3min.
4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720
4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008
4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600
6 CPUs https://cirrus-ci.com/task/6678321684545536
8 CPUs https://cirrus-ci.com/task/6264854121021440
See also:
https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4h...@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686
8 jobs 7min https://cirrus-ci.com/task/6186376667332608
//-os-only: freebsd
---
.cirrus.yml | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index aa381432050..3473d792561 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -63,11 +63,9 @@ task:
name: FreeBSD - 13 - Meson
env:
- # FreeBSD on GCP is slow when running with larger number of CPUS /
- # jobs. Using one more job than cpus seems to work best.
- CPUS: 2
- BUILD_JOBS: 3
- TEST_JOBS: 3
+ CPUS: 4
+ BUILD_JOBS: 4
+ TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST
@@ -94,8 +92,6 @@ task:
fingerprint_key: ccache/freebsd
reupload_on_changes: true
- # Workaround around performance issues due to 32KB block size
- repartition_script: src/tools/ci/gcp_freebsd_repartition.sh
create_user_script: |
pw useradd postgres
chown -R postgres:postgres .
--
2.25.1
>From 81f5e76049bc831f651e8e398eee17f0a9b9e26a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Sun, 13 Nov 2022 16:32:41 -0600
Subject: [PATCH 11/11] cirrus: split linux and move the 'only_if:' line ..
..above the big 'env' blocks
This allows (for example) only running only tests under autoconf.
---
.cirrus.yml | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 3473d792561..0a280f2cd71 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -61,6 +61,7 @@ on_failure_meson: &on_failure_meson
task:
name: FreeBSD - 13 - Meson
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
env:
CPUS: 4
@@ -71,8 +72,6 @@ task:
CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST
CFLAGS: -Og -ggdb
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
-
compute_engine_instance:
image_project: $IMAGE_PROJECT
image: family/pg-ci-freebsd-13
@@ -173,8 +172,6 @@ task:
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
-
compute_engine_instance:
image_project: $IMAGE_PROJECT
image: family/pg-ci-bullseye
@@ -210,6 +207,7 @@ task:
matrix:
- name: Linux - Debian Bullseye - Autoconf
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux-autoconf.*'
configure_script: |
su postgres <<-EOF
@@ -234,6 +232,7 @@ task:
<<: *on_failure_ac
- name: Linux - Debian Bullseye - Meson
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux-meson.*'
env:
CCACHE_MAXSIZE: "2400M" # tests two different builds
@@ -300,6 +299,7 @@ task:
task:
name: macOS - Ventura - Meson
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
env:
CPUS: 4 # always get that much for cirrusci macOS instances
@@ -320,8 +320,6 @@ task:
CFLAGS: -Og -ggdb
CXXFLAGS: -Og -ggdb
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
-
macos_instance:
image: ghcr.io/cirruslabs/macos-ventura-base:latest
#cpus: 4
@@ -409,6 +407,7 @@ task:
task:
name: Windows - Server 2019, VS 2019 - Meson
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
env:
# Half the allowed per-user CPU cores
@@ -435,8 +434,6 @@ task:
# 0x8001 is SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX
CIRRUS_WINDOWS_ERROR_MODE: 0x8001
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
-
windows_container:
image: $CONTAINER_REPO/windows_ci_vs_2019:latest
cpu: $CPUS
@@ -481,6 +478,9 @@ task:
task:
name: CompilerWarnings
+ # task that did not run, count as a success, so we need to recheck Linux'
+ # condition here ...
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
# To limit unnecessary work only run this once the normal linux test succeeds
depends_on:
@@ -500,10 +500,6 @@ task:
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
- # task that did not run, count as a success, so we need to recheck Linux'
- # condition here ...
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
-
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
cpu: $CPUS
@@ -623,14 +619,13 @@ task:
task:
name: Documentation
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
+ skip: "!changesInclude('.cirrus.yml', 'doc/**')"
env:
CPUS: 1
BUILD_JOBS: 1
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*'
- skip: "!changesInclude('.cirrus.yml', 'doc/**')"
-
container:
image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest
cpu: $CPUS
--
2.25.1
