On Mon, Nov 21, 2022 at 02:45:42PM -0800, Andres Freund wrote: > > > > + ninja -C build |tee build/meson-logs/build.txt > > > > + REM Since pipes lose exit status of the preceding command, rerun > > > > compilation, > > > > + REM without the pipe exiting now if it fails, rather than trying > > > > to run checks > > > > + ninja -C build > nul > > > > > > This seems mighty grotty :(. but I guess it's quick enough not worry > > > about, > > > and I can't come up with a better plan. > > > > > > It doesn't seem quite right to redirect into meson-logs/ to me, my > > > interpretation is that that's "meson's namespace". Why not just store it > > > in > > > build/? > > > > I put it there so it'd be included with the build artifacts. > > Wouldn't just naming it build-warnings.log suffice? I don't think we > want to actually upload build.txt - it already is captured.
Originally, I wanted the input and the output to be available as files and not just in cirrus' web GUI, but maybe that's not important anymore. I rewrote it again. > > I'm not sure if this ought to be combined with/before/after your "move > > compilerwarnings task to meson" patch? (Regarding that patch: I > > mentioned that it shouldn't use ccache -C, and it should use > > meson_log_artifacts.) > > TBH, I'm not quite sure a separate docs task does really still make > sense after the SanityCheck task. It's worth building the docs even if > some flappy test fails, but I don't think we should try to build the > docs if the code doesn't even compile, in all likelihood a lot more is > wrong in that case. It'd be okay either way. I had split it out to 1) isolate the changes in the "upload changed docs as artifacts" patch; and, 2) so the docs artifacts are visible in a cfbot link called "Documentation"; and, 3) so the docs task runs without a dependency on "Linux", since (as you said) docs/errors are worth showing/reviewing/reporting/addressing separately from test errors (perhaps similar to compiler warnings...). I shuffled my branch around and sending now the current "docs" patches, but I suppose this is waiting on the "convert CompilerWarnings task to meson" patch. -- Justin
>From e7eb22b85dcfe503810ea0f89f48833f04f80d3f Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Wed, 25 May 2022 21:53:22 -0500 Subject: [PATCH 01/10] cirrus/windows: add compiler_warnings_script I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index f31923333ef..6ce4f393e29 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -547,12 +547,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.25.1
>From 79586f7ac46ce48648b150b6ad08987578c730f5 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 11 Nov 2022 21:14:39 -0600 Subject: [PATCH 02/10] cirrus/macos: switch to "macos_instance" / M1.. This uses an M1 ARM processor rather than intel. See: https://cirrus-ci.org/blog/2022/11/08/sunsetting-intel-macos-instances/ https://github.com/justinpryzby/postgres/runs/9446478561 | warning We are sunsetting support for Intel-based macOS instances! Please migrate before January 1st 2023. ci-os-only: macos --- .cirrus.yml | 10 ++++++---- src/test/kerberos/t/001_auth.pl | 14 ++++++++++++-- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 6ce4f393e29..ed05c4adf87 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -391,7 +391,7 @@ task: name: macOS - Monterey - Meson env: - CPUS: 12 # always get that much for cirrusci macOS instances + CPUS: 4 # always get that much for cirrusci macOS instances BUILD_JOBS: $CPUS # Test performance regresses noticably when using all cores. 8 seems to # work OK. See @@ -412,8 +412,10 @@ task: depends_on: SanityCheck only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' - osx_instance: - image: monterey-base + macos_instance: + image: ghcr.io/cirruslabs/macos-monterey-base:latest + #cpus: 4 + #memory: 8G sysinfo_script: | id @@ -459,7 +461,7 @@ task: ccache_cache: folder: $CCACHE_DIR configure_script: | - brewpath="/usr/local" + brewpath="/opt/homebrew" PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}" for pkg in icu4c krb5 openldap openssl zstd ; do diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl index a2bc8a5351e..1f6e45b372e 100644 --- a/src/test/kerberos/t/001_auth.pl +++ b/src/test/kerberos/t/001_auth.pl @@ -34,8 +34,18 @@ my ($krb5_bin_dir, $krb5_sbin_dir); if ($^O eq 'darwin') { - $krb5_bin_dir = '/usr/local/opt/krb5/bin'; - $krb5_sbin_dir = '/usr/local/opt/krb5/sbin'; + if (-d "/usr/local/opt/krb5") + { + # old + $krb5_bin_dir = '/usr/local/opt/krb5/bin'; + $krb5_sbin_dir = '/usr/local/opt/krb5/sbin'; + } + elsif (-d "/opt/homebrew/opt/krb5") + { + # new + $krb5_bin_dir = '/opt/homebrew/opt/krb5/bin'; + $krb5_sbin_dir = '/opt/homebrew/opt/krb5/sbin'; + } } elsif ($^O eq 'freebsd') { -- 2.25.1
>From 9d340b707897985d18c4e6f10fb9da41b14e59e9 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 4 Nov 2022 15:43:56 -0500 Subject: [PATCH 03/10] cirrus/macos: update to macos ventura ci-os-only: macos --- .cirrus.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index ed05c4adf87..7c6e3691093 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -388,7 +388,7 @@ task: task: - name: macOS - Monterey - Meson + name: macOS - Ventura - Meson env: CPUS: 4 # always get that much for cirrusci macOS instances @@ -413,7 +413,7 @@ task: only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' macos_instance: - image: ghcr.io/cirruslabs/macos-monterey-base:latest + image: ghcr.io/cirruslabs/macos-ventura-base:latest #cpus: 4 #memory: 8G -- 2.25.1
>From 766f9c2b8114ace0431877cbb1fc8214aeeb8e5b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 24 Jun 2022 00:09:12 -0500 Subject: [PATCH 04/10] cirrus/freebsd: run with more CPUs+RAM and do not repartition There was some historic problem where tests under freebsd took 8+ minutes (and before 4a288a37f took 15 minutes). This reduces test time from 10min to 3min. 4 CPUs 4 tests https://cirrus-ci.com/task/4880240739614720 4 CPUs 6 tests https://cirrus-ci.com/task/4664440120410112 https://cirrus-ci.com/task/4586784884523008 4 CPUs 8 tests https://cirrus-ci.com/task/5001995491737600 6 CPUs https://cirrus-ci.com/task/6678321684545536 8 CPUs https://cirrus-ci.com/task/6264854121021440 See also: https://www.postgresql.org/message-id/flat/20220310033347.hgxk4pyarzq4h...@alap3.anarazel.de#f36c0b17e33e31e7925e7e5812998686 8 jobs 7min https://cirrus-ci.com/task/6186376667332608 //-os-only: freebsd --- .cirrus.yml | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 7c6e3691093..6aaaf62d6a9 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -131,11 +131,9 @@ task: name: FreeBSD - 13 - Meson env: - # FreeBSD on GCP is slow when running with larger number of CPUS / - # jobs. Using one more job than cpus seems to work best. - CPUS: 2 - BUILD_JOBS: 3 - TEST_JOBS: 3 + CPUS: 4 + BUILD_JOBS: 4 + TEST_JOBS: 6 CCACHE_DIR: /tmp/ccache_dir CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST @@ -160,8 +158,6 @@ task: ccache_cache: folder: $CCACHE_DIR - # Workaround around performance issues due to 32KB block size - repartition_script: src/tools/ci/gcp_freebsd_repartition.sh create_user_script: | pw useradd postgres chown -R postgres:postgres . -- 2.25.1
>From b36213a7801d354e8f47189d19b7c9166b1ad22b Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 13 Nov 2022 14:03:27 -0600 Subject: [PATCH 05/10] cirrus: clean up typos --- .cirrus.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 6aaaf62d6a9..3eda0268161 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -47,7 +47,7 @@ on_failure_meson: &on_failure_meson # In theory it'd be nice to upload the junit files meson generates, so that # cirrus will nicely annotate the commit. Unfortunately the files don't - # contain identifieable file + line numbers right now, so the annotations + # contain identifiable file + line numbers right now, so the annotations # don't end up useful. We could probably improve on that with a some custom # conversion script, but ... meson_log_artifacts: @@ -74,7 +74,7 @@ task: # no options enabled, should be small CCACHE_MAXSIZE: "150M" - # Container starts up quickly, but is slower at runtime, particularly for + # Containers start up quickly, but is slower at runtime, particularly for # tests. Good for the briefly running sanity check. container: image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest @@ -590,7 +590,7 @@ task: CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 - # for some reason mingw plpython cannot find it's installation without this + # for some reason mingw plpython cannot find its installation without this PYTHONHOME: C:/msys64/ucrt64 # prevents MSYS bash from resetting error mode MSYS: winjitdebug @@ -645,7 +645,7 @@ task: # Use larger ccache cache, as this task compiles with multiple compilers / # flag combinations - CCACHE_MAXSIZE: "1GB" + CCACHE_MAXSIZE: "1G" CCACHE_DIR: "/tmp/ccache_dir" LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES -- 2.25.1
>From 7133474e6280538439f43f799726275ce69ee2cd Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sat, 26 Feb 2022 19:34:35 -0600 Subject: [PATCH 06/10] cirrus: build docs as a separate task.. This will run the doc build if any docs have changed, even if Linux fails, to allow catch doc build failures. This automatically shows up as a separate "column" on cfbot. Also, in the future, this will hopefully upload each patch's changed HTML docs as an artifact, for easy review. XXX: it ought to be possible to skip (re)building docs if they haven't changed since the last build. But that doesn't appear to be working. https://cirrus-ci.com/task/4523764229603328 skip: "!changesInclude('.cirrus.yml', 'doc/**')" ci-os-only: html --- .cirrus.yml | 48 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 34 insertions(+), 14 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 3eda0268161..5cb18032779 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -736,20 +736,6 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - ### - # Verify docs can be built - ### - # XXX: Only do this if there have been changes in doc/ since last build - always: - docs_build_script: | - time ./configure \ - --cache gcc.cache \ - CC="ccache gcc" \ - CXX="ccache g++" \ - CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} -C doc - ### # Verify headerscheck / cpluspluscheck succeed # @@ -773,3 +759,37 @@ task: always: upload_caches: ccache + + +### +# Verify docs can be built +# changesInclude() will skip this task if none of the commits since +# CIRRUS_LAST_GREEN_CHANGE touched any relevant files. The comparison appears +# to be like "git log a..b -- ./file", not "git diff a..b -- ./file" +### + +task: + name: Documentation + + env: + CPUS: 1 + BUILD_JOBS: 1 + + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*' + + container: + image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest + cpu: $CPUS + memory: 2G + + # Exercise HTML and other docs: + ninja_docs_build_script: | + mkdir build-ninja + cd build-ninja + time meson setup + time ninja docs + + # Again, with autoconf + docs_build_script: | + time ./configure + make -s -j${BUILD_JOBS} -C doc -- 2.25.1
>From 0686c222fdf36bd2b844cbe9c86d522dfc030fc8 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Tue, 19 Jul 2022 12:38:45 -0500 Subject: [PATCH 07/10] cirrus/warnings: use a single/common 'always' block ci-os-only: warnings --- .cirrus.yml | 40 +++++++++++++++++----------------------- 1 file changed, 17 insertions(+), 23 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 5cb18032779..0bc7c5e3796 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -664,9 +664,6 @@ task: clang -v export - ccache_cache: - folder: $CCACHE_DIR - setup_additional_packages_script: | #apt-get update #DEBIAN_FRONTEND=noninteractive apt-get -y install ... @@ -681,8 +678,11 @@ task: # different compilers to build with different combinations of dtrace on/off # and cassert on/off. - # gcc, cassert off, dtrace on always: + ccache_cache: + folder: $CCACHE_DIR + + # gcc, cassert off, dtrace on gcc_warning_script: | time ./configure \ --cache gcc.cache \ @@ -692,8 +692,7 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - # gcc, cassert on, dtrace off - always: + # gcc, cassert on, dtrace off gcc_a_warning_script: | time ./configure \ --cache gcc.cache \ @@ -703,8 +702,7 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - # clang, cassert off, dtrace off - always: + # clang, cassert off, dtrace off clang_warning_script: | time ./configure \ --cache clang.cache \ @@ -713,8 +711,7 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - # clang, cassert on, dtrace on - always: + # clang, cassert on, dtrace on clang_a_warning_script: | time ./configure \ --cache clang.cache \ @@ -725,8 +722,7 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - # cross-compile to windows - always: + # cross-compile to windows mingw_cross_warning_script: | time ./configure \ --host=x86_64-w64-mingw32 \ @@ -736,16 +732,15 @@ task: make -s -j${BUILD_JOBS} clean time make -s -j${BUILD_JOBS} world-bin - ### - # Verify headerscheck / cpluspluscheck succeed - # - # - Don't use ccache, the files are uncacheable, polluting ccache's - # cache - # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose - # - XXX have to disable ICU to avoid errors: - # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de - ### - always: + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Don't use ccache, the files are uncacheable, polluting ccache's + # cache + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + # - XXX have to disable ICU to avoid errors: + # https://postgr.es/m/20220323002024.f2g6tivduzrktgfa%40alap3.anarazel.de + ### headers_headerscheck_script: | time ./configure \ ${LINUX_CONFIGURE_FEATURES} \ @@ -757,7 +752,6 @@ task: headers_cpluspluscheck_script: | time make -s cpluspluscheck EXTRAFLAGS='-fmax-errors=10' - always: upload_caches: ccache -- 2.25.1
>From 994c0df0b98fab4f10ac987f13302e0eb3aa07a9 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sat, 26 Feb 2022 19:39:10 -0600 Subject: [PATCH 08/10] cirrus: upload changed html docs as artifacts This could be done on the client side (cfbot). One advantage of doing it here is that fewer docs are uploaded - many patches won't upload docs at all. https://www.postgresql.org/message-id/flat/20220409021853.gp24...@telsasoft.com https://www.postgresql.org/message-id/CAB8KJ=i4qmeuopq+pcsmbzgd4o-xv0fcnc+q1x7hn9hsdvk...@mail.gmail.com https://cirrus-ci.com/task/5396696388599808 ci-os-only: html --- .cirrus.yml | 31 ++++++++++++++++++++++++++++--- src/tools/ci/copy-changed-docs | 29 +++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 3 deletions(-) create mode 100755 src/tools/ci/copy-changed-docs diff --git a/.cirrus.yml b/.cirrus.yml index 0bc7c5e3796..2f2c09027d8 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -27,6 +27,14 @@ env: TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl + # The commit that this branch is rebased on. There's no easy way to find this. + # This does the right thing for cfbot, which always squishes all patches into a single commit. + # And does the right thing for any 1-patch commits. + # Patch series manually submitted to cirrus would benefit from setting this to the + # number of patches in the series (or directly to the commit the series was rebased on). + #BASE_COMMIT: HEAD~1 + # For demo purposes: + BASE_COMMIT: 3d14e171e9e2236139e8976f3309a588bcc8683b # What files to preserve in case tests fail on_failure_ac: &on_failure_ac @@ -756,7 +764,7 @@ task: ### -# Verify docs can be built +# Verify docs can be built, and upload changed docs as artifacts # changesInclude() will skip this task if none of the commits since # CIRRUS_LAST_GREEN_CHANGE touched any relevant files. The comparison appears # to be like "git log a..b -- ./file", not "git diff a..b -- ./file" @@ -764,18 +772,21 @@ task: task: name: Documentation + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*' + #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')" env: CPUS: 1 BUILD_JOBS: 1 - only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*' - container: image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest cpu: $CPUS memory: 2G + sysinfo_script: | + git diff --name-only "$BASE_COMMIT" + # Exercise HTML and other docs: ninja_docs_build_script: | mkdir build-ninja @@ -787,3 +798,17 @@ task: docs_build_script: | time ./configure make -s -j${BUILD_JOBS} -C doc + cp -r doc new-docs + + # Build HTML docs from the base commit. + git checkout "$BASE_COMMIT" -- doc + #html-stamp + make -s -C doc clean + make -s -C doc html + cp -r doc old-docs + + copy_changed_docs_script: | + src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs" + + html_docs_artifacts: + paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css'] diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs new file mode 100755 index 00000000000..1c921a8df6f --- /dev/null +++ b/src/tools/ci/copy-changed-docs @@ -0,0 +1,29 @@ +#! /bin/sh +# Copy HTML which differ between $old and $new into $outdir +set -e + +old=$1 +new=$2 +outdir=$3 + +# The index is large and changes often +skippages="bookindex.html" + +mkdir "$outdir" +cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir" + +changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` || + [ $? -eq 1 ] + +for f in $changed +do + # Avoid removed files + [ -f "$f" ] || continue + + echo "$f" |grep -Ew "$skippages" >/dev/null && + continue + + cp -v "$f" "$outdir" +done + +exit 0 -- 2.25.1
>From 3bd2db1aba704bb2fd1d8ebd5cec349770ee5e21 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Wed, 19 Oct 2022 20:33:28 -0500 Subject: [PATCH 09/10] WIP: show changed docs with meson ci-os-only: html docs --- .cirrus.yml | 27 ++++++++++----------------- src/tools/ci/copy-changed-docs | 3 ++- 2 files changed, 12 insertions(+), 18 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 2f2c09027d8..338c6c2be16 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -776,8 +776,7 @@ task: #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')" env: - CPUS: 1 - BUILD_JOBS: 1 + CPUS: 2 container: image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest @@ -793,22 +792,16 @@ task: cd build-ninja time meson setup time ninja docs + cp -r doc ../new-docs - # Again, with autoconf - docs_build_script: | - time ./configure - make -s -j${BUILD_JOBS} -C doc - cp -r doc new-docs - - # Build HTML docs from the base commit. - git checkout "$BASE_COMMIT" -- doc - #html-stamp - make -s -C doc clean - make -s -C doc html - cp -r doc old-docs - - copy_changed_docs_script: | - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs" + # Re-build docs from the base commit. + git checkout "$BASE_COMMIT" -- ../doc + ninja clean + time ninja doc/src/sgml/html + cp -r doc ../old-docs + + copy_changed_docs_script: + - src/tools/ci/copy-changed-docs "old-docs" "new-docs" "html_docs" html_docs_artifacts: paths: ['html_docs/*.html', 'html_docs/*.png', 'html_docs/*.css'] diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs index 1c921a8df6f..598e639f695 100755 --- a/src/tools/ci/copy-changed-docs +++ b/src/tools/ci/copy-changed-docs @@ -10,7 +10,8 @@ outdir=$3 skippages="bookindex.html" mkdir "$outdir" -cp "$new"/src/sgml/html/*.css "$new"/src/sgml/html/*.svg "$outdir" +cp doc/src/sgml/*.css "$outdir" +cp doc/src/sgml/images/*.svg "$outdir" changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` || [ $? -eq 1 ] -- 2.25.1
>From 8086a65429b81fbbb2886f5bfa4495ff019b59bf Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 28 Feb 2022 23:18:19 -0600 Subject: [PATCH 10/10] +html index file This allows linking to the artifacts from the last successful build, which itself allows *not* rebuilding when sources haven't changed. //freebsd //-os-only: html, --- .cirrus.yml | 2 +- src/tools/ci/copy-changed-docs | 42 ++++++++++++++++++++++++++++++++-- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 338c6c2be16..5e1612b583e 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -773,7 +773,7 @@ task: task: name: Documentation only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(docs|html).*' - #skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')" + skip: "!changesInclude('.cirrus.yml', 'doc/**', 'src/tools/ci/copy-changed-docs')" env: CPUS: 2 diff --git a/src/tools/ci/copy-changed-docs b/src/tools/ci/copy-changed-docs index 598e639f695..c60428e1ae1 100755 --- a/src/tools/ci/copy-changed-docs +++ b/src/tools/ci/copy-changed-docs @@ -5,6 +5,10 @@ set -e old=$1 new=$2 outdir=$3 +branch=$CIRRUS_BRANCH + +# The index is large and changes often +skippages="bookindex.html" # The index is large and changes often skippages="bookindex.html" @@ -13,6 +17,19 @@ mkdir "$outdir" cp doc/src/sgml/*.css "$outdir" cp doc/src/sgml/images/*.svg "$outdir" +# The index is useful to allow a static link (not specific to a cirrus run) to the artifacts for the most-recent, successful CI run for a branch +branchurl=https://api.cirrus-ci.com/v1/artifact/github/$CIRRUS_REPO_FULL_NAME/$CIRRUS_TASK_NAME/html_docs/html_docs/00-index.html?branch=$branch + +index="$outdir/00-index.html" +cat >"$index" <<EOF +<html> +<head><title>Index of docs changed since: $BASE_COMMIT</title></head> +<body> +<!-- A link to documentation for the most recent successful build: $branchurl --> +<h1>Index of docs changed since: $BASE_COMMIT</h1> +<ul> +EOF + changed=`git diff --no-index --name-only "$old"/src/sgml/html "$new"/src/sgml/html` || [ $? -eq 1 ] @@ -24,7 +41,28 @@ do echo "$f" |grep -Ew "$skippages" >/dev/null && continue - cp -v "$f" "$outdir" -done + cp -v "$f" "$outdir" >&2 + fn=${f##*/} + # ?branch=... is needed when accessing the artifacts for the static link for the branch + # It's not needed and ignored if accessing artifacts for *this* CI run + echo "<li><a href='$fn?branch=$branch'>$fn</a>" +done >>"$index" + +github=https://github.com/$CIRRUS_REPO_FULL_NAME/commit +cirrus=https://cirrus-ci.com/build + +cat >>"$index" <<EOF +</ul> +<hr> +<code> +<br>This file was written on: `date --rfc-822 --utc` +<br>CIRRUS_CHANGE_TITLE: $CIRRUS_CHANGE_TITLE +<br>CIRRUS_CHANGE_IN_REPO: <a href="$github/$CIRRUS_CHANGE_IN_REPO">$CIRRUS_CHANGE_IN_REPO</a> +<br>CIRRUS_BUILD_ID: <a href="$cirrus/$CIRRUS_BUILD_ID">$CIRRUS_BUILD_ID</a> +<br>CIRRUS_LAST_GREEN_CHANGE: <a href="$github/$CIRRUS_LAST_GREEN_CHANGE">$CIRRUS_LAST_GREEN_CHANGE</a> +<br>CIRRUS_LAST_GREEN_BUILD_ID: <a href="$cirrus/$CIRRUS_LAST_GREEN_BUILD_ID">$CIRRUS_LAST_GREEN_BUILD_ID</a> +</code> +</body></html> +EOF exit 0 -- 2.25.1