On Sat, Sep 03, 2022 at 12:52:54AM +0300, Melih Mutlu wrote:
> Justin Pryzby <pry...@telsasoft.com>, 19 Ağu 2022 Cum, 05:34 tarihinde şunu
> yazdı:
>
> > Inline notes about changes since the last version.
> >
> > On Thu, Jul 28, 2022 at 05:44:28PM -0500, Justin Pryzby wrote:
> > > I think the "only_if" should allow separately running one but not both
> > of the
> > > windows instances, like:
> > >
> > > + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' ||
> > > $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw64'
> > >
> > > I'm not sure, but maybe this task should only run "by request", and omit
> > > the
> > > first condition:
> > >
> > > + only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw64'
> >
> > The patch shouldn't say this during development, or else cfbot doesn't run
> > it..
> > Oops.
>
> Actually, making MinGW task optional for now might make sense. Due to
> windows resource limits on Cirrus CI and slow builds on Windows, adding
> this task as non-optional may not be an efficient decision
> I think that continuing with this patch by changing MinGW to optional for
> now, instead of waiting for more resource on Cirrus or faster builds on
> Windows, could be better. I don't see any harm.
I agree that maybe it should be optional if merged to postgres.
But cfbot should run the Mingw task for this patch's own commitfest
entry. But right now (because cfbot doesn't include the original commit
message/s), it doesn't get run :(
> + only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.* ||
> $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
>
> Added this line to allow run only mingw task or run all tasks including
> mingw.
>
> What do you all think about this change? Does it make sense?
You're allowing to write "ci-os-include: mingw" to "opt-in" to the task,
without opting-out of all the other tasks (and without enumerating all
the tasks by writing "ci-os-only: mingw,windows,macos,freebsd,linux".
That makes sense, and the logic looks right. But that still has to be
commented during development to be run by cfbot.
Also, the first half was missing a closing quote.
https://cirrus-ci.com/build/5874178241855488
> > + setup_additional_packages_script: |
> > > + REM C:\msys64\usr\bin\pacman.exe -S --noconfirm busybox
> >
> > This should include choco, too.
>
> Added pacman.exe line. Do we really need choco here? I don't think mingw
> would require any package via choco.
I guess choco isn't needed.
> Also is ending pacman.exe line with busybox intentional? I just added that
> line with "..." at the end instead of any package name.
Yeah, the busybox part was unintentional.
> > > for item in `find "$sourcetree" -name Makefile -print -o -name
> > > GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do
> > > - filename=`expr "$item" : "$sourcetree\(.*\)"`
> > > - if test ! -f "${item}.in"; then
> > > - if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ;
> > > else
> > > - ln -fs "$item" "$buildtree/$filename" || exit 1
> > > - fi
> > > - fi
> > > + filename=${item#$sourcetree}
> > > + [ -e "$buildtree/$filename" ] && continue
> >
> > I fixed this to check for ".in" files as intended.
> >
> > It'd be a lot better if the image didn't take so long to start. :(
>
> One question would be that should this patch include "prep_buildtree"? It
> doesn't seem to me like it's directly related to adding MinGW into CI but
> more like an improvement for builds on Windows.
> Maybe we can make it a seperate patch if it's necessary.
I don't know what direction that idea is going, but it makes working
with this patch a bit easier when configure is less slow. Fine with me
to split it into a separate patch :)
> Sharing a new version of the patch. It also moves the above line so that it
> will apply to mingw task too. Otherwise mingw task was failing.
I saw that but hadn't tracked it down yet. Do you know if the tar
failures were from a TAP test added since you first posted the mingw
patch, or ??
Also: your original patch said --host=x86_64-w64-mingw32, but the task
is called MinGW64. The most recent patches don't use --host at all, and
were building for a 32 bit environment, even though the OS image says
MSYSTEM=UCRT64.
Also: right now src/test and src/interfaces/*/test aren't being built
during the build phase, which means that they're 1) not compiled in
parallel; and 2) not cached. This isn't specific to MinGW. Other than
compiling those dirs specifically, one option is to put
"always: upload_caches: ccache" after "test_world_script" (in that case,
if the CI instance is rescheduled during tests, the compilation won't be
pushed to cache). Actually, it seems better to compile stuff during
"build" or else any compilation warnings should up in the middle of
"check-world.."
Also: I'm having second thoughts about the loop around ./configure. It
could happen that a cached configure would succeed, but then the build
would later fail, and it wouldn't fix itself. I think a slow configure
is okay for an "opt-in" task.
Also: my backtrace call was using a path to cygwin rather than msys.
This seemed to work before, but doesn't seem to be working now...
--
Justin
>From 94023eb23329eb53cb0b164487afb85925e89c51 Mon Sep 17 00:00:00 2001
From: Melih Mutlu <m.melihmu...@gmail.com>
Date: Fri, 2 Sep 2022 23:10:02 +0300
Subject: [PATCH 1/2] Added Windows with MinGW environment in Cirrus CI
CI task on Windows Server with MinGW has been added as optional.
---
.cirrus.yml | 103 ++++++++++++++++++++++++++------
src/tools/ci/cores_backtrace.sh | 17 +++++-
2 files changed, 100 insertions(+), 20 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index a9193c2c34f..60f9ae2b65c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -34,6 +34,7 @@ on_failure: &on_failure
- "**/*.log"
- "**/*.diffs"
- "**/regress_log_*"
+ - "**/*.stackdump"
type: text/plain
task:
@@ -339,13 +340,33 @@ task:
cores_script: src/tools/ci/cores_backtrace.sh macos "${HOME}/cores"
+WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE
+ env:
+ # Half the allowed per-user CPU cores
+ CPUS: 4
+ # The default working dir is in a directory msbuild complains about
+ CIRRUS_WORKING_DIR: "c:/cirrus"
+
+ # Avoids port conflicts between concurrent tap test runs
+ PG_TEST_USE_UNIX_SOCKETS: 1
+
+ # git's tar doesn't deal with drive letters, see
+ # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net
+ TAR: "c:/windows/system32/tar.exe"
+
+ sysinfo_script: |
+ chcp
+ systeminfo
+ powershell -Command get-psdrive -psprovider filesystem
+ set
+
task:
+ << : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, VS 2019
- env:
- # Half the allowed per-user CPU cores
- CPUS: 4
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ env:
# Our windows infrastructure doesn't have test concurrency above the level
# of a single vcregress test target. Due to that, it's useful to run prove
# with multiple jobs. For the other tasks it isn't, because two sources
@@ -355,15 +376,9 @@ task:
# likely can be improved upon further.
PROVE_FLAGS: -j10 --timer
- # The default cirrus working dir is in a directory msbuild complains about
- CIRRUS_WORKING_DIR: "c:/cirrus"
# Avoid re-installing over and over
NO_TEMP_INSTALL: 1
- # git's tar doesn't deal with drive letters, see
- # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net
- TAR: "c:/windows/system32/tar.exe"
- # Avoids port conflicts between concurrent tap test runs
- PG_TEST_USE_UNIX_SOCKETS: 1
+
PG_REGRESS_SOCK_DIR: "c:/cirrus/"
# -m enables parallelism
# verbosity:minimal + Summary reduce verbosity, while keeping a summary of
@@ -387,8 +402,6 @@ task:
# currently have a tool for that...
CIRRUS_ESCAPING_PROCESSES: 1
- only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
-
windows_container:
#image: $CONTAINER_REPO/windows_ci_vs_2019:latest
#image: cirrusci/windowsservercore:visualstudio2019-2021.12.07
@@ -398,12 +411,6 @@ task:
cpu: $CPUS
memory: 4G
- sysinfo_script: |
- chcp
- systeminfo
- powershell -Command get-psdrive -psprovider filesystem
- set
-
setup_additional_packages_script: |
REM 3min
REM choco install -y --no-progress --version=1.0.0 visualstudio2022-workload-vctools --install-args="--add Microsoft.VisualStudio.Component.VC.CLI.Support"
@@ -475,6 +482,66 @@ task:
path: "crashlog-*.txt"
type: text/plain
+task:
+ << : *WINDOWS_ENVIRONMENT_BASE
+ name: Windows - Server 2019, MinGW64
+
+ only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.* || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+
+ windows_container:
+ image: $CONTAINER_REPO/windows_ci_mingw64:latest
+ cpu: $CPUS
+ memory: 4G
+ env:
+ CCACHE_DIR: C:/msys64/ccache
+ BUILD_DIR: "%CIRRUS_WORKING_DIR%/build"
+
+ ccache_cache:
+ folder: ${CCACHE_DIR}
+
+ setup_additional_packages_script: |
+ REM C:\msys64\usr\bin\pacman.exe -S --noconfirm ...
+
+ mingw_info_script:
+ - C:\msys64\usr\bin\dash.exe -lc "where gcc"
+ - C:\msys64\usr\bin\dash.exe -lc "gcc --version"
+ - C:\msys64\usr\bin\dash.exe -lc "where perl"
+ - C:\msys64\usr\bin\dash.exe -lc "perl --version"
+
+ configure_script:
+ # Try to configure with the cache file, and retry without if it fails, in case the flags changed.
+ - C:\msys64\usr\bin\dash.exe -lc "mkdir %BUILD_DIR% &&
+ cd %BUILD_DIR% && for i in 1 2; do
+ %CIRRUS_WORKING_DIR%/configure
+ --cache-file=${CCACHE_DIR}/configure.cache
+ --enable-cassert
+ --enable-debug
+ --enable-tap-tests
+ --with-icu
+ --with-libxml
+ --with-libxslt
+ --with-lz4
+ CC='ccache gcc'
+ CXX='ccache g++'
+ CFLAGS='-Og -ggdb -pipe'
+ CXXFLAGS='-Og -ggdb' && break;
+ rm -v ${CCACHE_DIR}/configure.cache;
+ done
+ "
+
+ build_script:
+ C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}"
+
+ upload_caches: ccache
+
+ tests_script:
+ - set "NoDefaultCurrentDirectoryInExePath=0"
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install"
+
+ on_failure:
+ <<: *on_failure
+ cores_script:
+ - C:\tools\cygwin\bin\dash.exe --login -c "cd '%cd%' && src/tools/ci/cores_backtrace.sh msys ."
task:
name: CompilerWarnings
diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh
index 28d3cecfc67..93e90f284e6 100755
--- a/src/tools/ci/cores_backtrace.sh
+++ b/src/tools/ci/cores_backtrace.sh
@@ -10,11 +10,24 @@ directory=$2
case $os in
freebsd|linux|macos)
- ;;
+ ;;
+
+ msys)
+ # XXX Evidently I don't know how to write two arguments here without pathname expansion later, other than eval.
+ #findargs='-name "*.stackdump"'
+ for corefile in $(find "$directory" -type f -name "*.stackdump") ; do
+ binary=`basename "$corefile" .stackdump`
+ echo;echo;
+ echo "dumping ${corefile} for ${binary}"
+ awk '/^0/{print $2}' $corefile |addr2line -f -i -e ./src/backend/postgres.exe
+ done
+ exit 0
+ ;;
+
*)
echo "unsupported operating system ${os}"
exit 1
- ;;
+ ;;
esac
first=1
--
2.17.1
>From 794874a6f0845f3d3a53613e1a6cf326f7ae83f9 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryz...@telsasoft.com>
Date: Thu, 28 Jul 2022 10:57:26 -0500
Subject: [PATCH 2/2] f!
ci-os-only: mingw
---
.cirrus.yml | 14 ++++++++++----
config/prep_buildtree | 20 ++++++++------------
2 files changed, 18 insertions(+), 16 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 60f9ae2b65c..f7cd6de93d1 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -486,7 +486,8 @@ task:
<< : *WINDOWS_ENVIRONMENT_BASE
name: Windows - Server 2019, MinGW64
- only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.* || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ #XXX only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+ only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
windows_container:
image: $CONTAINER_REPO/windows_ci_mingw64:latest
@@ -517,6 +518,7 @@ task:
--enable-cassert
--enable-debug
--enable-tap-tests
+ --host=x86_64-w64-mingw64
--with-icu
--with-libxml
--with-libxslt
@@ -530,18 +532,22 @@ task:
"
build_script:
- C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}"
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} world-bin"
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/test"
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/interfaces/ecpg/test"
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/interfaces/libpq/test"
upload_caches: ccache
- tests_script:
+ test_world_script:
- set "NoDefaultCurrentDirectoryInExePath=0"
- C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install"
on_failure:
<<: *on_failure
cores_script:
- - C:\tools\cygwin\bin\dash.exe --login -c "cd '%cd%' && src/tools/ci/cores_backtrace.sh msys ."
+ - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% src/tools/ci/cores_backtrace.sh msys %BUILD_DIR%/tmp_install"
+
task:
name: CompilerWarnings
diff --git a/config/prep_buildtree b/config/prep_buildtree
index a0eabd3dee2..97bf65967f1 100644
--- a/config/prep_buildtree
+++ b/config/prep_buildtree
@@ -26,20 +26,16 @@ buildtree=`cd ${2:-'.'} && pwd`
# If we did, it would interfere with installation of prebuilt docs from
# the source tree, if a VPATH build is done from a distribution tarball.
# See bug #5595.
-for item in `find "$sourcetree" -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) | grep -v "$sourcetree/doc/src/sgml/\+"`; do
- subdir=`expr "$item" : "$sourcetree\(.*\)"`
- if test ! -d "$buildtree/$subdir"; then
- mkdir -p "$buildtree/$subdir" || exit 1
- fi
-done
+( cd "$sourcetree" && find . -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) |grep -v "doc/src/sgml/\+" ) |
+ ( cd "$buildtree" && xargs mkdir -p )
for item in `find "$sourcetree" -name Makefile -print -o -name GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do
- filename=`expr "$item" : "$sourcetree\(.*\)"`
- if test ! -f "${item}.in"; then
- if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ; else
- ln -fs "$item" "$buildtree/$filename" || exit 1
- fi
- fi
+ [ -e "$item.in" ] && echo "skipping: $item" &&
+ continue
+
+ filename=${item#$sourcetree}
+ cmp "$item" "$buildtree/$filename" >/dev/null 2>&1 ||
+ ln -fs "$item" "$buildtree/$filename"
done
exit 0
--
2.17.1
