Re: Adding CI to our tree

2022-09-06 Thread Thomas Munro
On Wed, Oct 6, 2021 at 5:01 PM Thomas Munro wrote: > BTW, on those two OSes there are some messages like this each time a > submake dumps its output to the log: > > [03:36:16.591] fcntl(): Bad file descriptor > > It seems worth putting up with these compared to the alternatives of > either not

Re: Adding CI to our tree

2022-03-30 Thread Andres Freund
Hi, Now that zstd is used, enable it in CI. I plan to commit this shortly, unless somebody sees a reason not to do so. Greetings, Andres Freund >From 51cc830e2e82516966fe1c84cd134b1858a89bab Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 26 Mar 2022 12:36:04 -0700 Subject: [PATCH v1]

Re: Adding CI to our tree

2022-03-23 Thread Justin Pryzby
On Thu, Mar 24, 2022 at 09:52:39AM +1300, Thomas Munro wrote: > On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > > -Og > > Adding this to CXXFLAGS caused a torrent of warnings from g++ about > LLVM headers, which I also see on my local system for LLVM 11 and LLVM > 14: Yes, I mentioned

Re: Adding CI to our tree

2022-03-23 Thread Thomas Munro
On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > -Og Adding this to CXXFLAGS caused a torrent of warnings from g++ about LLVM headers, which I also see on my local system for LLVM 11 and LLVM 14: [19:47:11.047] /usr/lib/llvm-11/include/llvm/ADT/Twine.h: In member function ‘llvm::CallInst*

Re: Adding CI to our tree

2022-03-23 Thread Andres Freund
Hi, On 2022-03-22 23:14:23 -0500, Justin Pryzby wrote: > On Fri, Mar 18, 2022 at 03:45:03PM -0700, Andres Freund wrote: > > Pushed 0001, 0002. Only change I made was to add > > Thanks - is there any reason not to do the MSVC compiler warnings one, too ? Purely a lack of round tuits. IIRC I

Re: Adding CI to our tree

2022-03-22 Thread Justin Pryzby
On Fri, Mar 18, 2022 at 03:45:03PM -0700, Andres Freund wrote: > Pushed 0001, 0002. Only change I made was to add Thanks - is there any reason not to do the MSVC compiler warnings one, too ? I see that it'll warn about issues with at least 3 patches (including one of yours). -- Justin

Re: Adding CI to our tree

2022-03-18 Thread Andres Freund
Hi, Pushed 0001, 0002. Only change I made was to add DEBIAN_FRONTEND=noninteractive to the apt-get invocations, because some packages will fail / complain verbosely if there's no interactive prompt during installation. Greetings, Andres Freund

Re: Adding CI to our tree

2022-03-10 Thread Justin Pryzby
See attached, or at https://github.com/justinpryzby/postgres/runs/5503079878 >From c631c3d9bdb8325aaaecc5dcdfac46eca7bd907a Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 17 Jan 2022 00:53:04 -0600 Subject: [PATCH 1/7] cirrus: include hints how to install OS packages.. This is useful

Re: Adding CI to our tree

2022-03-10 Thread Justin Pryzby
On Thu, Mar 10, 2022 at 12:50:15PM -0800, Andres Freund wrote: > > - setup_cores_script: | > > + setup_os_script: | > > mkdir -m 770 /tmp/cores > > chown root:postgres /tmp/cores > > sysctl kern.corefile='/tmp/cores/%N.%P.core' > > +#pkg install -y ... > > Would you mind if I

Re: Adding CI to our tree

2022-03-10 Thread Andres Freund
Hi, On 2022-03-02 14:50:58 -0600, Justin Pryzby wrote: > From 883edaa653bcf7f1a2369d8edf46eaaac1ba0ba2 Mon Sep 17 00:00:00 2001 > From: Justin Pryzby > Date: Mon, 17 Jan 2022 00:53:04 -0600 > Subject: [PATCH 1/7] cirrus: include hints how to install OS packages.. > > This is useful for patches

Re: Adding CI to our tree

2022-03-09 Thread Thomas Munro
On Thu, Mar 10, 2022 at 4:33 PM Andres Freund wrote: > On 2022-03-10 15:43:16 +1300, Thomas Munro wrote: > > I'm confused. > > The "terrible IO wait" thing was before we reduced the number of CPUs and > concurrent jobs. It makes sense to me that with just two CPUs we're CPU bound, > in which case

Re: Adding CI to our tree

2022-03-09 Thread Andres Freund
Hi, On 2022-03-10 15:43:16 +1300, Thomas Munro wrote: > Wow, I see the effect on Cirrus -- test_world ran in 8:55 instead of > 12:43 when I tried (terrible absolute times, but fantastic > improvement!). Hmm, on my local FreeBSD 13 box I saw 5:07 -> 5:03 > with this change. My working theory had

Re: Adding CI to our tree

2022-03-09 Thread Thomas Munro
On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > On Wed, Mar 09, 2022 at 10:12:54AM -0800, Andres Freund wrote: > > On 2022-03-09 11:47:23 -0600, Justin Pryzby wrote: > > > I'm curious what you think of this patch. > > > > > > It makes check-world on freebsd over 30% faster - saving 5min. >

Re: Adding CI to our tree

2022-03-09 Thread Justin Pryzby
On Wed, Mar 09, 2022 at 10:12:54AM -0800, Andres Freund wrote: > On 2022-03-09 11:47:23 -0600, Justin Pryzby wrote: > > I'm curious what you think of this patch. > > > > It makes check-world on freebsd over 30% faster - saving 5min. > > That's nice! While -Og makes interactive debugging

Re: Adding CI to our tree

2022-03-09 Thread Andres Freund
Hi, On 2022-03-09 11:47:23 -0600, Justin Pryzby wrote: > I'm curious what you think of this patch. > > It makes check-world on freebsd over 30% faster - saving 5min. That's nice! While -Og makes interactive debugging noticeably harder IME, it's not likely to be a large enough difference just

Re: Adding CI to our tree

2022-03-09 Thread Justin Pryzby
I'm curious what you think of this patch. It makes check-world on freebsd over 30% faster - saving 5min. Apparently gcc -Og was added in gcc 4.8 (c. 2013). On Wed, Mar 02, 2022 at 02:50:58PM -0600, Justin Pryzby wrote: > From d180953d273c221a30c5e9ad8d74b1b4dfc60bd1 Mon Sep 17 00:00:00 2001 >

Re: Adding CI to our tree (ccache)

2022-03-07 Thread Justin Pryzby
On Mon, Mar 07, 2022 at 11:10:54AM -0800, Andres Freund wrote: > On 2022-03-06 10:16:54 -0600, Justin Pryzby wrote: > > On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > > > I tried to use it, but saw that no caching was happening, and debugged > > > it. Which yielded that it can't

Re: Adding CI to our tree (ccache)

2022-03-07 Thread Andres Freund
Hi, On 2022-03-06 10:16:54 -0600, Justin Pryzby wrote: > On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > > I tried to use it, but saw that no caching was happening, and debugged > > it. Which yielded that it can't be used due to the way output files are > > specified (and due to

Re: Adding CI to our tree (ccache)

2022-03-06 Thread Justin Pryzby
On Fri, Mar 04, 2022 at 05:30:03PM -0800, Andres Freund wrote: > I tried to use it, but saw that no caching was happening, and debugged > it. Which yielded that it can't be used due to the way output files are > specified (and due to multiple files, but that can be prevented with an > msbuild

Re: Adding CI to our tree (ccache)

2022-03-04 Thread Andres Freund
Hi, On 2022-03-03 22:56:15 -0600, Justin Pryzby wrote: > On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > > On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > > > Have you tried to use the yet-to-be-released ccache with MSVC ? > > > > Yes, it doesn't work, because it requires

Re: Adding CI to our tree (ccache)

2022-03-03 Thread Justin Pryzby
On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > > Have you tried to use the yet-to-be-released ccache with MSVC ? > > Yes, it doesn't work, because it requires cl.exe to be used in a specific way > (only a single input file,

Re: Adding CI to our tree

2022-03-02 Thread Justin Pryzby
On Mon, Feb 28, 2022 at 02:58:02PM -0600, Justin Pryzby wrote: > I still think that if "Build Docs" is a separate cirrus task, it should > rebuild > docs on every CI run, even if they haven't changed, for any patch that touches > docs/. It'll be confusing if cfbot shows 5 green circles and 4 of

Re: Adding CI to our tree

2022-02-28 Thread Justin Pryzby
On Sat, Feb 26, 2022 at 08:08:38PM -0800, Andres Freund wrote: > On 2022-02-26 21:10:57 -0600, Justin Pryzby wrote: > > If someone renames or removes an xref target, shouldn't CI fail on its next > > run for a patch which tries to reference it ? > > Why wouldn't it? I suppose you're right - I

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
Hi, On 2022-02-26 21:10:57 -0600, Justin Pryzby wrote: > I did git commit --amend --no-edit and repushed to github to trigger a new CI > run, and it did this: https://github.com/justinpryzby/postgres/runs/5347878714 > > This is in a branch with changes to doc. I wasn't intending it to skip >

Re: Adding CI to our tree

2022-02-26 Thread Justin Pryzby
On Sat, Feb 26, 2022 at 06:50:00PM -0800, Andres Freund wrote: > Hi, > > On 2022-02-26 20:43:52 -0600, Justin Pryzby wrote: > > This doesn't do the right thing - I just tried. > > https://cirrus-ci.org/guide/writing-tasks/#environment-variables > > | changesInclude function can be very useful for

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
Hi, On 2022-02-26 20:43:52 -0600, Justin Pryzby wrote: > This doesn't do the right thing - I just tried. > https://cirrus-ci.org/guide/writing-tasks/#environment-variables > | changesInclude function can be very useful for skipping some tasks when no > changes to sources have been made since the

Re: Adding CI to our tree

2022-02-26 Thread Justin Pryzby
On Sat, Feb 26, 2022 at 05:09:08PM -0800, Andres Freund wrote: > > XXX: if this is run in the same task, the configure flags should probably be > > consistent ? > > What do you mean? I mean that commit to run CompilerWarnings unconditionally built docs with different flags than the other stuff

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
On 2022-02-26 17:09:08 -0800, Andres Freund wrote: > You could put the script in src/tools/ci and call it from the script to avoid > the quoting issues. Might also be a good idea for the bulk of the docs / coverage stuff, even if there are no quoting issues.

Re: Adding CI to our tree

2022-02-26 Thread Andres Freund
Hi, > Subject: [PATCH 2/7] cirrus: upload changed html docs as artifacts I still think the determination of the base branch needs to be resolved before this can be considered. > Always run doc build; to allow them to be shown in cfbot, they should not be > skipped if the linux build fails. >

Re: Adding CI to our tree

2022-02-25 Thread Justin Pryzby
This is the other half of my CI patches, which are unrelated to the TAP ones on the other thread. >From 88c01c09ee26db2817629265fc12b2dbcd8c9a91 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 17 Jan 2022 00:53:04 -0600 Subject: [PATCH 1/7] cirrus: include hints how to install OS

Re: Adding CI to our tree (ccache)

2022-02-20 Thread Justin Pryzby
On Sun, Feb 20, 2022 at 12:47:31PM -0800, Andres Freund wrote: > > Did you ever try to use clcache (or others) ? > > > > When I tried, it refused to cache because of our debug settings > > (DebugInformationFormat) - which seem to be enabled even in release mode. > > > I wonder if that'll be an

Re: Adding CI to our tree (ccache)

2022-02-20 Thread Andres Freund
Hi, On 2022-02-20 13:36:55 -0600, Justin Pryzby wrote: > Have you tried to use the yet-to-be-released ccache with MSVC ? Yes, it doesn't work, because it requires cl.exe to be used in a specific way (only a single input file, specific output file naming). Which would require a decent amount of

Re: Adding CI to our tree (ccache)

2022-02-20 Thread Justin Pryzby
Have you tried to use the yet-to-be-released ccache with MSVC ? Also, do you know about msbuild /outputResultsCache ? When I tried that, it gave a bunch of error. https://cirrus-ci.com/task/5697497241747456 |[16:35:13.605] 1>c:\cirrus\pgsql.sln.metaproj : error : MSB4252: Project

Re: Adding CI to our tree

2022-02-16 Thread Peter Eisentraut
On 13.02.22 09:30, Andres Freund wrote: BTW, docs can be built in parallel, and CI is using BUILD_JOBS: 4. /usr/bin/xmllint --path . --noout --valid postgres.sgml /usr/bin/xmllint --path . --noout --valid postgres.sgml /usr/bin/xsltproc --path . --stringparam pg.version '15devel' stylesheet.xsl

Re: Adding CI to our tree

2022-02-15 Thread Andres Freund
Hi, On February 15, 2022 10:12:36 PM PST, Justin Pryzby wrote: >On Sun, Feb 13, 2022 at 01:53:19PM -0800, Andres Freund wrote: >> Hi, >> >> On 2022-02-13 15:31:20 -0600, Justin Pryzby wrote: >> > Oh - I suppose you're right. That's an unfortunate consequence of running >> > a >> > single

Re: Adding CI to our tree

2022-02-15 Thread Justin Pryzby
On Sun, Feb 13, 2022 at 01:53:19PM -0800, Andres Freund wrote: > Hi, > > On 2022-02-13 15:31:20 -0600, Justin Pryzby wrote: > > Oh - I suppose you're right. That's an unfortunate consequence of running a > > single prove instance without chdir. > > I don't think it's chdir that's relevant (that

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-13 15:42:13 -0600, Justin Pryzby wrote: > > Note that prove unfortunately serializes the test output to be in the order > > it > > started them, even when tests run concurrently. Extremely unhelpful, but ... > > Are you sure ? Somewhat. I think it's a question of the prove

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-13 15:31:20 -0600, Justin Pryzby wrote: > Oh - I suppose you're right. That's an unfortunate consequence of running a > single prove instance without chdir. I don't think it's chdir that's relevant (that changes into the source directory after all). It's the TESTDIR environment

Re: Adding CI to our tree

2022-02-13 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 02:26:25PM -0800, Andres Freund wrote: > On 2022-02-12 16:06:40 -0600, Justin Pryzby wrote: > > I had some success with that, but it doesn't seem to be significantly > > faster - > > it looks a lot like the tests are not actually running in parallel. Note that the total

Re: Adding CI to our tree

2022-02-13 Thread Justin Pryzby
On Sun, Feb 13, 2022 at 01:23:16PM -0800, Andres Freund wrote: > If you're seeing this on windows on one of your test branches, that's much > more likely to be caused by the alltaptests stuff, than by the change in > artifact instruction. Oh - I suppose you're right. That's an unfortunate

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-13 15:02:50 -0600, Justin Pryzby wrote: > On Sat, Feb 12, 2022 at 04:24:20PM -0800, Andres Freund wrote: > > > What I am excited about is that some of your other changes showed that we > > > don't need separate *_artifacts for separate directories anymore. That > > > used to > > >

Re: Adding CI to our tree

2022-02-13 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 04:24:20PM -0800, Andres Freund wrote: > > What I am excited about is that some of your other changes showed that we > > don't need separate *_artifacts for separate directories anymore. That used > > to > > be the case, but an array of paths is now supported. Putting log,

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-13 15:09:20 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-02-13 12:13:17 -0500, Tom Lane wrote: > >> Right. Can we set things up so that it's not too painful to inject > >> custom build options into a CI build? > > > What kind of injection are you thinking about? >

Re: Adding CI to our tree

2022-02-13 Thread Tom Lane
Andres Freund writes: > On 2022-02-13 12:13:17 -0500, Tom Lane wrote: >> Right. Can we set things up so that it's not too painful to inject >> custom build options into a CI build? > What kind of injection are you thinking about? That's exactly what needs to be decided. > A patch author can

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-13 12:13:17 -0500, Tom Lane wrote: > Robert Haas writes: > > This is exactly why I'm not a huge fan of having ci stuff in the tree. > > It supposes that there's one right way to do a build, but in reality, > > different people want and indeed need to use different options for all

Re: Adding CI to our tree

2022-02-13 Thread Tom Lane
Robert Haas writes: > This is exactly why I'm not a huge fan of having ci stuff in the tree. > It supposes that there's one right way to do a build, but in reality, > different people want and indeed need to use different options for all > kinds of reasons. That's the whole value of having things

Re: Adding CI to our tree

2022-02-13 Thread Robert Haas
On Sun, Feb 13, 2022 at 3:30 AM Andres Freund wrote: > > There's other things that it'd be nice to enable, but maybe these don't > > need to > > be on by default. Maybe just have a list of optional compiler flags (and > > hints > > when they're useful). Like WRITE_READ_PARSE_PLAN_TREES. > > I

Re: Adding CI to our tree

2022-02-13 Thread Andres Freund
Hi, On 2022-02-12 23:19:38 -0600, Justin Pryzby wrote: > On Sat, Feb 12, 2022 at 05:20:08PM -0800, Andres Freund wrote: > > What was the reason behind moving the docs stuff from the compiler warnings > > task to linux? > > I wanted to build docs even if the linux task fails. To allow CFBOT to

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 05:20:08PM -0800, Andres Freund wrote: > On 2022-02-03 23:04:04 -0600, Justin Pryzby wrote: > > > I'm a bit worried about the increased storage and runtime overhead due to > > > the > > > docs changes. We probably can make it a good bit cheaper though. > > > > If you mean

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
On 2022-02-12 20:47:04 -0600, Justin Pryzby wrote: > While rebasing, I noticed an error. > > You wrote **/.diffs, but should be **/*.diffs Embarassing. Thanks for noticing! Pushed the fix...

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 04:24:20PM -0800, Andres Freund wrote: > > > e5286ede1b4 cirrus: avoid unnecessary double star ** > > > > Can't get excited about this, but whatever. > > > > What I am excited about is that some of your other changes showed that we > > don't need separate *_artifacts for

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, Alvaro adding you because the "branch" discussion in the MERGE thread. On 2022-02-03 23:04:04 -0600, Justin Pryzby wrote: > > I'm a bit worried about the increased storage and runtime overhead due to > > the > > docs changes. We probably can make it a good bit cheaper though. > > If you

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, On 2022-02-03 11:57:18 -0800, Andres Freund wrote: > > 95793675633 cirrus: spell ccache_maxsize > > Yep, will apply with a bunch of your other changes, if you answer a question > or two... Pushed. > > e5286ede1b4 cirrus: avoid unnecessary double star ** > > Can't get excited about this,

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 16:06:40 -0600, Justin Pryzby wrote: > I had some success with that, but it doesn't seem to be significantly faster - > it looks a lot like the tests are not actually running in parallel. Note that prove unfortunately serializes the test output to be in the order it started

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Tue, Jan 18, 2022 at 05:16:26PM -0800, Andres Freund wrote: > On 2022-01-18 15:08:47 -0600, Justin Pryzby wrote: > > On Mon, Jan 17, 2022 at 12:16:19PM -0800, Andres Freund wrote: > > > I think it might still be worth adding stopgap way of running all tap > > > tests on > > > windows though.

Re: Adding CI to our tree

2022-02-05 Thread Andres Freund
Hi, On 2022-02-03 23:04:04 -0600, Justin Pryzby wrote: > > I assume this doesn't yet work to a meaningful degree? Last time I checked > > there were quite a few tests that needed to be invoked in a specific > > directory. > > It works - tap_check() does chdir(). Ah, I thought you'd implemented

Re: Adding CI to our tree

2022-02-03 Thread Andres Freund
Hi, On February 3, 2022 9:04:04 PM PST, Justin Pryzby wrote: >On Thu, Feb 03, 2022 at 11:57:18AM -0800, Andres Freund wrote: >> On 2022-02-02 21:58:28 -0600, Justin Pryzby wrote: >> > FYI: I've rebased these against your cirrus/windows changes. >> >> What's the idea behind >> #echo 'deb

Re: Adding CI to our tree

2022-02-03 Thread Justin Pryzby
On Thu, Feb 03, 2022 at 11:57:18AM -0800, Andres Freund wrote: > On 2022-02-02 21:58:28 -0600, Justin Pryzby wrote: > > FYI: I've rebased these against your cirrus/windows changes. > > Did you put then on a dedicated branch, or only intermixed with other changes? Yes it's intermixed (because I

Re: Adding CI to our tree

2022-02-03 Thread Andres Freund
Hi, On 2022-02-02 21:58:28 -0600, Justin Pryzby wrote: > FYI: I've rebased these against your cirrus/windows changes. Did you put then on a dedicated branch, or only intermixed with other changes? > A recent cirrus result is here; this has other stuff in the branch, so you can > see the

Re: Adding CI to our tree

2022-02-02 Thread Justin Pryzby
On Tue, Jan 18, 2022 at 03:08:47PM -0600, Justin Pryzby wrote: > On Mon, Jan 17, 2022 at 12:16:19PM -0800, Andres Freund wrote: > > I think it might still be worth adding stopgap way of running all tap tests > > on > > windows though. Having a vcregress.pl function to find all directories with >

Re: pg_basebackup fsyncs some files despite --no-sync (was: Adding CI to our tree)

2022-01-23 Thread Andres Freund
Hi, On 2022-01-21 12:00:57 -0800, Andres Freund wrote: > The attached patch adds no-sync handling to the manifest rename, as well as > one case in the directory wal method. Pushed that. Greetings, Andres Freund

pg_basebackup fsyncs some files despite --no-sync (was: Adding CI to our tree)

2022-01-21 Thread Andres Freund
Hi, On 2022-01-18 20:16:46 -0800, Andres Freund wrote: > I noticed a few other sources of "unnecessary" fsyncs. The most frequent > being the durable_rename() of backup_manifest in pg_basebackup.c. Manifests > are > surprisingly large, 135k for a freshly initdb'd cluster. Robert, I assume the

Re: Adding CI to our tree

2022-01-19 Thread Tom Lane
Andres Freund writes: > I'm doubtful that it's good that we use BAIL_OUT so liberally, because it > prevents further tests from running (i.e. if 001 bails, 002+ doesn't run), > which doesn't generally seem like the right thing to do after a single test > fails. But that's really independent of

Re: Adding CI to our tree

2022-01-19 Thread Andres Freund
Hi, On 2022-01-19 15:05:44 -0500, Tom Lane wrote: > And the cause of that is obvious: Cluster::start thinks that if "pg_ctl > start" failed, there couldn't possibly be a postmaster running. So it > doesn't bother to update self->_pid, and then the END hook thinks there is > nothing to do. Ah,

Re: Adding CI to our tree

2022-01-19 Thread Tom Lane
I wrote: > This test attempt revealed another problem too: the standby never > shut down, and thus the calling "make" never quit, until I intervened > manually. I'm not sure why. I see that Cluster::promote uses > system_or_bail() to run "pg_ctl promote" ... could it be that > BAIL_OUT causes

Re: Adding CI to our tree

2022-01-18 Thread Tom Lane
Andres Freund writes: > On 2022-01-18 21:50:07 -0500, Tom Lane wrote: >> This actually causes parallel check-world to fail altogether on florican's >> host, because the initial fsync of the recovered primary takes more than 3 >> minutes when there's conflicting I/O traffic, causing pg_ctl to time

Re: Adding CI to our tree

2022-01-18 Thread Tom Lane
Andres Freund writes: > On 2022-01-18 21:50:07 -0500, Tom Lane wrote: >> There is no reason for this script to be overriding Cluster.pm's >> fsync = off setting. >> This appears to go back to 917dc7d23 of 2016, so I think it just >> predates our recognition that we should disable fsync in routine

Re: Adding CI to our tree

2022-01-18 Thread Andres Freund
Hi, On 2022-01-18 21:50:07 -0500, Tom Lane wrote: > I just found one thing making check-world slower than it ought to be: > src/test/recovery/t/008_fsm_truncation.pl does > > $node_primary->append_conf( > 'postgresql.conf', qq{ > fsync = on > wal_log_hints = on > max_prepared_transactions

Re: Adding CI to our tree

2022-01-18 Thread Tom Lane
I just found one thing making check-world slower than it ought to be: src/test/recovery/t/008_fsm_truncation.pl does $node_primary->append_conf( 'postgresql.conf', qq{ fsync = on wal_log_hints = on max_prepared_transactions = 5 autovacuum = off }); There is no reason for this script to

Re: Adding CI to our tree

2022-01-18 Thread Andres Freund
Hi, On 2022-01-18 15:08:47 -0600, Justin Pryzby wrote: > On Mon, Jan 17, 2022 at 12:16:19PM -0800, Andres Freund wrote: > > I think it might still be worth adding stopgap way of running all tap tests > > on > > windows though. Having a vcregress.pl function to find all directories with > > t/ >

Re: Adding CI to our tree

2022-01-18 Thread Justin Pryzby
On Mon, Jan 17, 2022 at 12:16:19PM -0800, Andres Freund wrote: > I think it might still be worth adding stopgap way of running all tap tests on > windows though. Having a vcregress.pl function to find all directories with t/ > and run the tests there, shouldn't be a lot of code... I started doing

Re: Adding CI to our tree

2022-01-18 Thread Andrew Dunstan
On 1/18/22 12:44, Andres Freund wrote: > Hi, > > On 2022-01-18 11:20:08 -0500, Andrew Dunstan wrote: >> Sure, very doable, although we will still need some special logic for >> src/test - there are security implication from running the ssl, ldap and >> kerberos tests by default. See its

Re: Adding CI to our tree

2022-01-18 Thread Andres Freund
Hi, On 2022-01-18 11:20:08 -0500, Andrew Dunstan wrote: > Sure, very doable, although we will still need some special logic for > src/test - there are security implication from running the ssl, ldap and > kerberos tests by default. See its Makefile. ISTM that we should move the PG_TEST_EXTRA

Re: Adding CI to our tree

2022-01-18 Thread Andrew Dunstan
On 1/18/22 08:06, Peter Eisentraut wrote: > On 17.01.22 22:13, Andrew Dunstan wrote: >> Well, the logic we use for TAP tests is we run them for the following if >> they have a t subdirectory, subject to configuration settings like >> PG_TEST_EXTRA: >> >>   src/test/bin/* >>   contrib/* >>   

Re: Adding CI to our tree

2022-01-18 Thread Peter Eisentraut
On 17.01.22 22:13, Andrew Dunstan wrote: Well, the logic we use for TAP tests is we run them for the following if they have a t subdirectory, subject to configuration settings like PG_TEST_EXTRA:  src/test/bin/*  contrib/*  src/test/modules/*  src/test/* As long as any new TAP test gets

Re: Adding CI to our tree

2022-01-17 Thread Andrew Dunstan
On 1/17/22 13:19, Andres Freund wrote: > Hi, > > On 2022-01-17 10:25:12 -0500, Andrew Dunstan wrote: >> The buildfarm is moving in the opposite direction, to disaggregate >> steps. > I'm a bit confused as to where you want changes to vcregress.pl > going. Upthread you argued against adding more

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 14:30:53 -0500, Tom Lane wrote: > Andres Freund writes: > > I think it's not actually that hard, with something like I described in the > > email upthread, with each tests going into a prescribed location, and the > > on-disk status being inspectable in an automated way.

Re: Adding CI to our tree

2022-01-17 Thread Tom Lane
Andres Freund writes: > I think it's not actually that hard, with something like I described in the > email upthread, with each tests going into a prescribed location, and the > on-disk status being inspectable in an automated way. check-world could invoke > a command to summarize the tests at

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 13:50:04 -0500, Robert Haas wrote: > On Mon, Jan 17, 2022 at 1:19 PM Andres Freund wrote: > > FWIW, to me this shouldn't require a lot of separate manual test > > invocations. And continuing to have lots of granular test invocations from > > the > > buildfarm client is *bad*,

Re: Adding CI to our tree

2022-01-17 Thread Tom Lane
Robert Haas writes: > I have a lot of sympathy with Andrew here, actually. If you just do > 'make check-world' and assume that will cover everything, you get one > giant output file. That is not great at all. Yeah. I agree with Andrew that we want output that is more modular, not less so. But

Re: Adding CI to our tree

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 1:19 PM Andres Freund wrote: > FWIW, to me this shouldn't require a lot of separate manual test > invocations. And continuing to have lots of granular test invocations from the > buildfarm client is *bad*, because it requires constantly syncing up the set > of test

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 10:25:12 -0500, Andrew Dunstan wrote: > The buildfarm is moving in the opposite direction, to disaggregate > steps. I'm a bit confused as to where you want changes to vcregress.pl going. Upthread you argued against adding more granular targets to vcregress. But this seems to be

Re: Adding CI to our tree

2022-01-17 Thread Andrew Dunstan
On 1/14/22 18:54, Justin Pryzby wrote: > On Fri, Jan 14, 2022 at 03:34:11PM -0800, Andres Freund wrote: >> Hi, >> >> On 2022-01-13 15:27:40 -0500, Andrew Dunstan wrote: >>> I can probably adjust to whatever we decide to do. But I think we're >>> really just tinkering at the edges here. What I

Re: Adding CI to our tree

2022-01-14 Thread Justin Pryzby
On Fri, Jan 14, 2022 at 03:34:11PM -0800, Andres Freund wrote: > Hi, > > On 2022-01-13 15:27:40 -0500, Andrew Dunstan wrote: > > I can probably adjust to whatever we decide to do. But I think we're > > really just tinkering at the edges here. What I think we really need is > > the moral

Re: Adding CI to our tree

2022-01-14 Thread Andres Freund
Hi, On 2022-01-13 15:27:40 -0500, Andrew Dunstan wrote: > I can probably adjust to whatever we decide to do. But I think we're > really just tinkering at the edges here. What I think we really need is > the moral equivalent of `make check-world` in one invocation of > vcregress.pl. I agree

Re: Adding CI to our tree

2022-01-13 Thread Andrew Dunstan
On 1/13/22 13:55, Andres Freund wrote: >> It needs to avoid running on the buildfarm, right ? > I guess so. It currently appears to have its own logic for finding contrib > (and other) tap tests: > > foreach my $testdir (glob("$pgsql/contrib/*")) > { > next unless

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
Hi, On 2022-01-13 13:06:42 -0600, Justin Pryzby wrote: > Question: are data-checksums tested at all ? The only thing I can find is > that > some buildfarm members *might* exercise it during installcheck. There's some coverage via src/bin/pg_basebackup/t/010_pg_basebackup.pl and

Re: Adding CI to our tree

2022-01-13 Thread Justin Pryzby
On Thu, Jan 13, 2022 at 10:55:27AM -0800, Andres Freund wrote: > I'll probably apply that part and 0002 separately. I've hacked on it a bit more now.. Question: are data-checksums tested at all ? The only thing I can find is that some buildfarm members *might* exercise it during installcheck.

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
Hi, On 2022-01-10 16:07:48 -0600, Justin Pryzby wrote: > On Sun, Jan 09, 2022 at 11:57:44AM -0800, Andres Freund wrote: > > On 2022-01-09 13:16:50 -0600, Justin Pryzby wrote: > > > diff --git a/contrib/test_decoding/Makefile > > > b/contrib/test_decoding/Makefile > > > index

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
Hi, On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > I plan to push this after another cycle of tests passing (and driving > over the bay bridge...) unless I hear protests. I noticed that it's harder to see compile warnings on msvc in CI than it was at an earlier point. There used to be a

Re: Adding CI to our tree

2022-01-10 Thread Justin Pryzby
On Sun, Jan 09, 2022 at 11:57:44AM -0800, Andres Freund wrote: > On 2022-01-09 13:16:50 -0600, Justin Pryzby wrote: > > diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile > > index 9a31e0b8795..14fd847ba7f 100644 > > --- a/contrib/test_decoding/Makefile > > +++

Re: Adding CI to our tree

2022-01-09 Thread Andres Freund
Hi, On 2021-10-02 12:59:09 -0700, Andres Freund wrote: > On 2021-10-02 11:05:20 -0400, Tom Lane wrote: > > I don't know enough about Windows to evaluate 0001, but I'm a little > > worried about it because it looks like it's changing our *production* > > error handling on that platform. > > Yea.

Re: Adding CI to our tree

2022-01-09 Thread Andres Freund
Hi, On 2022-01-09 13:16:50 -0600, Justin Pryzby wrote: > I noticed a patch failing in cfbot everywhere except windows: > > https://commitfest.postgresql.org/36/3476/ > | Invalid relcache when ADD PRIMARY KEY USING INDEX > > It's because vcregress skips tests which have NO_INSTALLCHECK=1. > Is

Re: Adding CI to our tree

2022-01-09 Thread Justin Pryzby
I noticed a patch failing in cfbot everywhere except windows: https://commitfest.postgresql.org/36/3476/ | Invalid relcache when ADD PRIMARY KEY USING INDEX It's because vcregress skips tests which have NO_INSTALLCHECK=1. Is it desirable to enable more module/contrib tests for windows CI ?

Re: Adding CI to our tree

2021-12-31 Thread Alvaro Herrera
On 2021-Dec-30, Andres Freund wrote: > On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > > I plan to push this after another cycle of tests passing (and driving > > over the bay bridge...) unless I hear protests. > > Pushed. > > Marked CF entry as committed. I tried it using the column

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > I plan to push this after another cycle of tests passing (and driving > over the bay bridge...) unless I hear protests. Pushed. Marked CF entry as committed.

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
Hi, On 2021-12-30 20:28:46 -0600, Justin Pryzby wrote: > [ language fixes] Thanks! > script uses a pseudo-tty, which do support locking. > => which does This didn't seem right either way - it's pseudo-ttys that don't support locking, so plural seemed appropriate? I changed it to "script uses

Re: Adding CI to our tree

2021-12-30 Thread Justin Pryzby
commit message: agithub the the the buildfarm. => the access too. => to # Due to that it using concurrency within prove is helpful. => Due to that, it's useful to run prove with multiple jobs. further details src/tools/ci/README => further details , see src/tools/ci/README script uses a

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
Hi, Attached is v5 of the CI patch. Not a lot of changes: - a bunch of copy-editing, wrote a commit message etc - use ccache for CXX/CLANG in the CompilerWarnings task, I had missed that when making the task use all --with-* flags - switch linux to use ossp-uuid. I tried to switch macos at

Re: Adding CI to our tree

2021-12-29 Thread Daniel Gustafsson
> On 29 Dec 2021, at 21:17, Andres Freund wrote: > On 2021-12-20 11:21:05 -0800, Andres Freund wrote: >> Attached is v4 of the CI patch. > > I'd like to push this - any objections? It's not disruptive to anything but > cfbot, so we can incrementally improve it further. No objection, I'm +1 on

  1   2   >