Re: Mark all GUC variable as PGDLLIMPORT

2022-02-12 Thread Julien Rouhaud
Hi, On Sat, Feb 12, 2022 at 07:59:56PM -0800, Andres Freund wrote: > > On 2021-08-23 14:53:34 +0800, Julien Rouhaud wrote: > > So since the non currently explicitly exported GUC global variables > > shouldn't > > be accessible by third-party code, I'm attaching a POC patch that does the > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-12 Thread Dilip Kumar
On Sun, Feb 13, 2022 at 10:12 AM Dilip Kumar wrote: > I have done performance testing with different template DB sizes and different amounts of dirty shared buffers and I think as expected the bigger the dirty shared buffer the checkpoint approach becomes costly and OTOH the larger the template

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-12 Thread Andres Freund
Hi, On 2022-02-13 18:07:30 +1300, Thomas Munro wrote: > On Sun, Feb 13, 2022 at 5:50 PM Andres Freund wrote: > >> > +# required for 027_stream_regress.pl > >> > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery > >> > +export REGRESS_OUTPUTDIR > >> > >> Why do we need this? > > > > The

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 05:20:08PM -0800, Andres Freund wrote: > On 2022-02-03 23:04:04 -0600, Justin Pryzby wrote: > > > I'm a bit worried about the increased storage and runtime overhead due to > > > the > > > docs changes. We probably can make it a good bit cheaper though. > > > > If you mean

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-12 Thread Tom Lane
Thomas Munro writes: > I thought it was a goal that VPATH builds shouldn't pollute the source > tree, but the Make macro prove_check is explicitly doing so by > default. Perhaps *that* should be fixed? Indeed. That seems broken by definition. More generally, I thought we'd established a

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-12 Thread Thomas Munro
On Sun, Feb 13, 2022 at 5:50 PM Andres Freund wrote: > On 2022-01-18 11:20:16 +0900, Michael Paquier wrote: > > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/bin/pg_upgrade > > +export REGRESS_OUTPUTDIR > > I don't really understand why 027_stream_regress.pl is using this (and thus > not why it's

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-12 Thread Andres Freund
Hi, On 2022-01-18 11:20:16 +0900, Michael Paquier wrote: > +# required for 002_pg_upgrade.pl > +REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX) > +export REGRESS_SHLIB It seems weird to propagate this into multiple places. Why don't we define that centrally? Although it's

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-12 Thread Dilip Kumar
On Sat, Feb 12, 2022 at 2:38 AM Alvaro Herrera wrote: > > On 2022-Feb-11, Robert Haas wrote: > > > What I find difficult about doing that is that this is all a bunch of > > technical details that users may have difficulty understanding. If we > > say WAL_LOG or WAL_LOG_DATA, a reasonably but not

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Tom Lane
Michael Paquier writes: > On Sat, Feb 12, 2022 at 05:16:03PM -0500, Tom Lane wrote: >> I think adding an explicit PGAC_PATH_PROGS would be worth the cycles. > Well, this somebody is I, and the buildfarm did not blow up on any of > that so that looked rather fine. Eh? copperhead for one is

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-12 Thread Andres Freund
Hi, On 2022-01-18 11:20:16 +0900, Michael Paquier wrote: > On Sat, Jan 15, 2022 at 01:52:39PM +0800, Julien Rouhaud wrote: > > libpq environment variable PGHOST has a non-local server value: > > C:/Users/ContainerAdministrator/AppData/Local/Temp/FhBIlsw6SV > > Failure, exiting > > not ok 3 - run

Re: Mark all GUC variable as PGDLLIMPORT

2022-02-12 Thread Andres Freund
Hi, On 2021-08-23 14:53:34 +0800, Julien Rouhaud wrote: > So since the non currently explicitly exported GUC global variables shouldn't > be accessible by third-party code, I'm attaching a POC patch that does the > opposite of v1: enforce that restriction using a new pg_attribute_hidden() >

Re: Fix overflow in DecodeInterval

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 21:16:05 -0500, Joseph Koshakow wrote: > I've attached the patch below. Any reason for using int return types? > +/* As above, but initial val produces years */ > +static int > +AdjustYears(int val, struct pg_tm *tm, int multiplier) > +{ > + int years;

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
On 2022-02-12 20:47:04 -0600, Justin Pryzby wrote: > While rebasing, I noticed an error. > > You wrote **/.diffs, but should be **/*.diffs Embarassing. Thanks for noticing! Pushed the fix...

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-12 Thread Andres Freund
On 2022-02-13 12:36:13 +0900, Masahiko Sawada wrote: > Actually, I'm working on simplifying and improving radix tree > implementation for PG16 dev cycle. From the discussion so far I think > it's better to have a data structure that can be used for > general-purpose and is also good for storing

using file cloning in create database / initdb

2022-02-12 Thread Andres Freund
Hi, This thread started at https://www.postgresql.org/message-id/20220213021746.GM31460%40telsasoft.com but is mostly independent, so I split the thread off On 2022-02-12 20:17:46 -0600, Justin Pryzby wrote: > On Sat, Feb 12, 2022 at 06:00:44PM -0800, Andres Freund wrote: > > I bet using COW

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-12 Thread Masahiko Sawada
On Sun, Feb 13, 2022 at 11:02 AM Andres Freund wrote: > > Hi, > > On 2022-02-11 13:47:01 +0100, Matthias van de Meent wrote: > > Today I noticed the inefficiencies of our dead tuple storage once > > again, and started theorizing about a better storage method; which is > > when I remembered that

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Michael Paquier
On Sat, Feb 12, 2022 at 05:16:03PM -0500, Tom Lane wrote: > It looks to me like somebody figured it didn't need any more support > than gzip/bzip2, which is wrong on a couple of grounds: > * hardly any modern platforms lack those, unlike lz4 > * we don't invoke either one of them during testing,

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 04:24:20PM -0800, Andres Freund wrote: > > > e5286ede1b4 cirrus: avoid unnecessary double star ** > > > > Can't get excited about this, but whatever. > > > > What I am excited about is that some of your other changes showed that we > > don't need separate *_artifacts for

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-12 Thread Justin Pryzby
On Sat, Feb 12, 2022 at 06:00:44PM -0800, Andres Freund wrote: > Hi, > > On 2022-02-11 16:19:12 -0500, Robert Haas wrote: > > I somewhat hope we never end up with THREE strategies for creating a new > > database, but now that I think about it, we might. Somebody might want to > > use a fancy FS

Re: Fix overflow in DecodeInterval

2022-02-12 Thread Joseph Koshakow
On Fri, Feb 11, 2022 at 4:58 PM Joseph Koshakow wrote: > > Tom Lane writes: > > Joseph Koshakow writes: > > > The attached patch fixes an overflow bug in DecodeInterval when applying > > > the units week, decade, century, and millennium. The overflow check logic > > > was modelled after the

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-12 Thread Andres Freund
Hi, On 2022-02-11 13:47:01 +0100, Matthias van de Meent wrote: > Today I noticed the inefficiencies of our dead tuple storage once > again, and started theorizing about a better storage method; which is > when I remembered that this thread exists, and that this thread > already has amazing

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-12 Thread Andres Freund
Hi, On 2022-02-11 16:19:12 -0500, Robert Haas wrote: > I somewhat hope we never end up with THREE strategies for creating a new > database, but now that I think about it, we might. Somebody might want to > use a fancy FS primitive that clones a directory at the FS level, or > something. I think

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 11:47:20 -0500, Tom Lane wrote: > Alexander Lakhin writes: > > 11.02.2022 05:22, Andres Freund wrote: > >> Over in another thread I made some wild unsubstantiated guesses that the > >> windows issues could have been made much more likely by a somewhat odd bit > >> of > >> code

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, Alvaro adding you because the "branch" discussion in the MERGE thread. On 2022-02-03 23:04:04 -0600, Justin Pryzby wrote: > > I'm a bit worried about the increased storage and runtime overhead due to > > the > > docs changes. We probably can make it a good bit cheaper though. > > If you

Re: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 1173, ...

2022-02-12 Thread Andres Freund
Hi, On February 12, 2022 3:57:28 PM PST, Thomas Munro wrote: >Hi, > >I don't know what happened here, but I my little script that scrapes >for build farm assertion failures hasn't seen this one before now. >It's on a 15 year old macOS release and compiled with 15 year old GCC >version, for

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, On 2022-02-03 11:57:18 -0800, Andres Freund wrote: > > 95793675633 cirrus: spell ccache_maxsize > > Yep, will apply with a bunch of your other changes, if you answer a question > or two... Pushed. > > e5286ede1b4 cirrus: avoid unnecessary double star ** > > Can't get excited about this,

FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 1173, ...

2022-02-12 Thread Thomas Munro
Hi, I don't know what happened here, but I my little script that scrapes for build farm assertion failures hasn't seen this one before now. It's on a 15 year old macOS release and compiled with 15 year old GCC version, for some added mystery.

Re: Adding CI to our tree

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 16:06:40 -0600, Justin Pryzby wrote: > I had some success with that, but it doesn't seem to be significantly faster - > it looks a lot like the tests are not actually running in parallel. Note that prove unfortunately serializes the test output to be in the order it started

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Tom Lane
Robert Haas writes: > On Sat, Feb 12, 2022 at 5:09 PM Andres Freund wrote: >> I don't think there's an actual configure check for the lz4 binary? Looks >> like >> a static assignment in src/Makefile.global.in to me. > Oh. That seems kind of dumb. It looks to me like somebody figured it didn't

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Robert Haas
On Sat, Feb 12, 2022 at 5:09 PM Andres Freund wrote: > On 2022-02-12 07:24:46 -0500, Robert Haas wrote: > > You may be right, but I'm not entirely convinced. $ENV{'LZ4'} here is > > being set by make, and make is setting it to whatever configure found. > > If configure found a version of the lz4

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Andres Freund
On 2022-02-12 07:24:46 -0500, Robert Haas wrote: > You may be right, but I'm not entirely convinced. $ENV{'LZ4'} here is > being set by make, and make is setting it to whatever configure found. > If configure found a version of the lz4 program that doesn't actually > work, isn't that configure's

Re: Adding CI to our tree

2022-02-12 Thread Justin Pryzby
On Tue, Jan 18, 2022 at 05:16:26PM -0800, Andres Freund wrote: > On 2022-01-18 15:08:47 -0600, Justin Pryzby wrote: > > On Mon, Jan 17, 2022 at 12:16:19PM -0800, Andres Freund wrote: > > > I think it might still be worth adding stopgap way of running all tap > > > tests on > > > windows though.

Re: buildfarm warnings

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 16:42:03 -0500, Tom Lane wrote: > Another new one that maybe should be silenced is > > /mnt/resource/bf/build/skink-master/HEAD/pgsql.build/../pgsql/src/backend/access/heap/vacuumlazy.c:1129:41: > warning: 'freespace' may be used uninitialized in this function >

Re: Race condition in TransactionIdIsInProgress

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 13:25:58 +, Simon Riggs wrote: > On Fri, 11 Feb 2022 at 19:08, Andres Freund wrote: > > > On 2022-02-11 13:48:59 +, Simon Riggs wrote: > > > On Fri, 11 Feb 2022 at 08:48, Simon Riggs > > > wrote: > > > > > > > > On Fri, 11 Feb 2022 at 06:11, Andres Freund wrote: > >

Re: buildfarm warnings

2022-02-12 Thread Tom Lane
Justin Pryzby writes: > Is there any check for warnings from new code, other than those buildfarm > members with -Werror ? I periodically grep the buildfarm logs for interesting warnings. There are a lot of uninteresting ones :-(, so I'm not sure how automatable that'd be. I do use a

Re: refactoring basebackup.c

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 15:12:21 -0600, Justin Pryzby wrote: > I think they would've been visible in the CI environment, too. Yea, but only if you looked carefully enough. The postgres github repo has CI enabled, and it's green. But the windows build step does show the warnings:

buildfarm warnings

2022-02-12 Thread Justin Pryzby
Is there any check for warnings from new code, other than those buildfarm members with -Werror ? It'd be better to avoid warnings, allowing members to use -Werror, rather than to allow/ignore warnings, which preclude that possibility. A circular problem. I checked for warnings on master during

Re: refactoring basebackup.c

2022-02-12 Thread Justin Pryzby
The LZ4 patches caused new compiler warnings. It's the same issue that was fixed at 71ce8 for gzip. I think they would've been visible in the CI environment, too. https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=wrasse=2022-02-12%2005%3A08%3A48=make

Re: Use generation context to speed up tuplesorts

2022-02-12 Thread Tomas Vondra
On 1/20/22 08:36, Ronan Dunklau wrote: > Le jeudi 20 janvier 2022, 02:31:15 CET David Rowley a écrit : >> On Tue, 18 Jan 2022 at 19:45, Julien Rouhaud wrote: >>> I'm also a bit confused about which patch(es) should actually be reviewed >>> in that thread. My understanding is that the original

Re: Use generation context to speed up tuplesorts

2022-02-12 Thread Tomas Vondra
On 1/25/22 13:34, Ronan Dunklau wrote: > > ... > > I've run the same 1-to-32-columns sort benchmark, using both glibc malloc and > jemalloc, with the standard aset memory manager or with David's generation v2 > patch. To use jemalloc, I just use a LD_PRELOAD env variable before starting >

Re: PGroonga index-only scan problem with yesterday’s PostgreSQL updates

2022-02-12 Thread Anders Kaseorg
On 2/12/22 09:25, Tom Lane wrote: I don't know that we'd go that far ... maybe if another bad problem turns up. In the meantime, though, I do have a modest suggestion: it would not be too hard to put some defenses in place against another such bug. The faulty commit was in our tree for a month

[PATCH] Add tests for psql tab completion

2022-02-12 Thread Matheus Alcantara
Hi hackers. I'm attaching a patch that add some new test cases for tab completion of psql. This is my first patch that I'm sending here so let me know if I'm doing something wrong. -- Matheus AlcantaraFrom 6af6b972960f4e9017f1c311ee4b13a96d5f66a1 Mon Sep 17 00:00:00 2001 From: Matheus

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Andres Freund
Hi, On 2022-02-12 13:29:54 +0900, Michael Paquier wrote: > On Sat, Feb 12, 2022 at 09:49:47AM +0900, Michael Paquier wrote: > > I guess that it would be better to revert the TAP test and rework all > > that for the time being, then. > > And this part is done. You wrote in the commit message

Re: logical decoding and replication of sequences

2022-02-12 Thread Tomas Vondra
On 2/12/22 01:34, Tomas Vondra wrote: > On 2/10/22 19:17, Tomas Vondra wrote: >> I've polished & pushed the first part adding sequence decoding >> infrastructure etc. Attached are the two remaining parts. >> >> I plan to wait a day or two and then push the test_decoding part. The >> last part (for

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Christoph Berg
Re: Tom Lane > Ah. That seems a bit problematic anyway ... if the version-dependent > pg_configs don't all return identical results, what's the > non-version-dependent one supposed to do? It still returns a correct --includedir for client apps that need it. (Unfortunately many need

Re: Re: PGroonga index-only scan problem with yesterday’s PostgreSQL updates

2022-02-12 Thread Tom Lane
Anders Kaseorg writes: > On 2/11/22 15:57, Tom Lane wrote: >> Possibly same issue I just fixed? >> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=e5691cc9170bcd6c684715c2755d919c5a16fea2 > Yeah, that does seem to fix my test cases. Thanks! > Any chance this will make it

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Tom Lane
Christoph Berg writes: > I was confusing that with this: The problem that led to the pg_config > patch years ago was that we have a /usr/bin/pg_config in > (non-major-version-dependant) libpq-dev, and > /usr/lib/postgresql/NN/bin/pg_config in the individual > postgresql-server-dev-NN packages,

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-02-12 Thread Tom Lane
Alexander Lakhin writes: > 11.02.2022 05:22, Andres Freund wrote: >> Over in another thread I made some wild unsubstantiated guesses that the >> windows issues could have been made much more likely by a somewhat odd bit of >> code in PQisBusy(): >>

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Christoph Berg
Re: Tom Lane > Christoph Berg writes: > > I think the "bug" here is that vanilla PG doesn't support installing > > in FHS locations with /usr/lib and /usr/share split, hence the Debian > > patch. > > I'm confused by this statement. An out-of-the-box installation > produces trees under

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Tom Lane
Christoph Berg writes: > I think the "bug" here is that vanilla PG doesn't support installing > in FHS locations with /usr/lib and /usr/share split, hence the Debian > patch. I'm confused by this statement. An out-of-the-box installation produces trees under $PREFIX/lib and $PREFIX/share. What

Re: Race condition in TransactionIdIsInProgress

2022-02-12 Thread Simon Riggs
On Fri, 11 Feb 2022 at 19:08, Andres Freund wrote: > On 2022-02-11 13:48:59 +, Simon Riggs wrote: > > On Fri, 11 Feb 2022 at 08:48, Simon Riggs > > wrote: > > > > > > On Fri, 11 Feb 2022 at 06:11, Andres Freund wrote: > > > > > > > Looks lik syncrep will make this a lot worse, because it

Re: pgsql: Add suport for server-side LZ4 base backup compression.

2022-02-12 Thread Robert Haas
On Fri, Feb 11, 2022 at 11:39 PM Michael Paquier wrote: > Perhaps you'd better check that 'decompress_program' can be executed > and skip things if the command is not around? Note that > 020_pg_receivewal.pl does an extra lz4 --version as a safety measure, > but 008_untar.pl does not. You may

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-02-12 Thread Bharath Rupireddy
On Thu, Feb 10, 2022 at 9:55 PM Peter Geoghegan wrote: > > On Sun, Feb 6, 2022 at 7:45 AM Robert Haas wrote: > > For what it's worth, I am generally in favor of having something like > > this in PostgreSQL. I think it's wrong of us to continue assuming that > > everyone has command-line access.

Re: Accommodate startup process in a separate ProcState array slot instead of in MaxBackends slots.

2022-02-12 Thread Bharath Rupireddy
On Fri, Feb 11, 2022 at 7:56 PM Yura Sokolov wrote: > > В Сб, 16/10/2021 в 16:37 +0530, Bharath Rupireddy пишет: > > On Thu, Oct 14, 2021 at 10:56 AM Fujii Masao > > wrote: > > > On 2021/10/12 15:46, Bharath Rupireddy wrote: > > > > On Tue, Oct 12, 2021 at 5:37 AM Fujii Masao > > > > wrote: >

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-02-12 Thread Alexander Lakhin
Hello Andres, 11.02.2022 05:22, Andres Freund wrote: > Over in another thread I made some wild unsubstantiated guesses that the > windows issues could have been made much more likely by a somewhat odd bit of > code in PQisBusy(): > > https://postgr.es/m/1959196.1644544971%40sss.pgh.pa.us > >

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-12 Thread Christoph Berg
Re: Michael Paquier > On Fri, Feb 11, 2022 at 10:48:11AM +0100, Christoph Berg wrote: > > It never caused any problem in the 12+ years we have been doing this, > > but Debian is patching pg_config not to be relocatable since we are > > installing into /usr/lib/postgresql/NN