Re: Handle infinite recursion in logical replication setup

2022-04-26 Thread Peter Smith
Here are my review comments for v10-0001 and v10-0002. (I did not yet look at the v10-0002 TAP tests. I will check those separately later). = v10-0001 comments = 1.1 src/include/catalog/pg_subscription.h @@ -70,6 +70,8 @@

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Tom Lane
Michael Paquier writes: > On Wed, Apr 27, 2022 at 12:36:10PM +0800, Rui Zhao wrote: >> Do you have interest in adding a test like one in my patch? > I have studied the test case you are proposing, and I am afraid that > it is too expensive as designed. That was my feeling too. It's certainly a

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Michael Paquier
On Wed, Apr 27, 2022 at 12:36:10PM +0800, Rui Zhao wrote: > Do you have interest in adding a test like one in my patch? I have studied the test case you are proposing, and I am afraid that it is too expensive as designed. And it is actually racy as you expect the restart point to take longer

Re: Fwd: range of composite types!

2022-04-26 Thread Tom Lane
Jian He writes: >> for that means the following sql queries should return* false:* >> select mytyperange ( >> (1,'2022-01-01')::mytype, >> (8, '2022-01-31')::mytype, '[]') @> (2, '2020-01-19')::mytype; Why should that return false? The comparison rules for composites say that you compare the

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Michael Paquier
On Tue, Apr 26, 2022 at 08:26:09PM -0700, Nathan Bossart wrote: > On Wed, Apr 27, 2022 at 10:43:53AM +0900, Kyotaro Horiguchi wrote: >> At Tue, 26 Apr 2022 11:33:49 -0700, Nathan Bossart >> wrote in >>> I suspect we'll start seeing this problem more often once end-of-recovery >>> checkpoints

Fwd: range of composite types!

2022-04-26 Thread Jian He
Hello. Just wondering if this is possible or not.. -- Forwarded message - From: Jian He Date: Tue, Apr 26, 2022 at 2:46 PM Subject: range of composite types! To: pgsql-general range of composite types. I found this would be a great idea!!! Question on stackoverflow

Re: bogus: logical replication rows/cols combinations

2022-04-26 Thread Amit Kapila
On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra wrote: > > On 4/25/22 17:48, Alvaro Herrera wrote: > > > The desired result on subscriber is: > > > > table uno; > > a │ b │ c > > ┼───┼─── > > 1 │ 2 │ > > -1 │ │ 4 > > > > > > Thoughts? > > > > I'm not quite sure which of the two behaviors

Re:Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Rui Zhao
Kyotaro'spatch seems good to me and fixes the test case in my patch. Do you have interest in adding a test like one in my patch? + LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); +/* * Remember the prior checkpoint's redo ptr for * UpdateCheckPointDistanceEstimate()

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Bharath Rupireddy
On Wed, Apr 27, 2022 at 8:45 AM Tom Lane wrote: > > I wrote: > > Thomas Munro writes: > >> BTW If you had your local change from debug.patch (upthread), that'd > >> defeat the patch. I mean this: > > >> +if(!*errormsg) > >> + *errormsg = "decode_queue_head is null"; > > > Oh!

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Nathan Bossart
On Wed, Apr 27, 2022 at 10:43:53AM +0900, Kyotaro Horiguchi wrote: > At Tue, 26 Apr 2022 11:33:49 -0700, Nathan Bossart > wrote in >> I suspect we'll start seeing this problem more often once end-of-recovery >> checkpoints are removed [0]. Would you mind creating a commitfest entry >> for this

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
I wrote: > Thomas Munro writes: >> BTW If you had your local change from debug.patch (upthread), that'd >> defeat the patch. I mean this: >> +if(!*errormsg) >> + *errormsg = "decode_queue_head is null"; > Oh! Okay, I'll retry without that. I've now done several runs with

Re: Fix primary crash continually with invalid checkpoint after promote

2022-04-26 Thread Michael Paquier
On Wed, Apr 27, 2022 at 11:24:11AM +0900, Kyotaro Horiguchi wrote: > Zhao Rui's proposal is retension of WAL files according to (the wrong > content of) control file. > > Aside from the fact that it may let slots be invalidated ealier, It's > not great that an acutally performed restartpoint is

Re: Fix primary crash continually with invalid checkpoint after promote

2022-04-26 Thread Kyotaro Horiguchi
At Tue, 26 Apr 2022 15:47:13 -0400, Tom Lane wrote in > "=?ISO-8859-1?B?WmhhbyBSdWk=?=" <875941...@qq.com> writes: > > Newly promoted primary may leave an invalid checkpoint. > > In function CreateRestartPoint, control file is updated and old wals are > > removed. But in some situations,

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
Thomas Munro writes: > BTW If you had your local change from debug.patch (upthread), that'd > defeat the patch. I mean this: > +if(!*errormsg) > + *errormsg = "decode_queue_head is null"; Oh! Okay, I'll retry without that. regards, tom lane

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
Thomas Munro writes: > Ok, that's the same for me. Next question: why does the patch I > posted not help? I improved the instrumentation a bit, and it looks like what is happening is that loc > read_upto, causing that code to "break" independently of wait_for_wal. success.log is from "make

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Thomas Munro
On Wed, Apr 27, 2022 at 1:54 PM Thomas Munro wrote: > On Wed, Apr 27, 2022 at 1:47 PM Tom Lane wrote: > > Thomas Munro writes: > > > Hrmph... Are you sure you rebuilt the contrib module? Assuming so, > > > maybe it's failing in a different way for you and me. For me, it > > > always fails

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
Thomas Munro writes: > On Wed, Apr 27, 2022 at 12:25 PM Tom Lane wrote: >> Not sure what we're doing differently, but plain "make check" in >> contrib/pg_walinspect fails pretty consistently for me on gcc23. >> I tried it again just now and got five failures in five attempts. > I tried on the

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Thomas Munro
On Wed, Apr 27, 2022 at 1:47 PM Tom Lane wrote: > Thomas Munro writes: > > Hrmph... Are you sure you rebuilt the contrib module? Assuming so, > > maybe it's failing in a different way for you and me. For me, it > > always fails after this break is reached in xlogutil.c: > > > /*

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
Thomas Munro writes: > Hrmph... Are you sure you rebuilt the contrib module? Assuming so, > maybe it's failing in a different way for you and me. For me, it > always fails after this break is reached in xlogutil.c: > /* If asked, let's not wait for future WAL. */ > if

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Kyotaro Horiguchi
At Tue, 26 Apr 2022 11:33:49 -0700, Nathan Bossart wrote in > On Wed, Mar 16, 2022 at 10:24:44AM +0900, Kyotaro Horiguchi wrote: > > While discussing on additional LSNs in checkpoint log message, > > Fujii-san pointed out [2] that there is a case where > > CreateRestartPoint leaves

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread Michael Paquier
On Tue, Apr 26, 2022 at 01:15:05PM -0500, David Christensen wrote: > True. :-) This does seem like a tool geared towards "expert mode", so > maybe we just assume if you need it you know what you're doing? This is definitely an expert mode toy. -- Michael signature.asc Description: PGP signature

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Thomas Munro
On Wed, Apr 27, 2022 at 12:25 PM Tom Lane wrote: > Thomas Munro writes: > > I think it's a bug in pg_walinspect, so I'll move the discussion back > > here. Here's one rather simple way to fix it, that has survived > > running the test a thousand times (using a recipe that failed for me > >

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Tom Lane
Thomas Munro writes: > I think it's a bug in pg_walinspect, so I'll move the discussion back > here. Here's one rather simple way to fix it, that has survived > running the test a thousand times (using a recipe that failed for me > quite soon, after 20-100 attempts or so; I never figured out how

Re: WIP: WAL prefetch (another approach)

2022-04-26 Thread Thomas Munro
On Tue, Apr 26, 2022 at 6:11 PM Thomas Munro wrote: > I will poke some more tomorrow to try to confirm this and try to come > up with a fix. Done, and moved over to the pg_walinspect commit thread to reach the right eyeballs:

Re: pgsql: Add contrib/pg_walinspect.

2022-04-26 Thread Thomas Munro
On Tue, Apr 26, 2022 at 5:36 PM Michael Paquier wrote: > On Tue, Apr 26, 2022 at 01:25:14AM -0400, Tom Lane wrote: > > I've been wondering if the issue could be traced to topminnow's unusual > > hardware properties, specifically that it has MAXALIGN 8 even though > > it's only a 32-bit machine

Re: Building Postgres with lz4 on Visual Studio

2022-04-26 Thread Robert Haas
On Wed, Apr 13, 2022 at 10:22 AM Melih Mutlu wrote: > What I realized is that c:\lz4\lib\liblz4.lib actually does not exist. > The latest versions of lz4, downloaded from [2], do not contain \liblz4.lib > anymore, as opposed to what's written here [3]. Also there isn't a lib folder > too. > >

Re: avoid multiple hard links to same WAL file after a crash

2022-04-26 Thread Nathan Bossart
Here is an attempt at creating something that can be back-patched. 0001 simply replaces calls to durable_rename_excl() with durable_rename() and is intended to be back-patched. 0002 removes the definition of durable_rename_excl() and is _not_ intended for back-patching. I imagine 0002 will need

Re: Fix primary crash continually with invalid checkpoint after promote

2022-04-26 Thread Tom Lane
"=?ISO-8859-1?B?WmhhbyBSdWk=?=" <875941...@qq.com> writes: > Newly promoted primary may leave an invalid checkpoint. > In function CreateRestartPoint, control file is updated and old wals are > removed. But in some situations, control file is not updated, old wals are > still removed. Thus

Re: Possible corruption by CreateRestartPoint at promotion

2022-04-26 Thread Nathan Bossart
On Wed, Mar 16, 2022 at 10:24:44AM +0900, Kyotaro Horiguchi wrote: > While discussing on additional LSNs in checkpoint log message, > Fujii-san pointed out [2] that there is a case where > CreateRestartPoint leaves unrecoverable database when concurrent > promotion happens. That corruption is

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:42 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:24:52AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy > > wrote: > >> Thanks for working on this. I'm just thinking if we can use these FPIs > >> to repair the corrupted

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:54 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:11:10AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > >> I don't think that there is any need to rely on a new logic if there > >> is already some code in place able

Re: DBT-5 Stored Procedure Development (2022)

2022-04-26 Thread Peter Geoghegan
On Tue, Apr 26, 2022 at 10:36 AM Mark Wong wrote: > I'm afraid not. I'm guessing that pulling in egen 1.14 would address > that. Maybe it would make sense to put that on the top of todo list if > this project is accepted... Wouldn't it be a prerequisite here? I don't actually have any reason

Re: DBT-5 Stored Procedure Development (2022)

2022-04-26 Thread Mark Wong
On Tue, Apr 19, 2022 at 05:20:50PM -0700, Peter Geoghegan wrote: > On Tue, Apr 19, 2022 at 11:31 AM Mark Wong wrote: > > As some of tasks proposed are actually in place, one other task could be > > updating egen (the TPC supplied code.) The kit was last developed again > > 1.12 and 1.14 is

Re: orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Pavel Stehule
út 26. 4. 2022 v 18:36 odesílatel Bruce Momjian napsal: > On Tue, Apr 26, 2022 at 06:17:41PM +0200, Pavel Stehule wrote: > > > > út 26. 4. 2022 v 14:19 odesílatel Ashutosh Sharma > > > napsal: > > > > Hi, > > > > The below files in orafce contrib module are generated at build time. > >

Re: orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Bruce Momjian
On Tue, Apr 26, 2022 at 06:17:41PM +0200, Pavel Stehule wrote: > > út 26. 4. 2022 v 14:19 odesílatel Ashutosh Sharma > napsal: > > Hi, > > The below files in orafce contrib module are generated at build time. > However, these are checked into the repository. Shouldn't these files >

Re: orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Pavel Stehule
út 26. 4. 2022 v 14:19 odesílatel Ashutosh Sharma napsal: > Hi, > > The below files in orafce contrib module are generated at build time. > However, these are checked into the repository. Shouldn't these files > be removed from the repository and added to the .gitignore file so > that they get

Re: Cryptohash OpenSSL error queue in FIPS enabled builds

2022-04-26 Thread Daniel Gustafsson
> On 26 Apr 2022, at 03:55, Michael Paquier wrote: > > On Tue, Apr 26, 2022 at 12:07:32AM +0200, Daniel Gustafsson wrote: >> In this particular codepath I think we can afford clearing it on the way out, >> with a comment explaining why. It's easily reproducible and adding a call >> and >> a

Re: orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Ashutosh Sharma
Sure, I'll do that. Thanks, Ashutosh On Tue, Apr 26, 2022 at 5:51 PM Daniel Gustafsson wrote: > > > On 26 Apr 2022, at 14:19, Ashutosh Sharma wrote: > > > The below files in orafce contrib module are generated at build time. > > However, these are checked into the repository. Shouldn't these

Re: double inclusion of '-p' flex flag

2022-04-26 Thread Ashutosh Sharma
On Tue, Apr 26, 2022 at 5:55 PM John Naylor wrote: > > On Tue, Apr 26, 2022 at 7:16 PM Ashutosh Sharma wrote: > > > > Hi, > > > > I see that we have included '-p' flex flag twice in the commands used > > to generate the scanner files. See below: > > > > src/backend/parser/Makefile:60:

Re: [Patch] ALTER SYSTEM READ ONLY

2022-04-26 Thread Amul Sul
On Sat, Apr 23, 2022 at 1:34 PM Bharath Rupireddy wrote: > > On Mon, Mar 15, 2021 at 12:56 PM Amul Sul wrote: > > > > > > It is a very minor change, so I rebased the patch. Please take a look, if > > > that works for you. > > > > > > > Thanks, I am getting one more failure for the vacuumlazy.c.

Re: double inclusion of '-p' flex flag

2022-04-26 Thread John Naylor
On Tue, Apr 26, 2022 at 7:16 PM Ashutosh Sharma wrote: > > Hi, > > I see that we have included '-p' flex flag twice in the commands used > to generate the scanner files. See below: > > src/backend/parser/Makefile:60:scan.c: FLEXFLAGS = -CF -p -p > src/backend/utils/adt/Makefile:122:

Re: orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Daniel Gustafsson
> On 26 Apr 2022, at 14:19, Ashutosh Sharma wrote: > The below files in orafce contrib module are generated at build time. > However, these are checked into the repository. Shouldn't these files > be removed from the repository and added to the .gitignore file so > that they get ignored in the

orafce: some of the build time generated files are not present in .gitignore and also checked into the repository

2022-04-26 Thread Ashutosh Sharma
Hi, The below files in orafce contrib module are generated at build time. However, these are checked into the repository. Shouldn't these files be removed from the repository and added to the .gitignore file so that they get ignored in the future commits. sqlparse.c sqlscan.c sqlparse.h -- With

double inclusion of '-p' flex flag

2022-04-26 Thread Ashutosh Sharma
Hi, I see that we have included '-p' flex flag twice in the commands used to generate the scanner files. See below: src/backend/parser/Makefile:60:scan.c: FLEXFLAGS = -CF -p -p src/backend/utils/adt/Makefile:122:jsonpath_scan.c: FLEXFLAGS = -CF -p -p src/bin/psql/Makefile:61:

Re: variable filename for psql \copy

2022-04-26 Thread Jiří Fejfar
Dear Daniel, David On Mon, 25 Apr 2022 at 18:07, David G. Johnston wrote: > On Mon, Apr 25, 2022 at 1:24 AM Jiří Fejfar wrote: > >> contrib_regression=# copy (select 1) to :'afile'; >> > > Hopefully you realize that COPY is going to place that file on the server, > not send it to the psql

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-04-26 Thread Andrey Borodin
> 25 апр. 2022 г., в 21:48, Nathan Bossart > написал(а): > > I'm personally in > favor of just adding a GUC that can be enabled to block canceling > synchronous replication waits +1. I think it's the only option to provide quorum commit guarantees. Best regards, Andrey Borodin.

Re: Perform streaming logical transactions by background workers and parallel apply

2022-04-26 Thread vignesh C
On Fri, Apr 8, 2022 at 2:44 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, April 6, 2022 1:20 PM Amit Kapila > wrote: > > > In this email, I would like to discuss allowing streaming logical > > transactions (large in-progress transactions) by background workers > > and parallel apply in

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-04-26 Thread Laurenz Albe
On Mon, 2022-04-25 at 19:51 +0530, Bharath Rupireddy wrote: > With synchronous replication typically all the transactions (txns) > first locally get committed, then streamed to the sync standbys and > the backend that generated the transaction will wait for ack from sync > standbys. While waiting

Re: WIP: WAL prefetch (another approach)

2022-04-26 Thread Thomas Munro
On Tue, Apr 26, 2022 at 6:11 AM Tom Lane wrote: > I believe that the WAL prefetch patch probably accounts for the > intermittent errors that buildfarm member topminnow has shown > since it went in, eg [1]: > > diff -U3 > /home/nm/ext4/HEAD/pgsql/contrib/pg_walinspect/expected/pg_walinspect.out

RE: Skipping schema changes in publication

2022-04-26 Thread osumi.takami...@fujitsu.com
On Thursday, April 21, 2022 12:15 PM vignesh C wrote: > Updated patch by changing the syntax to use EXCEPT instead of SKIP. Hi This is my review comments on the v2 patch. (1) gram.y I think we can make a unified function that merges preprocess_alltables_pubobj_list with