Re: parallel vacuum comments

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 10:51 AM Masahiko Sawada wrote: > > I've attached updated patches. The first patch just moves common > function for index bulk-deletion and cleanup to vacuum.c. And the > second patch moves parallel vacuum code to vacuumparallel.c. The > comments I got so far are incorporat

Re: sequences vs. synchronous replication

2021-12-17 Thread Tomas Vondra
On 12/18/21 05:52, Tom Lane wrote: Tomas Vondra writes: The problem is exactly the same as in [1] - the aborted transaction generated WAL, but RecordTransactionAbort() ignores that and does not update LogwrtResult.Write, with the reasoning that aborted transactions do not matter. But sequenc

Re: sequences vs. synchronous replication

2021-12-17 Thread Tom Lane
Tomas Vondra writes: > The problem is exactly the same as in [1] - the aborted transaction > generated WAL, but RecordTransactionAbort() ignores that and does not > update LogwrtResult.Write, with the reasoning that aborted transactions > do not matter. But sequences violate that, because we on

Re: Column Filtering in Logical Replication

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 3:16 PM houzj.f...@fujitsu.com wrote: > > On Friday, December 17, 2021 1:55 AM Alvaro Herrera > wrote: > > On 2021-Dec-16, houzj.f...@fujitsu.com wrote: > > > > > The patch ensures all columns of RT are in column list when > > > CREATE/ALTER publication, but it seems does

Re: Column Filtering in Logical Replication

2021-12-17 Thread Amit Kapila
On Sat, Dec 18, 2021 at 7:04 AM Alvaro Herrera wrote: > > On 2021-Dec-17, Tomas Vondra wrote: > > > On 12/17/21 22:07, Alvaro Herrera wrote: > > > So I've been thinking about this as a "security" item (you can see my > > > comments to that effect sprinkled all over this thread), in the sense > > >

Re: row filtering for logical replication

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 5:29 PM Greg Nancarrow wrote: > > On Fri, Dec 17, 2021 at 7:20 PM Ajin Cherian wrote: > > > > On Fri, Dec 17, 2021 at 5:46 PM Greg Nancarrow wrote: > > > > > So using the v47 patch-set, I still find that the UPDATE above results in > > > publication of an INSERT of (2,1)

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-17 Thread Peter Geoghegan
On Thu, Dec 16, 2021 at 10:46 PM Masahiko Sawada wrote: > > My emphasis here has been on making non-aggressive VACUUMs *always* > > advance relfrozenxid, outside of certain obvious edge cases. And so > > with all the patches applied, up to and including the opportunistic > > freezing patch, every

Re: Column Filtering in Logical Replication

2021-12-17 Thread Tomas Vondra
On 12/18/21 02:34, Alvaro Herrera wrote: On 2021-Dec-17, Tomas Vondra wrote: On 12/17/21 22:07, Alvaro Herrera wrote: So I've been thinking about this as a "security" item (you can see my comments to that effect sprinkled all over this thread), in the sense that if a publication "hides" som

Re: logical decoding and replication of sequences

2021-12-17 Thread Tomas Vondra
On 12/15/21 14:51, Tomas Vondra wrote: On 12/15/21 14:20, Amit Kapila wrote: On Tue, Dec 14, 2021 at 7:02 AM Tomas Vondra wrote: Hi, here's an updated version of the patches, dealing with almost all of the issues (at least in the 0001 and 0002 parts). The main changes: 1) I've removed th

sequences vs. synchronous replication

2021-12-17 Thread Tomas Vondra
Hi, while working on logical decoding of sequences, I ran into an issue with nextval() in a transaction that rolls back, described in [1]. But after thinking about it a bit more (and chatting with Petr Jelinek), I think this issue affects physical sync replication too. Imagine you have a pri

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
On 2021-Dec-17, Tomas Vondra wrote: > On 12/17/21 22:07, Alvaro Herrera wrote: > > So I've been thinking about this as a "security" item (you can see my > > comments to that effect sprinkled all over this thread), in the sense > > that if a publication "hides" some column, then the replica just wo

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tom Lane
Greg Stark writes: > But the bigger question is. Are we really concerned about this flaky > problem? Is it worth investing time and money on? I can get money to > go buy a G4 or G5 and spend some time on it. It just seems a bit... > niche. But if it's a real bug that represents something broken on

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
On Fri, 17 Dec 2021 at 18:40, Tom Lane wrote: > > Greg Stark writes: > > Hm. I seem to have picked a bad checkout. I took the last one before > > the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). > > FWIW, I think that's the first one *after* the revert. Doh But the bigger question is. Are

Re: Add id's to various elements in protocol.sgml

2021-12-17 Thread Brar Piening
On Dec 17, 2021 at 08:13, Peter Eisentraut wrote: On 15.12.21 16:59, Brar Piening wrote: On Dec 15, 2021 at 15:49, Alvaro Herrera wrote: On 2021-Dec-15, Brar Piening wrote: Since I can't argue towards some general utility for the xreflabels and don't have any other solid argument in favor of a

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tom Lane
Greg Stark writes: > Hm. I seem to have picked a bad checkout. I took the last one before > the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). FWIW, I think that's the first one *after* the revert. > 2021-12-17 17:51:51.688 EST [50955] LOG: background worker "parallel > worker" (PID 54073)

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tomas Vondra
On 12/17/21 23:56, Greg Stark wrote: Hm. I seem to have picked a bad checkout. I took the last one before the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). Or there's some incompatibility with the emulation and the IPC stuff parallel workers use. 2021-12-17 17:51:51.688 EST [50955] LOG: b

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
Hm. I seem to have picked a bad checkout. I took the last one before the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). Or there's some incompatibility with the emulation and the IPC stuff parallel workers use. 2021-12-17 17:51:51.688 EST [50955] LOG: background worker "parallel worker" (PID

Re: Column Filtering in Logical Replication

2021-12-17 Thread Tomas Vondra
On 12/17/21 22:07, Alvaro Herrera wrote: So I've been thinking about this as a "security" item (you can see my comments to that effect sprinkled all over this thread), in the sense that if a publication "hides" some column, then the replica just won't get access to it. But in reality that's m

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
On 2021-12-17 11:31:59 -0800, Andres Freund wrote: > > Don't we need the ulimit call for FreeBSD? > > I think the default core limits were different, I will check. Yep, freebsd has -c unlimited by default: https://cirrus-ci.com/task/6199382239346688?logs=sysinfo#L23 vs https://cirrus-ci.com/task/

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
So I've been thinking about this as a "security" item (you can see my comments to that effect sprinkled all over this thread), in the sense that if a publication "hides" some column, then the replica just won't get access to it. But in reality that's mistaken: the filtering that this patch impleme

Re: speed up text_position() for utf-8

2021-12-17 Thread John Naylor
Attached is a short patch series to develop some ideas of inlining pg_utf_mblen(). 0001 puts the main implementation of pg_utf_mblen() into an inline function and uses this in pg_mblen(). This is somewhat faster in the strpos tests, so that gives some measure of the speedup expected for other call

Re: Adding CI to our tree

2021-12-17 Thread Daniel Gustafsson
> On 17 Dec 2021, at 20:31, Andres Freund wrote: > On 2021-12-17 12:34:36 +0100, Peter Eisentraut wrote: >> I don't like that the -s option is used. I would like to see what commands >> are executed. > > I can change it - but it makes it *much* harder to spot compiler warnings. Having used Cir

Re: make tuplestore helper function

2021-12-17 Thread Justin Pryzby
On Mon, Dec 13, 2021 at 05:27:52PM -0500, Melanie Plageman wrote: > Done in attached v4 Thanks. I think these comments can be removed from the callers: /* it's a simple type, so don't use get_call_result_type */ You removed one call to tuplestore_donestoring(), but not the others. I guess you co

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tom Lane
Greg Stark writes: > I'm guessing I should do CC=/usr/bin/powerpc-apple-darwin9-gcc-4.2.1 > or maybe 4.0.1. What version is on your G4? $ gcc -v Using built-in specs. Target: powerpc-apple-darwin9 Configured with: /var/tmp/gcc/gcc-5493~1/src/configure --disable-checking -enable-werror --prefix=/

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
I have IBUILD:postgresql gsstark$ ls /usr/bin/*gcc* /usr/bin/gcc /usr/bin/gcc-4.0 /usr/bin/gcc-4.2 /usr/bin/i686-apple-darwin9-gcc-4.0.1 /usr/bin/i686-apple-darwin9-gcc-4.2.1 /usr/bin/powerpc-apple-darwin9-gcc-4.0.1 /usr/bin/powerpc-apple-darwin9-gcc-4.2.1 I'm guessing I should do CC=/usr/bin/pow

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
Hi, On 2021-12-17 09:36:05 -0500, Tom Lane wrote: > Andrew Dunstan writes: > > Maye I have missed it, but why are we using ccache here? That seems a > > bit pointless in an ephemeral instance. > > I believe Munro's cfbot tooling is able to save and re-use ccache > across successive instantiation

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
Hi, On 2021-12-17 12:34:36 +0100, Peter Eisentraut wrote: > On 13.12.21 22:12, Andres Freund wrote: > > Attached is an updated version of the CI patches. An example of a test run > > with the attached version of this > > https://cirrus-ci.com/build/6501998521483264 > > + only_if: $CIRRUS_CHANGE_

Re: In-placre persistance change of a relation

2021-12-17 Thread Justin Pryzby
On Fri, Dec 17, 2021 at 02:33:25PM +, Jakub Wartak wrote: > > Justin wrote: > > On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > > > As the thread didn't get a lot of traction, I've registered it into > > > current > > commitfest > > https://eur02.safelinks.protection.outlook.c

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Tom Lane
Greg Stark writes: > What tools and tool versions are you using to build? Is it just GCC for PPC? > There aren't any special build processes to make a fat binary involved? Nope, just "configure; make" using that macOS version's regular gcc. regards, tom lane

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
What tools and tool versions are you using to build? Is it just GCC for PPC? There aren't any special build processes to make a fat binary involved? On Thu, 16 Dec 2021 at 23:11, Tom Lane wrote: > > Greg Stark writes: > > But if you're interested and can explain the tests to run I can try to >

[PATCH] Add reloption for views to enable RLS

2021-12-17 Thread Christoph Heiss
Hi all! As part of a customer project we are looking to implement an reloption for views which when set, runs the subquery as invoked by the user rather than the view owner, as is currently the case. The rewrite rule's table references are then checked as if the user were referencing the table

Re: pg_upgrade should truncate/remove its logs before running

2021-12-17 Thread Justin Pryzby
On Thu, Dec 16, 2021 at 12:23:08PM +0100, Daniel Gustafsson wrote: > > On 16 Dec 2021, at 12:11, Peter Eisentraut > > wrote: > > > Could we make it write just one log file? Is having multiple log files > > better? > > Having individual .txt files from checks with additional > information > o

Re: Column Filtering in Logical Replication

2021-12-17 Thread Alvaro Herrera
On 2021-Dec-17, Rahila Syed wrote: > > This means that we need to support changing the column list of a > > table in a publication. I'm looking at implementing some form of > > ALTER PUBLICATION for that. > > I think right now the patch contains support only for ALTER > PUBLICATION.. ADD TABLE w

\d with triggers: more than one row returned by a subquery used as an expression

2021-12-17 Thread Justin Pryzby
I want to mention that the 2nd problem I mentioned here is still broken. https://www.postgresql.org/message-id/20210717010259.gu20...@telsasoft.com It happens if non-inheritted triggers on child and parent have the same name. On Fri, Jul 16, 2021 at 08:02:59PM -0500, Justin Pryzby wrote: > On Fri

Re: Addition of --no-sync to pg_upgrade for test speedup

2021-12-17 Thread Bruce Momjian
On Fri, Dec 17, 2021 at 10:21:04AM +0100, Peter Eisentraut wrote: > On 16.12.21 07:50, Michael Paquier wrote: > > As per $subject, avoiding the flush of the new cluster's data > > directory shortens a bint the runtime of the test. In some of my slow > > VMs, aka Windows, this shaves a couple of se

Re: Adding CI to our tree

2021-12-17 Thread Tom Lane
Andrew Dunstan writes: > Maye I have missed it, but why are we using ccache here? That seems a > bit pointless in an ephemeral instance. I believe Munro's cfbot tooling is able to save and re-use ccache across successive instantiations of a build instance. I've not looked at this code, but if it

RE: In-placre persistance change of a relation

2021-12-17 Thread Jakub Wartak
> Justin wrote: > On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > > As the thread didn't get a lot of traction, I've registered it into current > commitfest > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommitf > est.postgresql.org%2F36%2F3461%2F&data=04%7C01%

Re: Adding CI to our tree

2021-12-17 Thread Andrew Dunstan
On 12/13/21 16:12, Andres Freund wrote: > Hi, > > Attached is an updated version of the CI patches. An example of a test run > with the attached version of this > https://cirrus-ci.com/build/6501998521483264 > Maye I have missed it, but why are we using ccache here? That seems a bit pointless in

Re: Use generation context to speed up tuplesorts

2021-12-17 Thread Ronan Dunklau
Le vendredi 17 décembre 2021, 14:39:10 CET Tomas Vondra a écrit : > I wasn't really suggesting to investigate those other allocators in this > patch - it seems like a task requiring a pretty significant amount of > work/time. My point was that we should make it reasonably easy to add > tweaks for t

Re: Use generation context to speed up tuplesorts

2021-12-17 Thread Tomas Vondra
On 12/17/21 09:08, Ronan Dunklau wrote: > Le jeudi 16 décembre 2021, 18:00:56 CET Tomas Vondra a écrit : >> On 12/16/21 17:03, Ronan Dunklau wrote: >>> Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit : I will follow up with a benchmark of the test sorting a table with a widt

Re: speed up verifying UTF-8

2021-12-17 Thread John Naylor
I plan to push v25 early next week, unless there are further comments. -- John Naylor EDB: http://www.enterprisedb.com

Re: In-placre persistance change of a relation

2021-12-17 Thread Justin Pryzby
On Fri, Dec 17, 2021 at 09:10:30AM +, Jakub Wartak wrote: > I'm especially worried if I didn't screw up something/forgot something > related to the last one (rd->rd_smgr changes), but I'm getting "All 210 tests > passed". > As the thread didn't get a lot of traction, I've registered it into

Re: Column Filtering in Logical Replication

2021-12-17 Thread Rahila Syed
Hi, Thank you for updating the patch. The regression tests and tap tests pass with v9 patch. > > After working on this a little bit more, I realized that this is a bad > idea overall. It causes lots of complications and it's just not worth > it. So I'm back at my original thought that we need

Re: row filtering for logical replication

2021-12-17 Thread Greg Nancarrow
On Fri, Dec 17, 2021 at 7:20 PM Ajin Cherian wrote: > > On Fri, Dec 17, 2021 at 5:46 PM Greg Nancarrow wrote: > > > So using the v47 patch-set, I still find that the UPDATE above results in > > publication of an INSERT of (2,1), rather than an UPDATE of (1,1) to (2,1). > > This is according to t

Re: Adding CI to our tree

2021-12-17 Thread Peter Eisentraut
On 13.12.21 22:12, Andres Freund wrote: Attached is an updated version of the CI patches. An example of a test run with the attached version of this https://cirrus-ci.com/build/6501998521483264 + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-o

Re: Skipping logical replication transactions on subscriber side

2021-12-17 Thread Masahiko Sawada
On Fri, Dec 17, 2021 at 7:12 PM Amit Kapila wrote: > > On Fri, Dec 17, 2021 at 3:23 PM Peter Eisentraut > wrote: > > > > On 13.12.21 04:12, Greg Nancarrow wrote: > > > (ii) "Setting -1 means to reset the transaction ID" > > > > > > Shouldn't it be explained what resetting actually does and when i

Re: row filtering for logical replication

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 1:50 PM Ajin Cherian wrote: > > On Fri, Dec 17, 2021 at 5:46 PM Greg Nancarrow wrote: > > > So using the v47 patch-set, I still find that the UPDATE above results in > > publication of an INSERT of (2,1), rather than an UPDATE of (1,1) to (2,1). > > This is according to t

Re: Use generation context to speed up tuplesorts

2021-12-17 Thread Ronan Dunklau
Le vendredi 17 décembre 2021, 09:08:06 CET Ronan Dunklau a écrit : > It is my understanding that malloc will try to compact memory by moving it > around. So the memory should be actually be released to the kernel at some > point. In the meantime, malloc can reuse it for our next invocation (which >

Re: Skipping logical replication transactions on subscriber side

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 3:23 PM Peter Eisentraut wrote: > > On 13.12.21 04:12, Greg Nancarrow wrote: > > (ii) "Setting -1 means to reset the transaction ID" > > > > Shouldn't it be explained what resetting actually does and when it can > > be, or is needed to be, done? Isn't it automatically reset

Re: row filtering for logical replication

2021-12-17 Thread Amit Kapila
On Fri, Dec 17, 2021 at 4:11 AM Peter Smith wrote: > > PSA the v47* patch set. > Few comments on v47-0002: === 1. The handling to find rowfilter for ancestors in RelationGetInvalidRowFilterCol seems complex. It seems you are accumulating non-partition relations as well in topr

Re: psql format output

2021-12-17 Thread Peter Eisentraut
On 15.12.21 20:58, Florian Koch wrote: I realized that the output of "\df+ func_name" has a formatting problem when a lot of arguments are used. The field 'Arguments data types' gets very long and destroys the whole formatting in the console. The field 'Source code' is most of the time multi-li

Re: Skipping logical replication transactions on subscriber side

2021-12-17 Thread Peter Eisentraut
On 13.12.21 04:12, Greg Nancarrow wrote: (ii) "Setting -1 means to reset the transaction ID" Shouldn't it be explained what resetting actually does and when it can be, or is needed to be, done? Isn't it automatically reset? I notice that negative values (other than -1) seem to be regarded as val

RE: Column Filtering in Logical Replication

2021-12-17 Thread houzj.f...@fujitsu.com
On Friday, December 17, 2021 1:55 AM Alvaro Herrera wrote: > On 2021-Dec-16, houzj.f...@fujitsu.com wrote: > > > The patch ensures all columns of RT are in column list when > > CREATE/ALTER publication, but it seems doesn't prevent user from > > changing the replica identity or dropping the inde

Re: Addition of --no-sync to pg_upgrade for test speedup

2021-12-17 Thread Peter Eisentraut
On 16.12.21 07:50, Michael Paquier wrote: As per $subject, avoiding the flush of the new cluster's data directory shortens a bint the runtime of the test. In some of my slow VMs, aka Windows, this shaves a couple of seconds even if the bulk of the time is still spent on the main regression test

Re: Transparent column encryption

2021-12-17 Thread Peter Eisentraut
On 17.12.21 01:41, Jacob Champion wrote: (And I think the client should be able to enforce encryption in the first place, before I distract you too much with other stuff.) Yes, this is a useful point that I have added to my notes.

RE: In-placre persistance change of a relation

2021-12-17 Thread Jakub Wartak
> Kyotaro wrote: > > In this version, I got rid of the "CLEANUP FORK"s, and added a new > > system "Smgr marks". The mark files have the name of the > > corresponding fork file followed by ".u" (which means Uncommitted.). > > "Uncommited"-marked main fork means the same as the > CLEANUP2_FORKNUM >

Re: [PoC] Delegating pg_ident to a third party

2021-12-17 Thread Peter Eisentraut
On 17.12.21 00:48, Jacob Champion wrote: WDYT? (My responses here will be slower than usual. Hope you all have a great end to the year!) Looks interesting. I wonder whether putting this into pg_ident.conf is sensible. I suspect people will want to eventually add more features around this,

Re: row filtering for logical replication

2021-12-17 Thread Ajin Cherian
On Fri, Dec 17, 2021 at 5:46 PM Greg Nancarrow wrote: > So using the v47 patch-set, I still find that the UPDATE above results in > publication of an INSERT of (2,1), rather than an UPDATE of (1,1) to (2,1). > This is according to the 2nd UPDATE rule below, from patch 0003. > > + * old-row (no m

Re: Use generation context to speed up tuplesorts

2021-12-17 Thread Ronan Dunklau
Le jeudi 16 décembre 2021, 18:00:56 CET Tomas Vondra a écrit : > On 12/16/21 17:03, Ronan Dunklau wrote: > > Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit : > >> I will follow up with a benchmark of the test sorting a table with a > >> width > >> varying from 1 to 32 columns. > > >

Re: pg_dump versus ancient server versions

2021-12-17 Thread Greg Stark
On Fri, 22 Oct 2021 at 19:27, Robert Haas wrote: > > Another thing to think about in that regard: how likely is that > PostgreSQL 7.4 and PostgreSQL 15 both compile and run on the same > operating system? I suspect the answer is "not very." I seem to recall > Greg Stark trying to compile really ol