Re: Column Filtering in Logical Replication

2022-03-23 Thread Tomas Vondra
On 3/21/22 12:55, Amit Kapila wrote: > On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra > wrote: >> >> ... >> >> However, while looking at how pgoutput, I realized one thing - for row >> filters we track them "per operation", depending on which operations are >> defined for a given publication. Should

Re: Column Filtering in Logical Replication

2022-03-23 Thread Tomas Vondra
On 3/21/22 12:28, Amit Kapila wrote: > On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra > wrote: >> >> Ah, thanks for reminding me - it's hard to keep track of all the issues >> in threads as long as this one. >> >> BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith >> proposed to g

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Jacob Champion
On Thu, 2022-03-17 at 18:33 -0400, Tom Lane wrote: > Yeah. It seems to me that putting the auth info into struct Port was > a fairly random thing to do in the first place, and we are now dealing > with the fallout of that. > > I think what we ought to do here is separate out the data that we thin

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Thomas Munro
On Thu, Mar 24, 2022 at 9:53 AM Peter Eisentraut wrote: > On 21.03.22 05:55, Thomas Munro wrote: > > [04:30:50.630] pg_waldump.c:963:26: error: format ‘%u’ expects > > argument of type ‘unsigned int *’, but argument 3 has type ‘ForkNumber > > *’ [-Werror=format=] > > [04:30:50.630] 963 | if (sscan

Re: ubsan

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 13:12:34 -0700, Andres Freund wrote: > I'm planning to enable it on two of mine. Looks like gcc and clang find > slightly different things, so I was intending to enable it on one of each. Originally I'd planned to mix them into existing members, but I think it'd be better to hav

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 5:52 PM Justin Pryzby wrote: > Also because the library may not be compiled with threading. A few days ago, > I > tried to rebase the original "parallel workers" patch over the COMPRESS DETAIL > patch but then couldn't test it, even after trying various versions of the >

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Tom Lane
Jacob Champion writes: > On Thu, 2022-03-17 at 18:33 -0400, Tom Lane wrote: >> I think what we ought to do here is separate out the data that we think >> parallel workers need access to. It does not seem wise to say "workers >> can access fields A,B,C of MyPort but not fields X,Y,Z". I do not ha

Re: ubsan

2022-03-23 Thread Tom Lane
Andres Freund writes: > Running with asan found an existing use-after-free bug in pg_waldump (*), a > bug in > dshash_seq_next() next that probably can't be hit in HEAD and a bug in my > shared memory stats patch. I count that as a success. Nice! regards, tom lane

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Jacob Champion
On Wed, 2022-03-23 at 19:00 -0400, Tom Lane wrote: > Hm. I was more envisioning getting the "sharable" info out of Port > entirely, although I'm not quite sure where it should go instead. If it helps, I can move the substruct out and up to a new global struct (MyProcShared?). At this point I thin

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Justin Pryzby
On Wed, Mar 23, 2022 at 04:34:04PM -0400, Robert Haas wrote: > be, spawning threads inside the PostgreSQL backend. Short of cats and > dogs living together, it's hard to think of anything more terrifying, > because the PostgreSQL backend is very much not thread-safe. However, > a lot of the things

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 18:31:12 -0400, Robert Haas wrote: > On Wed, Mar 23, 2022 at 5:14 PM Andres Freund wrote: > > The most likely source of problem would errors thrown while zstd threads are > > alive. Should make sure that that can't happen. > > > > What is the lifetime of the threads zstd spawns?

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Andres Freund
On 2022-03-23 18:07:01 -0500, Justin Pryzby wrote: > Did you try this on windows at all ? Really should get zstd installed in the windows cf environment... > It's probably no surprise that zstd implements threading differently there. Worth noting that we have a few of our own threads running on

Re: [PATCH] Accept IP addresses in server certificate SANs

2022-03-23 Thread Jacob Champion
On Wed, 2022-03-23 at 14:20 +0900, Kyotaro Horiguchi wrote: > I tried to write out the doc part. What do you think about it? I like it, thanks! I've applied that in v10, with a tweak to two iPAddress spellings and a short expansion of the condition in the Note, and I've added you as a co-author t

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 23:06:14 +, Jacob Champion wrote: > On Wed, 2022-03-23 at 19:00 -0400, Tom Lane wrote: > > Hm. I was more envisioning getting the "sharable" info out of Port > > entirely, although I'm not quite sure where it should go instead. > > If it helps, I can move the substruct out

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-03-23 Thread Thomas Munro
On Wed, Feb 16, 2022 at 12:11 PM Nathan Bossart wrote: > On Tue, Feb 15, 2022 at 10:37:58AM -0800, Nathan Bossart wrote: > > On Tue, Feb 15, 2022 at 10:10:34AM -0800, Andres Freund wrote: > >> I generally think it'd be a good exercise to go through an use > >> get_dirent_type() in nearly all ReadD

Re: Fix overflow in DecodeInterval

2022-03-23 Thread Joseph Koshakow
On Mon, Mar 21, 2022 at 8:31 PM Tom Lane wrote: > This isn't applying per the cfbot; looks like it got sideswiped > by 9e9858389. Here's a quick rebase. I've not reviewed it, but > I did notice (because git was in my face about this) that it's > got whitespace issues. Please try to avoid unnece

Re: Add pg_freespacemap extension sql test

2022-03-23 Thread Michael Paquier
On Wed, Mar 23, 2022 at 10:45:19AM -0300, Fabrízio de Royes Mello wrote: > On Wed, Mar 23, 2022 at 3:05 AM Michael Paquier wrote: >> Another thing that itched me is that we >> could also test more with indexes, particularly with btree, BRIN and >> hash (the latter should not have a FSM with 10 pa

pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, Starting with the below commit, pg_stat_reset_single_function_counters, pg_stat_reset_single_table_counters don't just reset the stats for the individual function, but also set pg_stat_database.stats_reset. commit 4c468b37a281941afd3bf61c782b20def8c17047 Author: Magnus Hagander Date: 2011-

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 17:55:16 -0700, Andres Freund wrote: > Maybe I just don't understand what these reset functions are intended for? > Their introduction [3] didn't explain much either. To me the behaviour of > resetting pg_stat_database.stats_reset but nothing else in pg_stat_database > makes them

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Andres Freund
On 2022-03-24 11:54:15 +1300, Thomas Munro wrote: > Erm, is that really OK? C says "Each enumerated type shall be > compatible with char, a signed integer type, or an > unsigned integer type. The choice of type is implementation-defined, > but shall be capable of representing the values of all the

Re: [PATCH] Add native windows on arm64 support

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 16:30:58 +1300, Thomas Munro wrote: > On Tue, Mar 22, 2022 at 11:30 PM Julien Rouhaud wrote: > > On Tue, Mar 22, 2022 at 09:37:46AM +, Niyas Sait wrote: > > > Yes, we could look into providing a build machine. Do you have any > > > reference to what the CI system looks like

Re: Fix overflow in DecodeInterval

2022-03-23 Thread Tom Lane
Joseph Koshakow writes: > Sorry about that, I didn't have my IDE set up quite right and > noticed a little too late that I had some auto-formatting turned > on. Thanks for doing the rebase, did it end up fixing > the whitespace issues? If not I'll go through the patch and try > and fix them all.

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread David G. Johnston
On Wed, Mar 23, 2022 at 5:55 PM Andres Freund wrote: > > Starting with the below commit, pg_stat_reset_single_function_counters, > pg_stat_reset_single_table_counters don't just reset the stats for the > individual function, but also set pg_stat_database.stats_reset. > > commit 4c468b37a281941afd

Re: [PoC] Reducing planning time when tables have many partitions

2022-03-23 Thread David Rowley
On Fri, 18 Mar 2022 at 23:32, Yuya Watari wrote: > I found a problem that planning takes too much time when the tables > have many child partitions. According to my observation, the planning > time increases in the order of O(n^2). Here, n is the number of child > partitions. I attached the patch

Re: freeing bms explicitly

2022-03-23 Thread Amit Kapila
On Wed, Mar 23, 2022 at 9:30 AM Zhihong Yu wrote: > > On Tue, Mar 22, 2022 at 8:45 PM Amit Kapila wrote: >> >> On Tue, Mar 22, 2022 at 3:39 AM Zhihong Yu wrote: >> >> Your patch looks good to me. I have found one more similar instance in >> the same file and changed that as well accordingly. Let

Re: Support isEmptyStringInfo

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 10:13:43 -0400, Tom Lane wrote in > Robert Haas writes: > > I think that the code is perfectly readable as it is and that this > > change makes it less so. > > Yeah, after a quick look through this patch I'm unimpressed too. > The new code is strictly longer, and it requires

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-03-23 Thread Yugo NAGATA
On Wed, 23 Mar 2022 14:26:54 -0400 Tom Lane wrote: > Tatsuo Ishii writes: > > The patch Pushed. Thank you! > > My hoary animal prairiedog doesn't like this [1]: > > # Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) > got an error in command 3 \\(SQL\\) of script 0;

Re: Column Filtering in Logical Replication

2022-03-23 Thread Amit Kapila
On Thu, Mar 24, 2022 at 4:11 AM Tomas Vondra wrote: > > On 3/21/22 12:28, Amit Kapila wrote: > > On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra > > wrote: > >> > >> Ah, thanks for reminding me - it's hard to keep track of all the issues > >> in threads as long as this one. > >> > >> BTW do you have

RE: logical replication empty transactions

2022-03-23 Thread houzj.f...@fujitsu.com
On Tuesday, March 22, 2022 7:50 PM Amit Kapila wrote: > On Tue, Mar 22, 2022 at 7:25 AM houzj.f...@fujitsu.com > wrote: > > > > > On Monday, March 21, 2022 6:01 PM Amit Kapila > > > > > > wrote: > > > > Oh, sorry, I posted the wrong patch, here is the correct one. > > > > The test change looks

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 18:47:58 -0700, David G. Johnston wrote: > It also seems that each tracked object type needs to have its own > last_reset field (we could choose to name it stats_reset too, leaving > pg_stat_database.last_reset as the only anomaly) added as an implied > behavior needed for such i

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-23 Thread Bharath Rupireddy
On Wed, Mar 23, 2022 at 10:16 AM Bharath Rupireddy wrote: > > On Tue, Mar 22, 2022 at 8:12 PM Andres Freund wrote: > > > Do you mean like this? > > > ereport(LOG, > > > /* translator: the placeholders show checkpoint options */ > > > (errmsg("%s starting:%s%s%s%s%s%s%s%s", > >

Re: Failed transaction statistics to measure the logical replication progress

2022-03-23 Thread Amit Kapila
On Thu, Mar 3, 2022 at 8:58 AM osumi.takami...@fujitsu.com wrote: > This patch introduces two new subscription statistics columns (apply_commit_count and apply_rollback_count) to the pg_stat_subscription_stats view for counting cumulative transactions commits/rollbacks for a particular subscripti

Checking pgwin32_is_junction() errors

2022-03-23 Thread Thomas Munro
Hi, The comment for pgwin32_is_junction() says "Assumes the file exists, so will return false if it doesn't (since a nonexistent file is not a junction)". In fact that's the behaviour for any kind of error, and although we set errno in that case, no caller ever checks it. I think it'd be better

RE: logical replication empty transactions

2022-03-23 Thread shiy.f...@fujitsu.com
On Thursday, March 24, 2022 11:19 AM Hou, Zhijie/侯 志杰 wrote: > > Attach the new version patch which include the following changes: > > - Fix a typo > - Change the requestreply flag of the newly added WalSndKeepalive to false, > because the subscriber can judge whether it's necessary to post a

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-23 Thread Thomas Munro
On Tue, Mar 8, 2022 at 12:58 AM Michael Paquier wrote: > Junction points are directories, no? Are you sure that this works > correctly on WIN32? It seems to me that we'd better use readlink() > only for entries in pg_tlbspc/ that are PGFILETYPE_LNK on non-WIN32 > and pgwin32_is_junction() on WIN

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-23 Thread Etsuro Fujita
On Sat, Mar 5, 2022 at 7:32 PM Etsuro Fujita wrote: > Attached is an updated version. In the 0002 patch I introduced a macro for building an abort command in preparation for the parallel abort patch (0003), but I moved it to 0003. Attached is a new patch set. The new version of 0002 is just a c

Re: Extensible Rmgr for Table AMs

2022-03-23 Thread Jeff Davis
On Fri, 2022-02-04 at 22:56 +0800, Julien Rouhaud wrote: > > Right, which I guess raises another question: if the maximum is > > UINT8_MAX, which BTW I find perfectly reasonable, why are we not > > just > > defining this as an array of size 256? There's no point in adding > > code > > complexity to

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-03-23 Thread Tatsuo Ishii
>> My hoary animal prairiedog doesn't like this [1]: >> >> # Failed test 'concurrent update with retrying stderr /(?s-xim:client >> (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not >> serialize access due to concurrent update\\b.*\\g1)/' >> # at t/001_pgbench_with_ser

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 21:36:09 +0530, Bharath Rupireddy wrote in > Here's a refactoring patch that basically moves the pg_waldump's > functions and stats structures to xlogreader.h/.c so that the > pg_walinspect can reuse them. If it looks okay, I will send the > pg_walinspect patches based on it.

Re: [PoC] Let libpq reject unexpected authentication requests

2022-03-23 Thread Laurenz Albe
On Wed, 2022-03-23 at 21:31 +, Jacob Champion wrote: > On Mon, 2022-03-07 at 11:44 +0100, Laurenz Albe wrote: > > I am all for the idea, but you implemented the reverse of proposal 2. > > > > Wouldn't it be better to list the *rejected* authentication methods? > > Then we could have "password"

Re: Remove an unnecessary errmsg_plural in dependency.c

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 17:39:52 +0100, Peter Eisentraut wrote in > On 23.03.22 17:33, Bharath Rupireddy wrote: > > It looks like the following errmsg_plural in dependency.c is > > unnecessary as numReportedClient > 1 always and numNotReportedClient > > can never be < 0. Therefore plural version of

Re: Removing unneeded self joins

2022-03-23 Thread Andrey V. Lepikhov
On 3/22/22 05:58, Andres Freund wrote: Hi, On 2022-03-04 15:47:47 +0500, Andrey Lepikhov wrote: Also, in new version of the patch I fixed one stupid bug: checking a self-join candidate expression operator - we can remove only expressions like F(arg1) = G(arg2). This CF entry currently fails t

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Laurenz Albe
On Wed, 2022-03-23 at 17:55 -0700, Andres Freund wrote: > Starting with the below commit, pg_stat_reset_single_function_counters, > pg_stat_reset_single_table_counters don't just reset the stats for the > individual function, but also set pg_stat_database.stats_reset. I see the point in the fine-g

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 10:50 PM Dilip Kumar wrote: > > On Wed, Mar 23, 2022 at 10:37 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > > > I could not see any reason for it to fail, and I could not reproduce > > > it either. Is it possible to access

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-24 06:27:48 +0100, Laurenz Albe wrote: > On Wed, 2022-03-23 at 17:55 -0700, Andres Freund wrote: > > Starting with the below commit, pg_stat_reset_single_function_counters, > > pg_stat_reset_single_table_counters don't just reset the stats for the > > individual function, but also s

Re: shared-memory based stats collector - v67

2022-03-23 Thread Kyotaro Horiguchi
(nice work about ubsan) At Wed, 23 Mar 2022 10:42:03 -0700, Andres Freund wrote in > Hi, > > On 2022-03-23 17:27:50 +0900, Kyotaro Horiguchi wrote: > > At Mon, 21 Mar 2022 14:30:17 -0700, Andres Freund > > wrote in > > Before there, InitWalRecovery changes the state to > > DB_IN_ARCHIVE_RECO

Re: PG DOCS - logical replication filtering

2022-03-23 Thread Peter Smith
On Fri, Mar 11, 2022 at 9:37 AM Euler Taveira wrote: > > On Fri, Mar 4, 2022, at 12:41 AM, Peter Smith wrote: > > PSA patch v3 to address all comments received so far. > > Peter, > > I started reviewing this documentation for row filter and I noticed that I was > changing too much lines to submit

Re: ubsan

2022-03-23 Thread Noah Misch
On Wed, Mar 23, 2022 at 03:58:09PM -0400, Tom Lane wrote: > Andres Freund writes: > > I think we should backpatch both, based on the reasoning in > > 46ab07ffda9d6c8e63360ded2d4568aa160a7700 ? > > Yeah, I suppose. Is anyone going to step up and run a buildfarm > member with ubsan enabled? thorn

Re: Fast COPY FROM based on batch insert

2022-03-23 Thread Andrey V. Lepikhov
On 3/22/22 06:54, Etsuro Fujita wrote: On Fri, Jun 4, 2021 at 5:26 PM Andrey Lepikhov wrote: We still have slow 'COPY FROM' operation for foreign tables in current master. Now we have a foreign batch insert operation And I tried to rewrite the patch [1] with this machinery. The patch has been

<    1   2