Re: Logical replication - schema change not invalidating the relation cache

2022-03-11 Thread vignesh C
On Fri, Dec 3, 2021 at 3:21 PM vignesh C wrote: > > On Fri, Dec 3, 2021 at 1:13 PM Michael Paquier wrote: > > > > On Thu, Aug 26, 2021 at 09:00:39PM +0530, vignesh C wrote: > > > The previous patch was failing because of the recent test changes made > > > by commit 201a76183e2 which unified new

Re: Printing backtrace of postgres processes

2022-03-11 Thread vignesh C
On Wed, Mar 9, 2022 at 9:26 PM Justin Pryzby wrote: > > rebased to appease cfbot. > > + couple of little fixes as 0002. Thanks for rebasing and fixing a few issues. I have taken all your changes except for mcxtfuncs changes as those changes were not done as part of this patch. Attached v19 patch

Re: Handle infinite recursion in logical replication setup

2022-03-11 Thread vignesh C
On Fri, Mar 11, 2022 at 4:28 PM kuroda.hay...@fujitsu.com wrote: > > Hi Vegnesh, > > While considering about second problem, I was very confusing about it. > I'm happy if you answer my question. > > > To handle this if user has specified only_local option, we could throw > > a warning or error

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Dilip Kumar
On Sat, Mar 12, 2022 at 1:55 AM Robert Haas wrote: > Responding to this specific issue.. > The changes to createdb_failure_params make me a little nervous. I > think we'd be in real trouble if we failed before completing both > DropDatabaseBuffers() and ForgetDatabaseSyncRequests(). However, it

Re: Handle infinite recursion in logical replication setup

2022-03-11 Thread vignesh C
On Wed, Mar 2, 2022 at 3:59 PM Amit Kapila wrote: > > On Wed, Feb 23, 2022 at 11:59 AM vignesh C wrote: > > > ... > ... > > I have attached a basic patch for this, if the idea is accepted, I > > will work further to test more scenarios, add documentation, and test > > and post an updated patch.

Re: Column Filtering in Logical Replication

2022-03-11 Thread Amit Kapila
On Fri, Mar 11, 2022 at 6:20 PM Tomas Vondra wrote: > > On 3/11/22 10:52, Amit Kapila wrote: > > On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra > > wrote: > >> > >> On 3/10/22 20:10, Tomas Vondra wrote: > >>> > >>> > >>> FWIW I think the reason is pretty simple - pgoutput_row_filter_init is > >>>

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Jeff Davis
On Thu, 2022-03-10 at 15:54 -0500, Stephen Frost wrote: > The standard is basically that all of the functions it brings are > written to enforce the PG privilege system and you aren't able to use > the extension to bypass those privileges. In some cases that means > that Every extension should

Re: range_agg with multirange inputs

2022-03-11 Thread Paul Jungwirth
On 3/10/22 14:07, Chapman Flack wrote: When I apply this patch, I get a func.sgml with two entries for range_intersect_agg(anymultirange). Arg, fixed. In range_agg_transfn, you've changed the message in the "must be called with a range or multirange"; that seems like another good candidate

Re: Issue with pg_stat_subscription_stats

2022-03-11 Thread Amit Kapila
On Sat, Mar 12, 2022 at 2:14 AM Melanie Plageman wrote: > > So, I noticed that pg_stat_reset_subscription_stats() wasn't working > properly, and, upon further investigation, I'm not sure the view > pg_stat_subscription_stats is being properly populated. > I have tried the below scenario based on

Re: On login trigger: take three

2022-03-11 Thread Andres Freund
Hi, On 2022-02-15 21:07:15 +1100, Greg Nancarrow wrote: > Subject: [PATCH v25] Add a new "login" event and login event trigger support. > > The login event occurs when a user logs in to the system. I think this needs a HUGE warning in the docs about most event triggers needing to check

Re: role self-revocation

2022-03-11 Thread Mark Dilger
> On Mar 11, 2022, at 4:56 PM, Stephen Frost wrote: > > First … I outlined a fair bit of further description in the message you just > responded to but neglected to include in your response, which strikes me as > odd that you’re now asking for further explanation. > When it comes to

Re: refactoring basebackup.c

2022-03-11 Thread Andres Freund
Hi, On 2022-03-11 10:19:29 -0500, Robert Haas wrote: > Thanks for the report. The problem here is that, when the output is > standard output (-D -), pg_basebackup can only produce a single output > file, so the manifest gets injected into the tar file on the client > side rather than being

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, On Fri, Mar 11, 2022 at 19:03 Mark Dilger wrote: > > On Mar 11, 2022, at 2:46 PM, Stephen Frost wrote: > > > > I do think that’s reasonable … and believe I suggested it about 3 > messages ago in this thread. ;) (step #3 I think it was? Or maybe 4). > > Yes, and you mentioned it to

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Andres Freund
Hi, On 2022-03-11 22:42:42 +0200, Heikki Linnakangas wrote: > Have you been able to create a test case for that? The largest record I can > think of is a commit record with a huge number of subtransactions, dropped > relations, and shared inval messages. I'm not sure if you can overflow a >

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-11 Thread David Zhang
Applied patches v6-0002 and v6-0003 to master branch, and the `make check` test is ok. Here is my test result in 10 times average on 3 virtual machines: before the patches: abort.1 = 2.5473 ms abort.2 = 4.1572 ms after the patches with OPTIONS (ADD parallel_abort 'true'): abort.1 =

Re: Kerberos delegation support in libpq and postgres_fdw

2022-03-11 Thread Stephen Frost
Greetings, On Fri, Mar 11, 2022 at 18:55 Jacob Champion wrote: > On Mon, 2022-02-28 at 20:28 -0500, Stephen Frost wrote: > > Will add to the CF for consideration. > > GSSAPI newbie here, so caveat lector. No worries, thanks for your interest! > diff --git a/src/backend/libpq/auth.c

Re: wal_compression=zstd

2022-03-11 Thread Michael Paquier
On Fri, Mar 11, 2022 at 03:49:00PM -0600, Justin Pryzby wrote: > While rebasing, I realized this should have bumped XLOG_PAGE_MAGIC. > > Also, there's a dangling "and". Right. I'll address that a bit later today. Thanks! -- Michael signature.asc Description: PGP signature

Re: role self-revocation

2022-03-11 Thread Mark Dilger
> On Mar 11, 2022, at 2:46 PM, Stephen Frost wrote: > > I do think that’s reasonable … and believe I suggested it about 3 messages > ago in this thread. ;) (step #3 I think it was? Or maybe 4). Yes, and you mentioned it to me off-list. I'm soliciting a more concrete specification for

Re: Kerberos delegation support in libpq and postgres_fdw

2022-03-11 Thread Jacob Champion
On Mon, 2022-02-28 at 20:28 -0500, Stephen Frost wrote: > Will add to the CF for consideration. GSSAPI newbie here, so caveat lector. > diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c > index efc53f3135..6f820a34f1 100644 > --- a/src/backend/libpq/auth.c > +++

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, On Fri, Mar 11, 2022 at 12:32 Mark Dilger wrote: > > > > On Mar 11, 2022, at 8:48 AM, Stephen Frost wrote: > > > > I agree that it would have an impact on backwards compatibility to > > change how WITH ADMIN works- but it would also get us more in line with > > what the SQL standard

Re: generic plans and "initial" pruning

2022-03-11 Thread Zhihong Yu
Hi, w.r.t. v5-0003-Teach-AcquireExecutorLocks-to-skip-locking-pruned.patch : (pruning steps containing expressions that can be computed before before the executor proper has started) the word 'before' was repeated. For ExecInitParallelPlan(): + char *execlockrelsinfo_data; + char

Re: wal_compression=zstd

2022-03-11 Thread Justin Pryzby
While rebasing, I realized this should have bumped XLOG_PAGE_MAGIC. Also, there's a dangling "and". +The supported methods are pglz, +lz4 (if PostgreSQL +was compiled with --with-lz4) and +zstd (if PostgreSQL +was compiled with --with-zstd) and +

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 3:42 PM Heikki Linnakangas wrote: > Have you been able to create a test case for that? The largest record I > can think of is a commit record with a huge number of subtransactions, > dropped relations, and shared inval messages. I'm not sure if you can > overflow a uint32

Re: On login trigger: take three

2022-03-11 Thread Robert Haas
On Tue, Feb 15, 2022 at 5:07 AM Greg Nancarrow wrote: > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. I tried this: rhaas=# create function on_login_proc() returns event_trigger as $$begin perform pg_sleep(1000); end$$ language plpgsql;

Issue with pg_stat_subscription_stats

2022-03-11 Thread Melanie Plageman
So, I noticed that pg_stat_reset_subscription_stats() wasn't working properly, and, upon further investigation, I'm not sure the view pg_stat_subscription_stats is being properly populated. I don't think subscriptionStatHash will be created properly and that the reset timestamp won't be

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Heikki Linnakangas
On 11/03/2022 17:42, Matthias van de Meent wrote: Hi, Xlogreader limits the size of what it considers valid xlog records to MaxAllocSize; but this is not currently enforced in the XLogRecAssemble API. This means it is possible to assemble a record that postgresql cannot replay. Oops, that

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Robert Haas
On Thu, Mar 10, 2022 at 9:38 PM Kyotaro Horiguchi wrote: > It seems to me too rigorous that pg_get_wal_records_info/stats() > reject future LSNs as end-LSN and I think WARNING or INFO and stop at > the real end-of-WAL is more kind to users. I think the same with the > restriction that start and

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 5:21 AM Dilip Kumar wrote: > Changes, 1) it take Robert's patch as first refactoring patch 2) > Rebase other new relmapper apis on top of that in 0002 3) Some code > refactoring in main patch 0005 and also one problem fix, earlier in > wal log method I have removed

Re: Proposal for internal Numeric to Uint64 conversion function.

2022-03-11 Thread Tom Lane
Amul Sul writes: > On Wed, Feb 16, 2022 at 4:50 PM Peter Eisentraut > wrote: >> On 16.02.22 06:00, Amul Sul wrote: >>> Currently, numeric_pg_lsn is the only one that accepts the Numeric >>> value and converts to uint64 and that is the reason all the type >>> conversion code is embedded into it.

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-11 Thread Nathan Bossart
+CheckpointStats.repl_map_cutoff_lsn = cutoff; Could we set repl_map_cutoff_lsn closer to where it is calculated? Right now, it's set at the bottom of the function just before the directory is freed. Is there a strong reason to do it there? +"logical rewrite mapping

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-11 Thread Nathan Bossart
On Fri, Mar 11, 2022 at 02:00:37PM -0500, Chapman Flack wrote: > v7 looks good to me. I have repeated the installcheck-world and given > the changed documentation sections one last read-through, and will > change the status to RfC. Thanks for reviewing! -- Nathan Bossart Amazon Web Services:

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-11 Thread Chapman Flack
On 03/10/22 19:38, Nathan Bossart wrote: > On Thu, Mar 10, 2022 at 07:13:14PM -0500, Chapman Flack wrote: >> +postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn); > > Ah, good catch. I made this change in v7. I considered doing something v7 looks good to me. I have repeated

Re: refactoring basebackup.c

2022-03-11 Thread Robert Haas
On Tue, Feb 15, 2022 at 11:26 AM Robert Haas wrote: > On Wed, Feb 9, 2022 at 8:41 AM Abhijit Menon-Sen wrote: > > It took me a while to assimilate these patches, including the backup > > targets one, which I hadn't looked at before. Now that I've wrapped my > > head around how to put the pieces

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 1:10 PM Robert Haas wrote: > I don't think you've adequately considered temporary relations here. > It seems to be that ReadBufferWithoutRelcache() could not be safe on a > temprel, because we'd need a BackendId to access the underlying > storage. So I think that

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 5:21 AM Dilip Kumar wrote: > Changes, 1) it take Robert's patch as first refactoring patch 2) > Rebase other new relmapper apis on top of that in 0002 3) Some code > refactoring in main patch 0005 and also one problem fix, earlier in > wal log method I have removed

Re: refactoring basebackup.c

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 11:29 AM Justin Pryzby wrote: > Sounds right. OK, committed. > Also, I think the magic 8 for .gz should actually be a 7. > > I'm not sure why it tests for ".gz" but not ".tar.gz", which would help to > make > them all less magic. > > commit

Re: role self-revocation

2022-03-11 Thread Mark Dilger
> On Mar 11, 2022, at 8:48 AM, Stephen Frost wrote: > > I agree that it would have an impact on backwards compatibility to > change how WITH ADMIN works- but it would also get us more in line with > what the SQL standard says for how WITH ADMIN is supposed to work and > that seems worth the

Re: Collecting statistics about contents of JSONB columns

2022-03-11 Thread Mahendra Singh Thalor
On Fri, 4 Feb 2022 at 08:30, Tomas Vondra wrote: > > > > On 2/4/22 03:47, Tomas Vondra wrote: > > ./json-generate.py 3 2 8 1000 6 1000 > > Sorry, this should be (different order of parameters): > > ./json-generate.py 3 2 1000 8 6 1000 > Thanks, Tomas for this test case. Hi Hackers, For

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 11:34 AM Tom Lane wrote: > Note that either case would also involve making entries in pg_shdepend; > although for the case of roles owned by/granted to the bootstrap > superuser, we could omit those on the usual grounds that we don't need > to record dependencies on pinned

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Mar 11, 2022 at 11:12 AM Mark Dilger > wrote: > > This issue of how much backwards compatibility breakage we're willing to > > tolerate is just as important as questions about how we would want roles to > > work in a green-field

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Robert Haas writes: > > If we implement the link between the creating role and the created > > role as role ownership, then we are surely just going to add a > > rolowner column to pg_authid, and when the role is owned by nobody, I > > think we

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 11:12 AM Mark Dilger wrote: > This issue of how much backwards compatibility breakage we're willing to > tolerate is just as important as questions about how we would want roles to > work in a green-field development project. The sense I got a year ago, on > this list,

Re: role self-revocation

2022-03-11 Thread Tom Lane
Robert Haas writes: > If we implement the link between the creating role and the created > role as role ownership, then we are surely just going to add a > rolowner column to pg_authid, and when the role is owned by nobody, I > think we should always just store a valid OID in it, rather than >

Re: refactoring basebackup.c

2022-03-11 Thread Justin Pryzby
On Fri, Mar 11, 2022 at 10:19:29AM -0500, Robert Haas wrote: > So I think we should just refuse this command. Patch for that attached. Sounds right. Also, I think the magic 8 for .gz should actually be a 7. I'm not sure why it tests for ".gz" but not ".tar.gz", which would help to make them all

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 10:46 AM Tom Lane wrote: > The bootstrap superuser clearly must be a special case in some way. > I'm not convinced that that means there should be other special > cases. Maybe there is a use-case for other "unowned" roles, but in > exactly what way would that be different

Re: role self-revocation

2022-03-11 Thread Mark Dilger
> On Mar 11, 2022, at 7:58 AM, Robert Haas wrote: > > This kind of thing is always a judgement call. If we were talking > about breaking 'SELECT * from table', I'm sure it would be hard to > convince anybody to agree to do that at all, let alone with no > deprecation period. Fortunately,

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 10:37 AM David G. Johnston wrote: > I largely agree and am perfectly fine with going with the majority on this > point. My vote would just fall on the conservative side. But as so far no > one else seems to be overly concerned, nerfing CREATEROLE seems to be the >

Re: role self-revocation

2022-03-11 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 11, 2022 at 10:27 AM Stephen Frost wrote: >> I agree that there would be a recorded relationship (that is, one that >> we write into the catalog and keep around until and unless it's removed >> by an admin) between creating and created roles and that's probably

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Bharath Rupireddy
On Fri, Mar 11, 2022 at 8:22 AM Kyotaro Horiguchi wrote: > > > - The difference between pg_get_wal_record_info and _records_ other than > - the number of argument is the former accepts incorrect LSNs. > > The discussion is somewhat confused after some twists and turns.. It > should be something

Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Matthias van de Meent
Hi, Xlogreader limits the size of what it considers valid xlog records to MaxAllocSize; but this is not currently enforced in the XLogRecAssemble API. This means it is possible to assemble a record that postgresql cannot replay. Similarly; it is possible to repeatedly call XlogRegisterData() so

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 10:27 AM Stephen Frost wrote: > I agree that there would be a recorded relationship (that is, one that > we write into the catalog and keep around until and unless it's removed > by an admin) between creating and created roles and that's probably the > default when CREATE

Re: role self-revocation

2022-03-11 Thread David G. Johnston
On Fri, Mar 11, 2022 at 8:32 AM Stephen Frost wrote: > > Such scripts as will break will still > break in a pretty clear way with a clear answer as to how to fix them > and I don't think there's some kind of data corruption or something that > would happen. > > I largely agree and am perfectly

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Thu, Mar 10, 2022 at 3:01 PM Robert Haas wrote: > > > On Thu, Mar 10, 2022 at 4:00 PM David G. Johnston > > wrote: > > > I dislike changing the documented behavior of CREATEROLE to the degree > > suggested here. However,

Re: role self-revocation

2022-03-11 Thread David G. Johnston
On Fri, Mar 11, 2022 at 6:55 AM Robert Haas wrote: > On Thu, Mar 10, 2022 at 5:14 PM Tom Lane wrote: > > This seems reasonable in isolation, but > > > > (1) it implies a persistent relationship between creating and created > > roles. Whether you want to call that ownership or not, it sure

Re: role self-revocation

2022-03-11 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Mar 10, 2022 at 5:14 PM Tom Lane wrote: > > This seems reasonable in isolation, but > > > > (1) it implies a persistent relationship between creating and created > > roles. Whether you want to call that ownership or not, it sure

Re: refactoring basebackup.c

2022-03-11 Thread Robert Haas
On Thu, Mar 10, 2022 at 8:02 PM Justin Pryzby wrote: > I'm getting errors from pg_basebackup when using both -D- and > --compress=server-* > The issue seems to go away if I use --no-manifest. > > $ ./src/bin/pg_basebackup/pg_basebackup -h /tmp -Ft -D- --wal-method none > --compress=server-gzip

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Ashutosh Sharma
On Fri, Mar 11, 2022 at 8:21 PM Robert Haas wrote: > > On Fri, Mar 11, 2022 at 12:15 AM Ashutosh Sharma > wrote: > > Looks better, but why do you want to pass elevel to the > > load_relmap_file()? Are we calling this function from somewhere other > > than read_relmap_file()? If not, do we have

Re: generic plans and "initial" pruning

2022-03-11 Thread Amit Langote
On Fri, Mar 11, 2022 at 11:35 PM Amit Langote wrote: > Attached is v5, now broken into 3 patches: > > 0001: Some refactoring of runtime pruning code > 0002: Add a plan_tree_walker > 0003: Teach AcquireExecutorLocks to skip locking pruned relations Repeated the performance tests described in the

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Robert Haas
On Fri, Mar 11, 2022 at 12:15 AM Ashutosh Sharma wrote: > Looks better, but why do you want to pass elevel to the > load_relmap_file()? Are we calling this function from somewhere other > than read_relmap_file()? If not, do we have any plans to call this > function directly bypassing

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Ashutosh Sharma
Some comments on pg_walinspect-docc.patch this time: + + + pg_get_wal_record_info(in_lsn pg_lsn, lsn OUT pg_lsn, prev_lsn OUT pg_lsn, xid OUT xid, resource_manager OUT text, length OUT int4, total_length OUT int4, description OUT text, block_ref OUT text, data OUT bytea, data_len OUT

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Ashutosh Sharma
You may also need to add documentation to app-createdb.sgml. Currently you have just added to create_database.sgml. Also, I had a quick look at the new changes done in v13-0005-WAL-logged-CREATE-DATABASE.patch and they seemed fine to me although I haven't put much emphasis on the comments and

Re: role self-revocation

2022-03-11 Thread Robert Haas
On Thu, Mar 10, 2022 at 5:14 PM Tom Lane wrote: > This seems reasonable in isolation, but > > (1) it implies a persistent relationship between creating and created > roles. Whether you want to call that ownership or not, it sure walks > and quacks like ownership. I agree. It's been obvious to

Re: logical decoding and replication of sequences

2022-03-11 Thread Tomas Vondra
On 3/11/22 12:34, Amit Kapila wrote: > On Tue, Mar 8, 2022 at 11:59 PM Tomas Vondra > wrote: >> >> On 3/7/22 22:25, Tomas Vondra wrote: Interesting. I can think of one reason that might cause this - we log the first sequence increment after a checkpoint. So if a checkpoint

Re: Column Filtering in Logical Replication

2022-03-11 Thread Tomas Vondra
On 3/11/22 10:52, Amit Kapila wrote: > On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra > wrote: >> >> On 3/10/22 20:10, Tomas Vondra wrote: >>> >>> >>> FWIW I think the reason is pretty simple - pgoutput_row_filter_init is >>> broken. It assumes you can just do this >>> >>> rftuple =

Re: Column Filtering in Logical Replication

2022-03-11 Thread Tomas Vondra
sts and results with you in a later email. We are trying to >>> unify the column filter queries with row filter to make their behavior >>> the same and will share the findings once it is done. I hope if we are >>> able to achieve this that we will reduce the chances of bugs in

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Bharath Rupireddy
On Fri, Mar 11, 2022 at 8:08 AM Kyotaro Horiguchi wrote: > > > Attaching the v8 patch-set resolving above comments and some tests for > > checking function permissions. Please review it further. > > I played with this a bit, and would like to share some thoughts on it. Thanks a lot Kyotaro-san

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-11 Thread Julien Rouhaud
On Fri, Mar 11, 2022 at 04:59:11PM +0530, Nitin Jadhav wrote: > > That "throttled" flag should be the same as having or not a "force" in the > > flags. We should be consistent and report information the same way, so > > either > > a lot of flags (is_throttled, is_force...) or as now a single

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Ashutosh Sharma
On Fri, Mar 11, 2022 at 8:22 AM Kyotaro Horiguchi wrote: > > - The difference between pg_get_wal_record_info and _records_ other than > - the number of argument is the former accepts incorrect LSNs. > > The discussion is somewhat confused after some twists and turns.. It > should be something

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-11 Thread Ashutosh Sharma
On Fri, Mar 11, 2022 at 8:22 AM Kyotaro Horiguchi wrote: > > Sorry, some minor non-syntactical corrections. > > At Fri, 11 Mar 2022 11:38:22 +0900 (JST), Kyotaro Horiguchi > wrote in > > I played with this a bit, and would like to share some thoughts on it. > > > > It seems to me too rigorous

Re: logical decoding and replication of sequences

2022-03-11 Thread Amit Kapila
On Fri, Mar 11, 2022 at 5:04 PM Amit Kapila wrote: > > On Tue, Mar 8, 2022 at 11:59 PM Tomas Vondra > wrote: > > > > On 3/7/22 22:25, Tomas Vondra wrote: > > >> > > >> Interesting. I can think of one reason that might cause this - we log > > >> the first sequence increment after a checkpoint. So

RE: Skipping logical replication transactions on subscriber side

2022-03-11 Thread osumi.takami...@fujitsu.com
On Friday, March 11, 2022 5:20 PM Masahiko Sawada wrote: > I've attached an updated version patch. This patch can be applied on top of > the > latest disable_on_error patch[1]. Hi, thank you for the patch. I'll share my review comments on v13. (a) src/backend/commands/subscriptioncmds.c @@

Re: logical decoding and replication of sequences

2022-03-11 Thread Amit Kapila
On Tue, Mar 8, 2022 at 11:59 PM Tomas Vondra wrote: > > On 3/7/22 22:25, Tomas Vondra wrote: > >> > >> Interesting. I can think of one reason that might cause this - we log > >> the first sequence increment after a checkpoint. So if a checkpoint > >> happens in an unfortunate place, there'll be

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-11 Thread Nitin Jadhav
> > > > Ok. I agree that it is difficult to interpret it correctly. So even if > > say that a new checkpoint has been explicitly requested, the user may > > not understand that it affects current checkpoint behaviour unless the > > user knows the internals of the checkpoint. How about naming the

Re: On login trigger: take three

2022-03-11 Thread Daniel Gustafsson
> On 15 Feb 2022, at 11:07, Greg Nancarrow wrote: > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. Thanks for adopting this patch, I took another look at it today and have some comments. +This flag is used internally by PostgreSQL

RE: Handle infinite recursion in logical replication setup

2022-03-11 Thread kuroda.hay...@fujitsu.com
Hi Vegnesh, While considering about second problem, I was very confusing about it. I'm happy if you answer my question. > To handle this if user has specified only_local option, we could throw > a warning or error out while creating subscription in this case, we > could have a column

Fix typo in progress reporting doc

2022-03-11 Thread Nitin Jadhav
Hi, I have observed that the table naming conventions used in 'progress-reporting.html' are not consistent across different sections. For some cases "Phases" (Table 28.37. CREATE INDEX Phases) is used and for some cases "phases" (Table 28.35. ANALYZE phases) is used. I have attached a patch to

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-11 Thread Dilip Kumar
On Fri, Mar 11, 2022 at 11:52 AM Dilip Kumar wrote: > > On Thu, Mar 10, 2022 at 10:18 PM Robert Haas wrote: > > > > On Thu, Mar 10, 2022 at 6:02 AM Dilip Kumar wrote: > > > I have completely changed the logic for this refactoring. Basically, > > > write_relmap_file(), is already having

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-11 Thread Julien Rouhaud
On Fri, Mar 11, 2022 at 02:41:23PM +0530, Nitin Jadhav wrote: > > Ok. I agree that it is difficult to interpret it correctly. So even if > say that a new checkpoint has been explicitly requested, the user may > not understand that it affects current checkpoint behaviour unless the > user knows the

Re: Column Filtering in Logical Replication

2022-03-11 Thread Amit Kapila
On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra wrote: > > On 3/10/22 20:10, Tomas Vondra wrote: > > > > > > FWIW I think the reason is pretty simple - pgoutput_row_filter_init is > > broken. It assumes you can just do this > > > > rftuple = SearchSysCache2(PUBLICATIONRELMAP, > >

Re: BufferAlloc: don't take two simultaneous locks

2022-03-11 Thread Yura Sokolov
В Пт, 11/03/2022 в 15:49 +0900, Kyotaro Horiguchi пишет: > At Fri, 11 Mar 2022 15:30:30 +0900 (JST), Kyotaro Horiguchi > wrote in > > Thanks! I looked into dynahash part. > > > > struct HASHHDR > > { > > - /* > > - * The freelist can become a point of contention in high-concurrency

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-11 Thread Nitin Jadhav
> > I just wanted to avoid extra calculations just to show the progress in > > the view. Since it's a good metric, I have added an additional field > > named 'next_flags' to the view which holds all possible flag values of > > the next checkpoint. > > I still don't think that's ok. IIUC the only

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-03-11 Thread Nitin Jadhav
> > The current format matches with the server log message for the > > checkpoint start in LogCheckpointStart(). Just to be consistent, I > > have not changed the code. > > > > See below, how flags are shown in other sql functions like: > > ashu@postgres=# select * from

Re: BufferAlloc: don't take two simultaneous locks

2022-03-11 Thread Yura Sokolov
В Пт, 11/03/2022 в 15:30 +0900, Kyotaro Horiguchi пишет: > At Thu, 03 Mar 2022 01:35:57 +0300, Yura Sokolov > wrote in > > В Вт, 01/03/2022 в 10:24 +0300, Yura Sokolov пишет: > > > Ok, here is v4. > > > > And here is v5. > > > > First, there was compilation error in Assert in dynahash.c . > >

Re: WIP: WAL prefetch (another approach)

2022-03-11 Thread Julien Rouhaud
On Fri, Mar 11, 2022 at 06:31:13PM +1300, Thomas Munro wrote: > On Wed, Mar 9, 2022 at 7:47 PM Julien Rouhaud wrote: > > > > This could use XLogRecGetBlock? Note that this macro is for now never used. > > xlogreader.c also has some similar forgotten code that could use > > XLogRecMaxBlockId. > >

Re: BufferAlloc: don't take two simultaneous locks

2022-03-11 Thread Kyotaro Horiguchi
At Fri, 11 Mar 2022 15:49:49 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 11 Mar 2022 15:30:30 +0900 (JST), Kyotaro Horiguchi > wrote in > > Thanks! I looked into dynahash part. Then I looked into bufmgr part. It looks fine to me but I have some comments on code comments. >

Re: Skipping logical replication transactions on subscriber side

2022-03-11 Thread Masahiko Sawada
On Thu, Mar 10, 2022 at 9:02 PM Amit Kapila wrote: > > On Tue, Mar 1, 2022 at 8:31 PM Masahiko Sawada wrote: > > > > I've attached an updated patch along with two patches for cfbot tests > > since the main patch (0003) depends on the other two patches. Both > > 0001 and 0002 patches are the same