Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-01-27 Thread Kyotaro Horiguchi
At Tue, 25 Jan 2022 19:20:05 +, "Bossart, Nathan" wrote in > On 1/24/22, 9:16 PM, "Michael Paquier" wrote: > > Now, I also find confusing the state of CreateCheckpoint() once this > > patch gets applied. Now the code and comments imply that an > > end-of-recovery checkpoint is a shutdown

Re: Suppressing useless wakeups in walreceiver

2022-01-27 Thread Kyotaro Horiguchi
Hello. At Thu, 27 Jan 2022 23:50:04 +1300, Thomas Munro wrote in > While working on WaitEventSet-ifying various codepaths, I found it > strange that walreceiver wakes up 10 times per second while idle. > Here's a draft patch to compute the correct sleep time. Agree to the objective. However

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-27 Thread Laurenz Albe
On Thu, 2022-01-27 at 12:20 -0800, Peter Geoghegan wrote: > I can see why the nearby, similar vacthresh and anlthresh variables > (not shown here) are scaled based on pg_class.reltuples -- that makes > sense. But why should we follow that example here, with vacinsthresh? What would you suggest

Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-01-27 Thread Bharath Rupireddy
On Fri, Jan 21, 2022 at 11:07 AM Nitin Jadhav wrote: > > > I think the right choice to solve the *general* problem is the > > mentioned pg_stat_progress_checkpoints. > > > > We may want to *additionally* have the ability to log the progress > > specifically for the special cases when we're not

Re: [BUG]Update Toast data failure in logical replication

2022-01-27 Thread Dilip Kumar
On Tue, Jan 25, 2022 at 11:59 AM Amit Kapila wrote: > > On Tue, Jan 25, 2022 at 12:26 AM Euler Taveira wrote: > > > > I am not sure if your proposal is much different compared to v4 or how > much it improves the situation? I see you didn't consider > 'check_external_attr' parameter and I think

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-27 Thread Bharath Rupireddy
On Fri, Jan 28, 2022 at 11:16 AM Nathan Bossart wrote: > > I know I voted for "start=%X/%X, end=%X/%X," but looking at this again, I > wonder if it could be misleading. "start" is the redo location, and "end" > is the location of the checkpoint record, but I could understand why > someone might

RE: Support tab completion for upper character inputs in psql

2022-01-27 Thread tanghy.f...@fujitsu.com
On Friday, January 28, 2022 5:24 AM, Tom Lane wrote: > Here's a fleshed-out patch series for this idea. Thanks for you patch. I did some tests on it and here are something cases I feel we need to confirm whether they are suitable. 1) postgres=# create table atest(id int, "iD" int, "ID" int);

Re: Design of pg_stat_subscription_workers vs pgstats

2022-01-27 Thread Amit Kapila
On Fri, Jan 28, 2022 at 1:49 AM David G. Johnston wrote: > > On Thu, Jan 27, 2022 at 5:08 AM Amit Kapila wrote: >> >> On Thu, Jan 27, 2022 at 11:16 AM Andres Freund wrote: >> > >> > On 2022-01-25 20:27:07 +0900, Masahiko Sawada wrote: >> > >> > > There will be some challenges in a case where

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-27 Thread Nathan Bossart
On Fri, Jan 28, 2022 at 08:43:36AM +0530, Bharath Rupireddy wrote: > 2022-01-28 03:06:10.213 UTC [2409486] LOG: checkpoint starting: > immediate force wait > 2022-01-28 03:06:10.257 UTC [2409486] LOG: checkpoint complete: > start=0/14D9510, end=0/14D9548; wrote 4 buffers (0.0%); 0 WAL file(s) >

Re: Add connection active, idle time to pg_stat_activity

2022-01-27 Thread Kyotaro Horiguchi
At Fri, 28 Jan 2022 14:36:31 +0900 (JST), Kyotaro Horiguchi wrote in > Hi. > > At Thu, 27 Jan 2022 20:36:56 +0800, Julien Rouhaud wrote > in > > On Thu, Jan 27, 2022 at 11:43:26AM +0100, Sergey Dudoladov wrote: > > > > > > Per agreement with Rafia I have reworked the patch in the past days.

Re: Add connection active, idle time to pg_stat_activity

2022-01-27 Thread Kyotaro Horiguchi
At Mon, 29 Nov 2021 20:34:14 +0530, Kuntal Ghosh wrote in > active_time. But, I'm wondering why you need to distinguish between > idle and idle in transactions - what's the usage? Either the backend > is doing some work or it sits idle. Another useful information would I believe many people

Re: Add connection active, idle time to pg_stat_activity

2022-01-27 Thread Kyotaro Horiguchi
Hi. At Thu, 27 Jan 2022 20:36:56 +0800, Julien Rouhaud wrote in > On Thu, Jan 27, 2022 at 11:43:26AM +0100, Sergey Dudoladov wrote: > > > > Per agreement with Rafia I have reworked the patch in the past days. > > The new version 6 is now ready for review. > > Great, thanks a lot Sergey! > >

RE: Support escape sequence for cluster_name in postgres_fdw.application_name

2022-01-27 Thread r.takahash...@fujitsu.com
Hi, Thank you for developing this feature. I think adding escape sequence for cluster_name is useful too. > Is the reason for 'C' in upper-case to avoid possible conflict with > 'c' of log_line_prefix? I'm not sure that preventive measure is worth > doing. Looking the escape-sequence spec

Re: GUC flags

2022-01-27 Thread Justin Pryzby
On Wed, Jan 26, 2022 at 03:29:29PM +0900, Michael Paquier wrote: > On Tue, Jan 25, 2022 at 09:44:26PM -0600, Justin Pryzby wrote: > > It seems like an arbitrary and short-sighted policy to expose a handful of > > flags in the view for the purpose of retiring ./check_guc, but not expose > > other

Re: make MaxBackends available in _PG_init

2022-01-27 Thread Michael Paquier
On Thu, Jan 27, 2022 at 10:18:15AM -0800, Nathan Bossart wrote: > On Thu, Jan 27, 2022 at 09:56:04AM +0900, Michael Paquier wrote: >> Hmm. I have been looking at this patch, and the lack of centralized >> solution that could be used for other GUCs worries me like Fujii-san, >> even if this would

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Bharath Rupireddy
On Fri, Jan 28, 2022 at 8:54 AM Julien Rouhaud wrote: > > Hi, > > On Fri, Jan 28, 2022 at 08:17:39AM +0530, Bharath Rupireddy wrote: > > > > Thanks all for the comments. pg_stat_progress_checkpoint is being > > discussed in another thread [1]. > > > > [1] > >

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Julien Rouhaud
Hi, On Fri, Jan 28, 2022 at 08:17:39AM +0530, Bharath Rupireddy wrote: > > Thanks all for the comments. pg_stat_progress_checkpoint is being > discussed in another thread [1]. > > [1] >

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-27 Thread Bharath Rupireddy
On Thu, Jan 20, 2022 at 8:30 AM Kyotaro Horiguchi wrote: > > At Thu, 20 Jan 2022 00:36:32 +, "Bossart, Nathan" > wrote in > > On 1/3/22, 5:52 PM, "Kyotaro Horiguchi" wrote: > > > It seems to me "LSN" or just "location" is more confusing or > > > mysterious than "REDO LSN" for the average

Re: substring odd behavior

2022-01-27 Thread Julien Rouhaud
Hi, On Thu, Jan 27, 2022 at 07:54:49PM -0700, David G. Johnston wrote: > On Thu, Jan 27, 2022 at 7:22 PM Regina Obe wrote: > > > Is this intentional behavior? > > > > -- I can do this > > SELECT substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); > > > > -- But can't do this gives error syntax error

Re: substring odd behavior

2022-01-27 Thread Mark Dilger
> On Jan 27, 2022, at 7:06 PM, Tom Lane wrote: > > In short: you can call substring() with the SQL syntax, which is a > special-purpose production that does not involve any schema name, > or you can call it as an ordinary function with ordinary function > notation. You can't mix pieces of

Re: substring odd behavior

2022-01-27 Thread Tom Lane
"David G. Johnston" writes: > On Thu, Jan 27, 2022 at 7:22 PM Regina Obe wrote: >> Is this intentional behavior? >> -- I can do this >> SELECT substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); >> -- But can't do this gives error syntax error at or near "from" >> SELECT pg_catalog.substring('3.2.0'

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 11:39 AM Heikki Linnakangas wrote: > Hmm, if a relation is dropped, we use plain unlink() to delete it (at > the next checkpoint). Should we use durable_unlink() there, or otherwise > arrange to fsync() the parent directory? Hm. I think the latter might be a good

Re: substring odd behavior

2022-01-27 Thread David G. Johnston
On Thu, Jan 27, 2022 at 7:22 PM Regina Obe wrote: > Is this intentional behavior? > > -- I can do this > SELECT substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); > > -- But can't do this gives error syntax error at or near "from" > SELECT pg_catalog.substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); > >

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Bharath Rupireddy
On Fri, Jan 28, 2022 at 7:30 AM Julien Rouhaud wrote: > > Hi, > > On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote: > > > > I'd like to see the PID of the triggering process, but it is really > > not a information suitable in the control file... > > Yes that's something I would

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Kyotaro Horiguchi
At Fri, 28 Jan 2022 10:00:46 +0800, Julien Rouhaud wrote in > Hi, > > On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote: > > > > I'd like to see the PID of the triggering process, but it is really > > not a information suitable in the control file... > > Yes that's something

Re: pg_log_backend_memory_contexts() and log level

2022-01-27 Thread Fujii Masao
On 2022/01/27 12:45, Fujii Masao wrote: Thanks for the review! So barring any objection, I will commit the patch and backport it to v14 where pg_log_backend_memory_contexts() is added. Pushed. Thanks! Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development

substring odd behavior

2022-01-27 Thread Regina Obe
Is this intentional behavior? -- I can do this SELECT substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); -- But can't do this gives error syntax error at or near "from" SELECT pg_catalog.substring('3.2.0' from '[0-9]*\.([0-9]*)\.'); -- but can do SELECT pg_catalog.substring('3.2.0',

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Kyotaro Horiguchi
At Fri, 28 Jan 2022 10:41:28 +0900 (JST), Kyotaro Horiguchi wrote in > Sorry, the last message lacks one citation. > > At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud wrote > in > > On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote: > > > > > > What it's showing is the

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Julien Rouhaud
Hi, On Fri, Jan 28, 2022 at 10:38:53AM +0900, Kyotaro Horiguchi wrote: > > I'd like to see the PID of the triggering process, but it is really > not a information suitable in the control file... Yes that's something I would like too. But even if the PIDs could be store, I don't think that

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 12:36 PM Andres Freund wrote: > On 2022-01-28 00:39:22 +0200, Heikki Linnakangas wrote: > > On 28/01/2022 00:11, Thomas Munro wrote: > > > ... but we still never synchronize "base/5". According to our > > > project's reading of the POSIX tea leaves we should be doing that

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Kyotaro Horiguchi
Sorry, the last message lacks one citation. At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud wrote in > On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote: > > > > What it's showing is the "currently ongoing checkpoint or last completed > > checkpoint" kind. > > Ah after double

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Kyotaro Horiguchi
At Thu, 27 Jan 2022 19:09:29 +0800, Julien Rouhaud wrote in > On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote: > > > > What it's showing is the "currently ongoing checkpoint or last completed > > checkpoint" kind. > > Ah after double checking I see it's storing the information

Re: do only critical work during single-user vacuum?

2022-01-27 Thread Justin Pryzby
On Fri, Jan 21, 2022 at 05:41:58PM -0500, John Naylor wrote: > On Wed, Jan 19, 2022 at 5:26 PM Michael Paquier wrote: > > > > Could you avoid introducing a new grammar pattern in VACUUM? Any new > > option had better be within the parenthesized part as it is extensible > > at will with its set

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-27 Thread Nathan Bossart
On Thu, Jan 27, 2022 at 08:37:37PM +0530, Bharath Rupireddy wrote: > I'm still not clear how the REDO location can be treated as a start > LSN? Can someone throw some light one what this checkpoint's REDO > location is? It's the WAL insert location at the time the checkpoint began (i.e., where

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Andres Freund
Hi, On 2022-01-28 00:39:22 +0200, Heikki Linnakangas wrote: > On 28/01/2022 00:11, Thomas Munro wrote: > > ... but we still never synchronize "base/5". According to our > > project's reading of the POSIX tea leaves we should be doing that to > > nail down the directory entry. > > Really?

Re: A qsort template

2022-01-27 Thread John Naylor
Hi, I've run a few tests to get some feel for the effects of various comparators on Datums containing int32. I've attached the full results, as well as the (messy) patch which applies on top of 0012 to run the tests. I'll excerpt some of those results as I go through them here. For now, I only

Re: A test for replay of regression tests

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 12:03 PM Andres Freund wrote: > Revert "graceful shutdown" changes for Windows, in back branches only. FTR I'm actively working on a fix for that one for master now (see that other thread where the POC survived Alexander's torture testing).

Re: Replace uses of deprecated Python module distutils.sysconfig

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 17:53:02 -0500, Tom Lane wrote: > Andres Freund writes: > > Thanks! Looks pretty good so far. Including on machines that were broken in > > take 1... > > Just about all of the buildfarm has reported in now, and it's all good. > So now we need to discuss whether we want to

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 17:51:52 -0500, Andrew Dunstan wrote: > (Not actually fairywren, but equivalent) It's hung at > src/test/recovery/t/009_twophase.pl line 84: > > > $psql_rc = $cur_primary->psql('postgres', "COMMIT PREPARED > 'xact_009_1'"); That very likely is the socket-shutdown bug

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 14:36:32 -0800, Andres Freund wrote: > > On my windows test instance where I noticed this (w10, > > msys2/ucrt), check took 516s and this test took 685s. > > Hm. That's both excruciatingly slow. Way way slower than what I see here, also > w10, msys2/ucrt. Any chance the test

Re: Replace uses of deprecated Python module distutils.sysconfig

2022-01-27 Thread Tom Lane
Andres Freund writes: > Thanks! Looks pretty good so far. Including on machines that were broken in > take 1... Just about all of the buildfarm has reported in now, and it's all good. So now we need to discuss whether we want to back-patch this. Pros: avoid configure warning now (not worth

Re: A test for replay of regression tests

2022-01-27 Thread Andrew Dunstan
On 1/27/22 15:47, Andres Freund wrote: > Hi, > > On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote: >> fairywren is not happy with the recovery tests still. > Any more details? (Not actually fairywren, but equivalent) It's hung at src/test/recovery/t/009_twophase.pl line 84: $psql_rc =

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Heikki Linnakangas
On 28/01/2022 00:11, Thomas Munro wrote: On Fri, Jan 28, 2022 at 8:12 AM Thomas Munro wrote: On Fri, Jan 28, 2022 at 6:55 AM Heikki Linnakangas wrote: I think the simplest fix is to call register_dirty_segment() from mdcreate(). As in the attached. Thoughts? +1 [Testing] Erm, so now I

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 17:16:17 -0500, Andrew Dunstan wrote: > On crake (slowish fedora 34), a normal check run took 95s, and this test > took 114s. That's roughly what I see on msys after the fix. > On my windows test instance where I noticed this (w10, > msys2/ucrt), check took 516s and this test

Re: Design of pg_stat_subscription_workers vs pgstats

2022-01-27 Thread David G. Johnston
On Thu, Jan 27, 2022 at 2:15 PM Andres Freund wrote: > Another related thing is that using a 32bit xid for allowing skipping is a > bad > idea anyway - we shouldn't adding new interfaces with xid wraparound > dangers - > it's getting more and more common to have multiple wraparounds a day. An >

Re: A test for replay of regression tests

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 11:03 AM Andres Freund wrote: > That means every single psql started by 027_stream_regress.pl's pg_regress > takes 2s. Which of course adds up... That is very surprising, thanks. Will fix. I've been experimenting with reusing psql sessions and backends for queries in

Re: A test for replay of regression tests

2022-01-27 Thread Andrew Dunstan
On 1/27/22 15:47, Andres Freund wrote: > Hi, > > On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote: >> fairywren is not happy with the recovery tests still. > Any more details? I'll go back and get some. > > >> I have noticed on a different setup that this test adds 11 minutes to the >>

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 8:12 AM Thomas Munro wrote: > On Fri, Jan 28, 2022 at 6:55 AM Heikki Linnakangas wrote: > > I think the simplest fix is to call register_dirty_segment() from > > mdcreate(). As in the attached. Thoughts? > > +1 [Testing] Erm, so now I see my new table in checkpoint's

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
On 2022-01-27 14:03:51 -0800, Andres Freund wrote: > In my msys install a normal regress run takes 57s, 027_stream_regress.pl takes > 194s. > > That means every single psql started by 027_stream_regress.pl's pg_regress > takes 2s. Which of course adds up... Oh, forgot: After adding --host to the

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 12:47:08 -0800, Andres Freund wrote: > > I have noticed on a different setup that this test adds 11 minutes to the > > runtime of the recovery tests, effectively doubling it. The doubling is > > roughly true on faster setups, too > > Does a normal regress run take roughly that

Re: warn if GUC set to an invalid shared library

2022-01-27 Thread Justin Pryzby
On Sun, Jan 09, 2022 at 11:58:18AM -0800, Maciek Sakrejda wrote: > On Sat, Jan 8, 2022 at 2:07 PM Justin Pryzby wrote: > > Unfortunately, the output for dlopen() is not portable, which (I think) > > means > > most of what I wrote can't be made to work.. Since it doesn't work to call > >

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-27 Thread Peter Geoghegan
On Thu, Jan 27, 2022 at 12:20 PM Peter Geoghegan wrote: > Both VACUUM and ANALYZE update pg_class.reltuples. But this code seems > to assume that it's only something that VACUUM can ever do. Why > wouldn't we expect a plain ANALYZE to have actually been the last > thing to update

Re: Write visibility map during CLUSTER/VACUUM FULL

2022-01-27 Thread Justin Pryzby
On Sun, Dec 26, 2021 at 08:59:31PM -0600, Justin Pryzby wrote: > Rebased on 8e1fae193864527c931a704bd7908e4fbc983f5c. > > Would someone step up to "own" this patch ? > > If not, its CF entry may need to be closed (there's no status for "needs > author"). I'm planning to close this patch until

Re: A test for replay of regression tests

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 9:27 AM Andrew Dunstan wrote: > I have noticed on a different setup that this test adds 11 minutes to > the runtime of the recovery tests, effectively doubling it. The doubling > is roughly true on faster setups, too. At least I would like a simple > way to disable the

Re: Design of pg_stat_subscription_workers vs pgstats

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 13:18:51 -0700, David G. Johnston wrote: > Repeating myself here to try and keep complaints regarding > pg_stat_subscription_worker in one place. Thanks! > This is my specific email with respect to the pg_stat_scription_workers > design. > >

Re: A test for replay of regression tests

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 15:27:17 -0500, Andrew Dunstan wrote: > fairywren is not happy with the recovery tests still. Any more details? > I have noticed on a different setup that this test adds 11 minutes to the > runtime of the recovery tests, effectively doubling it. The doubling is > roughly true

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 8:17 AM Andres Freund wrote: > On 2022-01-28 08:01:01 +1300, Thomas Munro wrote: > > It might be possible to avoid that on xfs or pretty much any other > > file system. I wasn't following this closely, but even with ext4's > > recent fast commit changes, its fsync

Re: A test for replay of regression tests

2022-01-27 Thread Andrew Dunstan
On 1/21/22 16:22, Andrew Dunstan wrote: > On 1/21/22 13:58, Thomas Munro wrote: >> On Fri, Jan 21, 2022 at 3:42 PM Thomas Munro wrote: >>> Thanks. I added that and pushed. Let's see if fairywren likes it >>> when it comes back online. >> A watched pot never boils, but I wonder why Andrew's 4

Re: Two noncritical bugs of pg_waldump

2022-01-27 Thread Nathan Bossart
On Thu, Jan 27, 2022 at 01:27:36PM +0900, Kyotaro Horiguchi wrote: > At Thu, 27 Jan 2022 13:23:06 +0900 (JST), Kyotaro Horiguchi > wrote in >> So the issue there is neither EndRecPtr and ReadRecPtr always points >> to the current read LSN. The first proposal from Nathen was to use > > Mmm.

Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-27 Thread Peter Geoghegan
Commit b07642dbcd ("Trigger autovacuum based on number of INSERTs") taught autovacuum to run in response to INSERTs, which is now typically the dominant factor that drives vacuuming for an append-only table -- a very useful feature, certainly. This is driven by the following logic from

Re: Design of pg_stat_subscription_workers vs pgstats

2022-01-27 Thread David G. Johnston
On Thu, Jan 27, 2022 at 5:08 AM Amit Kapila wrote: > On Thu, Jan 27, 2022 at 11:16 AM Andres Freund wrote: > > > > On 2022-01-25 20:27:07 +0900, Masahiko Sawada wrote: > > > > > There will be some challenges in a case where updating > pg_subscription_rel > > > also failed too (what to report to

Re: Two noncritical bugs of pg_waldump

2022-01-27 Thread Nathan Bossart
On Thu, Jan 27, 2022 at 01:23:06PM +0900, Kyotaro Horiguchi wrote: > So the issue there is neither EndRecPtr and ReadRecPtr always points > to the current read LSN. The first proposal from Nathen was to use > currRecPtr but it was a private member. But after discussion, it > seems to me it is (at

Re: pgsql: Server-side gzip compression.

2022-01-27 Thread Robert Haas
On Tue, Jan 25, 2022 at 3:56 PM David Rowley wrote: > On Wed, 26 Jan 2022 at 07:12, Robert Haas wrote: > > wouldn't this same consideration apply to a very large number of other > > places in the code base? > > All of the other places are handled. See locations with "keep compiler quiet". > >

Re: Bug in ProcArrayApplyRecoveryInfo for snapshots crossing 4B, breaking replicas

2022-01-27 Thread Tomas Vondra
On 1/26/22 23:54, Michael Paquier wrote: On Wed, Jan 26, 2022 at 07:31:00PM +0100, Tomas Vondra wrote: I actually tried doing that, but I was not very happy with the result. The test has to call pg_resetwal, but then it also has to fake pg_xact data and so on, which seemed a bit ugly so did not

Unlogged relations and WAL-logging

2022-01-27 Thread Heikki Linnakangas
Unlogged relations are not WAL-logged, but creating the init-fork is. There are a few things around that seem sloppy: 1. In index_build(), we do this: /* * If this is an unlogged index, we may need to write out an init fork for * it -- but we must first check

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Andres Freund
Hi, On 2022-01-28 08:01:01 +1300, Thomas Munro wrote: > On Fri, Jan 28, 2022 at 7:28 AM Andres Freund wrote: > > On 2022-01-27 19:55:45 +0200, Heikki Linnakangas wrote: > > > I was not able to reproduce this without the tablespace on a different > > > virtual disk, I presume because ext4 orders

Re: refactoring basebackup.c

2022-01-27 Thread Robert Haas
On Thu, Jan 27, 2022 at 2:37 AM Dipesh Pandit wrote: > I have updated the patches to support server compression (gzip) for > plain format backup. Please find attached v4 patches. I made a pass over these patches today and made a bunch of minor corrections. New version attached. The two biggest

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 6:55 AM Heikki Linnakangas wrote: > I think the simplest fix is to call register_dirty_segment() from > mdcreate(). As in the attached. Thoughts? +1

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Thomas Munro
On Fri, Jan 28, 2022 at 7:28 AM Andres Freund wrote: > On 2022-01-27 19:55:45 +0200, Heikki Linnakangas wrote: > > I was not able to reproduce this without the tablespace on a different > > virtual disk, I presume because ext4 orders the writes so that the > > checkpoint implicitly always flushes

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-01-27 Thread Nathan Bossart
On Thu, Jan 27, 2022 at 02:06:40PM +0900, Michael Paquier wrote: > So, I have been checking this idea in details, and spotted what looks > like one issue in CreateRestartPoint(), as of: > /* >* Update pg_control, using current time. Check that it still shows >*

Re: Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Andres Freund
Hi, On 2022-01-27 19:55:45 +0200, Heikki Linnakangas wrote: > If you create an empty table, it is not fsync'd. As soon as you insert a row > to it, register_dirty_segment() gets called, and after that, the next > checkpoint will fsync it. But before that, the creation itself is never > fsync'd.

Re: make MaxBackends available in _PG_init

2022-01-27 Thread Nathan Bossart
On Thu, Jan 27, 2022 at 09:56:04AM +0900, Michael Paquier wrote: > Hmm. I have been looking at this patch, and the lack of centralized > solution that could be used for other GUCs worries me like Fujii-san, > even if this would prevent an incorrect use of MaxBackends in contexts > where it should

Creation of an empty table is not fsync'd at checkpoint

2022-01-27 Thread Heikki Linnakangas
If you create an empty table, it is not fsync'd. As soon as you insert a row to it, register_dirty_segment() gets called, and after that, the next checkpoint will fsync it. But before that, the creation itself is never fsync'd. That's obviously not great. The lack of an fsync is a bit hard to

Re: refactoring basebackup.c

2022-01-27 Thread Robert Haas
On Thu, Jan 27, 2022 at 12:08 PM tushar wrote: > On 1/27/22 10:17 PM, Robert Haas wrote: > > Cool. I committed that patch. > Thanks , Please refer to this scenario where the label is set to 0 for > server-gzip but the directory is still compressed > > [edb@centos7tushar bin]$ ./pg_basebackup

Re: refactoring basebackup.c

2022-01-27 Thread tushar
On 1/27/22 10:17 PM, Robert Haas wrote: Cool. I committed that patch. Thanks , Please refer to this scenario  where the label is set to  0 for server-gzip but the directory is still  compressed [edb@centos7tushar bin]$ ./pg_basebackup -t server:/tmp/11 --gzip --compress=0 -Xnone NOTICE: 

Re: refactoring basebackup.c

2022-01-27 Thread Robert Haas
On Thu, Jan 27, 2022 at 7:15 AM tushar wrote: > On 1/27/22 2:15 AM, Robert Haas wrote: > > The attached patch should fix this, too. > Thanks, the issues seem to be fixed now. Cool. I committed that patch. -- Robert Haas EDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences

2022-01-27 Thread Peter Eisentraut
On 27.01.22 00:32, Tomas Vondra wrote: On 1/26/22 14:01, Petr Jelinek wrote: I would not remove it altogether, there is plenty of consumers of this extension's output in the wild (even if I think it's unfortunate) that might not be interested in sequences, but changing it to off by default

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2022-01-27 Thread Bharath Rupireddy
On Thu, Jan 20, 2022 at 6:06 AM Bossart, Nathan wrote: > > On 1/3/22, 5:52 PM, "Kyotaro Horiguchi" wrote: > > It seems to me "LSN" or just "location" is more confusing or > > mysterious than "REDO LSN" for the average user. If we want to avoid > > being technically too detailed, we would use

Re: snapper and skink and fairywren (oh my!)

2022-01-27 Thread Andrew Dunstan
On 1/26/22 18:45, Andrew Dunstan wrote: > On 1/26/22 16:17, Robert Haas wrote >> 3. fairywren failed the last run in module-commit_tsCheck. It's unhappy >> because: >> > Intermittent failures give a false positive against the latest set of > commits. These failures started happening regularly

Re: Proposal: More structured logging

2022-01-27 Thread Ronan Dunklau
Le jeudi 27 janvier 2022, 08:15:01 CET Michael Paquier a écrit : > On Tue, Jan 18, 2022 at 06:46:03AM +0100, Ronan Dunklau wrote: > > Hum, there was a missing import in csvlog.c from the fix above. Sorry > > about > > that. > > + > > > You are also forgetting that the table listing all the

Re: generalized conveyor belt storage

2022-01-27 Thread Amul Sul
On Wed, Jan 5, 2022 at 1:12 AM Robert Haas wrote: > > On Wed, Dec 29, 2021 at 7:08 AM Amul Sul wrote: > > Thought patch is WIP, here are a few comments that I found while > > reading the patch and thought might help: > > > > + { > > + if (meta->cbm_oldest_index_segment == > > +

Re: Skipping logical replication transactions on subscriber side

2022-01-27 Thread Peter Eisentraut
On 26.01.22 05:05, Masahiko Sawada wrote: I think it is okay to clear after the first successful application of any transaction. What I was not sure was about the idea of giving WARNING/ERROR if the first xact to be applied is not the same as skip_xid. Do you prefer not to do anything in this

Re: BRIN summarization vs. WAL logging

2022-01-27 Thread Tomas Vondra
On 1/26/22 19:14, Alvaro Herrera wrote: On 2022-Jan-26, Robert Haas wrote: On Tue, Jan 25, 2022 at 10:12 PM Tomas Vondra wrote: 2) brin_summarize_range() Now, the issue I think is more serious, more likely to happen, and harder to fix. When summarizing a range, we write two WAL

Re: psql - add SHOW_ALL_RESULTS option

2022-01-27 Thread Peter Eisentraut
On 23.01.22 18:17, Fabien COELHO wrote: But I think if we are adding a libpq API function to work around a misbehavior in libpq, we might as well fix the misbehavior in libpq to begin with. Adding a new public libpq function is a significant step, needs documentation, etc. I'm not so sure.

Re: Output clause for Upsert aka INSERT...ON CONFLICT

2022-01-27 Thread David G. Johnston
On Thursday, January 27, 2022, Anand Sowmithiran wrote: > > However, the MS SQL server MERGE command also does 'delete' using the > 'when not matched' clause, is there an equivalent ? > PostgreSQL does not have a merge command feature. Just the subset of behavior that is INSERT…on conflict

Re: logical replication empty transactions

2022-01-27 Thread Ajin Cherian
On Thu, Jan 27, 2022 at 12:16 AM osumi.takami...@fujitsu.com wrote: > > On Tuesday, January 11, 2022 6:43 PM From: Ajin Cherian > wrote: > > Minor update to rebase the patch so that it applies clean on HEAD. > Hi, let me share some additional comments on v16. > > > (1) comment of

Re: logical replication empty transactions

2022-01-27 Thread Ajin Cherian
On Wed, Jan 26, 2022 at 8:33 PM osumi.takami...@fujitsu.com wrote: > > On Tuesday, January 11, 2022 6:43 PM Ajin Cherian wrote: > > Minor update to rebase the patch so that it applies clean on HEAD. > Hi, thanks for you rebase. > > Several comments. > > (1) the commit message > > " >

Re: Add connection active, idle time to pg_stat_activity

2022-01-27 Thread Julien Rouhaud
Hi, On Thu, Jan 27, 2022 at 11:43:26AM +0100, Sergey Dudoladov wrote: > > Per agreement with Rafia I have reworked the patch in the past days. > The new version 6 is now ready for review. Great, thanks a lot Sergey! The cfbot is happy with this new version:

Re: Fix uninitialized variable access (src/backend/utils/mmgr/freepage.c)

2022-01-27 Thread Greg Nancarrow
On Thu, Jan 27, 2022 at 6:32 PM Michael Paquier wrote: > > On Fri, Oct 01, 2021 at 05:03:04PM -0300, Ranier Vilela wrote: > > For me the assertion remains valid and usable. > > Well, I was looking at this thread again, and I still don't see what > we benefit from this change. One thing that

Re: refactoring basebackup.c

2022-01-27 Thread tushar
On 1/27/22 2:15 AM, Robert Haas wrote: The attached patch should fix this, too. Thanks, the issues seem to be fixed now. -- regards,tushar EnterpriseDB https://www.enterprisedb.com/ The Enterprise PostgreSQL Company

Re: Design of pg_stat_subscription_workers vs pgstats

2022-01-27 Thread Amit Kapila
On Thu, Jan 27, 2022 at 11:16 AM Andres Freund wrote: > > On 2022-01-25 20:27:07 +0900, Masahiko Sawada wrote: > > > There will be some challenges in a case where updating pg_subscription_rel > > also failed too (what to report to the user, etc.). And moreover, we don't > > want to consume space

psql: Rename results to result when only a single one is meant

2022-01-27 Thread Peter Eisentraut
While reviewing code related to supporting multiple result sets in psql, it is always confusing that in psql many variables of type PGresult* are named "results" (plural), as if there could be multiple. While it is ok in casual talk to consider a return from a query to be a bunch of stuff,

Re: RFC: Logging plan of the running query

2022-01-27 Thread Fujii Masao
On 2022/01/24 14:33, torikoshia wrote: As mentioned above, I updated the patch. Thanks for updating the patch! Here are another review comments: +LOG: plan of the query running on backend with PID 17793 is: This seems not the same as what actually logged. +

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Julien Rouhaud
On Thu, Jan 27, 2022 at 06:56:57PM +0800, Julien Rouhaud wrote: > > What it's showing is the "currently ongoing checkpoint or last completed > checkpoint" kind. Ah after double checking I see it's storing the information *after* the checkpoint completion, so it's indeed the last completed

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Julien Rouhaud
Hi, On Thu, Jan 27, 2022 at 10:53:32AM +0100, Sergey Dudoladov wrote: > Hi all, > > > Here's v2, rebased onto the latest master. > > I've reviewed this patch. The patch builds against the master (commit > e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests. > The patch does what

Suppressing useless wakeups in walreceiver

2022-01-27 Thread Thomas Munro
Hi, While working on WaitEventSet-ifying various codepaths, I found it strange that walreceiver wakes up 10 times per second while idle. Here's a draft patch to compute the correct sleep time. From 553d2dae8f8e7bb46ac73ca739aac03862f473cc Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 27

Re: Add connection active, idle time to pg_stat_activity

2022-01-27 Thread Sergey Dudoladov
Hello, > Without update in the next few > days this patch will be closed as Returned with Feedback, Thank you for the reminder, Julien. Per agreement with Rafia I have reworked the patch in the past days. The new version 6 is now ready for review. Regards, Sergey Dudoladov diff --git

Re: Support escape sequence for cluster_name in postgres_fdw.application_name

2022-01-27 Thread Fujii Masao
On 2022/01/27 17:10, Kyotaro Horiguchi wrote: At Tue, 25 Jan 2022 16:02:39 +0900, Fujii Masao wrote in Hi, Commit 6e0cb3dec1 allowed postgres_fdw.application_name to include escape sequences %a (application name), %d (database name), %u (user name) and %p (pid). In addition to them, I'd

Re: Is there a way (except from server logs) to know the kind of on-going/last checkpoint?

2022-01-27 Thread Sergey Dudoladov
Hi all, > Here's v2, rebased onto the latest master. I've reviewed this patch. The patch builds against the master (commit e9d4001ec592bcc9a3332547cb1b0211e8794f38) and passes all the tests. The patch does what it intends to do, namely store the kind of the last checkpoint in the control file

Re: libpq async duplicate error results

2022-01-27 Thread Peter Eisentraut
On 26.01.22 14:52, Fabien COELHO wrote: command = SELECT pg_terminate_backend(pg_backend_pid()); result 1 status = PGRES_FATAL_ERROR error message = "FATAL:  terminating connection due to administrator command " result 2 status = PGRES_FATAL_ERROR error message = "FATAL:  terminating

  1   2   >