Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Masahiro Ikeda
On 2021/03/23 11:40, Fujii Masao wrote: > > > On 2021/03/23 9:05, Masahiro Ikeda wrote: >> Yes. I attached the v5 patch based on v3 patch. >> I renamed SignalHandlerForUnsafeExit() and fixed the following comment. > > Thanks for updating the patch! > > When the startup process exits because

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-22 Thread Fujii Masao
On 2021/03/23 10:52, Thomas Munro wrote: On Tue, Mar 23, 2021 at 2:44 PM Fujii Masao wrote: I found 0001 patch was committed in de829ddf23, and which added new wait event WalrcvExit. This name seems not consistent with other wait events. I'm thinking it's better to rename it to

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread Fujii Masao
On 2021/03/23 3:59, James Coleman wrote: Are you saying we should only change the message for a single case: the case where we'd otherwise allow connections but EnableHotStandby is false? No. Let me clarify my opinion. At PM_STARTUP, "the database system is starting up" should be logged

Re: a verbose option for autovacuum

2021-03-22 Thread Masahiko Sawada
On Tue, Mar 23, 2021 at 1:31 PM Michael Paquier wrote: > > On Mon, Mar 22, 2021 at 12:17:37PM +0900, Masahiko Sawada wrote: > > I've updated the patch. I saved the index names at the beginning of > > heap_vacuum_rel() for autovacuum logging, and add indstats and > > nindexes to LVRelStats. Some

Re: Change default of checkpoint_completion_target

2021-03-22 Thread Michael Paquier
On Mon, Mar 22, 2021 at 01:11:00PM -0400, Stephen Frost wrote: > Unless there's anything further on this, I'll plan to commit it tomorrow > or Wednesday. Cool, looks fine to me. This version of the patch has forgotten to update one spot: src/backend/postmaster/checkpointer.c:double

Re: proposal - psql - use pager for \watch command

2021-03-22 Thread Pavel Stehule
út 23. 3. 2021 v 0:35 odesílatel Thomas Munro napsal: > On Sun, Mar 21, 2021 at 11:43 PM Pavel Stehule > wrote: > > so 20. 3. 2021 v 23:45 odesílatel Thomas Munro > napsal: > >> Thoughts? I put my changes into a separate patch for clarity, but > >> they need some more tidying up. > > > > yes,

Re: Proposal: Save user's original authenticated identity for logging

2021-03-22 Thread Michael Paquier
On Mon, Mar 22, 2021 at 06:22:52PM +0100, Magnus Hagander wrote: > The 0002/0001/whateveritisaftertherebase is tracked over at > https://www.postgresql.org/message-id/flat/92e70110-9273-d93c-5913-0bccb6562...@dunslane.net > isn't it? I've assumed the expectation is to have that one committed >

Re: replication cleanup code incorrect way to use of HTAB HASH_REMOVE ?

2021-03-22 Thread Amit Kapila
On Mon, Mar 22, 2021 at 7:57 AM Amit Kapila wrote: > > On Mon, Mar 22, 2021 at 3:20 AM Peter Smith wrote: > > > > On Sun, Mar 21, 2021 at 8:54 PM Amit Kapila wrote: > > > > > > On Sat, Mar 20, 2021 at 12:54 PM Peter Smith > > > wrote: > > > > > > > > PSA my patch to correct this by firstly

Re: Proposal: Save user's original authenticated identity for logging

2021-03-22 Thread Michael Paquier
On Mon, Mar 22, 2021 at 07:17:26PM +, Jacob Champion wrote: > v8's test_access lost the in-order log search from v7; I've added it > back in v9. The increased resistance to entropy seems worth the few > extra lines. Thoughts? I am not really sure that we need to bother about the ordering of

Re: [CLOBBER_CACHE]Server crashed with segfault 11 while executing clusterdb

2021-03-22 Thread Neha Sharma
On Tue, Mar 23, 2021 at 10:08 AM Amul Sul wrote: > On Mon, Mar 22, 2021 at 3:03 PM Amit Langote > wrote: > > > > On Mon, Mar 22, 2021 at 5:26 PM Amul Sul wrote: > > > In heapam_relation_copy_for_cluster(), begin_heap_rewrite() sets > > > rwstate->rs_new_rel->rd_smgr correctly but next line >

Re: Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tom Lane
Tomas Vondra writes: > On 3/23/21 3:13 AM, Tom Lane wrote: >> Hm. Both catalogs.sgml and pg_opclass.h say specifically that >> opckeytype should be zero if it's to be the same as the input >> column type. I don't think just dropping the enforcement of that >> is the right answer. > Yeah,

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > > with data_dest_cb callback. It is used for send text representation of a > > tuple to a custom destination. > > > > send text -> sending text > > I would say "It is used to send the text representation ..." I

Re: Type of wait events WalReceiverWaitStart and WalSenderWaitForWAL

2021-03-22 Thread Kyotaro Horiguchi
(I finally get to catch up here..) At Mon, 22 Mar 2021 13:59:47 +0900, Fujii Masao wrote in > > > On 2021/03/22 12:01, Kyotaro Horiguchi wrote: > > Mmm. I agree that it waits for WAL in most cases, but still WAL-wait > > is activity for me because it is not waiting for being cued by > >

Re: [CLOBBER_CACHE]Server crashed with segfault 11 while executing clusterdb

2021-03-22 Thread Amul Sul
On Mon, Mar 22, 2021 at 3:03 PM Amit Langote wrote: > > On Mon, Mar 22, 2021 at 5:26 PM Amul Sul wrote: > > In heapam_relation_copy_for_cluster(), begin_heap_rewrite() sets > > rwstate->rs_new_rel->rd_smgr correctly but next line > > tuplesort_begin_cluster() > > get called which cause the

Re: a verbose option for autovacuum

2021-03-22 Thread Michael Paquier
On Mon, Mar 22, 2021 at 12:17:37PM +0900, Masahiko Sawada wrote: > I've updated the patch. I saved the index names at the beginning of > heap_vacuum_rel() for autovacuum logging, and add indstats and > nindexes to LVRelStats. Some functions still have nindexes as a > function argument but it seems

Re: proposal - psql - use pager for \watch command

2021-03-22 Thread Pavel Stehule
po 22. 3. 2021 v 22:07 odesílatel Thomas Munro napsal: > On Tue, Mar 23, 2021 at 1:53 AM Pavel Stehule > wrote: > > po 22. 3. 2021 v 13:13 odesílatel Thomas Munro > napsal: > >> The problem is that Apple's /dev/tty device is defective, and doesn't > >> work in poll(). It always returns

Re: UPDATE ... SET (single_column) = row_constructor is a bit broken from V10 906bfcad7ba7c

2021-03-22 Thread Mahendra Singh Thalor
On Tue, 23 Mar 2021 at 02:43, Justin Pryzby wrote: > > On Mon, Mar 22, 2021 at 02:10:49PM +0530, Mahendra Singh Thalor wrote: > > Hi Hackers, > > > > Commit 906bfcad7ba7c has improved handling for "UPDATE ... SET > > (column_list) = row_constructor", but it has broken in some cases where it > >

Re: Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tomas Vondra
On 3/23/21 3:13 AM, Tom Lane wrote: > Tomas Vondra writes: >> while working on the new BRIN opclasses in [1], I stumbled on something >> I think is actually a bug in how CREATE OPERATOR CLASS handles the >> storage type. > > Hm. Both catalogs.sgml and pg_opclass.h say specifically that >

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Peter Geoghegan
On Mon, Mar 22, 2021 at 8:33 PM Peter Geoghegan wrote: > More concretely, maybe the new GUC is forced to be 1.05 of > vacuum_freeze_table_age. Of course that scheme is a bit arbitrary -- > but so is the existing 0.95 scheme. I meant to write 1.05 of autovacuum_vacuum_max_age. So just as

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Peter Geoghegan
On Mon, Mar 22, 2021 at 8:28 PM Peter Geoghegan wrote: > You seem to be concerned about a similar contradiction. In fact it's > *very* similar contradiction, because this new GUC is naturally a > "sibling GUC" of both vacuum_freeze_table_age and > autovacuum_vacuum_max_age (the "units" are the

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 13:22 -0400, Stephen Frost wrote: > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > > > Perhaps allowing to set unlogged tables to logged ones without writing > > > > WAL > > > > is a more elegant way to do

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Peter Geoghegan
On Mon, Mar 22, 2021 at 6:41 PM Masahiko Sawada wrote: > But we're not sure when the next anti-wraparound vacuum will take > place. Since the table is already vacuumed by a non-aggressive vacuum > with disabling index cleanup, an autovacuum will process the table > when the table gets modified

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > with data_dest_cb callback. It is used for send text representation of a > tuple to a custom destination. > > send text -> sending text I would say "It is used to send the text representation ..." > struct PgFdwModifyState

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-22 Thread Greg Stark
On Sun, 21 Mar 2021 at 18:16, Stephen Frost wrote: > > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > I also believe that the snapshotting behavior has advantages in terms > > of being able to perform multiple successive queries and get consistent > > results from them. Only the most

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Zhihong Yu
Hi, In the description: with data_dest_cb callback. It is used for send text representation of a tuple to a custom destination. send text -> sending text struct PgFdwModifyState *aux_fmstate; /* foreign-insert state, if * created */ +

RE: Disable WAL logging to speed up data loading

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > First- what are you expecting would actually happen during crash recovery in > this specific case with your proposed new WAL level? ... > I'm not suggesting it's somehow more crash safe- but it's at least very clear > what happens in such a case, to wit: the entire table is

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Fujii Masao
On 2021/03/23 9:05, Masahiro Ikeda wrote: Yes. I attached the v5 patch based on v3 patch. I renamed SignalHandlerForUnsafeExit() and fixed the following comment. Thanks for updating the patch! When the startup process exits because of recovery_target_action=shutdown, reaper() calls

Re: shared-memory based stats collector

2021-03-22 Thread Kyotaro Horiguchi
At Fri, 19 Mar 2021 14:27:38 -0700, Andres Freund wrote in > Hi, > > On 2021-03-10 20:26:56 -0800, Andres Freund wrote: > > > + * We expose this shared entry now. You might think that the > > > entry > > > + * can be removed by a concurrent backend, but since we are > > >

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Mon, Mar 22, 2021 at 08:38:37PM -0400, Bruce Momjian wrote: > > This particular patch (introducing the RelationIsPermanent() macro) > > seems like it'd be a nice thing to commit independently of the rest, > > reducing the size of this patch set..? > > Committed as suggested. Also, I have

Re: Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tom Lane
Tomas Vondra writes: > while working on the new BRIN opclasses in [1], I stumbled on something > I think is actually a bug in how CREATE OPERATOR CLASS handles the > storage type. Hm. Both catalogs.sgml and pg_opclass.h say specifically that opckeytype should be zero if it's to be the same as

RE: [PATCH] Tracking statements entry timestamp in pg_stat_statements

2021-03-22 Thread kuroda.hay...@fujitsu.com
Dear Andrei, I think the idea is good because the pg_stat_statements_info view cannot distinguish whether the specific statement is deallocated or not. But multiple calling of GetCurrentTimestamp() may cause some performance issues. How about adding a configuration parameter for controlling this

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Macros _() at the postgresExecForeignCopy routine: > if (PQputCopyEnd(conn, OK ? NULL : _("canceled by server")) <= 0) > > uses gettext. Under linux it is compiled ok, because (as i understood) > uses standard implementation of gettext: > objdump -t

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-22 Thread Thomas Munro
On Tue, Mar 23, 2021 at 2:44 PM Fujii Masao wrote: > I found 0001 patch was committed in de829ddf23, and which added new > wait event WalrcvExit. This name seems not consistent with other wait > events. I'm thinking it's better to rename it to WalReceiverExit. Thought? > Patch attached. Agreed,

Re: PostmasterIsAlive() in recovery (non-USE_POST_MASTER_DEATH_SIGNAL builds)

2021-03-22 Thread Fujii Masao
On 2021/03/02 10:10, Thomas Munro wrote: On Tue, Mar 2, 2021 at 12:00 AM Thomas Munro wrote: On Mon, Nov 16, 2020 at 8:56 PM Michael Paquier wrote: No objections with the two changes from pg_usleep() to WaitLatch() so they could be applied separately first. I thought about committing

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Masahiko Sawada
On Fri, Mar 19, 2021 at 3:36 AM Peter Geoghegan wrote: > > On Thu, Mar 18, 2021 at 3:32 AM Masahiko Sawada wrote: > > If we have the constant threshold of 1 billion transactions, a vacuum > > operation might not be an anti-wraparound vacuum and even not be an > > aggressive vacuum, depending on

Handling of opckeytype / CREATE OPERATOR CLASS (bug?)

2021-03-22 Thread Tomas Vondra
Hi, while working on the new BRIN opclasses in [1], I stumbled on something I think is actually a bug in how CREATE OPERATOR CLASS handles the storage type. If you look at built-in opclasses, there's a bunch of cases where (opcintype == opckeytype), for example the BRIN opclasses for inet data

Re: Type of wait events WalReceiverWaitStart and WalSenderWaitForWAL

2021-03-22 Thread Fujii Masao
On 2021/03/22 13:59, Fujii Masao wrote: Ok, so barring any objection, I will commit the patch that I posted upthread. Pushed! I'm waiting for other two patches to be reviewed :) Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA

Re: Wrong statistics for size of XLOG_SWITCH during pg_waldump.

2021-03-22 Thread Fujii Masao
On 2021/03/22 17:49, Kyotaro Horiguchi wrote: At Mon, 22 Mar 2021 14:07:43 +0900, Fujii Masao wrote in On 2021/03/22 14:03, shinya11.k...@nttdata.com wrote: Barring any objection, I will commit this. I think it's good except for a typo "thoes four bits" Thanks for the review!

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-22 Thread Bruce Momjian
On Mon, Mar 22, 2021 at 05:17:15PM -0700, Zhihong Yu wrote: > Hi, > For queryjumble.c : > > + * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group > > The year should be updated. > Same with queryjumble.h Thanks, fixed. -- Bruce Momjian https://momjian.us EDB

Re: create table like: ACCESS METHOD

2021-03-22 Thread Justin Pryzby
On Tue, Jan 19, 2021 at 03:03:31PM -0600, Justin Pryzby wrote: > On Wed, Dec 30, 2020 at 12:33:56PM +, Simon Riggs wrote: > > There are no tests for the new functionality, please could you add some? > > Did you look at the most recent patch? > > +CREATE ACCESS METHOD heapdup TYPE TABLE

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote: > > src/backend/access/gist/gistutil.c | 2 +- > > src/backend/access/heap/heapam_handler.c | 2 +- > > src/backend/catalog/pg_publication.c | 2 +- > > src/backend/commands/tablecmds.c | 10 +- > >

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-22 16:17:37 -0700, Andres Freund wrote: > and to reduce unnecessary diff noise This patch has just tons of stuff like: -/* - * Calculate function call usage and update stat counters. - * Called by the executor after invoking a function. +/* -- + *

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-22 Thread Zhihong Yu
Hi, For queryjumble.c : + * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group The year should be updated. Same with queryjumble.h Cheers On Mon, Mar 22, 2021 at 2:56 PM Bruce Momjian wrote: > On Sat, Mar 20, 2021 at 02:12:34PM +0800, Julien Rouhaud wrote: > > On Fri, Mar

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Masahiro Ikeda
On 2021/03/22 23:59, Fujii Masao wrote: > > > On 2021/03/20 13:40, Masahiro Ikeda wrote: >> Sorry, there is no evidence we should call it. >> I thought that to call FreeWaitEventSet() is a manner after using >> CreateWaitEventSet() and the performance impact to call it seems to be too >> small.

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Tom Lane
Justin Pryzby writes: > On Mon, Mar 22, 2021 at 03:47:58PM -0400, Robert Haas wrote: >> Are you sure you're looking at the patch I sent, >> toast-compression-guc-rmh.patch? I can't help wondering if you applied >> it to a dirty source tree or got the wrong file or something, because >> otherwise

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-22 Thread Peter Smith
On Mon, Mar 22, 2021 at 11:51 PM Amit Kapila wrote: > > I have incorporated all your changes and additionally made few more > changes (a) got rid of LogicalRepBeginPrepareData and instead used > LogicalRepPreparedTxnData, (b) made a number of changes in comments > and docs, (c) ran pgindent, (d)

Re: proposal - psql - use pager for \watch command

2021-03-22 Thread Thomas Munro
On Sun, Mar 21, 2021 at 11:43 PM Pavel Stehule wrote: > so 20. 3. 2021 v 23:45 odesílatel Thomas Munro > napsal: >> Thoughts? I put my changes into a separate patch for clarity, but >> they need some more tidying up. > > yes, your solution is much better. Hmm, there was a problem with it

Re: pg_upgrade failing for 200+ million Large Objects

2021-03-22 Thread Jan Wieck
On 3/22/21 5:36 PM, Zhihong Yu wrote: Hi, w.r.t. pg_upgrade_improvements.v2.diff. +       blobBatchCount = 0; +       blobInXact = false; The count and bool flag are always reset in tandem. It seems variable blobInXact is not needed. You are right. I will fix that. Thanks, Jan --

Re: Allow an alias to be attached directly to a JOIN ... USING

2021-03-22 Thread Tom Lane
Peter Eisentraut writes: > I think Tom's input on the guts of this patch would be most valuable, > since it intersects a lot with the parse namespace refactoring he did. I really didn't like the way you'd done that :-(. My primary complaint is that any one ParseNamespaceItem can describe only

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-22 12:02:39 +0900, Kyotaro Horiguchi wrote: > At Mon, 22 Mar 2021 09:55:59 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Thu, 18 Mar 2021 01:47:20 -0700, Andres Freund > > wrote in > > > Hi, > > > > > > On 2021-03-18 16:56:02 +0900, Kyotaro Horiguchi wrote: > > > > At Tue,

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 03:47:58PM -0400, Robert Haas wrote: > On Mon, Mar 22, 2021 at 2:10 PM Tom Lane wrote: > > Robert Haas writes: > > > I think this is significantly cleaner than what we have now, and I > > > also prefer it to your proposal. > > > > +1 in general. However, I suspect that

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-22 Thread Bruce Momjian
On Sat, Mar 20, 2021 at 02:12:34PM +0800, Julien Rouhaud wrote: > On Fri, Mar 19, 2021 at 12:53:18PM -0400, Bruce Momjian wrote: > > > > Well, given we don't really want to support multiple query id types > > being generated or displayed, the "error out" above should fix it. > > > > Let's do

Re: pg_upgrade failing for 200+ million Large Objects

2021-03-22 Thread Zhihong Yu
> > Hi, > w.r.t. pg_upgrade_improvements.v2.diff. + blobBatchCount = 0; + blobInXact = false; The count and bool flag are always reset in tandem. It seems variable blobInXact is not needed. Cheers

Re: UPDATE ... SET (single_column) = row_constructor is a bit broken from V10 906bfcad7ba7c

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 02:10:49PM +0530, Mahendra Singh Thalor wrote: > Hi Hackers, > > Commit 906bfcad7ba7c has improved handling for "UPDATE ... SET > (column_list) = row_constructor", but it has broken in some cases where it > was working prior to this commit. > After this commit query “DO

Re: proposal - psql - use pager for \watch command

2021-03-22 Thread Thomas Munro
On Tue, Mar 23, 2021 at 1:53 AM Pavel Stehule wrote: > po 22. 3. 2021 v 13:13 odesílatel Thomas Munro > napsal: >> The problem is that Apple's /dev/tty device is defective, and doesn't >> work in poll(). It always returns immediately with revents=POLLNVAL, >> but pspg assumes that data is

Re: Since '2001-09-09 01:46:40'::timestamp microseconds are lost when extracting epoch

2021-03-22 Thread Peter Eisentraut
On 19.03.21 21:06, Tom Lane wrote: I guess the immediate question is how much of a performance gap there is now between the float and numeric implementations. Attached are my test script and the full output. To summarize, for cases that don't do any interesting computation and where the

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 4:33 PM Robert Haas wrote: > On Mon, Mar 22, 2021 at 1:58 PM Justin Pryzby wrote: > > guc.c should not longer define this as extern: > > default_toast_compression_options > > Fixed. Fixed some more. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 1:58 PM Justin Pryzby wrote: > guc.c should not longer define this as extern: > default_toast_compression_options Fixed. > I think you should comment that default_toast_compression is an int as far as > guc.c is concerned, but storing one of the char value of

Re: postgres_fdw: IMPORT FOREIGN SCHEMA ... LIMIT TO (partition)

2021-03-22 Thread Bernd Helmle
Am Donnerstag, dem 21.01.2021 um 15:56 +0100 schrieb Matthias van de Meent: > Hi, > > Recently I was trying to copy some of the data of one database to > another through postgres_fdw, and found that it wouldn't import that > partition through IMPORT FOREIGN SCHEMA, even when I explicitly >

Re: Autovacuum worker doesn't immediately exit on postmaster death

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 1:48 PM Stephen Frost wrote: > Thanks for that. Attached is just a rebased version with a commit > message added. If there aren't any other concerns, I'll commit this in > the next few days and back-patch it. When it comes to 12 and older, > does anyone want to opine

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 2:10 PM Tom Lane wrote: > Robert Haas writes: > > I think this is significantly cleaner than what we have now, and I > > also prefer it to your proposal. > > +1 in general. However, I suspect that you did not try to compile > this without --with-lz4, because if you had

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-19 14:27:38 -0700, Andres Freund wrote: > Yep, it's not even particularly hard to hit: > > S0: CREATE TABLE a_table(); > S0: INSERT INTO a_table(); > S0: disconnect > S1: set a breakpoint to just after the dshash_release_lock(), with an > if objid == a_table_oid > S1: SELECT

Re: Proposal: Save user's original authenticated identity for logging

2021-03-22 Thread Jacob Champion
On Mon, 2021-03-22 at 15:16 +0900, Michael Paquier wrote: > On Fri, Mar 19, 2021 at 06:37:05PM +, Jacob Champion wrote: > > The same effect can be had by moving the log rotation to the top of the > > test that needs it, so I've done it that way in v7. > > After thinking more about 0001, I

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread James Coleman
On Mon, Mar 22, 2021 at 2:52 PM Fujii Masao wrote: > > > > On 2021/03/23 3:25, James Coleman wrote: > > On Mon, Mar 22, 2021 at 1:49 PM Fujii Masao > > wrote: > >> > >> > >> > >> On 2021/03/19 23:35, James Coleman wrote: > >>> Here's an updated patch; I think I've gotten what Alvaro suggested.

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread Fujii Masao
On 2021/03/23 3:25, James Coleman wrote: On Mon, Mar 22, 2021 at 1:49 PM Fujii Masao wrote: On 2021/03/19 23:35, James Coleman wrote: Here's an updated patch; I think I've gotten what Alvaro suggested. Thanks for updating the patch! But I was thinking that our consensus is something

Re: Proposal: Save user's original authenticated identity for logging

2021-03-22 Thread Jacob Champion
On Mon, 2021-03-22 at 18:22 +0100, Magnus Hagander wrote: > On Mon, Mar 22, 2021 at 7:16 AM Michael Paquier wrote: > > > > I have briefly looked at 0002 (0001 in the attached set), and it seems > > sane to me. I still need to look at 0003 (well, now 0002) in details, > > which is very sensible

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Andrey Lepikhov
On 5/3/21 21:54, tsunakawa.ta...@fujitsu.com wrote: I've managed to rebased it, although it took unexpectedly long. The patch is attached. It passes make check against core and postgres_fdw. I'll turn the CF status back to ready for committer shortly. Macros _() at the

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Peter Geoghegan
On Mon, Mar 22, 2021 at 7:05 AM Robert Haas wrote: > I agree. I was having trouble before understanding exactly what you > are proposing, but this makes sense to me and I agree it's a good > idea. Our ambition is for this to be one big multi-release umbrella project, with several individual

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread James Coleman
On Mon, Mar 22, 2021 at 1:49 PM Fujii Masao wrote: > > > > On 2021/03/19 23:35, James Coleman wrote: > > Here's an updated patch; I think I've gotten what Alvaro suggested. > > Thanks for updating the patch! But I was thinking that our consensus is > something like the attached patch. Could you

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Tom Lane
Robert Haas writes: > I think this is significantly cleaner than what we have now, and I > also prefer it to your proposal. +1 in general. However, I suspect that you did not try to compile this without --with-lz4, because if you had you'd have noticed the other uses of NO_LZ4_SUPPORT() that

Re: Fix pg_upgrade to preserve datdba

2021-03-22 Thread Jan Wieck
On 3/21/21 3:56 PM, Tom Lane wrote: Jan Wieck writes: So let's focus on the actual problem of running out of XIDs and memory while doing the upgrade involving millions of small large objects. Right. So as far as --single-transaction vs. --create goes, that's mostly a definitional problem.

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 01:38:36PM -0400, Robert Haas wrote: > On Mon, Mar 22, 2021 at 12:16 PM Robert Haas wrote: > > But, what about giving the default_toast_compression_method GUC an > > assign hook that sets a global variable of type "char" to the > > appropriate value? Then

Re: Nicer error when connecting to standby with hot_standby=off

2021-03-22 Thread Fujii Masao
On 2021/03/19 23:35, James Coleman wrote: Here's an updated patch; I think I've gotten what Alvaro suggested. Thanks for updating the patch! But I was thinking that our consensus is something like the attached patch. Could you check this patch? Regards, -- Fujii Masao Advanced Computing

Re: Autovacuum worker doesn't immediately exit on postmaster death

2021-03-22 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > On Fri, Dec 11, 2020 at 7:57 AM Stephen Frost wrote: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > > The if-we're-going-to-delay-anyway path in vacuum_delay_point seems > > > OK to add a touch more overhead to, though. > > > > Alright,

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 12:16 PM Robert Haas wrote: > But, what about giving the default_toast_compression_method GUC an > assign hook that sets a global variable of type "char" to the > appropriate value? Then GetDefaultToastCompression() goes away > entirely. That might be worth exploring.

Re: Wired if-statement in gen_partprune_steps_internal

2021-03-22 Thread Ryan Lambert
Should the status of this patch be updated to ready for comitter to get in line for Pg 14 deadline? *Ryan Lambert* On Sun, Mar 7, 2021 at 11:38 PM Amit Langote wrote: > On Fri, Mar 5, 2021 at 7:50 AM Ryan Lambert > wrote: > > On Wed, Mar 3, 2021 at 11:03 PM Amit Langote > wrote: > >> > >>

Re: Proposal: Save user's original authenticated identity for logging

2021-03-22 Thread Magnus Hagander
On Mon, Mar 22, 2021 at 7:16 AM Michael Paquier wrote: > > On Fri, Mar 19, 2021 at 06:37:05PM +, Jacob Champion wrote: > > The same effect can be had by moving the log rotation to the top of the > > test that needs it, so I've done it that way in v7. > > After thinking more about 0001, I have

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > > Perhaps allowing to set unlogged tables to logged ones without writing WAL > > > is a more elegant way to do that, but I cannot see how that would be any > > > more crash

Re: Protect syscache from bloating with negative cache entries

2021-03-22 Thread Bruce Momjian
On Thu, Jan 28, 2021 at 05:16:52PM +0900, Kyotaro Horiguchi wrote: > At Thu, 28 Jan 2021 16:50:44 +0900 (JST), Kyotaro Horiguchi > wrote in > > I was going to write in the doc something like "you can inspect memory > > consumption by catalog caches using pg_backend_memory_contexts", but > > all

Re: Change default of checkpoint_completion_target

2021-03-22 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > I had a look at the patch and the change and new documentation seem sensible > to me. Thanks! > I think this phrase may be a bit too idiomatic: > > +consistent I/O load while also leaving some slop for checkpoint > > Perhaps

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > Perhaps allowing to set unlogged tables to logged ones without writing WAL > > is a more elegant way to do that, but I cannot see how that would be any > > more crash safe than this patch. Besides, the feature doesn't exist yet. > > I'm

Re: Add docs stub for recovery.conf

2021-03-22 Thread Stephen Frost
Greetings, * Craig Ringer (craig.rin...@enterprisedb.com) wrote: > Pretty good to me. Thanks so much for your help and support with this. Thanks for helping me move it forward! > Index entries render as e.g. > > pg_xlogdump, The pg_xlogdump command > (see also pg_waldump) > >

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 11:13 AM Justin Pryzby wrote: > The first iteration was pretty rough, and there's still some question in my > mind about where default_toast_compression_options[] should be defined. If > it's in the header file, then I could use lengthof() - but then it probably > gets

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Tom Lane
Robert Haas writes: > On Mon, Mar 22, 2021 at 11:48 AM Tom Lane wrote: >> Possibly the former names should survive and the latter become >> wrappers around them, not sure. But we shouldn't be using the "4B" >> terminology anyplace except this part of postgres.h. > I would argue that it

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 11:48 AM Tom Lane wrote: > > Anyway, this particular macro name was chosen, it seems, for symmetry > > with VARDATA_4B_C, but if you want to change it to something else, I'm > > OK with that, too. > > After looking at postgres.h for a bit, I'm thinking that what these >

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Tom Lane
Robert Haas writes: > On Mon, Mar 22, 2021 at 10:41 AM Tom Lane wrote: >> Yeah, I thought about that too, but do we want to assume that >> VARRAWSIZE_4B_C is the correct way to get the decompressed size >> for all compression methods? > I think it's OK to assume this. OK, cool. >> (If so, I

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 10:41 AM Tom Lane wrote: > > Okay, the fix makes sense. In fact, IMHO, in general also this fix > > looks like an optimization, I mean when slicelength >= > > VARRAWSIZE_4B_C(value), then why do we need to allocate extra memory > > even in the case of pglz. So shall we

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Dilip Kumar
On Mon, Mar 22, 2021 at 8:11 PM Tom Lane wrote: > > Dilip Kumar writes: > > On Mon, Mar 22, 2021 at 5:22 AM Tom Lane wrote: > >> Also, after studying the documentation for LZ4_decompress_safe > >> and LZ4_decompress_safe_partial, I realized that liblz4 is also > >> counting on the *output*

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 11:05:19AM -0400, Robert Haas wrote: > On Mon, Mar 22, 2021 at 10:44 AM Justin Pryzby wrote: > > Thanks. I just realized that if you also push the GUC change, then the docs > > should change from to > > > > doc/src/sgml/config.sgml: > > default_toast_compression

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 09:46 -0400, Stephen Frost wrote: > > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > > From: Stephen Frost > > > > The argument here seems to stem from the idea that issueing a 'TRUNCATE'

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 10:44 AM Justin Pryzby wrote: > Thanks. I just realized that if you also push the GUC change, then the docs > should change from to > > doc/src/sgml/config.sgml: > default_toast_compression (string) I've now also committed your 0005. As for 0006, aside from the

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-22 Thread Fujii Masao
On 2021/03/20 13:40, Masahiro Ikeda wrote: Sorry, there is no evidence we should call it. I thought that to call FreeWaitEventSet() is a manner after using CreateWaitEventSet() and the performance impact to call it seems to be too small. (Please let me know if my understanding is wrong.)

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Laurenz Albe
On Mon, 2021-03-22 at 09:46 -0400, Stephen Frost wrote: > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > From: Stephen Frost > > > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > > > inside the transaction before starting the 'COPY' command is

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 10:41:33AM -0400, Robert Haas wrote: > On Sun, Mar 21, 2021 at 7:55 PM Justin Pryzby wrote: > > Rebased again. > > Thanks, Justin. I committed 0003 and 0004 together as > 226e2be3876d0bda3dc33d16dfa0bed246b7b74f. I also committed 0001 and > 0002 together as

Re: [HACKERS] Custom compression methods (mac+lz4.h)

2021-03-22 Thread Tom Lane
Dilip Kumar writes: > On Mon, Mar 22, 2021 at 5:22 AM Tom Lane wrote: >> Also, after studying the documentation for LZ4_decompress_safe >> and LZ4_decompress_safe_partial, I realized that liblz4 is also >> counting on the *output* buffer size to not be a lie. So we >> cannot pass it a number

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Sun, Mar 21, 2021 at 7:55 PM Justin Pryzby wrote: > Rebased again. Thanks, Justin. I committed 0003 and 0004 together as 226e2be3876d0bda3dc33d16dfa0bed246b7b74f. I also committed 0001 and 0002 together as 24f0e395ac5892cd12e8914646fe921fac5ba23d, but with some revisions, because your text

Re: PG13 fails to startup even though the current transaction is equal to the target transaction

2021-03-22 Thread Fujii Masao
On 2021/03/22 21:40, Sean Jezewski wrote: Hi Kyotaro - Thanks for the response. I think it boils down to your comment: > I'm not sure.  The direct cause of the "issue" is a promotion trigger > that came before reaching recovery target.  That won't happen if the > "someone" doesn't do

Re: Minimal logical decoding on standbys

2021-03-22 Thread Fabrízio de Royes Mello
On Thu, Mar 18, 2021 at 5:34 AM Drouvot, Bertrand wrote: > > Thanks! > > Just made minor changes to make it compiling again on current master (mainly had to take care of ResolveRecoveryConflictWithSnapshotFullXid() that has been introduced in e5d8a99903). > > Please find enclosed the new patch

Re: New IndexAM API controlling index vacuum strategies

2021-03-22 Thread Robert Haas
On Thu, Mar 18, 2021 at 9:42 PM Peter Geoghegan wrote: > The fact that we can *continually* reevaluate if an ongoing VACUUM is > at risk of taking too long is entirely the point here. We can in > principle end index vacuuming dynamically, whenever we feel like it > and for whatever reasons occur

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > > inside the transaction before starting the 'COPY' command is 'too hard'. > > > I could be wrong and perhaps

  1   2   >