Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 5:16 PM Tomas Vondra wrote: > > I wonder if there's a way to free the TOASTed data earlier, instead of > waiting until the end of the transaction (as this patch does). > IIUC we are anyway freeing the toasted data at the next insert/update/delete. We can try to free at

Re: Parallel Inserts in CREATE TABLE AS

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 8:53 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 7:37 PM Bharath Rupireddy > > > > I captured below information with the attached patch > > 0001-test-times-and-block-counts.patch applied on top of CTAS v23 > > patch set. Testing details are attached in the file named

Regarding the necessity of RelationGetNumberOfBlocks for every rescan / bitmap heap scan.

2021-05-28 Thread Andy Fan
Hi: I'm always confused about the following codes. static void initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) { ParallelBlockTableScanDesc bpscan = NULL; bool allow_strat; bool allow_sync; /* * Determine the number of blocks we have to scan. * * It is sufficient to do this once

Re: Skipping logical replication transactions on subscriber side

2021-05-28 Thread Masahiko Sawada
On Thu, May 27, 2021 at 7:04 PM Amit Kapila wrote: > > On Thu, May 27, 2021 at 1:46 PM Masahiko Sawada wrote: > > > > On Thu, May 27, 2021 at 2:48 PM Amit Kapila wrote: > > > > > > Okay, that makes sense but still not sure how will you identify if we > > > need to reset XID in case of failure

Re: Skipping logical replication transactions on subscriber side

2021-05-28 Thread Masahiko Sawada
On Thu, May 27, 2021 at 7:26 PM Amit Kapila wrote: > > On Thu, May 27, 2021 at 12:01 PM Masahiko Sawada > wrote: > > > > On Wed, May 26, 2021 at 6:11 PM Amit Kapila wrote: > > > > > > I agree with you that specifying XID could be easier and > > > understandable for users. I was thinking and

ANALYZE's dead tuple accounting can get confused

2021-05-28 Thread Peter Geoghegan
The accounting used by ANALYZE to count dead tuples in acquire_sample_rows() (actually in heapam_scan_analyze_next_tuple() these days) makes some dubious assumptions about how it should count dead tuples. This is something that I discussed with Masahiko in the context of our Postgres 14 work on

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Johannes Graën
On 28/05/2021 18.24, Tom Lane wrote: > Pavel Stehule writes: >> pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën >> napsal: >>> When trying to upgrade an existing database from version 10 to 13 I came >>> across a degression in some existing code used by clients. Further >>> investigations

Re: Add option --drop-cascade for pg_dump/restore

2021-05-28 Thread Greg Sabino Mullane
Overall the patch looks good, but I did notice a few small things: 1. In pg_dumpall.c, the section /* Add long options to the pg_dump argument list */, we are now passing along the --drop-cascade option. However, --clean is not passed in, so any call to pg_dumpall using --drop-cascade fails a

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-05-28 Thread Mark Dilger
> On May 27, 2021, at 11:06 PM, Noah Misch wrote: > > On Tue, May 25, 2021 at 01:33:54PM -0700, Mark Dilger wrote: >> v3-0001 adds a new pg_logical_replication role with permission to manage >> publications and subscriptions. > >> v3-0004 adds a new pg_database_security role with permission

Re: terminate called after throwing an instance of 'std::bad_alloc'

2021-05-28 Thread Justin Pryzby
There's a memory leak affecting JIT expressions, even when inlining, optimization, and tuple deforming are disabled. The server that got OOM after installing PG13.3 (because your patch wasn't applied) still gets OOM even with inline_above_cost=-1, optimize_above_cost=-1, and deforming=off.

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Tom Lane
Pavel Stehule writes: > pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën > napsal: >> When trying to upgrade an existing database from version 10 to 13 I came >> across a degression in some existing code used by clients. Further >> investigations showed that performance measures are similar in

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/28/21 4:22 PM, Andrew Dunstan wrote: > > On 5/28/21 6:35 AM, Tomas Vondra wrote: >> >>> >>> IMO the main benefit of having different dictionaries is that you >>> could have a small dictionary for small and very structured JSONB >>> fields (e.g. some time-series data), and a large one for

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Pavel Stehule
hI pá 28. 5. 2021 v 16:12 odesílatel Johannes Graën napsal: > Hi, > > When trying to upgrade an existing database from version 10 to 13 I came > across a degression in some existing code used by clients. Further > investigations showed that performance measures are similar in versions > 11 to

Re: Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Tomas Vondra
On 5/28/21 4:12 PM, Johannes Graën wrote: > Hi, > > When trying to upgrade an existing database from version 10 to 13 I came > across a degression in some existing code used by clients. Further > investigations showed that performance measures are similar in versions > 11 to 13, while in the

Re: be-secure-gssapi.c and auth.c with setenv() not compatible on Windows

2021-05-28 Thread Tom Lane
Michael Paquier writes: > We can do two things here: > 1) Switch be-secure-gssapi.c and auth.c to use putenv(). > 2) Backport into 12 and 13 the fallback implementation of setenv > introduced in 7ca37fb, and keep be-secure-gssapi.c as they are now. There's a lot of value in keeping the branches

Re: Command statistics system (cmdstats)

2021-05-28 Thread Mark Dilger
> On May 28, 2021, at 6:42 AM, Alvaro Herrera wrote: > On Mar 4, 2020, at 7:43 PM, Mark Dilger wrote: > as mentioned in [1], I have created an implementation of command counter statistics very similar in purpose to the one already pending in the commitfest going by

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Andrew Dunstan
On 5/28/21 6:35 AM, Tomas Vondra wrote: > >> >> IMO the main benefit of having different dictionaries is that you >> could have a small dictionary for small and very structured JSONB >> fields (e.g. some time-series data), and a large one for large / >> unstructured JSONB fields, without having

Degression (PG10 > 11, 12 or 13)

2021-05-28 Thread Johannes Graën
Hi, When trying to upgrade an existing database from version 10 to 13 I came across a degression in some existing code used by clients. Further investigations showed that performance measures are similar in versions 11 to 13, while in the original database on version 10 it's around 100 times

Re: Reduce lock level for ALTER TABLE ... ADD CHECK .. NOT VALID

2021-05-28 Thread Greg Sabino Mullane
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Looks fine to me The new status of this patch is: Ready for

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Etsuro Fujita
Horiguchi-san, On Fri, May 28, 2021 at 5:29 PM Kyotaro Horiguchi wrote: > At Fri, 28 May 2021 16:30:29 +0900, Etsuro Fujita > wrote in > > The root cause would > > be that we call classify_matching_subplans() to re-determine > > sync/async subplans when called from the first ExecAppend() after

Re: Command statistics system (cmdstats)

2021-05-28 Thread Alvaro Herrera
> >> On Mar 4, 2020, at 7:43 PM, Mark Dilger > >> wrote: > >> as mentioned in [1], I have created an implementation of command > >> counter statistics very similar in purpose to the one already > >> pending in the commitfest going by the name "pg_stat_sql". I don't > >> really care if this

Re: storing an explicit nonce

2021-05-28 Thread Bruce Momjian
On Thu, May 27, 2021 at 04:36:23PM -0400, Stephen Frost wrote: > At this point I'm wondering if it's just: > > dboid/relfilenode:block-offset > > and then we hash it to whatever size EVP_CIPHER_iv_length(AES-XTS-128) > (or -256, whatever we're using based on what was passed to initdb) > returns.

Re: [HACKERS] EvalPlanQual behaves oddly for FDW queries involving system columns

2021-05-28 Thread Etsuro Fujita
On Fri, May 28, 2021 at 8:25 AM Alvaro Herrera wrote: > On 2019-Mar-15, Etsuro Fujita wrote: > > (2019/03/04 12:10), Etsuro Fujita wrote: > > > (2019/03/02 3:57), Andres Freund wrote: > > > > FWIW, I pushed the EPQ patch, doing this conversion blindly. It'd be > > > > awesome if you'd check that

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Amit Kapila
On Fri, May 28, 2021 at 6:01 PM Tomas Vondra wrote: > > On 5/28/21 2:17 PM, Dilip Kumar wrote: > > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > > wrote: > >> On 5/27/21 6:36 AM, Dilip Kumar wrote: > >>> On Thu, May 27, 2021 at 9:47 AM Amit Kapila > >>> wrote: > > On Thu, May 27,

be-secure-gssapi.c and auth.c with setenv() not compatible on Windows

2021-05-28 Thread Michael Paquier
Hi all, Now that hamerkop has been fixed and that we have some coverage with builds of GSSAPI on Windows thanks to 02511066, the buildfarm has been complaining about a build failure on Windows for 12 and 13:

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Tomas Vondra
On 5/28/21 2:17 PM, Dilip Kumar wrote: > On Fri, May 28, 2021 at 5:16 PM Tomas Vondra > wrote: >> On 5/27/21 6:36 AM, Dilip Kumar wrote: >>> On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: True, but if you do this

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Dilip Kumar
On Fri, May 28, 2021 at 5:16 PM Tomas Vondra wrote: > On 5/27/21 6:36 AM, Dilip Kumar wrote: > > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: > >> > >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > >> > >> True, but if you do this clean-up in ReorderBufferCleanupTXN then you > >>

Re: Decoding speculative insert with toast leaks memory

2021-05-28 Thread Tomas Vondra
On 5/27/21 6:36 AM, Dilip Kumar wrote: > On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: >> >> On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: >> >> True, but if you do this clean-up in ReorderBufferCleanupTXN then you >> don't need to take care at separate places. Also, toast_hash is

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/27/21 4:15 AM, Andrew Dunstan wrote: > > On 5/26/21 5:29 PM, Bruce Momjian wrote: >> On Tue, May 25, 2021 at 01:55:13PM +0300, Aleksander Alekseev wrote: >>> Hi hackers, >>> >>> Back in 2016 while being at PostgresPro I developed the ZSON extension [1]. >>> The >>> extension introduces

Re: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread Dilip Kumar
On Fri, May 28, 2021 at 12:31 PM tanghy.f...@fujitsu.com wrote: > > On Friday, May 28, 2021 3:02 PM, tanghy.f...@fujitsu.com wrote: > > FYI. The problem also occurs in PG-13. I will try to check from which > > version it > > got introduced. > > I reproduced it in PG-10,11,12,13. > I think the

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/26/21 2:49 AM, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Matthias van de Meent writes: >>> I like the idea of the ZSON type, but I'm somewhat disappointed by its >>> current limitations: >> >> I've not read the code, so maybe this thought is

Re: Add ZSON extension to /contrib/

2021-05-28 Thread Tomas Vondra
On 5/26/21 6:43 PM, Matthias van de Meent wrote: > On Wed, 26 May 2021 at 12:49, Aleksander Alekseev > wrote: >> >> Hi hackers, >> >> Many thanks for your feedback, I very much appreciate it! >> >>> If the extension is mature enough, why make it an extension in >>> contrib, and not instead

RE: Parallel Inserts in CREATE TABLE AS

2021-05-28 Thread houzj.f...@fujitsu.com
From: Bharath Rupireddy Sent: Thursday, May 27, 2021 10:07 PM > On Thu, May 27, 2021 at 9:53 AM Bharath Rupireddy > wrote: > > > One idea to find this out could be that we have three counters for > > > each worker which counts the number of times each worker extended > > > the relation in bulk,

Re: Parallel INSERT SELECT take 2

2021-05-28 Thread Greg Nancarrow
On Mon, May 24, 2021 at 3:15 PM houzj.f...@fujitsu.com wrote: > > > Thanks for the comments and your descriptions looks good. > Attaching v5 patchset with all these changes. > A few other minor things I noticed: (1) error message wording when declaring a table SAFE for parallel DML

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Kyotaro Horiguchi
At Fri, 28 May 2021 16:30:29 +0900, Etsuro Fujita wrote in > On Wed, Mar 31, 2021 at 6:55 PM Etsuro Fujita wrote: > > On Tue, Mar 30, 2021 at 8:40 PM Etsuro Fujita > > wrote: > > > I'm happy with the patch, so I'll commit it if there are no objections. > > > > Pushed. > > I noticed that

Re: Race condition in recovery?

2021-05-28 Thread Kyotaro Horiguchi
(Sorry for being a bit off-topic) At Fri, 28 May 2021 12:18:35 +0900, Tatsuro Yamada wrote in > Hi Horiguchi-san, (Why me?) > In a project I helped with, I encountered an issue where > the archive command kept failing. I thought this issue was > related to the problem in this thread, so I'm

Re: Asynchronous Append on postgres_fdw nodes.

2021-05-28 Thread Etsuro Fujita
On Wed, Mar 31, 2021 at 6:55 PM Etsuro Fujita wrote: > On Tue, Mar 30, 2021 at 8:40 PM Etsuro Fujita wrote: > > I'm happy with the patch, so I'll commit it if there are no objections. > > Pushed. I noticed that rescan of async Appends is broken when do_exec_prune=false, leading to incorrect

RE: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread tanghy.f...@fujitsu.com
On Friday, May 28, 2021 3:02 PM, tanghy.f...@fujitsu.com wrote: > FYI. The problem also occurs in PG-13. I will try to check from which version > it > got introduced. I reproduced it in PG-10,11,12,13. I think the problem has been existing since Logical replication introduced in PG-10.

Re: RFC: Logging plan of the running query

2021-05-28 Thread torikoshia
On 2021-05-13 21:57, Dilip Kumar wrote: On Thu, May 13, 2021 at 5:18 PM Dilip Kumar wrote: On Thu, May 13, 2021 at 5:15 PM Bharath Rupireddy wrote: > > On Thu, May 13, 2021 at 5:14 PM Dilip Kumar wrote: > > > > On Thu, May 13, 2021 at 4:16 PM Bharath Rupireddy > > wrote: > > > > > > I'm

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-28 Thread Amit Kapila
On Thu, May 27, 2021 at 8:05 AM Ajin Cherian wrote: > > Thanks for confirmation. The problem seemed to be as you reported a > table not closed when a transaction was committed. > This seems to be because the function UpdateTwoPhaseState was > committing a transaction inside the function when the

Re: Bracket, brace, parenthesis

2021-05-28 Thread Kyotaro Horiguchi
At Thu, 27 May 2021 21:08:46 +0900, Michael Paquier wrote in > On Thu, May 27, 2021 at 03:20:10PM +0900, Kyotaro Horiguchi wrote: > > At Fri, 14 May 2021 10:04:57 -0400, Tom Lane wrote in > >> +1. I tend to write "square bracket" or "curly brace" when I want to > >> be extra clear, but I

Re: storing an explicit nonce

2021-05-28 Thread Neil Chen
On Fri, May 28, 2021 at 2:12 PM Neil Chen wrote: > > When the checksums are disabled, we can use these unused checksum values > as a counter to make sure we have different nonce even if we don't write > the new WAL record. > > Ah, well, I think I've figured it out for myself. In this way, we

Re: storing an explicit nonce

2021-05-28 Thread Neil Chen
On Thu, May 27, 2021 at 11:12 PM Bruce Momjian wrote: > > Well, the code now does write full page images for hint bit changes, so > it should work fine. > > Yes, indeed it works well and I'd tested it. But here I want to make clear my understanding of the argument, if there is any problem please

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-05-28 Thread Noah Misch
On Tue, May 25, 2021 at 01:33:54PM -0700, Mark Dilger wrote: > v3-0001 adds a new pg_logical_replication role with permission to manage > publications and subscriptions. > v3-0004 adds a new pg_database_security role with permission to perform many > actions that would otherwise require

Re: Race condition in recovery?

2021-05-28 Thread Kyotaro Horiguchi
Thanks! At Thu, 27 May 2021 15:05:44 -0400, Robert Haas wrote in > On Wed, May 26, 2021 at 8:49 PM Kyotaro Horiguchi > wrote: > > So in the mail [1] and [2] I tried to describe what's going on around > > the two issues. Although I haven't have a response to [2], can I > > think that we

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-28 Thread vignesh C
On Fri, May 28, 2021 at 9:14 AM Ajin Cherian wrote: > > On Wed, May 26, 2021 at 6:53 PM vignesh C wrote: > > > > On Tue, May 25, 2021 at 8:54 AM Ajin Cherian wrote: > > > > > > On Fri, May 21, 2021 at 6:43 PM Peter Smith wrote: > > > > > > > Fixed in v77-0001, v77-0002 > > > > > > Attaching a

RE: [BUG]Update Toast data failure in logical replication

2021-05-28 Thread tanghy.f...@fujitsu.com
On Friday, May 28, 2021 2:16 PM, tanghy.f...@fujitsu.com wrote: >I think I just found a bug in logical replication. Data couldn't be >synchronized while updating toast data. >Could anyone take a look at it? FYI. The problem also occurs in PG-13. I will try to check from which version it got