Re: pg_restore causing deadlocks on partitioned tables

2020-09-16 Thread Tom Lane
Amit Langote writes: > On Thu, Sep 17, 2020 at 3:40 AM Tom Lane wrote: >> I also did a bit more work on the comments. (Speaking of which, >> is there a better place to put the commentary you removed from >> InitResultRelInfo? It was surely wildly out of place there, >> but I'm wondering if

Re: pg_restore causing deadlocks on partitioned tables

2020-09-16 Thread Amit Langote
On Thu, Sep 17, 2020 at 3:40 AM Tom Lane wrote: > Amit Langote writes: > > Updated patch attached. > > Pushed with a little bit of fooling about. Thank you. > After looking at the > git history, I saw that the Assert we were wondering about used to > be just "Assert(constr)", and there were

RE: pg_restore causing deadlocks on partitioned tables

2020-09-16 Thread Domagoj Smoljanovic
Thanx Tom and Amit for the effort. Looking forward to try it out. D. -Original Message- From: Tom Lane Sent: 16. rujna 2020. 20:41 To: Amit Langote Cc: Alvaro Herrera ; Domagoj Smoljanovic ; pgsql-hack...@postgresql.org Subject: Re: pg_restore causing deadlocks on partitioned tables

Re: pg_restore causing deadlocks on partitioned tables

2020-09-16 Thread Tom Lane
Amit Langote writes: > Updated patch attached. Pushed with a little bit of fooling about. After looking at the git history, I saw that the Assert we were wondering about used to be just "Assert(constr)", and there were not run-time checks on whether constr is null. That was changed when

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Amit Langote
On Wed, Sep 16, 2020 at 2:41 AM Tom Lane wrote: > Amit Langote writes: > > On Tue, Sep 15, 2020 at 7:28 AM Tom Lane wrote: > >> AFAICS, it is utterly silly for InitResultRelInfo to be forcing > >> a partition qual to be computed when we might not need it. > >> We could flush

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Tom Lane
Amit Langote writes: > On Tue, Sep 15, 2020 at 7:28 AM Tom Lane wrote: >> AFAICS, it is utterly silly for InitResultRelInfo to be forcing >> a partition qual to be computed when we might not need it. >> We could flush ResultRelInfo.ri_PartitionCheck altogether and >> have anything that was

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Tom Lane
Amit Langote writes: > On Tue, Sep 15, 2020 at 10:47 PM Tom Lane wrote: >> Ah, right. That seems like a bug but we have not attempted to fix it. > IIRC, when this behavior was brought up as a bug in past discussions, > it was decided that it will be fixed when NOT NULL constraints are >

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Amit Langote
On Tue, Sep 15, 2020 at 10:47 PM Tom Lane wrote: > Amit Langote writes: > > On Tue, Sep 15, 2020 at 9:09 AM Tom Lane wrote: > >> I wrote a quick patch for this part. It seems pretty safe and probably > >> could be back-patched without fear. > > > The patch's theory that if the parent column

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Tom Lane
Amit Langote writes: > On Tue, Sep 15, 2020 at 9:09 AM Tom Lane wrote: >> I wrote a quick patch for this part. It seems pretty safe and probably >> could be back-patched without fear. > The patch's theory that if the parent column has NOT NULL set then it > must be set in child tables too does

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Amit Langote
On Tue, Sep 15, 2020 at 7:28 AM Tom Lane wrote: > I wrote: > > However, the deadlock report suggests, and manual experimentation > > confirms, that > > > (1) TRUNCATE on a partition tries to get AccessShareLock on the parent; > > The reason for this is that > > (a) ExecuteTruncateGuts calls

Re: pg_restore causing deadlocks on partitioned tables

2020-09-15 Thread Amit Langote
On Tue, Sep 15, 2020 at 9:09 AM Tom Lane wrote: > I wrote: > >> (2) ALTER TABLE ONLY ... ADD CONSTRAINT on a partition root tries to get > >> AccessExclusiveLock on all child partitions, despite the ONLY. > > > The cause of this seems to be that ATPrepSetNotNull is too dumb to > > avoid recursing

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Sep-14, Tom Lane wrote: >> AFAICS, it is utterly silly for InitResultRelInfo to be forcing >> a partition qual to be computed when we might not need it. >> We could flush ResultRelInfo.ri_PartitionCheck altogether and >> have anything that was reading it instead

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Alvaro Herrera
On 2020-Sep-14, Tom Lane wrote: > > (1) TRUNCATE on a partition tries to get AccessShareLock on the parent; > > The reason for this is that > > (a) ExecuteTruncateGuts calls InitResultRelInfo, because it might > need that to fire TRUNCATE triggers for the child relation. Hmm, this seems

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
I wrote: >> (2) ALTER TABLE ONLY ... ADD CONSTRAINT on a partition root tries to get >> AccessExclusiveLock on all child partitions, despite the ONLY. > The cause of this seems to be that ATPrepSetNotNull is too dumb to > avoid recursing to all the child tables when the parent is already >

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
I wrote: > However, the deadlock report suggests, and manual experimentation > confirms, that > (1) TRUNCATE on a partition tries to get AccessShareLock on the parent; The reason for this is that (a) ExecuteTruncateGuts calls InitResultRelInfo, because it might need that to fire TRUNCATE

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
I wrote: > If memory serves, which it may not given my undercaffeinated state, > we would not expect there to be a direct dependency link between the > constraint and the table data "object". What there should be is > dependencies forcing the data to be restored before the post-data > boundary

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Sep-14, Tom Lane wrote: >> Hm, this seems related to 2ba5b2db7, but not the same thing. >> Alvaro, any thoughts? > So apparently when we go to restore the table data for the partition, > the TRUNCATE deadlocks with the PK addition ... that's pretty odd; >

RE: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Domagoj Smoljanovic
Forgot to mention the versions: pg_restore (PostgreSQL) 12.4 source/ destination databases also 12.4 D. -Original Message- From: Alvaro Herrera Sent: 14. rujna 2020. 16:40 To: Tom Lane Cc: Domagoj Smoljanovic ; pgsql-hack...@postgresql.org Subject: Re: pg_restore causing deadlocks

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Alvaro Herrera
On 2020-Sep-14, Tom Lane wrote: > Domagoj Smoljanovic writes: > > I have pg_restore running in parallel (3 or more) and processing large > > amount of data that is in partitioned tables. However it seems that > > sometime deadlock appears when one process is trying to process primary key > >

Re: pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Tom Lane
Domagoj Smoljanovic writes: > I have pg_restore running in parallel (3 or more) and processing large amount > of data that is in partitioned tables. However it seems that sometime > deadlock appears when one process is trying to process primary key on parent > table while data still hasn’t

pg_restore causing deadlocks on partitioned tables

2020-09-14 Thread Domagoj Smoljanovic
Hi all. I tried searching for the response to this but couldn’t find any. Tried also posting to general but got no love there. I have pg_restore running in parallel (3 or more) and processing large amount of data that is in partitioned tables. However it seems that sometime deadlock appears