Re: generic plans and "initial" pruning

2022-02-27 Thread Amit Langote
Hi Andres, On Fri, Feb 11, 2022 at 10:29 AM Andres Freund wrote: > On 2022-02-10 17:13:52 +0900, Amit Langote wrote: > > The attached patch implements this idea. Sorry for the delay in > > getting this out and thanks to Robert for the off-list discussions on > > this. >

Re: generic plans and "initial" pruning

2022-03-07 Thread Amit Langote
On Fri, Feb 11, 2022 at 7:02 AM Robert Haas wrote: > On Thu, Feb 10, 2022 at 3:14 AM Amit Langote wrote: > > Maybe this should be more than one patch? Say: > > > > 0001 to add ExecutorPrep and the boilerplate, > > 0002 to teach plancache.c to use the new facility T

Re: generic plans and "initial" pruning

2022-03-11 Thread Amit Langote
On Fri, Mar 11, 2022 at 11:35 PM Amit Langote wrote: > Attached is v5, now broken into 3 patches: > > 0001: Some refactoring of runtime pruning code > 0002: Add a plan_tree_walker > 0003: Teach AcquireExecutorLocks to skip locking pruned relations Repeated the performance tests d

Re: simplifying foreign key/RI checks

2022-03-14 Thread Amit Langote
On Tue, Jan 18, 2022 at 3:30 PM Amit Langote wrote: > v13 is attached. I noticed that the recent 641f3dffcdf's changes to get_constraint_index() made it basically unusable for this patch's purposes. Reading in the thread that led to 641f3dffcdf why get_constraint_index() was chang

Re: support for MERGE

2022-03-14 Thread Amit Langote
't more readable to just have these two be the output arguments of ExecUpdateAct()? +redo_act: + /* XXX reinit ->crossPartUpdate here? */ Why not just make ExecUpdateAct() always set the flag, not only when a cross-partition update does occur? Will look some more in th

Re: generic plans and "initial" pruning

2022-03-14 Thread Amit Langote
made available via a new data structure that plancache.c has been made to pass down to the executor alongside the plan tree. If it does not, ExecInit[Merge]Append() does the pruning in the same way it does now. Such cases include initial pruning using only STABLE expressions that the planner doesn't bother to compute by itself lest the resulting plan may be cached, but no EXTERN parameters. -- Amit Langote EDB: http://www.enterprisedb.com

Re: support for MERGE

2022-03-15 Thread Amit Langote
On Mon, Mar 14, 2022 at 11:36 PM Amit Langote wrote: > On Sun, Mar 13, 2022 at 12:36 AM Alvaro Herrera > wrote: > > FYI I intend to get the ModifyTable split patch (0001+0003) pushed > > hopefully on Tuesday or Wednesday next week, unless something really > > ugly is f

Re: a misbehavior of partition row movement (?)

2022-03-19 Thread Amit Langote
s have foreign keys pointing into it or that it is the PK table in that relationship. Other than the comment, the code itself seems self-documenting with regard to what's being done (given the function/macro/variable naming), so maybe we're better off without additional commentary here. I've attached a delta patch on v16 for the above comment and a couple of other changes. -- Amit Langote EDB: http://www.enterprisedb.com v16_delta.diff Description: Binary data

Re: Allow batched insert during cross-partition updates

2022-03-21 Thread Amit Langote
On Tue, Mar 22, 2022 at 9:30 AM Andres Freund wrote: > On 2021-08-24 12:03:59 +0900, Amit Langote wrote: > > Tomas committed the bug-fix, so attaching a rebased version of the > > patch for $subject. > > This patch is in the current CF, but doesn't apply: > http://cf

Re: a misbehavior of partition row movement (?)

2022-03-21 Thread Amit Langote
Hi Alvaro, On Mon, Mar 21, 2022 at 2:58 AM Alvaro Herrera wrote: > On 2022-Mar-20, Amit Langote wrote: > > On Sun, Mar 20, 2022 at 5:13 AM Alvaro Herrera > > wrote: > > > On 2022-Mar-18, Zhihong Yu wrote: > > > > > + if (!partRel->rd_rel->relisp

Re: simplifying foreign key/RI checks

2022-03-21 Thread Amit Langote
On Mon, Mar 14, 2022 at 6:28 PM Zhihong Yu wrote: > On Mon, Mar 14, 2022 at 1:33 AM Amit Langote wrote: >> On Tue, Jan 18, 2022 at 3:30 PM Amit Langote wrote: >> > v13 is attached. >> >> I noticed that the recent 641f3dffcdf's changes to >> get_constrai

Re: generic plans and "initial" pruning

2022-03-22 Thread Amit Langote
On Tue, Mar 15, 2022 at 3:19 PM Amit Langote wrote: > On Tue, Mar 15, 2022 at 5:06 AM Robert Haas wrote: > > On Mon, Mar 14, 2022 at 3:38 PM Tom Lane wrote: > > > Also, while I've not spent much time at all reading this patch, > > > it seems rather desperately

Re: Skip partition tuple routing with constant partition key

2022-03-23 Thread Amit Langote
rned the same bound offset in succession a number of times. Maybe waiting a 1000 tuples to re-assess that is a bit too conservative, yes. I guess even as small a number as 10 is fine here? I've attached an updated version of the patch, though I haven't changed the threshold constant. --

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-01 Thread Amit Langote
On Tue, Dec 1, 2020 at 2:40 PM tsunakawa.ta...@fujitsu.com wrote: > From: Amit Langote > > Andrey's original patch had the flag to, as I understand it, make the > > partitioning case work correctly. When inserting into a > > non-partitioned table, there's only

Re: Huge memory consumption on partitioned table with FKs

2020-12-02 Thread Amit Langote
s in a given partition hierarchy would've originated. Actually, I had written a patch a few months back to do exactly that, a polished version of which I've attached with this email. Please take a look. -- Amit Langote EDB: http://www.enterprisedb.com 0001-ri_triggers.c-Use-root-constraint-OID-as-key-to-ri_q.patch Description: Binary data

Re: Huge memory consumption on partitioned table with FKs

2020-12-02 Thread Amit Langote
Hello, On Thu, Dec 3, 2020 at 10:15 AM Kyotaro Horiguchi wrote: > At Wed, 2 Dec 2020 22:02:50 +0900, Amit Langote > wrote in > > Hello, > > > > On Tue, Dec 1, 2020 at 9:40 AM Kyotaro Horiguchi > > wrote: > > > > > > At Mon, 30 Nov 2020

Re: Huge memory consumption on partitioned table with FKs

2020-12-02 Thread Amit Langote
On Thu, Dec 3, 2020 at 2:29 PM Kyotaro Horiguchi wrote: > At Thu, 3 Dec 2020 12:27:53 +0900, Amit Langote > wrote in > > On Thu, Dec 3, 2020 at 10:15 AM Kyotaro Horiguchi > > wrote: > > > I don't think you're missing something. As I (tried to..) mentioned i

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Amit Langote
On Thu, Dec 3, 2020 at 5:13 PM Kyotaro Horiguchi wrote: > At Thu, 3 Dec 2020 16:41:45 +0900, Amit Langote > wrote in > > Maybe I misread but I think you did in your email dated Dec 1 where you > > said: > > > > "After an off-list discussion, we confirmed t

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Amit Langote
d go away, perhaps replaced by an > already-looked-up-this-tuple hash sorta like what the cached nested loops > effort is doing. > > I've been meaning to give this a try when I got some spare time. This may > inspire me to try again. +1 for this line of work. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Huge memory consumption on partitioned table with FKs

2020-12-03 Thread Amit Langote
to this thread, but keeping constraint_id and constraint_root_id separate looked better for documenting the partitioning case as working differently from the regular table case. I guess a comment in ri_BuildQueryKey is enough for that though and it's not like we're using constraint_roo

Re: ModifyTable overheads in generic plans

2020-12-06 Thread Amit Langote
On Thu, Nov 12, 2020 at 5:04 PM Amit Langote wrote: > Attached new 0002 which does these adjustments. I went with > ri_RootTargetDesc to go along with ri_RelationDesc. > > Also, I have updated the original 0002 (now 0003) to make > GetChildToRootMap() use ri_RootTarget

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
with the fact that a partition's param info remains attached to the parent's RI_ConstraintInfo even after it's detached from the parent table using DETACH PARTITION. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
Hi Alvaro, On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > On 2020-Dec-07, Amit Langote wrote: > > > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > > wrote: > > > > Also, the comment that was in RI_ConstraintInfo now appears in > > > &

Re: Huge memory consumption on partitioned table with FKs

2020-12-07 Thread Amit Langote
On Tue, Dec 8, 2020 at 12:04 PM Kyotaro Horiguchi wrote: > At Tue, 8 Dec 2020 01:16:00 +0900, Amit Langote > wrote in > > Hi Alvaro, > > > > On Mon, Dec 7, 2020 at 23:48 Alvaro Herrera wrote: > > > > > On 2020-Dec-07, Amit Langote wrote: > > >

Re: Huge memory consumption on partitioned table with FKs

2020-12-08 Thread Amit Langote
On Mon, Dec 7, 2020 at 11:01 PM Amit Langote wrote: > On Fri, Dec 4, 2020 at 12:05 PM Kyotaro Horiguchi > wrote: > > > Also, the comment that was in RI_ConstraintInfo now appears in > > > RI_ConstraintParam, and the new struct (RI_ConstraintInfo) is now > >

Re: a misbehavior of partition row movement (?)

2020-12-14 Thread Amit Langote
On Fri, Nov 20, 2020 at 8:55 PM Amit Langote wrote: > On Sat, Oct 3, 2020 at 8:26 PM Amit Langote wrote: > > On Sat, Oct 3, 2020 at 8:15 PM Tomas Vondra > > wrote > > > I think we need to differentiate between master and backbranches. IMO we > > > should try

Re: a misbehavior of partition row movement (?)

2020-12-14 Thread Amit Langote
r for partitioned tables for non RI-cases as well. Maybe we could do that, but I don't know of a known issue where the root cause is our firing of a row trigger on the leaf partition instead of on the root partitioned table. -- Amit Langote EDB: http://www.enterprisedb.com

Re: a misbehavior of partition row movement (?)

2020-12-14 Thread Amit Langote
On Tue, Dec 15, 2020 at 12:43 PM Amit Langote wrote: > Quoting your original example: > > drop table a, b; > create table a (id serial, primary key (id)) partition by range (id); > create table b (id serial, primary key (id)) partition by range (id); > alter table b add const

Re: a misbehavior of partition row movement (?)

2020-12-21 Thread Amit Langote
ed difficulty of implementing it the other way around, that is, of firing AFTER UPDATE triggers in this case. As for the original issue of this thread, it happens to be fixed by firing the *internal* AFTER UPDATE triggers that are involved in enforcing the foreign key even if the UPDATE has been tur

Re: a misbehavior of partition row movement (?)

2020-12-21 Thread Amit Langote
On Tue, Dec 22, 2020 at 4:16 PM Amit Langote wrote: > On Mon, Dec 21, 2020 at 11:30 PM Arne Roland wrote: > > thanks for the quick reply! Sadly I have been busy and the second part of > > your patch does no longer apply in src/include/nodes/execnodes.h:497. > > I don&#

Re: ModifyTable overheads in generic plans

2020-12-22 Thread Amit Langote
On Mon, Dec 7, 2020 at 3:53 PM Amit Langote wrote: > > On Thu, Nov 12, 2020 at 5:04 PM Amit Langote wrote: > > Attached new 0002 which does these adjustments. I went with > > ri_RootTargetDesc to go along with ri_RelationDesc. > > > > Also, I have updated the or

Re: Huge memory consumption on partitioned table with FKs

2021-01-07 Thread Amit Langote
orrect? > > I have confirmed that this patch can be applied to HEAD (master), > and that check-world PASS. Thanks for checking. Indeed, it should have been added to the January commit-fest. I've added it to the March one: https://commitfest.postgresql.org/32/2930/ -- Amit Langote EDB: http://www.enterprisedb.com

Re: a misbehavior of partition row movement (?)

2021-01-08 Thread Amit Langote
current approach. Someone also said this during the discussion: "Regarding the trigger issue, I can't claim to have a terribly strong opinion on this. I think that practically anything we do here might upset somebody, but probably any halfway-reasonable thing we choose to do will be OK for most people." So what we've got is that "halfway-reasonable" thing, YMMV. :) -- Amit Langote EDB: http://www.enterprisedb.com

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-12 Thread Amit Langote
On Thu, Feb 11, 2021 at 4:43 PM Greg Nancarrow wrote: > On Thu, Feb 11, 2021 at 5:33 PM Greg Nancarrow wrote: > > On Tue, Feb 9, 2021 at 1:04 AM Amit Langote wrote: > > > > > > * I think that the concerns raised by Tsunakawa-san in: > > > > &g

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-15 Thread Amit Langote
aybe the condition above could be changed as this: /* * Check whether we support copying data out of the specified relation, * unless the caller also passed a non-NULL data_dest_cb, in which case, * the callback will take care of it */ if (rel != NULL && rel->rd_rel->relkind != RELKIND_RELATION && data_dest_cb == NULL) I just checked that this works or at least doesn't break any newly added tests. -- Amit Langote EDB: http://www.enterprisedb.com

Re: A reloption for partitioned tables - parallel_workers

2021-02-15 Thread Amit Langote
/* Generate a partial append path. */ appendpath = create_append_path(root, rel, NIL, partial_subpaths, NIL, NULL, parallel_workers, enable_parallel_append, -1); Note that the 'rel' in this code refers to the partitioned table for which an Append path is being considered, so compute_parallel_worker() using that 'rel' would use the partitioned table's rel_parallel_workers as you are trying to do. -- Amit Langote EDB: http://www.enterprisedb.com

Re: a misbehavior of partition row movement (?)

2021-02-15 Thread Amit Langote
Thank you for looking at this. On Mon, Feb 15, 2021 at 4:12 PM Masahiko Sawada wrote: > On Wed, Jan 20, 2021 at 7:04 PM Amit Langote wrote: > > This shows that the way we've made these triggers behave in general > > can cause some unintended behaviors for foreign keys during

Re: A reloption for partitioned tables - parallel_workers

2021-02-15 Thread Amit Langote
Hi Seamus, Please see my reply below. On Tue, Feb 16, 2021 at 1:35 AM Seamus Abshere wrote: > On Mon, Feb 15, 2021, at 3:53 AM, Amit Langote wrote: > > On Mon, Feb 15, 2021 at 5:28 PM Seamus Abshere wrote: > > > It turns out parallel_workers may be a useful reloption for cert

Re: A reloption for partitioned tables - parallel_workers

2021-02-15 Thread Amit Langote
Hi, On Tue, Feb 16, 2021 at 1:06 AM Laurenz Albe wrote: > On Mon, 2021-02-15 at 17:53 +0900, Amit Langote wrote: > > On Mon, Feb 15, 2021 at 5:28 PM Seamus Abshere wrote: > > > It turns out parallel_workers may be a useful reloption for certain uses > > > of partitio

Re: POC: postgres_fdw insert batching

2021-02-16 Thread Amit Langote
On Tue, Feb 16, 2021 at 1:36 AM Tomas Vondra wrote: > On 2/5/21 3:52 AM, Amit Langote wrote: > > Tsunakwa-san, > > > > On Mon, Jan 25, 2021 at 1:21 PM tsunakawa.ta...@fujitsu.com > > wrote: > >> From: Amit Langote > >>> Yes, it can be simplifie

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-16 Thread Amit Langote
On Mon, Feb 15, 2021 at 4:39 PM Greg Nancarrow wrote: > On Sat, Feb 13, 2021 at 12:17 AM Amit Langote wrote: > > On Thu, Feb 11, 2021 at 4:43 PM Greg Nancarrow wrote: > > > Actually, I tried adding the following in the loop that checks the > > > parallel-safety of e

Re: A reloption for partitioned tables - parallel_workers

2021-02-17 Thread Amit Langote
On Tue, Feb 16, 2021 at 11:02 PM Laurenz Albe wrote: > On Tue, 2021-02-16 at 16:29 +0900, Amit Langote wrote: > > > I am +1 on allowing to override the degree of parallelism on a parallel > > > append. If "parallel_workers" on the partitioned table is an optio

Re: POC: postgres_fdw insert batching

2021-02-17 Thread Amit Langote
On Wed, Feb 17, 2021 at 5:46 PM Amit Langote wrote: > On Wed, Feb 17, 2021 at 12:04 AM Tomas Vondra > wrote: > > On 2/16/21 10:25 AM, Amit Langote wrote: > > > On Tue, Feb 16, 2021 at 1:36 AM Tomas Vondra > > >> Thanks for the patch, it seems fine to me

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-17 Thread Amit Langote
On Wed, Feb 17, 2021 at 10:44 AM Greg Nancarrow wrote: > On Wed, Feb 17, 2021 at 12:19 AM Amit Langote wrote: > > On Mon, Feb 15, 2021 at 4:39 PM Greg Nancarrow wrote: > > > On Sat, Feb 13, 2021 at 12:17 AM Amit Langote > > > wrote: > > > > On Thu,

Re: POC: postgres_fdw insert batching

2021-02-17 Thread Amit Langote
tchSize() returned 0 later, > triggering an assert. > > It's probably better to require GetForeignModifyBatchSize() to always > return a valid batch size (>= 1). If batching is not supported, just > return 1. That makes sense. How about the attached? -- Amit Langote EDB: http://www.enterprisedb.com ForeignModifyBatchSize-sanity.patch Description: Binary data

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-17 Thread Amit Langote
On Thu, Feb 18, 2021 at 10:03 AM Greg Nancarrow wrote: > On Thu, Feb 18, 2021 at 12:34 AM Amit Langote wrote: > > All that is to say that we should move our code to add partition OIDs > > as plan dependencies to somewhere in set_plan_references(), which > >

Re: POC: postgres_fdw insert batching

2021-02-17 Thread Amit Langote
On Thu, Feb 18, 2021 at 4:36 AM Tomas Vondra wrote: > On 2/17/21 9:51 AM, Amit Langote wrote: > > On Wed, Feb 17, 2021 at 5:46 PM Amit Langote > > wrote: > >> Sorry, I hadn't shared enough details of my investigations when I > >> originally ran int

Re: A reloption for partitioned tables - parallel_workers

2021-02-18 Thread Amit Langote
ef struct PartitionedTableRdOptions { int32 vl_len_;/* varlena header (do not touch directly!) */ int parallel_workers; /* max number of parallel workers */ } PartitionedTableRdOptions; -- Amit Langote EDB: http://www.enterprisedb.com

Allow batched insert during cross-partition updates

2021-02-18 Thread Amit Langote
nal inserts, we'd need to make some changes to both the core code and postgres_fdw, which the attached patch implements. Details are in the commit message. -- Amit Langote EDB: http://www.enterprisedb.com v1-0001-Allow-batching-of-inserts-during-cross-partition-.patch Description: Binary data

Re: Allow batched insert during cross-partition updates

2021-02-18 Thread Amit Langote
On Thu, Feb 18, 2021 at 6:52 PM Amit Langote wrote: > > Based on the discussion at: > > https://www.postgresql.org/message-id/6929d485-2d2a-da46-3681-4a400a3d794f%40enterprisedb.com > > I'm posting the patch for $subject here in this new thread and I'll > add it t

Re: a misbehavior of partition row movement (?)

2021-02-18 Thread Amit Langote
the UPDATE reference page which has some notes on row movement or somewhere else. Do you have suggestions? Attaching rebased version of the patches for HEAD to appease the cfbot. -- Amit Langote EDB: http://www.enterprisedb.com v4-0001-Create-foreign-key-triggers-in-partitioned-tables.patch Descript

Re: A reloption for partitioned tables - parallel_workers

2021-02-18 Thread Amit Langote
On Tue, Feb 16, 2021 at 3:05 PM Amit Langote wrote: > On Tue, Feb 16, 2021 at 1:35 AM Seamus Abshere wrote: > > Here we go, my first patch... solves > > https://www.postgresql.org/message-id/7d6fdc20-857c-4cbe-ae2e-c0ff9520e...@www.fastmail.com > > Thanks for sending t

Re: A reloption for partitioned tables - parallel_workers

2021-02-19 Thread Amit Langote
On Fri, Feb 19, 2021 at 11:54 PM Seamus Abshere wrote: > On Fri, Feb 19, 2021, at 2:30 AM, Amit Langote wrote: > > Here is an updated version of the Seamus' patch that takes into > > account these and other comments received on this thread so far. > > Maybe warrants a

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread Amit Langote
On Fri, Feb 19, 2021 at 7:38 PM Amit Kapila wrote: > On Thu, Feb 18, 2021 at 11:05 AM Amit Langote wrote: > > > > > > It also occurred to me that we can avoid pointless adding of > > > > partitions if the final plan won't use parallelism. For that

Re: a misbehavior of partition row movement (?)

2021-02-21 Thread Amit Langote
On Fri, Feb 19, 2021 at 5:04 PM Masahiko Sawada wrote: > On Mon, Feb 15, 2021 at 10:37 PM Amit Langote wrote: > > Regarding the patch, I would have liked if it only prevented the > > update when the foreign key that would be violated by the component > > delete references th

Re: A reloption for partitioned tables - parallel_workers

2021-02-23 Thread Amit Langote
TableRdOptions +{ + int32 vl_len_; /* varlena header (do not touch directly!) */ + int parallel_workers; /* max number of parallel workers */ +} PartitionedTableRdOptions; + #define HEAP_MIN_FILLFACTOR 10 #define HEAP_DEFAULT_FILLFACTOR 100 -- Amit Langote EDB: http://www.enterprisedb.com

Re: [BUG] segfault during delete

2021-02-24 Thread Amit Langote
the block for populating the transition NEW TABLE, because without doing so that block too can crash similarly: if (!TupIsNull(newslot) && ((event == TRIGGER_EVENT_INSERT && insert_new_table) || (event == TRIGGER_EVENT_UPDATE && update_new_table))) I've attached a patch with my suggested fixes and also test cases. Please take a look. -- Amit Langote EDB: http://www.enterprisedb.com v2-0001-Fix-use-after-tree-bug-with-AfterTriggersTableDat.patch Description: Binary data

Re: [BUG] segfault during delete

2021-02-24 Thread Amit Langote
On Wed, Feb 24, 2021 at 6:12 PM Drouvot, Bertrand wrote: > On 2/24/21 9:12 AM, Amit Langote wrote: > > I've attached a patch with my suggested fixes and also test cases. > > Please take a look. > > I had a look and it looks good to me. Also the new regression tests ar

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
ses do the thing that produces a better plan [2]. While no compromises can be made for the former, whether or not to go for the latter probably involves some cost-benefit analysis, something we can probably revisit. I don't think we're compromising by not adding the partition OIDs when

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
Hi Greg, Replying to an earlier email in the thread because I think I found a problem relevant to the topic that was brought up. On Wed, Feb 17, 2021 at 10:34 PM Amit Langote wrote: > On Wed, Feb 17, 2021 at 10:44 AM Greg Nancarrow wrote: > > Is the use of "table_clo

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-25 Thread Amit Langote
On Fri, Feb 26, 2021 at 3:35 PM Greg Nancarrow wrote: > On Fri, Feb 26, 2021 at 4:07 PM Amit Langote wrote: > > The attached patch fixes this, although I am starting to have second > > thoughts about how we're tracking partitions in this patch. Wondering > > if we shou

Re: a misbehavior of partition row movement (?)

2021-02-25 Thread Amit Langote
such behaviour on a referencing table. > I think it is worth adding some explanation in this document. Thus, explaining > impact on referencing tables here, as it already describes behaviour of > UPDATE on a partitioned table. ISTM the description of the case that will now be prevented s

Re: [BUG] segfault during delete

2021-02-28 Thread Amit Langote
the test case > to pg11. It worked fine for me (with no code changes), but it seems good to > have it there just to make sure the buildfarm agrees with us on this. Ah, good call. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-28 Thread Amit Langote
On Mon, Mar 1, 2021 at 12:38 PM Greg Nancarrow wrote: > On Fri, Feb 26, 2021 at 5:50 PM Amit Langote wrote: > > > > On Fri, Feb 26, 2021 at 3:35 PM Greg Nancarrow wrote: > > > On Fri, Feb 26, 2021 at 4:07 PM Amit Langote > > > wrote: > > > >

Re: A reloption for partitioned tables - parallel_workers

2021-03-01 Thread Amit Langote
for partitioned tables, and I can see that it is actually wrong to access a partitioned table's parallel_workers through this macro as-is. Although I hadn't tested that, so thanks for pointing that out. > Like: > #define PartitionedTableGetParallelWorkers(relation, defaultpw) \ > xxx > (PartitionedTableRdOptions *) (relation)->rd_options)->parallel_workers : > (defaultpw)) I'm thinking it would be better to just modify the existing macro to check relkind to decide which struct pointer type to cast the value in rd_options to. I will post an updated patch later. -- Amit Langote EDB: http://www.enterprisedb.com

Re: simplifying foreign key/RI checks

2021-03-01 Thread Amit Langote
about the attached? >> > > Sorry for the delay. > I see that the changes were made as described. > Passes make check and make check-world yet again. > I'm marking this Ready For Committer unless someone objects. Thank you Corey for the review. -- Amit Langote EDB: http://www.enterprisedb.com

Re: A reloption for partitioned tables - parallel_workers

2021-03-01 Thread Amit Langote
rkers=0 on the parent table shouldn't really disable a regular (non-parallel-aware) Append running under Gather even if it does Parallel Append (parallel-aware)? So in this test case, there should have been a Gather atop Append, with individual partitions scanned using Parallel Seq Scan where app

Re: Increase value of OUTER_VAR

2021-03-03 Thread Amit Langote
y mean we'd not > need to revisit this for quite a while. +1 Also, I got reminded of this discussion from not so long ago: https://www.postgresql.org/message-id/flat/16302-e45634e2c0e34e97%40postgresql.org -- Amit Langote EDB: http://www.enterprisedb.com

Re: A reloption for partitioned tables - parallel_workers

2021-03-03 Thread Amit Langote
On Tue, Mar 2, 2021 at 5:47 PM Laurenz Albe wrote: > On Tue, 2021-03-02 at 11:23 +0900, Amit Langote wrote: > > +ALTER TABLE pagg_tab_ml SET (parallel_workers = 0); > > +EXPLAIN (COSTS OFF) > > +SELECT a FROM pagg_tab_ml WHERE b = 42; > > +

Re: Huge memory consumption on partitioned table with FKs

2021-03-03 Thread Amit Langote
Hi David, On Wed, Mar 3, 2021 at 10:21 PM David Steele wrote: > On 12/7/20 10:59 PM, Amit Langote wrote: > > On Tue, Dec 8, 2020 at 12:04 PM Kyotaro Horiguchi > > wrote: > >> At Tue, 8 Dec 2020 01:16:00 +0900, Amit Langote > >> wrote in > >>>

Re: Wired if-statement in gen_partprune_steps_internal

2021-03-03 Thread Amit Langote
On Tue, Oct 20, 2020 at 9:46 PM Amit Langote wrote: > On Tue, Oct 20, 2020 at 4:05 PM Andy Fan wrote: > > On Wed, Oct 14, 2020 at 11:26 AM Andy Fan wrote: > >> On Mon, Oct 12, 2020 at 4:37 PM Amit Langote > >> wrote: > >>> I think we should remov

Re: Extend more usecase for planning time partition pruning and init partition pruning.

2021-03-04 Thread Amit Langote
working with ScalarArrayOpExpr and BoolExpr to begin with? * Or maybe have you considered generalizing what build_implied_pruning_quals() does so that other places like indxpath.c can use the facility? -- Amit Langote EDB: http://www.enterprisedb.com

Re: A reloption for partitioned tables - parallel_workers

2021-03-05 Thread Amit Langote
On Fri, Mar 5, 2021 at 10:47 PM Laurenz Albe wrote: > On Wed, 2021-03-03 at 17:58 +0900, Amit Langote wrote: > > For example, with the attached PoC patch: > > I have incorporated your POC patch and added a regression test. > > I didn't test it thoroughly though. Thanks.

Re: Huge memory consumption on partitioned table with FKs

2021-03-07 Thread Amit Langote
On Fri, Mar 5, 2021 at 6:00 AM Tom Lane wrote: > Amit Langote writes: > > Updated patch attached. > > This claim seems false on its face: > > > All child constraints of a given foreign key constraint can use the > > same RI query and the resulting plan, that is,

Re: Wired if-statement in gen_partprune_steps_internal

2021-03-07 Thread Amit Langote
On Fri, Mar 5, 2021 at 7:50 AM Ryan Lambert wrote: > On Wed, Mar 3, 2021 at 11:03 PM Amit Langote wrote: >> >> Sorry, this seems to have totally slipped my mind. >> >> Attached is the patch I had promised. >> >> Also, I have updated the title of the CF ent

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-07 Thread Amit Langote
ireExecutorLocksOnPartitions() will do in the generic plan case. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Huge memory consumption on partitioned table with FKs

2021-03-08 Thread Amit Langote
EATE TABLE p (a int) partitioned by list(a) RESTRICTED". We can add these > limitation to restricted partitioned relation only. I think you'd agree that the topics you want to discuss deserve a separate discussion thread. You may refer to this discussion in that new thread if you thi

Re: Huge memory consumption on partitioned table with FKs

2021-03-08 Thread Amit Langote
On Mon, Mar 8, 2021 at 9:53 PM Andy Fan wrote: > On Mon, Mar 8, 2021 at 8:42 PM Amit Langote wrote: >> On Mon, Mar 8, 2021 at 8:39 PM Andy Fan wrote: >> > On Mon, Mar 8, 2021 at 3:43 PM Andy Fan wrote: >> >> My point below is a bit off-topic, but I want to

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-08 Thread Amit Langote
> 0001* patch or if you want to review it further? I just read through v25 and didn't find anything to complain about. -- Amit Langote EDB: http://www.enterprisedb.com

Re: simplifying foreign key/RI checks

2021-03-08 Thread Amit Langote
s is a cute idea, and the speedup is pretty impressive, > but I don't think it's anywhere near committable. I also wonder > whether we really want ri_triggers.c having its own copy of > low-level stuff like the tuple-locking code you copied. Seems > like a likely maintenance hazard, so maybe some more refactoring > is needed. Okay, I will see if there's a way to avoid copying too much code. -- Amit Langote EDB: http://www.enterprisedb.com

Re: TRUNCATE on foreign table

2021-03-08 Thread Amit Langote
) does. That leads to relids and relids_extra having different lengths, which trips the Assert in ExecuteTruncateGuts(). -- Amit Langote EDB: http://www.enterprisedb.com

Re: Huge memory consumption on partitioned table with FKs

2021-03-10 Thread Amit Langote
On Wed, Mar 10, 2021 at 8:37 AM Tom Lane wrote: > Amit Langote writes: > > On Fri, Mar 5, 2021 at 6:00 AM Tom Lane wrote: > >> This claim seems false on its face: > >>> All child constraints of a given foreign key constraint can use the > >>> same RI

Re: Huge memory consumption on partitioned table with FKs

2021-03-10 Thread Amit Langote
On Thu, Mar 11, 2021 at 4:25 AM Tom Lane wrote: > Amit Langote writes: > > On Wed, Mar 10, 2021 at 8:37 AM Tom Lane wrote: > >> Hmm. So, the key point is that the values coming from the partitioned > >> child table are injected into the test query as parameters, no

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-10 Thread Amit Langote
On Wed, Mar 10, 2021 at 6:18 PM Amit Kapila wrote: > On Mon, Mar 8, 2021 at 7:19 PM Amit Langote wrote: > > I just read through v25 and didn't find anything to complain about. > > Thanks a lot, pushed now! Amit L., your inputs are valuable for this work. Glad I could

Re: Allow batched insert during cross-partition updates

2021-03-11 Thread Amit Langote
or functions would be better, I will change the patch that way. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Allow batched insert during cross-partition updates

2021-03-11 Thread Amit Langote
is patch is for teaching those INSERTs to use batched mode if allowed, which is currently prohibited. So with this patch, if an UPDATE moves 10 rows from a local partition to a remote partition, then they will be inserted with a single INSERT command containing all 10 rows, instead of 10 separate IN

Re: Allow batched insert during cross-partition updates

2021-03-11 Thread Amit Langote
On Thu, Mar 11, 2021 at 8:36 PM wrote: > On Thursday, March 11, 2021 9:42 AM, Amit Langote > wrote: > > On Wed, Mar 10, 2021 at 9:30 PM Georgios gkokola...@protonmail.com wrote: > > > > > I continued looking a bit at the patch, yet I am either failing to see > &g

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-11 Thread Amit Langote
the patch for adding parallel_workers option to partitioned tables does? -- Amit Langote EDB: http://www.enterprisedb.com

Re: Allow batched insert during cross-partition updates

2021-03-15 Thread Amit Langote
Hi Georgios, On Fri, Mar 12, 2021 at 7:59 PM wrote: > On Friday, March 12, 2021 3:45 AM, Amit Langote > wrote: > > On Thu, Mar 11, 2021 at 8:36 PM gkokola...@pm.me wrote: > > > On Thursday, March 11, 2021 9:42 AM, Amit Langote amitlangot...@gmail.com > > > wro

Re: Allow batched insert during cross-partition updates

2021-03-16 Thread Amit Langote
Hi Georgios, On Tue, Mar 16, 2021 at 5:12 PM wrote: > On Tuesday, March 16, 2021 6:13 AM, Amit Langote > wrote: > > On Fri, Mar 12, 2021 at 7:59 PM gkokola...@pm.me wrote: > > > On Friday, March 12, 2021 3:45 AM, Amit Langote amitlangot...@gmail.com > > > wrote

Re: simplifying foreign key/RI checks

2021-03-16 Thread Amit Langote
On Mon, Mar 8, 2021 at 11:41 PM Amit Langote wrote: > On Thu, Mar 4, 2021 at 5:15 AM Tom Lane wrote: > > Lastly, ri_PerformCheck is pretty careful about not only which > > snapshot it uses, but which *pair* of snapshots it uses, because > > sometimes it needs to worry abo

crash during cascaded foreign key update

2021-03-16 Thread Amit Langote
) at postmaster.c:1736 #32 0x00891a4b in PostmasterMain (argc=3, argv=0x1b3cc20) at postmaster.c:1408 #33 0x0079360f in main (argc=3, argv=0x1b3cc20) at main.c:209 I haven't checked the failure in more detail yet other than that the failed Assert was added in 5db6df0c0117. -- Amit Langote EDB: http://www.enterprisedb.com

Re: crash during cascaded foreign key update

2021-03-16 Thread Amit Langote
On Tue, Mar 16, 2021 at 11:17 PM Tom Lane wrote: > Amit Langote writes: > > With HEAD (I think v12 and greater), I see $subject when trying out > > the following scenario: > > I wonder if this is related to > > https://www.postgresql.org/message-id/flat/89429.1584443

Re: Parallel Append can break run-time partition pruning

2020-10-26 Thread Amit Langote
On Sun, Oct 25, 2020 at 10:06 AM David Rowley wrote: > On Fri, 16 Oct 2020 at 03:01, Amit Langote wrote: > > > > Going over the last few emails, it seems that David held off from > > committing the patch, because of the lack of confidence in its > > robustne

Re: partition routing layering in nodeModifyTable.c

2020-10-27 Thread Amit Langote
On Tue, Oct 27, 2020 at 10:23 PM Heikki Linnakangas wrote: > On 23/10/2020 12:37, Amit Langote wrote: > > To explain these numbers a bit, "overheaul update/delete processing" > > patch improves the performance of that benchmark by allowing the > > updates to use

Re: partition routing layering in nodeModifyTable.c

2020-10-28 Thread Amit Langote
tion in > postgres_fdw. Ah, calling BeginDirectModify() itself lazily sounds like a good idea; see attached updated 0001 to see how that looks. While updating that patch, I realized that the ForeignScan.resultRelation that we introduced in 178f2d560d will now be totally useless. :-( -- Amit Langote EDB: htt

Re: partition routing layering in nodeModifyTable.c

2020-10-29 Thread Amit Langote
On Wed, Oct 28, 2020 at 12:02 PM Amit Langote wrote: > On Tue, Oct 27, 2020 at 10:23 PM Heikki Linnakangas wrote: > > But since this applies on top of the "overhaul update/delete processing" > > patch, let's tackle that patch set next. Could you rebase that, please?

Re: MINUS SIGN (U+2212) in EUC-JP encoding is mapped to FULLWIDTH HYPHEN-MINUS (U+FF0D) in UTF-8

2020-10-29 Thread Amit Langote
b/src/backend/utils/mb/Unicode/euc_jp_to_utf8.map - {0xa1dd, 0xe28892}, + {0xa1dd, 0xefbc8d}, --- a/src/backend/utils/mb/Unicode/utf8_to_euc_jp.map +++ b/src/backend/utils/mb/Unicode/utf8_to_euc_jp.map - {0xe28892, 0xa1dd}, + {0xefbc8d, 0xa1dd}, Can't tell what reason there was to do that, but there must have been some. Maybe the Japanese character sets prefer full-width hyphen minus (unicode U+FF0D) over mathematical minus sign (U+2212)? -- Amit Langote EDB: http://www.enterprisedb.com

Re: MINUS SIGN (U+2212) in EUC-JP encoding is mapped to FULLWIDTH HYPHEN-MINUS (U+FF0D) in UTF-8

2020-10-29 Thread Amit Langote
don't change authoritative mappings, but maybe can add some > one-way conversions for the convenience. Maybe UCS_TO_EUC_JP.pl could do something like the above. Are there other cases that were fixed like this in the past, either for euc_jp or sjis? -- Amit Langote EDB: http://www.enterprisedb.com

Re: partition routing layering in nodeModifyTable.c

2020-10-29 Thread Amit Langote
On Wed, Oct 28, 2020 at 4:46 PM Amit Langote wrote: > > On Tue, Oct 27, 2020 at 10:23 PM Heikki Linnakangas wrote: > > This patch looks reasonable to me at a quick glance. I'm a bit worried > > or unhappy about the impact on FDWs, though. It doesn't seem nice that

  1   2   3   4   5   6   7   8   9   10   >