Re: generic plans and "initial" pruning

2024-10-15 Thread Tom Lane
Robert Haas writes: > On Fri, Oct 11, 2024 at 3:30 AM Amit Langote wrote: >> Maybe just "relids" suffices with a comment updated like this: >> >> * relids RelOptInfo.relids of the parent plan node (e.g. Append >> * or MergeAppend) to which his PartitionPruneInf

Re: generic plans and "initial" pruning

2024-10-15 Thread Robert Haas
On Fri, Oct 11, 2024 at 3:30 AM Amit Langote wrote: > Maybe just "relids" suffices with a comment updated like this: > > * relids RelOptInfo.relids of the parent plan node (e.g. Append > * or MergeAppend) to which his PartitionPruneInfo node > *

Re: generic plans and "initial" pruning

2024-10-11 Thread Amit Langote
Robert, On Fri, Oct 11, 2024 at 5:15 AM Robert Haas wrote: > > Hi Amit, > > This is not a full review (sorry!) but here are a few comments. Thank you for taking a look. > In general, I don't have a problem with this direction. I thought > Tom's previous proposal of abandoning ExecInitNode() in

Re: generic plans and "initial" pruning

2024-10-10 Thread Robert Haas
Hi Amit, This is not a full review (sorry!) but here are a few comments. In general, I don't have a problem with this direction. I thought Tom's previous proposal of abandoning ExecInitNode() in medias res if we discover that we need to replan was doable and I still think that, but ISTM that this

Re: generic plans and "initial" pruning

2024-09-02 Thread Amit Langote
On Sat, Aug 31, 2024 at 9:30 PM Junwang Zhao wrote: > @@ -1241,7 +1244,7 @@ GetCachedPlan(CachedPlanSource *plansource, > ParamListInfo boundParams, > if (customplan) > { > /* Build a custom plan */ > - plan = BuildCachedPlan(plansource, qlist, boundParams, queryEnv); > + plan = BuildCachedP

Re: generic plans and "initial" pruning

2024-08-31 Thread Junwang Zhao
Hi, On Thu, Aug 29, 2024 at 9:34 PM Amit Langote wrote: > > On Fri, Aug 23, 2024 at 9:48 PM Amit Langote wrote: > > On Wed, Aug 21, 2024 at 10:10 PM Robert Haas wrote: > > > On Wed, Aug 21, 2024 at 8:45 AM Amit Langote > > > wrote: > > > > * The replanning aspect of the lock-in-the-executor d

Re: generic plans and "initial" pruning

2024-08-23 Thread Amit Langote
On Wed, Aug 21, 2024 at 10:10 PM Robert Haas wrote: > On Wed, Aug 21, 2024 at 8:45 AM Amit Langote wrote: > > * The replanning aspect of the lock-in-the-executor design would be > > simpler if a CachedPlan contained the plan for a single query rather > > than a list of queries, as previously ment

Re: generic plans and "initial" pruning

2024-08-21 Thread Robert Haas
On Wed, Aug 21, 2024 at 8:45 AM Amit Langote wrote: > * The replanning aspect of the lock-in-the-executor design would be > simpler if a CachedPlan contained the plan for a single query rather > than a list of queries, as previously mentioned. This is particularly > due to the requirements of the

Re: generic plans and "initial" pruning

2024-08-21 Thread Amit Langote
On Tue, Aug 20, 2024 at 11:53 PM Robert Haas wrote: > On Tue, Aug 20, 2024 at 9:00 AM Amit Langote wrote: > > I think we'd modify plancache.c to postpone the locking of only > > prunable relations (i.e., partitions), so we're looking at only a > > handful of concurrent modifications that are goin

Re: generic plans and "initial" pruning

2024-08-20 Thread Robert Haas
On Tue, Aug 20, 2024 at 9:00 AM Amit Langote wrote: > I think we'd modify plancache.c to postpone the locking of only > prunable relations (i.e., partitions), so we're looking at only a > handful of concurrent modifications that are going to cause execution > errors. That's because we disallow ma

Re: generic plans and "initial" pruning

2024-08-20 Thread Amit Langote
On Tue, Aug 20, 2024 at 3:21 AM Robert Haas wrote: > On Mon, Aug 19, 2024 at 1:52 PM Tom Lane wrote: > > Robert Haas writes: > > > But that seems somewhat incidental to what this thread is about. > > > > Perhaps. But if we're running into issues related to that, it might > > be good to set asid

Re: generic plans and "initial" pruning

2024-08-20 Thread Amit Langote
On Tue, Aug 20, 2024 at 1:39 AM Robert Haas wrote: > On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: > > So it is possible for the executor to try to run a plan that has > > become invalid since it was created, so... > > I'm not sure what the "so what" here is. I meant that if the executor h

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Mon, Aug 19, 2024 at 1:52 PM Tom Lane wrote: > Robert Haas writes: > > But that seems somewhat incidental to what this thread is about. > > Perhaps. But if we're running into issues related to that, it might > be good to set aside the long-term goal for a bit and come up with > a cleaner answ

Re: generic plans and "initial" pruning

2024-08-19 Thread Tom Lane
Robert Haas writes: > But that seems somewhat incidental to what this thread is about. Perhaps. But if we're running into issues related to that, it might be good to set aside the long-term goal for a bit and come up with a cleaner answer for intra-session locking. That could allow the pruning

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Mon, Aug 19, 2024 at 12:54 PM Tom Lane wrote: > What the examples here are showing is that AcquireExecutorLocks > is incomplete because it only provides defenses against DDL > initiated by other sessions, not by our own session. We have > CheckTableNotInUse but I'm not sure if it could be appl

Re: generic plans and "initial" pruning

2024-08-19 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: >> So it is possible for the executor to try to run a plan that has >> become invalid since it was created, so... > I'm not sure what the "so what" here is. The fact that there are holes in our protections against that do

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: > So it is possible for the executor to try to run a plan that has > become invalid since it was created, so... I'm not sure what the "so what" here is. > One perhaps crazy idea [1]: > > What if we remove AcquireExecutorLocks() and move the res

Re: generic plans and "initial" pruning

2024-08-16 Thread Amit Langote
On Fri, Aug 16, 2024 at 12:35 AM Robert Haas wrote: > On Thu, Aug 15, 2024 at 8:57 AM Amit Langote wrote: > > TBH, it's more of a hunch that people who are not involved in this > > development might find the new reality, whereby the execution is not > > racefree until ExecutorRun(), hard to reaso

Re: generic plans and "initial" pruning

2024-08-15 Thread Robert Haas
On Thu, Aug 15, 2024 at 8:57 AM Amit Langote wrote: > TBH, it's more of a hunch that people who are not involved in this > development might find the new reality, whereby the execution is not > racefree until ExecutorRun(), hard to reason about. I'm confused by what you mean here by "racefree". A

Re: generic plans and "initial" pruning

2024-08-14 Thread Robert Haas
On Mon, Aug 12, 2024 at 8:54 AM Amit Langote wrote: > 1. I went through many iterations of the changes to ExecInitNode() to > return a partially initialized PlanState tree when it detects that the > CachedPlan was invalidated after locking a child table and to > ExecEndNode() to account for the Pl

Re: generic plans and "initial" pruning

2024-08-12 Thread Amit Langote
On Thu, Jun 20, 2024 at 2:09 AM Alvaro Herrera wrote: > I hope we can get this new executor code in 18. Thanks for doing the benchmark, Alvaro, and sorry for the late reply. Yes, I'm hoping to get *some* version of this into v18. I've been thinking how to move this forward and I'm starting to t

Re: generic plans and "initial" pruning

2024-06-19 Thread Alvaro Herrera
I had occasion to run the same benchmark you described in the initial email in this thread. To do so I applied patch series v49 on top of 07cb29737a4e, which is just one that happened to have the same date as v49. I then used a script like this (against a server having plan_cache_mode=force_gener

Re: generic plans and "initial" pruning

2024-05-20 Thread Amit Langote
On Sun, May 19, 2024 at 9:39 AM David Rowley wrote: > For #1, the locks taken for SELECT queries are less likely to conflict > with other locks obtained by PostgreSQL, but at least at the moment if > someone is getting deadlocks with a DDL type operation, they can > change their query or DDL scrip

Re: generic plans and "initial" pruning

2024-05-18 Thread David Rowley
On Sun, 19 May 2024 at 13:27, Tom Lane wrote: > > David Rowley writes: > > 1. No ability to control the order that the locks are obtained. The > > order in which the locks are taken will be at the mercy of the plan > > the planner chooses. > > I do not think I buy this argument, because plancache

Re: generic plans and "initial" pruning

2024-05-18 Thread Tom Lane
David Rowley writes: > With the caveat of not yet having looked at the latest patch, my > thoughts are that having the executor startup responsible for taking > locks is a bad idea and I don't think we should go down this path. OK, it's certainly still up for argument, but ... > 1. No ability to

Re: generic plans and "initial" pruning

2024-05-18 Thread David Rowley
On Fri, 20 Jan 2023 at 08:39, Tom Lane wrote: > I spent some time re-reading this whole thread, and the more I read > the less happy I got. We are adding a lot of complexity and introducing > coding hazards that will surely bite somebody someday. And after awhile > I had what felt like an epipha

Re: generic plans and "initial" pruning

2024-04-08 Thread Amit Langote
Hi Andrey, On Sun, Mar 31, 2024 at 2:03 PM Andrey M. Borodin wrote: > > On 6 Dec 2023, at 23:52, Robert Haas wrote: > > > > I hope that it's at least somewhat useful. > > > On 5 Jan 2024, at 15:46, vignesh C wrote: > > > > There is a leak reported > > Hi Amit, > > this is a kind reminder that

Re: generic plans and "initial" pruning

2024-03-30 Thread Andrey M. Borodin
> On 6 Dec 2023, at 23:52, Robert Haas wrote: > > I hope that it's at least somewhat useful. > > On 5 Jan 2024, at 15:46, vignesh C wrote: > > There is a leak reported Hi Amit, this is a kind reminder that some feedback on your patch[0] is waiting for your reply. Thank you for your w

Re: generic plans and "initial" pruning

2024-01-05 Thread vignesh C
On Mon, 20 Nov 2023 at 10:00, Amit Langote wrote: > > On Thu, Sep 28, 2023 at 5:26 PM Amit Langote wrote: > > On Tue, Sep 26, 2023 at 10:06 PM Amit Langote > > wrote: > > > After sleeping on this, I think we do need the checks after all the > > > ExecInitNode() calls too, because we have many i

Re: generic plans and "initial" pruning

2023-12-06 Thread Robert Haas
Reviewing 0001: Perhaps ExecEndCteScan needs an adjustment. What if node->leader was never set? Other than that, I think this is in good shape. Maybe there are other things we'd want to adjust here, or maybe there aren't, but there doesn't seem to be any good reason to bundle more changes into th

Re: generic plans and "initial" pruning

2023-09-25 Thread Amit Langote
On Wed, Sep 6, 2023 at 11:20 PM Robert Haas wrote: > On Wed, Sep 6, 2023 at 5:12 AM Amit Langote wrote: > > Attached updated patches. Thanks for the review. > > I think 0001 looks ready to commit. I'm not sure that the commit > message needs to mention future patches here, since this code cleanu

Re: generic plans and "initial" pruning

2023-09-06 Thread Robert Haas
On Wed, Sep 6, 2023 at 5:12 AM Amit Langote wrote: > Attached updated patches. Thanks for the review. I think 0001 looks ready to commit. I'm not sure that the commit message needs to mention future patches here, since this code cleanup seems like a good idea regardless, but if you feel otherwis

Re: generic plans and "initial" pruning

2023-09-05 Thread Robert Haas
On Tue, Sep 5, 2023 at 3:13 AM Amit Langote wrote: > Attached 0001 removes unnecessary cleanup calls from ExecEnd*() routines. It also adds a few random Assert()s to verify that unrelated pointers are not NULL. I suggest that it shouldn't do that. The commit message doesn't mention the removal o

Re: generic plans and "initial" pruning

2023-08-28 Thread Robert Haas
On Fri, Aug 11, 2023 at 9:50 AM Amit Langote wrote: > After removing the unnecessary cleanup code from most node types’ ExecEnd* > functions, one thing I’m tempted to do is remove the functions that do > nothing else but recurse to close the outerPlan, innerPlan child nodes. We > could instead

Re: generic plans and "initial" pruning

2023-08-11 Thread Amit Langote
On Fri, Aug 11, 2023 at 14:31 Amit Langote wrote: > On Wed, Aug 9, 2023 at 1:05 AM Robert Haas wrote: > > On Tue, Aug 8, 2023 at 10:32 AM Amit Langote > wrote: > > > But should ExecInitNode() subroutines return the partially initialized > > > PlanState node or NULL on detecting invalidation? I

Re: generic plans and "initial" pruning

2023-08-08 Thread Robert Haas
On Tue, Aug 8, 2023 at 10:32 AM Amit Langote wrote: > But should ExecInitNode() subroutines return the partially initialized > PlanState node or NULL on detecting invalidation? If I'm > understanding how you think this should be working correctly, I think > you mean the former, because if it were

Re: generic plans and "initial" pruning

2023-08-08 Thread Amit Langote
On Tue, Aug 8, 2023 at 12:36 AM Robert Haas wrote: > On Thu, Aug 3, 2023 at 4:37 AM Amit Langote wrote: > > Here's a patch set where the refactoring to move the ExecutorStart() > > calls to be closer to GetCachedPlan() (for the call sites that use a > > CachedPlan) is extracted into a separate pa

Re: generic plans and "initial" pruning

2023-08-07 Thread Robert Haas
On Mon, Aug 7, 2023 at 11:44 AM Tom Lane wrote: > Right, I doubt that changing that is going to work out well. > Hash joins might have issues with it too. I thought about the case, because Hash and Hash Join are such closely intertwined nodes, but I don't see any problem there. It doesn't really

Re: generic plans and "initial" pruning

2023-08-07 Thread Tom Lane
Robert Haas writes: > Second, I wondered whether the ordering of cleanup operations could be > an issue. Right now, a node can position cleanup code before, after, > or both before and after recursing to child nodes, whereas with this > design change, the cleanup code will always be run before rec

Re: generic plans and "initial" pruning

2023-08-07 Thread Robert Haas
On Thu, Aug 3, 2023 at 4:37 AM Amit Langote wrote: > Here's a patch set where the refactoring to move the ExecutorStart() > calls to be closer to GetCachedPlan() (for the call sites that use a > CachedPlan) is extracted into a separate patch, 0002. Its commit > message notes an aspect of this ref

Re: generic plans and "initial" pruning

2023-07-18 Thread Thom Brown
On Tue, 18 Jul 2023, 08:26 Amit Langote, wrote: > Hi Thom, > > On Tue, Jul 18, 2023 at 1:33 AM Thom Brown wrote: > > On Thu, 13 Jul 2023 at 13:59, Amit Langote > wrote: > > > In an absolutely brown-paper-bag moment, I realized that I had not > > > updated src/backend/executor/README to reflect

Re: generic plans and "initial" pruning

2023-07-17 Thread Thom Brown
On Thu, 13 Jul 2023 at 13:59, Amit Langote wrote: > In an absolutely brown-paper-bag moment, I realized that I had not > updated src/backend/executor/README to reflect the changes to the > executor's control flow that this patch makes. That is, after > scrapping the old design back in January wh

Re: generic plans and "initial" pruning

2023-07-03 Thread Daniel Gustafsson
> On 8 Jun 2023, at 16:23, Amit Langote wrote: > > Here is a new version. The local planstate variable in the hunk below is shadowing the function parameter planstate which cause a compiler warning: @@ -1495,18 +1556,15 @@ ExecEndPlan(PlanState *planstate, EState *estate) ListCell *l;

Re: generic plans and "initial" pruning

2023-04-05 Thread Amit Langote
On Tue, Apr 4, 2023 at 10:29 PM Amit Langote wrote: > On Tue, Apr 4, 2023 at 6:41 AM Tom Lane wrote: > > A few concrete thoughts: > > > > * I understand that your plan now is to acquire locks on all the > > originally-named tables, then do permissions checks (which will > > involve only those tab

Re: generic plans and "initial" pruning

2023-04-04 Thread Amit Langote
On Tue, Apr 4, 2023 at 6:41 AM Tom Lane wrote: > Amit Langote writes: > > [ v38 patchset ] > > I spent a little bit of time looking through this, and concluded that > it's not something I will be wanting to push into v16 at this stage. > The patch doesn't seem very close to being committable on i

Re: generic plans and "initial" pruning

2023-04-03 Thread Tom Lane
Amit Langote writes: > [ v38 patchset ] I spent a little bit of time looking through this, and concluded that it's not something I will be wanting to push into v16 at this stage. The patch doesn't seem very close to being committable on its own terms, and even if it was now is not a great time in

Re: generic plans and "initial" pruning

2023-03-02 Thread Amit Langote
On Wed, Feb 8, 2023 at 7:31 PM Amit Langote wrote: > On Tue, Feb 7, 2023 at 23:38 Andres Freund wrote: >> The tests seem to frequently hang on freebsd: >> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3478 > > Thanks for the heads up. I’ve noticed this one too, thoug

Re: generic plans and "initial" pruning

2023-02-08 Thread Amit Langote
On Tue, Feb 7, 2023 at 23:38 Andres Freund wrote: > Hi, > > On 2023-02-03 22:01:09 +0900, Amit Langote wrote: > > I've added a test case under src/modules/delay_execution by adding a > > new ExecutorStart_hook that works similarly as > > delay_execution_planner(). The test works by allowing a co

Re: generic plans and "initial" pruning

2023-02-07 Thread Andres Freund
Hi, On 2023-02-03 22:01:09 +0900, Amit Langote wrote: > I've added a test case under src/modules/delay_execution by adding a > new ExecutorStart_hook that works similarly as > delay_execution_planner(). The test works by allowing a concurrent > session to drop an object being referenced in a cach

Re: generic plans and "initial" pruning

2023-02-03 Thread Amit Langote
On Thu, Feb 2, 2023 at 11:49 PM Amit Langote wrote: > On Fri, Jan 27, 2023 at 4:01 PM Amit Langote wrote: > > I didn't actually go with calling the plancache on every lock taken on > > a relation, that is, in ExecGetRangeTableRelation(). One thing about > > doing it that way that I didn't quite

Re: generic plans and "initial" pruning

2023-02-02 Thread Amit Langote
On Fri, Jan 27, 2023 at 4:01 PM Amit Langote wrote: > On Fri, Jan 20, 2023 at 12:52 PM Amit Langote wrote: > > Alright, I'll try to get something out early next week. Thanks for > > all the pointers. > > Sorry for the delay. Attached is what I've come up with so far. > > I didn't actually go wi

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 12:58 PM Tom Lane wrote: > Amit Langote writes: > > On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: > >> It might be possible to incorporate this pointer into PlannedStmt > >> instead of passing it separately. > > > Yeah, that would be less churn. Though, I wonder if yo

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
Amit Langote writes: > On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: >> It might be possible to incorporate this pointer into PlannedStmt >> instead of passing it separately. > Yeah, that would be less churn. Though, I wonder if you still hold > that PlannedStmt should not be scribbled upon

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: > Amit Langote writes: > > On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: > >> I had what felt like an epiphany: the whole problem arises because the > >> system is wrongly factored. We should get rid of AcquireExecutorLocks > >> altogether, all

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
Amit Langote writes: > On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: >> I had what felt like an epiphany: the whole problem arises because the >> system is wrongly factored. We should get rid of AcquireExecutorLocks >> altogether, allowing the plancache to hand back a generic plan that >> it's

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: > I spent some time re-reading this whole thread, and the more I read > the less happy I got. Thanks a lot for your time on this. > We are adding a lot of complexity and introducing > coding hazards that will surely bite somebody someday. And aft

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
I spent some time re-reading this whole thread, and the more I read the less happy I got. We are adding a lot of complexity and introducing coding hazards that will surely bite somebody someday. And after awhile I had what felt like an epiphany: the whole problem arises because the system is wron

Re: generic plans and "initial" pruning

2022-12-21 Thread Tom Lane
Alvaro Herrera writes: > This version of the patch looks not entirely unreasonable to me. I'll > set this as Ready for Committer in case David or Tom or someone else > want to have a look and potentially commit it. I will have a look during the January CF. regards, tom l

Re: generic plans and "initial" pruning

2022-12-21 Thread Amit Langote
On Wed, Dec 21, 2022 at 7:18 PM Alvaro Herrera wrote: > This version of the patch looks not entirely unreasonable to me. I'll > set this as Ready for Committer in case David or Tom or someone else > want to have a look and potentially commit it. Thank you, Alvaro. -- Thanks, Amit Langote EDB:

Re: generic plans and "initial" pruning

2022-12-21 Thread Alvaro Herrera
This version of the patch looks not entirely unreasonable to me. I'll set this as Ready for Committer in case David or Tom or someone else want to have a look and potentially commit it. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/

Re: generic plans and "initial" pruning

2022-12-15 Thread Amit Langote
On Wed, Dec 14, 2022 at 5:35 PM Amit Langote wrote: > I have moved the original functionality of GetCachedPlan() to > GetCachedPlanInternal(), turning the former into a sort of controller > as described shortly. The latter's CheckCachedPlan() part now only > locks the "minimal" set of, non-prunab

Re: generic plans and "initial" pruning

2022-12-14 Thread Amit Langote
On Tue, Dec 13, 2022 at 2:24 AM Alvaro Herrera wrote: > On 2022-Dec-12, Amit Langote wrote: > > I started feeling like putting all the new logic being added > > by this patch into plancache.c at the heart of GetCachedPlan() and > > tweaking its API in kind of unintuitive ways may not have been suc

Re: generic plans and "initial" pruning

2022-12-12 Thread Alvaro Herrera
On 2022-Dec-12, Amit Langote wrote: > I started feeling like putting all the new logic being added > by this patch into plancache.c at the heart of GetCachedPlan() and > tweaking its API in kind of unintuitive ways may not have been such a > good idea to begin with. So I started thinking again ab

Re: generic plans and "initial" pruning

2022-12-12 Thread Amit Langote
On Fri, Dec 9, 2022 at 8:37 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > > Pruning will be done afresh on every fetch of a given cached plan when > > CheckCachedPlan() is called on it, so the part_prune_results_list part > > will be discarded and rebuilt as many times as the

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > Pruning will be done afresh on every fetch of a given cached plan when > CheckCachedPlan() is called on it, so the part_prune_results_list part > will be discarded and rebuilt as many times as the plan is executed. > You'll find a description around CachedPlan

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
On Fri, Dec 9, 2022 at 7:49 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera > > wrote: > > > Remind me again why is part_prune_results_list not part of struct > > > CachedPlan then? I tried to understand that based on comments upt

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera wrote: > > Remind me again why is part_prune_results_list not part of struct > > CachedPlan then? I tried to understand that based on comments upthread, > > but I was unable to find anything. > > It used to be

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera > > wrote: > > > I find the API of GetCachedPlans a little weird after this patch. > > > David, in his Apr 7 reply on this thread, also sounded to suggest

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera wrote: > > I find the API of GetCachedPlans a little weird after this patch. > David, in his Apr 7 reply on this thread, also sounded to suggest > something similar. > > Hmm, I was / am not so sure if GetCachedP

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
Thanks for the review. On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera wrote: > I find the API of GetCachedPlans a little weird after this patch. I > think it may be better to have it return a pointer of a new struct -- > one that contains both the CachedPlan pointer and the list of pruning > resu

Re: generic plans and "initial" pruning

2022-12-06 Thread Alvaro Herrera
I find the API of GetCachedPlans a little weird after this patch. I think it may be better to have it return a pointer of a new struct -- one that contains both the CachedPlan pointer and the list of pruning results. (As I understand, the sole caller that isn't interested in the pruning results,

Re: generic plans and "initial" pruning

2022-12-04 Thread Amit Langote
On Mon, Dec 5, 2022 at 12:00 PM Amit Langote wrote: > On Fri, Dec 2, 2022 at 7:40 PM Amit Langote wrote: > > Thought it might be good for PartitionPruneResult to also have > > root_parent_relids that matches with the corresponding > > PartitionPruneInfo. ExecInitPartitionPruning() does a sanity

Re: generic plans and "initial" pruning

2022-12-04 Thread Amit Langote
On Fri, Dec 2, 2022 at 7:40 PM Amit Langote wrote: > On Thu, Dec 1, 2022 at 9:43 PM Amit Langote wrote: > > On Thu, Dec 1, 2022 at 8:21 PM Alvaro Herrera > > wrote: > > > On 2022-Dec-01, Amit Langote wrote: > > > > Hmm, how about keeping the [Merge]Append's parent relation's RT index > > > > in

Re: generic plans and "initial" pruning

2022-12-02 Thread Amit Langote
On Thu, Dec 1, 2022 at 9:43 PM Amit Langote wrote: > On Thu, Dec 1, 2022 at 8:21 PM Alvaro Herrera wrote: > > On 2022-Dec-01, Amit Langote wrote: > > > Hmm, how about keeping the [Merge]Append's parent relation's RT index > > > in the PartitionPruneInfo and passing it down to > > > ExecInitPartit

Re: generic plans and "initial" pruning

2022-12-01 Thread Amit Langote
On Thu, Dec 1, 2022 at 8:21 PM Alvaro Herrera wrote: > On 2022-Dec-01, Amit Langote wrote: > > Hmm, how about keeping the [Merge]Append's parent relation's RT index > > in the PartitionPruneInfo and passing it down to > > ExecInitPartitionPruning() from ExecInit[Merge]Append() for > > cross-checki

Re: generic plans and "initial" pruning

2022-12-01 Thread Alvaro Herrera
On 2022-Dec-01, Amit Langote wrote: > Hmm, how about keeping the [Merge]Append's parent relation's RT index > in the PartitionPruneInfo and passing it down to > ExecInitPartitionPruning() from ExecInit[Merge]Append() for > cross-checking? Both Append and MergeAppend already have a > 'apprelids' f

Re: generic plans and "initial" pruning

2022-11-30 Thread Amit Langote
Hi Alvaro, Thanks for looking at this one. On Thu, Dec 1, 2022 at 3:12 AM Alvaro Herrera wrote: > Looking at 0001, I wonder if we should have a crosscheck that a > PartitionPruneInfo you got from following an index is indeed constructed > for the relation that you think it is: previously, you we

Re: generic plans and "initial" pruning

2022-11-30 Thread Alvaro Herrera
Looking at 0001, I wonder if we should have a crosscheck that a PartitionPruneInfo you got from following an index is indeed constructed for the relation that you think it is: previously, you were always sure that the prune struct is for this node, because you followed a pointer that was set up in

Re: generic plans and "initial" pruning

2022-11-07 Thread Amit Langote
On Thu, Oct 27, 2022 at 11:41 AM Amit Langote wrote: > On Mon, Oct 17, 2022 at 6:29 PM Amit Langote wrote: > > On Wed, Oct 12, 2022 at 4:36 PM Amit Langote > > wrote: > > > On Fri, Jul 29, 2022 at 1:20 PM Amit Langote > > > wrote: > > > > On Thu, Jul 28, 2022 at 1:27 AM Robert Haas > > > >

Re: generic plans and "initial" pruning

2022-10-26 Thread Amit Langote
On Mon, Oct 17, 2022 at 6:29 PM Amit Langote wrote: > On Wed, Oct 12, 2022 at 4:36 PM Amit Langote wrote: > > On Fri, Jul 29, 2022 at 1:20 PM Amit Langote > > wrote: > > > On Thu, Jul 28, 2022 at 1:27 AM Robert Haas wrote: > > > > 0001 adds es_part_prune_result but does not use it, so maybe th

Re: generic plans and "initial" pruning

2022-10-17 Thread Amit Langote
On Wed, Oct 12, 2022 at 4:36 PM Amit Langote wrote: > On Fri, Jul 29, 2022 at 1:20 PM Amit Langote wrote: > > On Thu, Jul 28, 2022 at 1:27 AM Robert Haas wrote: > > > 0001 adds es_part_prune_result but does not use it, so maybe the > > > introduction of that field should be deferred until it's n

Re: generic plans and "initial" pruning

2022-07-29 Thread Robert Haas
On Fri, Jul 29, 2022 at 12:47 PM Tom Lane wrote: > > I'm just uncertain whether what Amit has implemented is the > > least-annoying way to go about it... any thoughts on that, > > specifically as it pertains to this patch? > > I haven't looked at this patch at all. I'll try to make some > time fo

Re: generic plans and "initial" pruning

2022-07-29 Thread Tom Lane
Robert Haas writes: > ... it's > always struck me as a little unfortunate that we basically test > whether a var is equal by testing whether the varno and varlevelsup > are equal. That only works if you assume that you can never end up > comparing two vars from thoroughly unrelated parts of the tr

Re: generic plans and "initial" pruning

2022-07-29 Thread Robert Haas
On Fri, Jul 29, 2022 at 11:04 AM Tom Lane wrote: > We could probably make that work, but I'm skeptical that it would > really be an improvement overall, for a couple of reasons. > > (1) The need for merge-rangetables-and-renumber-Vars logic doesn't > go away. It just moves from setrefs.c to the r

Re: generic plans and "initial" pruning

2022-07-29 Thread Tom Lane
Robert Haas writes: > That's not quite my question, though. Why do we ever build a non-flat > range table in the first place? Like, instead of assigning indexes > relative to the current subquery level, why not just assign them > relative to the whole query from the start? We could probably make

Re: generic plans and "initial" pruning

2022-07-29 Thread Robert Haas
On Fri, Jul 29, 2022 at 12:55 AM Tom Lane wrote: > It would not be profitable to flatten the range table before we've > done remove_useless_joins. We'd end up with useless entries from > subqueries that ultimately aren't there. We could perhaps do it > after we finish that phase, but I don't rea

Re: generic plans and "initial" pruning

2022-07-28 Thread Tom Lane
Amit Langote writes: > On Thu, Jul 28, 2022 at 1:27 AM Robert Haas wrote: >> I wonder whether it's really necessary to added the PartitionPruneInfo >> objects to a list in PlannerInfo first and then roll them up into >> PlannerGlobal later. I know we do that for range table entries, but >> I've n

Re: generic plans and "initial" pruning

2022-07-28 Thread Amit Langote
On Thu, Jul 28, 2022 at 1:27 AM Robert Haas wrote: > On Tue, Jul 26, 2022 at 11:01 PM Amit Langote wrote: > > Needed to be rebased again, over 2d04277121f this time. Thanks for looking. > 0001 adds es_part_prune_result but does not use it, so maybe the > introduction of that field should be def

Re: generic plans and "initial" pruning

2022-07-27 Thread Robert Haas
On Tue, Jul 26, 2022 at 11:01 PM Amit Langote wrote: > Needed to be rebased again, over 2d04277121f this time. 0001 adds es_part_prune_result but does not use it, so maybe the introduction of that field should be deferred until it's needed for something. I wonder whether it's really necessary to

Re: generic plans and "initial" pruning

2022-07-26 Thread Amit Langote
On Wed, Jul 13, 2022 at 4:03 PM Amit Langote wrote: > On Wed, Jul 13, 2022 at 3:40 PM Amit Langote wrote: > > Rebased over 964d01ae90c. > > Sorry, left some pointless hunks in there while rebasing. Fixed in > the attached. Needed to be rebased again, over 2d04277121f this time. -- Thanks, Ami

Re: generic plans and "initial" pruning

2022-07-13 Thread Amit Langote
On Wed, Jul 13, 2022 at 3:40 PM Amit Langote wrote: > Rebased over 964d01ae90c. Sorry, left some pointless hunks in there while rebasing. Fixed in the attached. -- Thanks, Amit Langote EDB: http://www.enterprisedb.com v19-0001-Move-PartitioPruneInfo-out-of-plan-nodes-into-Pl.patch Descriptio

Re: generic plans and "initial" pruning

2022-07-12 Thread Amit Langote
Rebased over 964d01ae90c. -- Thanks, Amit Langote EDB: http://www.enterprisedb.com v18-0002-Optimize-AcquireExecutorLocks-by-locking-only-un.patch Description: Binary data v18-0001-Move-PartitioPruneInfo-out-of-plan-nodes-into-Pl.patch Description: Binary data

Re: generic plans and "initial" pruning

2022-07-05 Thread Jacob Champion
On Fri, May 27, 2022 at 1:09 AM Amit Langote wrote: > 0001 contains the mechanical changes of moving PartitionPruneInfo out > of Append/MergeAppend into a list in PlannedStmt. > > 0002 is the main patch to "Optimize AcquireExecutorLocks() by locking > only unpruned partitions". This patchset will

Re: generic plans and "initial" pruning

2022-05-27 Thread Zhihong Yu
On Fri, May 27, 2022 at 1:10 AM Amit Langote wrote: > On Mon, Apr 11, 2022 at 12:53 PM Zhihong Yu wrote: > > On Sun, Apr 10, 2022 at 8:05 PM Amit Langote > wrote: > >> Sending v15 that fixes that to keep the cfbot green for now. > > > > Hi, > > > > + /* RT index of the partitione

Re: generic plans and "initial" pruning

2022-04-10 Thread Zhihong Yu
On Sun, Apr 10, 2022 at 8:05 PM Amit Langote wrote: > On Fri, Apr 8, 2022 at 8:45 PM Amit Langote > wrote: > > Most looked fine changes to me except a couple of typos, so I've > > adopted those into the attached new version, even though I know it's > > too late to try to apply it. > > > > + * XX

Re: generic plans and "initial" pruning

2022-04-10 Thread Amit Langote
On Fri, Apr 8, 2022 at 8:45 PM Amit Langote wrote: > Most looked fine changes to me except a couple of typos, so I've > adopted those into the attached new version, even though I know it's > too late to try to apply it. > > + * XXX is it worth doing a bms_copy() on glob->minLockRelids if > + * glo

Re: generic plans and "initial" pruning

2022-04-08 Thread Amit Langote
Hi David, On Fri, Apr 8, 2022 at 8:16 PM David Rowley wrote: > On Fri, 8 Apr 2022 at 17:49, Amit Langote wrote: > > Attached updated patch with these changes. > Thanks for making the changes. I started looking over this patch but > really feel like it needs quite a few more iterations of what w

Re: generic plans and "initial" pruning

2022-04-08 Thread David Rowley
On Fri, 8 Apr 2022 at 17:49, Amit Langote wrote: > Attached updated patch with these changes. Thanks for making the changes. I started looking over this patch but really feel like it needs quite a few more iterations of what we've just been doing to get it into proper committable shape. There se

Re: generic plans and "initial" pruning

2022-04-07 Thread Amit Langote
On Thu, Apr 7, 2022 at 9:41 PM David Rowley wrote: > On Thu, 7 Apr 2022 at 20:28, Amit Langote wrote: > > Here's an updated version. In Particular, I removed > > part_prune_results list from PortalData, in favor of anything that > > needs to look at the list can instead get it from the CachedPla

Re: generic plans and "initial" pruning

2022-04-07 Thread David Rowley
On Thu, 7 Apr 2022 at 20:28, Amit Langote wrote: > Here's an updated version. In Particular, I removed > part_prune_results list from PortalData, in favor of anything that > needs to look at the list can instead get it from the CachedPlan > (PortalData.cplan). This makes things better in 2 ways:

  1   2   >