Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-08-19 Thread Ashutosh Bapat
On Mon, Aug 19, 2024 at 6:43 PM Ashutosh Bapat wrote: > > Sorry for the delay in my response. > > On Wed, May 29, 2024 at 9:34 AM Ashutosh Bapat > wrote: > > > > Documenting some comments from todays' patch review session > > I forgot to mention back then that both of the suggestions below came >

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-08-19 Thread Ashutosh Bapat
Sorry for the delay in my response. On Wed, May 29, 2024 at 9:34 AM Ashutosh Bapat wrote: > > Documenting some comments from todays' patch review session I forgot to mention back then that both of the suggestions below came from Tom Lane. > 1. Instead of a nested hash table, it might be better

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-05-28 Thread Ashutosh Bapat
On Mon, Feb 19, 2024 at 5:17 AM Ashutosh Bapat wrote: > On Mon, Feb 19, 2024 at 4:35 AM Tomas Vondra > wrote: > > > > Hi, > > > > After taking a look at the patch optimizing SpecialJoinInfo allocations, > > I decided to take a quick look at this one too. I don't have any > > specific comments on

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-19 Thread Ashutosh Bapat
On Mon, Feb 19, 2024 at 4:35 AM Tomas Vondra wrote: > > Hi, > > After taking a look at the patch optimizing SpecialJoinInfo allocations, > I decided to take a quick look at this one too. I don't have any > specific comments on the code yet, but it seems quite a bit more complex > than the other pa

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-18 Thread Andrei Lepikhov
On 19/2/2024 06:05, Tomas Vondra wrote: However, this is a somewhat extreme example - it's joining 5 tables, each with 1000 partitions, using a partition-wise join. It reduces the amount of memory, but the planning time is still quite high (and essentially the same as without the patch). So it's

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-18 Thread Tomas Vondra
Hi, After taking a look at the patch optimizing SpecialJoinInfo allocations, I decided to take a quick look at this one too. I don't have any specific comments on the code yet, but it seems quite a bit more complex than the other patch ... it's introducing a HTAB into the optimizer, surely that ha

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-01-29 Thread Ashutosh Bapat
On Sat, Jan 27, 2024 at 8:26 AM vignesh C wrote: > > On Tue, 31 Oct 2023 at 10:48, Ashutosh Bapat > wrote: > > > > On Thu, Sep 14, 2023 at 4:39 PM Ashutosh Bapat > > wrote: > > > > > > The patch set is thus > > > 0001 - patch used to measure memory used during planning > > > > > > 0002 - Patch t

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-01-26 Thread vignesh C
On Tue, 31 Oct 2023 at 10:48, Ashutosh Bapat wrote: > > On Thu, Sep 14, 2023 at 4:39 PM Ashutosh Bapat > wrote: > > > > The patch set is thus > > 0001 - patch used to measure memory used during planning > > > > 0002 - Patch to free intermediate Relids computed by > > adjust_child_relids_multileve

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-11-24 Thread Ashutosh Bapat
On Fri, Nov 24, 2023 at 3:56 PM Alena Rybakina wrote: > > On 24.11.2023 13:20, Alena Rybakina wrote: > > Hi! Thank you for your work on the subject, I think it's a really useful > feature. > > I've reviewed your patch and I have a few questions. > > First of all, have you thought about creating a

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-11-24 Thread Alena Rybakina
On 24.11.2023 13:20, Alena Rybakina wrote: Hi! Thank you for your work on the subject, I think it's a really useful feature. I've reviewed your patch and I have a few questions. First of all, have you thought about creating a gun parameter to display memory scheduling information? I agree t

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-11-24 Thread Alena Rybakina
Hi! Thank you for your work on the subject, I think it's a really useful feature. I've reviewed your patch and I have a few questions. First of all, have you thought about creating a gun parameter to display memory scheduling information? I agree that this is an important feature, but I think

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-10-30 Thread Ashutosh Bapat
On Thu, Sep 14, 2023 at 4:39 PM Ashutosh Bapat wrote: > > The patch set is thus > 0001 - patch used to measure memory used during planning > > 0002 - Patch to free intermediate Relids computed by > adjust_child_relids_multilevel(). I didn't test memory consumption for > multi-level partitioning. B

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-09-14 Thread Ashutosh Bapat
Hi All, On Fri, Aug 11, 2023 at 6:24 PM Ashutosh Bapat wrote: > > Obtaining child clauses from parent clauses by translation and > tracking the translations is less complex and may be more efficient > too. I will post a patch on those lines soon. > PFA patch set to add infrastructure to track Re

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-11 Thread Ashutosh Bapat
I spent some time on 4th point below but also looked at other points. Here's what I have found so far On Thu, Jul 27, 2023 at 7:35 PM Ashutosh Bapat wrote: > > 1. The patch uses RestrictInfo::required_relids as the key for > searching child RelOptInfos. I am not sure which of the two viz. > requi

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 10:12 AM David Rowley wrote: > > On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat > wrote: > > 0001 - to measure memory consumption during planning. This is the same > > one as attached to [1]. > I have started a separate thread to discuss this patch. I am taking this discussi

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread David Rowley
On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat wrote: > 0001 - to measure memory consumption during planning. This is the same > one as attached to [1]. I see you're recording the difference in the CurrentMemoryContext of palloc'd memory before and after planning. That won't really alert us to pro

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread Ashutosh Bapat
On Tue, Aug 8, 2023 at 4:08 PM Richard Guo wrote: > > > I haven't looked into the details but with 0002 patch I came across a > crash while planning the query below. > > regression=# set enable_partitionwise_join to on; > SET > regression=# EXPLAIN (COSTS OFF) > SELECT * FROM prt1 t1, prt2 t2 > WH

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread Richard Guo
On Thu, Jul 27, 2023 at 10:06 PM Ashutosh Bapat < ashutosh.bapat@gmail.com> wrote: > 0002 - WIP patch to avoid repeated translations of RestrictInfo. > The WIP patch avoids repeated translations by tracking the child for > which a RestrictInfo is translated and reusing the same translation > e

Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-07-27 Thread Ashutosh Bapat
Hi All, Following up on [1] ... A restrictlist is a list of RestrictInfo nodes, each representing one clause, applicable to a set of joins. When partitionwise join is used as a strategy, the restrictlists for child join are obtained by translating the restrictlists for the parent in try_partitio