Re: Add GUC to tune glibc's malloc implementation.

2023-08-01 Thread Daniel Gustafsson
> On 29 Jun 2023, at 00:31, Andres Freund wrote: > On 2023-06-28 07:26:03 +0200, Ronan Dunklau wrote: >> I see it as a way to have *some* sort of control over the malloc >> implementation we use, instead of tuning our allocations pattern on top of it >> while treating it entirely as a black box. A

Re: Add GUC to tune glibc's malloc implementation.

2023-06-28 Thread Andres Freund
Hi, On 2023-06-28 07:26:03 +0200, Ronan Dunklau wrote: > I see it as a way to have *some* sort of control over the malloc > implementation we use, instead of tuning our allocations pattern on top of it > while treating it entirely as a black box. As for the tuning, I proposed > earlier to replace

Re: Add GUC to tune glibc's malloc implementation.

2023-06-27 Thread Ronan Dunklau
Le mardi 27 juin 2023, 20:17:46 CEST Andres Freund a écrit : > > Yes this is probably much more appropriate, but a much larger change with > > greater risks of regression. Especially as we have to make sure we're not > > overfitting our own code for a specific malloc implementation, to the > > detr

Re: Add GUC to tune glibc's malloc implementation.

2023-06-27 Thread Andres Freund
Hi, On 2023-06-27 08:35:28 +0200, Ronan Dunklau wrote: > Le lundi 26 juin 2023, 23:03:48 CEST Andres Freund a écrit : > > Hi, > > > > On 2023-06-26 08:38:35 +0200, Ronan Dunklau wrote: > > > I hope what I'm trying to achieve is clearer that way. Maybe this patch is > > > not the best way to go abo

Re: Add GUC to tune glibc's malloc implementation.

2023-06-26 Thread Ronan Dunklau
Le mardi 27 juin 2023, 08:35:28 CEST Ronan Dunklau a écrit : > I re-attached the simple script I used. I've run this script with different > values for glibc_malloc_max_trim_threshold. I forgot to add that it was using default parametrers except for work_mem, set to 32M, and max_parallel_workers_

Re: Add GUC to tune glibc's malloc implementation.

2023-06-26 Thread Ronan Dunklau
Le lundi 26 juin 2023, 23:03:48 CEST Andres Freund a écrit : > Hi, > > On 2023-06-26 08:38:35 +0200, Ronan Dunklau wrote: > > I hope what I'm trying to achieve is clearer that way. Maybe this patch is > > not the best way to go about this, but since the memory allocator > > behaviour can have such

Re: Add GUC to tune glibc's malloc implementation.

2023-06-26 Thread Andres Freund
Hi, On 2023-06-26 08:38:35 +0200, Ronan Dunklau wrote: > I hope what I'm trying to achieve is clearer that way. Maybe this patch is not > the best way to go about this, but since the memory allocator behaviour can > have such an impact it's a bit sad we have to leave half the performance on > the

Re: Add GUC to tune glibc's malloc implementation.

2023-06-26 Thread Andres Freund
Hi, On 2023-06-22 15:35:12 +0200, Ronan Dunklau wrote: > This can cause problems. Let's take for example a table with 10k rows, and 32 > columns (as defined by a bench script David Rowley shared last year when > discussing the GenerationContext for tuplesort), and execute the following > query,

Re: Add GUC to tune glibc's malloc implementation.

2023-06-25 Thread Ronan Dunklau
Le vendredi 23 juin 2023, 22:55:51 CEST Peter Eisentraut a écrit : > On 22.06.23 15:35, Ronan Dunklau wrote: > > The thing is, by default, those parameters are adjusted dynamically by the > > glibc itself. It starts with quite small thresholds, and raises them when > > the program frees some memory

Re: Add GUC to tune glibc's malloc implementation.

2023-06-23 Thread Peter Eisentraut
On 22.06.23 15:35, Ronan Dunklau wrote: The thing is, by default, those parameters are adjusted dynamically by the glibc itself. It starts with quite small thresholds, and raises them when the program frees some memory, up to a certain limit. This patch proposes a new GUC allowing the user to adj

Re: Add GUC to tune glibc's malloc implementation.

2023-06-22 Thread Tom Lane
Ronan Dunklau writes: > Le jeudi 22 juin 2023, 15:49:36 CEST Tom Lane a écrit : >> Aren't these same settings controllable via environment variables? >> I could see adding some docs suggesting that you set thus-and-such >> values in the postmaster's startup script. Admittedly, the confusion >> ar

Re: Add GUC to tune glibc's malloc implementation.

2023-06-22 Thread Ronan Dunklau
Le jeudi 22 juin 2023, 15:49:36 CEST Tom Lane a écrit : > This seems like a pretty awful idea, mainly because there's no way > to have such a GUC mean anything on non-glibc platforms, which is > going to cause confusion or worse. I named the GUC glibc_malloc_max_trim_threshold, I hope this is enou

Re: Add GUC to tune glibc's malloc implementation.

2023-06-22 Thread Tom Lane
Ronan Dunklau writes: > Following some conversation with Tomas at PGCon, I decided to resurrect this > topic, which was previously discussed in the context of moving tuplesort to > use GenerationContext: https://www.postgresql.org/message-id/ > 8046109.NyiUUSuA9g%40aivenronan This seems like a

Add GUC to tune glibc's malloc implementation.

2023-06-22 Thread Ronan Dunklau
Hello, Following some conversation with Tomas at PGCon, I decided to resurrect this topic, which was previously discussed in the context of moving tuplesort to use GenerationContext: https://www.postgresql.org/message-id/ 8046109.NyiUUSuA9g%40aivenronan The idea for this patch is that the beha