Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-29 Thread Önder Kalacı
Hi David, Tom, all, > I've just pushed the disable byref Datums patch I posted earlier. I > only made a small adjustment to make use of the TupleDescAttr() macro. > Önder, thank you for the report. > > With this commit, I re-run the query patterns where we observed the problem, all looks good now

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-29 Thread Peter Geoghegan
On Thu, Sep 29, 2022 at 7:10 AM Tom Lane wrote: > TBH, I think this is completely ridiculous over-optimization. > There's exactly zero evidence that a second copy of the function > would improve performance, or do anything but contribute to code > bloat (which does have a distributed performance c

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-29 Thread Ronan Dunklau
Le jeudi 29 septembre 2022, 16:10:03 CEST Tom Lane a écrit : > Ronan Dunklau writes: > >> Yeah, I think the same rules around scope apply as > >> tuplesort_gettupleslot() with copy==false. We could do it by adding a > >> copy flag to the existing function, but I'd rather not add the > >> branchin

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-29 Thread Tom Lane
Ronan Dunklau writes: >> Yeah, I think the same rules around scope apply as >> tuplesort_gettupleslot() with copy==false. We could do it by adding a >> copy flag to the existing function, but I'd rather not add the >> branching to that function. It's probably just better to duplicate it >> and ad

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-29 Thread Ronan Dunklau
> I've just pushed the disable byref Datums patch I posted earlier. I > only made a small adjustment to make use of the TupleDescAttr() macro. > Önder, thank you for the report. Thank you David for taking care of this. > Yeah, I think the same rules around scope apply as > tuplesort_gettupleslot(

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Peter Geoghegan
On Wed, Sep 28, 2022 at 9:59 PM David Rowley wrote: > select b from t1 order by b offset 100; > > Master: > latency average = 344.763 ms > > Patched: > latency average = 268.374 ms > > about 28% faster. That's more like it! -- Peter Geoghegan

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread David Rowley
On Thu, 29 Sept 2022 at 14:32, Peter Geoghegan wrote: > > On Wed, Sep 28, 2022 at 6:13 PM David Rowley wrote: > > Master: > > latency average = 313.197 ms > > > > Patched: > > latency average = 304.335 ms > > > > So not a very impressive speedup there (about 3%) > > Worth a try, at least. Having

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Peter Geoghegan
On Wed, Sep 28, 2022 at 6:13 PM David Rowley wrote: > Master: > latency average = 313.197 ms > > Patched: > latency average = 304.335 ms > > So not a very impressive speedup there (about 3%) Worth a try, at least. Having a more consistent interface is valuable in itself too. -- Peter Geoghegan

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread David Rowley
On Thu, 29 Sept 2022 at 12:07, Peter Geoghegan wrote: > Also potentially relevant: the 2017 commit fa117ee4 anticipated adding > a "copy" argument to tuplesort_getdatum() (the same commit added such > a "copy" argument to tuplesort_gettupleslot()). I see that that still > hasn't happened to tuples

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Michael Paquier
On Wed, Sep 28, 2022 at 07:35:07PM -0400, Tom Lane wrote: > Michael Paquier writes: >> Wouldn't it be better to have 3a58176 reflect the non-optimization >> path in the EXPLAIN output of a new regression test if none of the >> existing tests are able to show any difference? > > This decision is n

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread David Rowley
On Thu, 29 Sept 2022 at 12:30, Michael Paquier wrote: > > On Thu, Sep 29, 2022 at 11:58:17AM +1300, David Rowley wrote: > > I've just pushed the disable byref Datums patch I posted earlier. I > > only made a small adjustment to make use of the TupleDescAttr() macro. > > Önder, thank you for the re

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Tom Lane
Michael Paquier writes: > Wouldn't it be better to have 3a58176 reflect the non-optimization > path in the EXPLAIN output of a new regression test if none of the > existing tests are able to show any difference? This decision is not visible in EXPLAIN in any case. regards

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Michael Paquier
On Thu, Sep 29, 2022 at 11:58:17AM +1300, David Rowley wrote: > I've just pushed the disable byref Datums patch I posted earlier. I > only made a small adjustment to make use of the TupleDescAttr() macro. > Önder, thank you for the report. Wouldn't it be better to have 3a58176 reflect the non-opti

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Peter Geoghegan
On Wed, Sep 28, 2022 at 4:00 PM Peter Geoghegan wrote: > I am reminded of the discussion that led to bugfix commit c2d4eb1b > some years back. Also potentially relevant: the 2017 commit fa117ee4 anticipated adding a "copy" argument to tuplesort_getdatum() (the same commit added such a "copy" argu

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Peter Geoghegan
On Wed, Sep 28, 2022 at 12:57 PM Tom Lane wrote: > David Rowley writes: > > I'm wondering if the best way to fix it if doing it that way would be > > to invent tuplesort_getdatum_nocopy() which would be the same as > > tuplesort_getdatum() except it wouldn't do the datumCopy for byref > > types.

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread David Rowley
On Thu, 29 Sept 2022 at 08:57, Tom Lane wrote: > > David Rowley writes: > > I'm wondering if the best way to fix it if doing it that way would be > > to invent tuplesort_getdatum_nocopy() which would be the same as > > tuplesort_getdatum() except it wouldn't do the datumCopy for byref > > types.

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Tom Lane
David Rowley writes: > I'm wondering if the best way to fix it if doing it that way would be > to invent tuplesort_getdatum_nocopy() which would be the same as > tuplesort_getdatum() except it wouldn't do the datumCopy for byref > types. Yeah, perhaps. We'd need a clear spec on how long the Datu

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread David Rowley
Thanks for investigating this and finding the guilty commit. On Thu, 29 Sept 2022 at 07:34, Tom Lane wrote: > After looking at that for a little while, I wonder if we shouldn't > fix this by restricting the Datum-sort path to be used only with > pass-by-value data types. That'd require only a mi

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Tom Lane
I wrote: > and bisecting fingers this commit as the guilty party: > commit 91e9e89dccdfdf4216953d3d8f5515dcdef177fb > Author: David Rowley > Date: Thu Jul 22 14:03:19 2021 +1200 > Make nodeSort.c use Datum sorts for single column sorts After looking at that for a little while, I wonder if

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Tom Lane
=?UTF-8?B?w5ZuZGVyIEthbGFjxLE=?= writes: > With PG 15 (rc1 or beta4), I'm observing an interesting memory pattern. Yup, that is a leak. valgrind'ing it blames this call chain: ==00:00:16:12.228 4011013== 790,404,056 bytes in 60,800,312 blocks are definitely lost in loss record 1,108 of 1,108 =

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Ranier Vilela
Em qua., 28 de set. de 2022 às 14:24, Önder Kalacı escreveu: > Hi, > > Thanks for replying so quickly! > > I run your test here with a fix attached. >> >> Can you retake your test with the patch attached? >> >> >> Unfortunately, with the patch, I still see the memory usage increase and > get the

Re: A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Önder Kalacı
Hi, Thanks for replying so quickly! I run your test here with a fix attached. > > Can you retake your test with the patch attached? > > > Unfortunately, with the patch, I still see the memory usage increase and get the OOMs Thanks, Onder KALACI

A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Ranier Vilela
>CREATE TABLE t1 (a text); >CREATE TABLE t2 (a text); >-- make the text a little large by adding 1000 >INSERT INTO t1 SELECT (1000+i%1000)::text FROM >generate_series(0,1000) i; >-- make the text a little large by adding 1000 >INSERT INTO t2 SELECT (1000+i%1000

A potential memory leak on Merge Join when Sort node is not below Materialize node

2022-09-28 Thread Önder Kalacı
Hi hackers, With PG 15 (rc1 or beta4), I'm observing an interesting memory pattern. I have not seen a similar discussion on the mailing list. If I missed that, please refer me there. The problem that I'm going to explain does not happen on PG 13/14. It seems like there is a memory leak(?) with $t