Re: [HACKERS] Multiple sorts in a query

2009-05-21 Thread Simon Riggs
On Wed, 2009-05-20 at 23:01 -0400, Zdenek Kotala wrote: We use default one. I did not tested difference between them, but IIRC that Jignesh did some testing with umem. I will ask him. However if you give me test scenario I can test it. Talk with Dimitri from Sun who is doing scalability

Re: [HACKERS] Multiple sorts in a query

2009-05-20 Thread Simon Riggs
On Tue, 2009-05-19 at 22:19 +0200, Zdenek Kotala wrote: Chuck McDevitt píše v út 19. 05. 2009 v 09:33 -0700: Solaris default malloc always uses sbrk(), and never ever tried to reduce the sbrk point. If you want a malloc that uses mmap, there is an non-default malloc that does that

Re: [HACKERS] Multiple sorts in a query

2009-05-20 Thread Simon Riggs
On Tue, 2009-05-19 at 16:49 -0400, Greg Stark wrote: Well I'm just saying if you realloc a x kilobyte block into a 2x block and the allocator can't expand it and has to copy then it seems inevitable. OK, understood. So there is grounds at least for an investigation into how that works

Re: [HACKERS] Multiple sorts in a query

2009-05-20 Thread Andres Freund
On 05/20/2009 10:14 AM, Simon Riggs wrote: On Tue, 2009-05-19 at 22:19 +0200, Zdenek Kotala wrote: Chuck McDevitt píše v út 19. 05. 2009 v 09:33 -0700: What I heart is that standard malloc is not good, but it is still here for compatibility reason with old application which depends on some

Re: realloc overhead (was [HACKERS] Multiple sorts in a query)

2009-05-20 Thread pg
So at least transiently we use 3x the size of the actual array. I was conjecturing, prior to investigation. Are you saying you know this/have seen this already? Well I'm just saying if you realloc a x kilobyte block into a 2x block and the allocator can't expand it and has to copy then it

Re: [HACKERS] Multiple sorts in a query

2009-05-20 Thread Zdenek Kotala
Simon Riggs píše v st 20. 05. 2009 v 09:14 +0100: What I heart is that standard malloc is not good, but it is still here for compatibility reason with old application which depends on some functionality. Which one is used in the default PostgreSQL build for Solaris? If you use

[HACKERS] Multiple sorts in a query

2009-05-19 Thread Simon Riggs
Just wanted to check some thoughts about how memory allocation works in complex queries. Been thinking some more about recent Solaris testing results that *seemed* to show issues with multiple concurrent queries that have multiple sorts. If we have a query that uses multiple sorts, we may have a

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Martijn van Oosterhout
On Tue, May 19, 2009 at 12:32:13PM +0100, Simon Riggs wrote: If we allocate large chunks of memory we use malloc(). So complex queries can have multiple mallocs, followed by multiple reallocs. That in itself seems likely to end up with roughly double memory use, since realloc won't work

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Simon Riggs
On Tue, 2009-05-19 at 09:17 -0400, Merlin Moncure wrote: On Tue, May 19, 2009 at 7:44 AM, Martijn van Oosterhout The threshold is dynamic apparently, but starts at 128KB. I just read an article that suggests assuming that can be dangerous (by one of the authors of jemalloc)...an

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Greg Stark
On Tue, May 19, 2009 at 12:32 PM, Simon Riggs si...@2ndquadrant.com wrote: If we have a query that uses multiple sorts, we may have a top-level sort, with child nodes that contain sorts also. In some cases we may find with sub-nodes that have both inner and outer sub-trees that contain sorts

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Merlin Moncure
On Tue, May 19, 2009 at 7:44 AM, Martijn van Oosterhout klep...@svana.org wrote: On Tue, May 19, 2009 at 12:32:13PM +0100, Simon Riggs wrote: If we allocate large chunks of memory we use malloc(). So complex queries can have multiple mallocs, followed by multiple reallocs. That in itself seems

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Chuck McDevitt
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Simon Riggs Sent: Tuesday, May 19, 2009 4:32 AM To: pgsql-hackers Subject: [HACKERS] Multiple sorts in a query Just wanted to check some thoughts about how

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Simon Riggs
On Tue, 2009-05-19 at 09:33 -0700, Chuck McDevitt wrote: Is it possible that Solaris's default malloc isn't appropriate for repeated use in complex queries that use multiple sorts? http://developers.sun.com/solaris/articles/multiproc/multiproc.html and recent OpenSolaris bug reports.

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Simon Riggs
On Tue, 2009-05-19 at 13:52 +0100, Greg Stark wrote: So at least transiently we use 3x the size of the actual array. I was conjecturing, prior to investigation. Are you saying you know this/have seen this already? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Greg Stark
Well I'm just saying if you realloc a x kilobyte block into a 2x block and the allocator can't expand it and has to copy then it seems inevitable. -- Greg On 19 May 2009, at 14:11, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2009-05-19 at 13:52 +0100, Greg Stark wrote: So at

Re: [HACKERS] Multiple sorts in a query

2009-05-19 Thread Zdenek Kotala
Chuck McDevitt píše v út 19. 05. 2009 v 09:33 -0700: Solaris default malloc always uses sbrk(), and never ever tried to reduce the sbrk point. If you want a malloc that uses mmap, there is an non-default malloc that does that (libumem or something?) There are severals memory