Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-07-06 Thread Stephen Frost
Jeff, * Jeff Janes (jeff.ja...@gmail.com) wrote: > I was going to add another item to make nodeHash.c use the new huge > allocator, but after looking at it just now it was not clear to me that it > even has such a limitation. nbatch is limited by MaxAllocSize, but > nbuckets doesn't seem to be.

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-27 Thread Jeff Janes
On Sat, Jun 22, 2013 at 12:46 AM, Stephen Frost wrote: > Noah, > > * Noah Misch (n...@leadboat.com) wrote: > > This patch introduces MemoryContextAllocHuge() and repalloc_huge() that > check > > a higher MaxAllocHugeSize limit of SIZE_MAX/2. > > Nice! I've complained about this limit a few diffe

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-27 Thread Noah Misch
On Wed, Jun 26, 2013 at 03:48:23PM -0700, Jeff Janes wrote: > On Mon, May 13, 2013 at 7:26 AM, Noah Misch wrote: > > This patch introduces MemoryContextAllocHuge() and repalloc_huge() that > > check > > a higher MaxAllocHugeSize limit of SIZE_MAX/2. Chunks don't bother > > recording > > whether t

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-26 Thread Jeff Janes
On Mon, May 13, 2013 at 7:26 AM, Noah Misch wrote: > A memory chunk allocated through the existing palloc.h interfaces is > limited > to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE() > need > not check its own 1 GiB limit, and algorithms that grow a buffer by > doubling > n

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-24 Thread Stephen Frost
On Monday, June 24, 2013, Noah Misch wrote: > On Sat, Jun 22, 2013 at 03:46:49AM -0400, Stephen Frost wrote: > > * Noah Misch (n...@leadboat.com ) wrote: > > > The next limit faced by sorts is > > > INT_MAX concurrent tuples in memory, which limits helpful work_mem to > about > > > 150 GiB when so

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-24 Thread Noah Misch
On Sat, Jun 22, 2013 at 11:36:45AM +0100, Simon Riggs wrote: > On 13 May 2013 15:26, Noah Misch wrote: > I'm concerned that people will accidentally use MaxAllocSize. Can we > put in a runtime warning if someone tests AllocSizeIsValid() with a > larger value? I don't see how we could. To preemp

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-24 Thread Noah Misch
On Sat, Jun 22, 2013 at 03:46:49AM -0400, Stephen Frost wrote: > * Noah Misch (n...@leadboat.com) wrote: > > The next limit faced by sorts is > > INT_MAX concurrent tuples in memory, which limits helpful work_mem to about > > 150 GiB when sorting int4. > > That's frustratingly small. :( I could a

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 3:46 AM, Stephen Frost wrote: > I'm not a huge fan of moving directly to INT_MAX. Are we confident that > everything can handle that cleanly..? I feel like it might be a bit > safer to shy a bit short of INT_MAX (say, by 1K). Maybe it would be better to stick with INT_MA

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > On 22 June 2013 08:46, Stephen Frost wrote: > >>The next limit faced by sorts is > >> INT_MAX concurrent tuples in memory, which limits helpful work_mem to about > >> 150 GiB when sorting int4. > > > > That's frustratingly small. :( > > But that has

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 22 June 2013 08:46, Stephen Frost wrote: >>The next limit faced by sorts is >> INT_MAX concurrent tuples in memory, which limits helpful work_mem to about >> 150 GiB when sorting int4. > > That's frustratingly small. :( > But that has nothing to do with this patch, right? And is easily fixed,

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 13 May 2013 15:26, Noah Misch wrote: > A memory chunk allocated through the existing palloc.h interfaces is limited > to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE() need > not check its own 1 GiB limit, and algorithms that grow a buffer by doubling > need not check for

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
Noah, * Noah Misch (n...@leadboat.com) wrote: > This patch introduces MemoryContextAllocHuge() and repalloc_huge() that check > a higher MaxAllocHugeSize limit of SIZE_MAX/2. Nice! I've complained about this limit a few different times and just never got around to addressing it. > This was ma

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-05-13 Thread Pavel Stehule
+1 Pavel Dne 13.5.2013 16:29 "Noah Misch" napsal(a): > A memory chunk allocated through the existing palloc.h interfaces is > limited > to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE() > need > not check its own 1 GiB limit, and algorithms that grow a buffer by > doubling