On Monday, June 24, 2013, Noah Misch wrote: > On Sat, Jun 22, 2013 at 03:46:49AM -0400, Stephen Frost wrote: > > * Noah Misch (n...@leadboat.com <javascript:;>) wrote: > > > The next limit faced by sorts is > > > INT_MAX concurrent tuples in memory, which limits helpful work_mem to > about > > > 150 GiB when sorting int4. > > > > That's frustratingly small. :( > > I could appreciate a desire to remove that limit. The way to do that is to > audit all uses of "int" variables in tuplesort.c and tuplestore.c, changing > them to Size where they can be used as indexes into the memtuples array.
Right, that's about what I figured would need to be done. > Nonetheless, this new limit is about 50x the current limit; you need an > (unpartitioned) table of 2B+ rows to encounter it. I'm happy with that. Definitely better but I could see cases with that many tuples in the not-too-distant future, esp. when used with MinMax indexes... > > > ! if (memtupsize * grow_ratio < INT_MAX) > > > ! newmemtupsize = (int) (memtupsize * grow_ratio); > > > ! else > > > ! newmemtupsize = INT_MAX; > > > > > > /* We won't make any further enlargement attempts */ > > > state->growmemtuples = false; > > > > I'm not a huge fan of moving directly to INT_MAX. Are we confident that > > everything can handle that cleanly..? I feel like it might be a bit > > safer to shy a bit short of INT_MAX (say, by 1K). Perhaps that's overly > > paranoid, but there's an awful lot of callers and some loop which +2's > > and then overflows would suck, eg: > > Where are you seeing "an awful lot of callers"? The code that needs to be > correct with respect to the INT_MAX limit is all in > tuplesort.c/tuplestore.c. > Consequently, I chose to verify that code rather than add a safety factor. > (I > did add an unrelated safety factor to repalloc_huge() itself.) Ok, I was thinking this code was used beyond tuplesort (I was thinking it was actually associated with palloc). Apologies for the confusion. :) > > Also, could this be used to support hashing larger sets..? If we change > > NTUP_PER_BUCKET to one, we could end up wanting to create a hash table > > larger than INT_MAX since, with 8-byte pointers, that'd only be around > > 134M tuples. > > The INT_MAX limit is an internal limit of tuplesort/tuplestore; other > consumers of the huge allocation APIs are only subject to that limit if > they > find reasons to enforce it on themselves. (Incidentally, the internal > limit > in question is INT_MAX tuples, not INT_MAX bytes.) There's other places where we use integers for indexes into arrays of tuples (at least hashing is another area..) and those are then also subject to INT_MAX, which was really what I was getting at. We might move the hashing code to use the _huge functions and would then need to adjust that code to use Size for the index into the hash table array of pointers. Thanks, Stephen
