> On June 19, 2015, 5:06 p.m., Jason Altekruse wrote: > > exec/java-exec/src/main/codegen/templates/FixedValueVectors.java, line 92 > > <https://reviews.apache.org/r/35484/diff/1/?file=985297#file985297line92> > > > > The reallocation loops will run infinitely if we have it the maximum > > buffer size. This can be reproduced by reading a JSON file with a lot of > > large lists. This could be fixed by re-intriducing the code we had before > > re-allocation, but only in the case where we hit one of these limits. > > Allocations should be able to fail, and we need to make sure the operators > > can handle this case. > > > > We should fail with an OOM error in these cases until we can split > > lists across batches or find another way around the fact that the allocator > > tracks buffer lengths with ints. > > Jason Altekruse wrote: > *hit the maximum buffer size, *re-introducing the code > > Hanifi Gunes wrote: > I doubt if it is a good idea to fail with OOM. Instead we should consider > setting the upper bound to max allowed by the allocator (Int.MAX) and ensure > that this invariant holds across vector impls. > > Jason Altekruse wrote: > The problem is that the consumers of this interface are currently working > under the assumption that a request to write to a vector will not fail unless > we run out of memory. We need to go back and update the operators to stop > their processing when we cannot do a write into the vector, or "emulate" the > case they expect, which is OOM.
Sounds good but an operation on the buffer that goes beyond its boundaries does not translate into OOM but perhaps to an IOOB. It is only when allocator cannot supply an allocation demand that we should expect an OOM. I am inclined to throw an IOOB here for the case we go beyond the max buffer capacity. - Hanifi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35484/#review88551 ----------------------------------------------------------- On June 17, 2015, 6:54 p.m., Hanifi Gunes wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35484/ > ----------------------------------------------------------- > > (Updated June 17, 2015, 6:54 p.m.) > > > Review request for drill, Mehant Baid and Venki Korukanti. > > > Repository: drill-git > > > Description > ------- > > DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent > overflows > Vectors > - set an upper bound on # of bytes to allocate > - > TestValueVector.java > - Add unit tests > > > Diffs > ----- > > exec/java-exec/src/main/codegen/templates/FixedValueVectors.java > 7103a17108693d47839212c418d11d13fbb8f6f4 > exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java > bd41e10d3f69e13d0f8c426460af5e9a09d93fd9 > > exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java > ec409a3fc59616708226aa500ccab1680cd261f6 > exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java > 10bdf0752632c7577b9a6eb445c7101ec1a24730 > > exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java > 037c8c6d3da94acf5c2ca300ce617338cacb0fb0 > > Diff: https://reviews.apache.org/r/35484/diff/ > > > Testing > ------- > > all > > > Thanks, > > Hanifi Gunes > >
