> On June 19, 2015, 5:06 p.m., Jason Altekruse wrote: > > exec/java-exec/src/main/codegen/templates/FixedValueVectors.java, line 92 > > <https://reviews.apache.org/r/35484/diff/1/?file=985297#file985297line92> > > > > The reallocation loops will run infinitely if we have it the maximum > > buffer size. This can be reproduced by reading a JSON file with a lot of > > large lists. This could be fixed by re-intriducing the code we had before > > re-allocation, but only in the case where we hit one of these limits. > > Allocations should be able to fail, and we need to make sure the operators > > can handle this case. > > > > We should fail with an OOM error in these cases until we can split > > lists across batches or find another way around the fact that the allocator > > tracks buffer lengths with ints. > > Jason Altekruse wrote: > *hit the maximum buffer size, *re-introducing the code > > Hanifi Gunes wrote: > I doubt if it is a good idea to fail with OOM. Instead we should consider > setting the upper bound to max allowed by the allocator (Int.MAX) and ensure > that this invariant holds across vector impls.
The problem is that the consumers of this interface are currently working under the assumption that a request to write to a vector will not fail unless we run out of memory. We need to go back and update the operators to stop their processing when we cannot do a write into the vector, or "emulate" the case they expect, which is OOM. - Jason ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35484/#review88551 ----------------------------------------------------------- On June 17, 2015, 6:54 p.m., Hanifi Gunes wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35484/ > ----------------------------------------------------------- > > (Updated June 17, 2015, 6:54 p.m.) > > > Review request for drill, Mehant Baid and Venki Korukanti. > > > Repository: drill-git > > > Description > ------- > > DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent > overflows > Vectors > - set an upper bound on # of bytes to allocate > - > TestValueVector.java > - Add unit tests > > > Diffs > ----- > > exec/java-exec/src/main/codegen/templates/FixedValueVectors.java > 7103a17108693d47839212c418d11d13fbb8f6f4 > exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java > bd41e10d3f69e13d0f8c426460af5e9a09d93fd9 > > exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java > ec409a3fc59616708226aa500ccab1680cd261f6 > exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java > 10bdf0752632c7577b9a6eb445c7101ec1a24730 > > exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java > 037c8c6d3da94acf5c2ca300ce617338cacb0fb0 > > Diff: https://reviews.apache.org/r/35484/diff/ > > > Testing > ------- > > all > > > Thanks, > > Hanifi Gunes > >
