In addition to test*VectorReallocation errors I am seeing frequently on my linux VM, today I saw the following error multiple times on my VM and once on my Macbook:
Tests in error: > TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป IllegalState > Failure... > java.lang.IllegalStateException: Failure while closing accountor. > Expected private and shared pools to be set to initial values. However, > one or more were not. Stats are > zone init allocated delta > private 0 0 0 > shared 3221225472 3214166779 7058693. > at > org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) > ~[classes/:na] > at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390) > ~[classes/:na] > at > org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185) > ~[classes/:na] > at > org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75) > ~[classes/:na] > at com.google.common.io.Closeables.close(Closeables.java:77) > ~[guava-14.0.1.jar:na] > at com.google.common.io.Closeables.closeQuietly(Closeables.java:108) > ~[guava-14.0.1.jar:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294) > ~[classes/:na] > at > org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332) > ~[classes/:na] On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <chriswesti...@gmail.com> wrote: > Given that the difference is just > > > java.lang.Exception: Unexpected exception, > > expected<org.apache.drill.exec.exception.OversizedAllocationException> > but > > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > The question of "what constitutes an oversized allocation?" comes to mind. > Is this test fragile relative to being run in different environments? > I haven't seen the test so how is the determination that something is > oversized made? It seems like that criterion sometimes fails, and we get an > OOM because whatever the request is is still very large. > > > On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgu...@maprtech.com> wrote: > > > I don't seem to be able to re-prod this. Let me look at this and update > you > > all. > > > > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche < > > adene...@maprtech.com> > > wrote: > > > > > I didn't make any change, I'm running 2 forks (the default). I got > those > > > errors 3 times now, 2 on a linux VM and 1 on a linux physical node > > > > > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <hgu...@maprtech.com> > > wrote: > > > > > > > Did you tighten your memory settings? How many forks are you running > > > with? > > > > I bet you are truly running out of memory while executing this > > particular > > > > test case. > > > > > > > > -H+ > > > > > > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam < > skat...@maprtech.com> > > > > wrote: > > > > > > > > > b2bbd99 committed on July 6th introduced the test. > > > > > > > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <jinfengn...@gmail.com> > > > wrote: > > > > > > > > > > > > In that case, we probably need do binary search to figure out > > which > > > > > recent > > > > > > patch is causing this problem. > > > > > > > > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche < > > > > > adene...@maprtech.com> > > > > > > wrote: > > > > > > > > > > > >> Just got those errors on master too > > > > > >> > > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche < > > > > > adene...@maprtech.com > > > > > >>> > > > > > >> wrote: > > > > > >> > > > > > >>> I'm seeing those errors intermittently when building my private > > > > > branch, I > > > > > >>> don't believe I made any change that would have caused them. > > Anyone > > > > > seen > > > > > >>> them too ? > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > > > > >>>> Time elapsed: 2.043 sec <<< ERROR! > > > > > >>>> java.lang.Exception: Unexpected exception, > > > > > >>>> > > > > > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > > > > >> but > > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > > > > >>>> at > > > > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125) > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > > > > >>>> Time elapsed: 0.436 sec <<< ERROR! > > > > > >>>> java.lang.Exception: Unexpected exception, > > > > > >>>> > > > > > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > > > > >> but > > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85) > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) > > > > > >>>> Time elapsed: 0.788 sec <<< ERROR! > > > > > >>>> java.lang.Exception: Unexpected exception, > > > > > >>>> > > > > > expected<org.apache.drill.exec.exception.OversizedAllocationException> > > > > > >> but > > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> > > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) > > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372) > > > > > >>>> at > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142) > > > > > >>> > > > > > >>> > > > > > >>> Thanks > > > > > >>> -- > > > > > >>> > > > > > >>> Abdelhakim Deneche > > > > > >>> > > > > > >>> Software Engineer > > > > > >>> > > > > > >>> <http://www.mapr.com/> > > > > > >>> > > > > > >>> > > > > > >>> Now Available - Free Hadoop On-Demand Training > > > > > >>> < > > > > > >> > > > > > > > > > > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > >>> > > > > > >>> > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Abdelhakim Deneche > > > > > >> > > > > > >> Software Engineer > > > > > >> > > > > > >> <http://www.mapr.com/> > > > > > >> > > > > > >> > > > > > >> Now Available - Free Hadoop On-Demand Training > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Abdelhakim Deneche > > > > > > Software Engineer > > > > > > <http://www.mapr.com/> > > > > > > > > > Now Available - Free Hadoop On-Demand Training > > > < > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > > > > > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
