I found what was causing those errors on my VM. I exported a MAVEN_OPTS that limits the direct memory to 8GB and forget to unset it, this was causing the build to fail because it couldn't allocate more direct memory, for some reason it was always failing on those TestValueVector unit tests.
On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <[email protected]> wrote: > I am still seeing the TestValueVector unit tests fail on my linux machine > (2/3 of my builds) > > On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche < > [email protected]> wrote: > >> You can safely ignore the TestImpersonationMetadata error, it's most >> likely caused by my change >> >> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche < >> [email protected]> wrote: >> >>> In addition to test*VectorReallocation errors I am seeing frequently on >>> my linux VM, today I saw the following error multiple times on my VM and >>> once on my Macbook: >>> >>> Tests in error: >>>> TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป >>>> IllegalState Failure... >>>> java.lang.IllegalStateException: Failure while closing accountor. >>>> Expected private and shared pools to be set to initial values. However, >>>> one or more were not. Stats are >>>> zone init allocated delta >>>> private 0 0 0 >>>> shared 3221225472 3214166779 7058693. >>>> at >>>> org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) >>>> ~[classes/:na] >>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390) >>>> ~[classes/:na] >>>> at >>>> org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185) >>>> ~[classes/:na] >>>> at >>>> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75) >>>> ~[classes/:na] >>>> at com.google.common.io.Closeables.close(Closeables.java:77) >>>> ~[guava-14.0.1.jar:na] >>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108) >>>> ~[guava-14.0.1.jar:na] >>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294) >>>> ~[classes/:na] >>>> at >>>> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332) >>>> ~[classes/:na] >>> >>> >>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <[email protected]> >>> wrote: >>> >>>> Given that the difference is just >>>> >>>> > java.lang.Exception: Unexpected exception, >>>> > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> but >>>> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> >>>> >>>> The question of "what constitutes an oversized allocation?" comes to >>>> mind. >>>> Is this test fragile relative to being run in different environments? >>>> I haven't seen the test so how is the determination that something is >>>> oversized made? It seems like that criterion sometimes fails, and we >>>> get an >>>> OOM because whatever the request is is still very large. >>>> >>>> >>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <[email protected]> >>>> wrote: >>>> >>>> > I don't seem to be able to re-prod this. Let me look at this and >>>> update you >>>> > all. >>>> > >>>> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche < >>>> > [email protected]> >>>> > wrote: >>>> > >>>> > > I didn't make any change, I'm running 2 forks (the default). I got >>>> those >>>> > > errors 3 times now, 2 on a linux VM and 1 on a linux physical node >>>> > > >>>> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <[email protected]> >>>> > wrote: >>>> > > >>>> > > > Did you tighten your memory settings? How many forks are you >>>> running >>>> > > with? >>>> > > > I bet you are truly running out of memory while executing this >>>> > particular >>>> > > > test case. >>>> > > > >>>> > > > -H+ >>>> > > > >>>> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam < >>>> [email protected]> >>>> > > > wrote: >>>> > > > >>>> > > > > b2bbd99 committed on July 6th introduced the test. >>>> > > > > >>>> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni < >>>> [email protected]> >>>> > > wrote: >>>> > > > > > >>>> > > > > > In that case, we probably need do binary search to figure out >>>> > which >>>> > > > > recent >>>> > > > > > patch is causing this problem. >>>> > > > > > >>>> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche < >>>> > > > > [email protected]> >>>> > > > > > wrote: >>>> > > > > > >>>> > > > > >> Just got those errors on master too >>>> > > > > >> >>>> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche < >>>> > > > > [email protected] >>>> > > > > >>> >>>> > > > > >> wrote: >>>> > > > > >> >>>> > > > > >>> I'm seeing those errors intermittently when building my >>>> private >>>> > > > > branch, I >>>> > > > > >>> don't believe I made any change that would have caused them. >>>> > Anyone >>>> > > > > seen >>>> > > > > >>> them too ? >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) >>>> > > > > >>>> Time elapsed: 2.043 sec <<< ERROR! >>>> > > > > >>>> java.lang.Exception: Unexpected exception, >>>> > > > > >>>> >>>> > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> >>>> > > > > >> but >>>> > > > > >>>> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) >>>> > > > > >>>> at >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>>> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) >>>> > > > > >>>> at >>>> > > > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125) >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) >>>> > > > > >>>> Time elapsed: 0.436 sec <<< ERROR! >>>> > > > > >>>> java.lang.Exception: Unexpected exception, >>>> > > > > >>>> >>>> > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> >>>> > > > > >> but >>>> > > > > >>>> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) >>>> > > > > >>>> at >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>>> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85) >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector) >>>> > > > > >>>> Time elapsed: 0.788 sec <<< ERROR! >>>> > > > > >>>> java.lang.Exception: Unexpected exception, >>>> > > > > >>>> >>>> > > > >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException> >>>> > > > > >> but >>>> > > > > >>>> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658) >>>> > > > > >>>> at >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>>> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372) >>>> > > > > >>>> at >>>> > > > > >>>> >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142) >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >>> Thanks >>>> > > > > >>> -- >>>> > > > > >>> >>>> > > > > >>> Abdelhakim Deneche >>>> > > > > >>> >>>> > > > > >>> Software Engineer >>>> > > > > >>> >>>> > > > > >>> <http://www.mapr.com/> >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >>> Now Available - Free Hadoop On-Demand Training >>>> > > > > >>> < >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>> > > > > >>> >>>> > > > > >>> >>>> > > > > >> >>>> > > > > >> >>>> > > > > >> >>>> > > > > >> -- >>>> > > > > >> >>>> > > > > >> Abdelhakim Deneche >>>> > > > > >> >>>> > > > > >> Software Engineer >>>> > > > > >> >>>> > > > > >> <http://www.mapr.com/> >>>> > > > > >> >>>> > > > > >> >>>> > > > > >> Now Available - Free Hadoop On-Demand Training >>>> > > > > >> < >>>> > > > > >> >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>> > > > > >>> >>>> > > > > >> >>>> > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > >>>> > > Abdelhakim Deneche >>>> > > >>>> > > Software Engineer >>>> > > >>>> > > <http://www.mapr.com/> >>>> > > >>>> > > >>>> > > Now Available - Free Hadoop On-Demand Training >>>> > > < >>>> > > >>>> > >>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>> > > > >>>> > > >>>> > >>>> >>> >>> >>> >>> -- >>> >>> Abdelhakim Deneche >>> >>> Software Engineer >>> >>> <http://www.mapr.com/> >>> >>> >>> Now Available - Free Hadoop On-Demand Training >>> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available> >>> >> >> >> >> -- >> >> Abdelhakim Deneche >> >> Software Engineer >> >> <http://www.mapr.com/> >> >> >> Now Available - Free Hadoop On-Demand Training >> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available> >> > > > > -- > > Abdelhakim Deneche > > Software Engineer > > <http://www.mapr.com/> > > > Now Available - Free Hadoop On-Demand Training > <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available> > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
