Re: anyone seen these errors on master ?

Daniel Barclay Tue, 15 Sep 2015 13:13:45 -0700

I recently saw some failures that may be related.  Some test got a
different allocation exception than it expected.  I think that it was
expecting something like OversizeAllocationException but got an
OutOfMemoryError or OutOfMemoryException.


Daniel

Abdel Hakim Deneche wrote:

I didn't go into the details of the failures, but the exception was caused
by netty's not being able to allocate a new chunk of direct memory.

On Tue, Sep 15, 2015 at 10:21 AM, Jacques Nadeau <jacq...@dremio.com> wrote:

I've seen sporadic failures with those tests. I'm wondering whether they
need to be redesigned. Limiting direct memory to 8gb shouldn't fail the
test.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <
adene...@maprtech.com>
wrote:

I found what was causing those errors on my VM. I exported a MAVEN_OPTS
that limits the direct memory to 8GB and forget to unset it, this was
causing the build to fail because it couldn't allocate more direct

memory,

for some reason it was always failing on those TestValueVector unit

tests.


On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <
adene...@maprtech.com>
wrote:

I am still seeing the TestValueVector unit tests fail on my linux

machine

(2/3 of my builds)

On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche <
adene...@maprtech.com> wrote:

You can safely ignore the TestImpersonationMetadata error, it's most
likely caused by my change

On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche <
adene...@maprtech.com> wrote:

In addition to test*VectorReallocation errors I am seeing frequently

on

my linux VM, today I saw the following error multiple times on my VM

and

once on my Macbook:

Tests in error:

   TestImpersonationMetadata>BaseTestQuery.closeClient:239 »
IllegalState Failure...
java.lang.IllegalStateException: Failure while closing accountor.
Expected private and shared pools to be set to initial values.

However,

one or more were not.  Stats are
zone init allocated delta
private 0 0 0
shared 3221225472 3214166779 7058693.
at

org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)

~[classes/:na]
at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
~[classes/:na]
at

org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)

~[classes/:na]
at

org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)

~[classes/:na]
at com.google.common.io.Closeables.close(Closeables.java:77)
~[guava-14.0.1.jar:na]
at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
~[guava-14.0.1.jar:na]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
~[classes/:na]
at

org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)

~[classes/:na]



On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <

chriswesti...@gmail.com>

wrote:

Given that the difference is just

java.lang.Exception: Unexpected exception,

expected<org.apache.drill.exec.exception.OversizedAllocationException>

but

was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>


The question of "what constitutes an oversized allocation?" comes to
mind.
Is this test fragile relative to being run in different

environments?

I haven't seen the test so how is the determination that something

is

oversized made? It seems like that criterion sometimes fails, and we
get an
OOM because whatever the request is is still very large.


On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgu...@maprtech.com>
wrote:

I don't seem to be able to re-prod this. Let me look at this and

update you

all.

On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
adene...@maprtech.com>
wrote:

I didn't make any change, I'm running 2 forks (the default). I

got

those

errors 3 times now, 2 on a linux VM and 1 on a linux physical

node


On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <

hgu...@maprtech.com

wrote:

Did you tighten your memory settings? How many forks are you

running

with?

I bet you are truly running out of memory while executing this

particular

test case.

-H+

On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam <

skat...@maprtech.com>

wrote:

b2bbd99 committed on July 6th introduced the test.

On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <

jinfengn...@gmail.com>

wrote:


In that case,  we probably need do binary search to figure

out

which

recent

patch is causing this problem.

On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche <

adene...@maprtech.com>

wrote:

Just got those errors on master too

On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche <

adene...@maprtech.com

wrote:

I'm seeing those errors intermittently when building my

private

branch, I

don't believe I made any change that would have caused

them.

Anyone

seen

them too ?

testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)

Time elapsed: 2.043 sec  <<< ERROR!
java.lang.Exception: Unexpected exception,

expected<org.apache.drill.exec.exception.OversizedAllocationException>

but

was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>

at java.nio.Bits.reserveMemory(Bits.java:658)
at

java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)

at

java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)

at

io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)

at

io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)

at

io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)

at

io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)

at

org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)

at

org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)

testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)

Time elapsed: 0.436 sec  <<< ERROR!
java.lang.Exception: Unexpected exception,

expected<org.apache.drill.exec.exception.OversizedAllocationException>

but

was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>

at java.nio.Bits.reserveMemory(Bits.java:658)
at

java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)

at

java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)

at

io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)

at

io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)

at

io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)

at

io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)

at

org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)

at

org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)

at

org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)

testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)

Time elapsed: 0.788 sec  <<< ERROR!
java.lang.Exception: Unexpected exception,

expected<org.apache.drill.exec.exception.OversizedAllocationException>

but

was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>

at java.nio.Bits.reserveMemory(Bits.java:658)
at

java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)

at

java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)

at

io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)

at

io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)

at

io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)

at

io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)

at

io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)

at

org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)

at

org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)

at

org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)



Thanks
--

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

   <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

   <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

   <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

   <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available




--

Abdelhakim Deneche

Software Engineer

   <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<

http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available



--
Daniel Barclay
MapR Technologies

Re: anyone seen these errors on master ?

Reply via email to