I've seen sporadic failures with those tests. I'm wondering whether they
need to be redesigned. Limiting direct memory to 8gb shouldn't fail the
test.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <
[email protected]>
wrote:
I found what was causing those errors on my VM. I exported a MAVEN_OPTS
that limits the direct memory to 8GB and forget to unset it, this was
causing the build to fail because it couldn't allocate more direct
memory,
for some reason it was always failing on those TestValueVector unit
tests.
On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <
[email protected]>
wrote:
I am still seeing the TestValueVector unit tests fail on my linux
machine
(2/3 of my builds)
On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche <
[email protected]> wrote:
You can safely ignore the TestImpersonationMetadata error, it's most
likely caused by my change
On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche <
[email protected]> wrote:
In addition to test*VectorReallocation errors I am seeing frequently
on
my linux VM, today I saw the following error multiple times on my VM
and
once on my Macbook:
Tests in error:
TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป
IllegalState Failure...
java.lang.IllegalStateException: Failure while closing accountor.
Expected private and shared pools to be set to initial values.
However,
one or more were not. Stats are
zone init allocated delta
private 0 0 0
shared 3221225472 3214166779 7058693.
at
org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
~[classes/:na]
at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
~[classes/:na]
at
org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)
~[classes/:na]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
~[classes/:na]
at com.google.common.io.Closeables.close(Closeables.java:77)
~[guava-14.0.1.jar:na]
at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
~[guava-14.0.1.jar:na]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
~[classes/:na]
at
org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)
~[classes/:na]
On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <
[email protected]>
wrote:
Given that the difference is just
java.lang.Exception: Unexpected exception,
expected<org.apache.drill.exec.exception.OversizedAllocationException>
but
was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
The question of "what constitutes an oversized allocation?" comes to
mind.
Is this test fragile relative to being run in different
environments?
I haven't seen the test so how is the determination that something
is
oversized made? It seems like that criterion sometimes fails, and we
get an
OOM because whatever the request is is still very large.
On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <[email protected]>
wrote:
I don't seem to be able to re-prod this. Let me look at this and
update you
all.
On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
[email protected]>
wrote:
I didn't make any change, I'm running 2 forks (the default). I
got
those
errors 3 times now, 2 on a linux VM and 1 on a linux physical
node
On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <
[email protected]
wrote:
Did you tighten your memory settings? How many forks are you
running
with?
I bet you are truly running out of memory while executing this
particular
test case.
-H+
On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam <
[email protected]>
wrote:
b2bbd99 committed on July 6th introduced the test.
On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <
[email protected]>
wrote:
In that case, we probably need do binary search to figure
out
which
recent
patch is causing this problem.
On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche <
[email protected]>
wrote:
Just got those errors on master too
On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche <
[email protected]
wrote:
I'm seeing those errors intermittently when building my
private
branch, I
don't believe I made any change that would have caused
them.
Anyone
seen
them too ?
testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
Time elapsed: 2.043 sec <<< ERROR!
java.lang.Exception: Unexpected exception,
expected<org.apache.drill.exec.exception.OversizedAllocationException>
but
was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
at java.nio.Bits.reserveMemory(Bits.java:658)
at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
at
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
at
io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
at
io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
at
org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)
at
org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)
testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
Time elapsed: 0.436 sec <<< ERROR!
java.lang.Exception: Unexpected exception,
expected<org.apache.drill.exec.exception.OversizedAllocationException>
but
was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
at java.nio.Bits.reserveMemory(Bits.java:658)
at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
at
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
at
io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
at
io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
at
org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)
at
org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)
at
org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)
testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
Time elapsed: 0.788 sec <<< ERROR!
java.lang.Exception: Unexpected exception,
expected<org.apache.drill.exec.exception.OversizedAllocationException>
but
was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
at java.nio.Bits.reserveMemory(Bits.java:658)
at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
at
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
at
io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
at
io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
at
org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
at
org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)
at
org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)
Thanks
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<
http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
