Robert Hou created DRILL-7120:
---------------------------------
Summary: Query fails with ChannelClosedException
Key: DRILL-7120
URL: https://issues.apache.org/jira/browse/DRILL-7120
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.16.0
Reporter: Robert Hou
Assignee: Gautam Parai
Fix For: 1.16.0
TPCH query 5 fails at sf100. Here is the query:
{noformat}
select
n.n_name,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue
from
customer c,
orders o,
lineitem l,
supplier s,
nation n,
region r
where
c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and l.l_suppkey = s.s_suppkey
and c.c_nationkey = s.s_nationkey
and s.s_nationkey = n.n_nationkey
and n.n_regionkey = r.r_regionkey
and r.r_name = 'EUROPE'
and o.o_orderdate >= date '1997-01-01'
and o.o_orderdate < date '1997-01-01' + interval '1' year
group by
n.n_name
order by
revenue desc;
{noformat}
This is the error from drillbit.log:
{noformat}
2019-03-04 17:46:38,684 [23822b0a-b7bd-0b79-b905-1438f5b1d039:frag:6:64] INFO
o.a.d.e.w.fragment.FragmentExecutor -
23822b0a-b7bd-0b79-b905-1438f5b1d039:6:64: State change requested RUNNING -->
FINISHED
2019-03-04 17:46:38,684 [23822b0a-b7bd-0b79-b905-1438f5b1d039:frag:6:64] INFO
o.a.d.e.w.f.FragmentStatusReporter - 23822b0a-b7bd-0b79-b905-1438f5b1d039:6:64:
State to report: FINISHED
2019-03-04 18:17:51,454 [BitServer-13] WARN
o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming
stream due to memory limits. Current Allocation: 262144.
2019-03-04 18:17:51,454 [BitServer-13] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2019-03-04 18:17:51,463 [BitServer-13] ERROR o.a.d.exec.rpc.RpcExceptionHandler
- Exception in RPC communication. Connection: /10.10.120.104:31012 <-->
/10.10.120.106:53048 (data server). Closing connection.
io.netty.handler.codec.DecoderException:
org.apache.drill.exec.exception.OutOfMemoryException: Failure allocating buffer.
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271)
~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure
allocating buffer.
at
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
at
org.apache.drill.exec.memory.AllocationManager.<init>(AllocationManager.java:84)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:258)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:211)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
io.netty.buffer.ExpandableByteBuf.capacity(ExpandableByteBuf.java:43)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
at
io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:270)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:867)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:859)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:94)
~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
... 18 common frames omitted
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate
16777216 byte(s) of direct memory (used: 107374182400, max: 107374182400)
at
io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:510)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:464)
~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:766)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:742)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:226)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:169)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
at
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:169)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
at
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:201)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
at
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:65)
~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
... 30 common frames omitted
2019-03-04 18:17:51,464 [BitServer-13] INFO
o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining
95971 byte(s) in buffer.
2019-03-04 18:17:51,472 [BitServer-13] WARN
o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming
stream due to memory limits. Current Allocation: 335872.
{noformat}
This query completes when Statistics is enabled.
I am using commit 469be17597e7b7c6bc1de9863dcb6c5604a55f0c.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
