[jira] [Updated] (DRILL-5191) OutOfMemoryException - TPC-DS query4 on SF100

2017-01-17 Thread Khurram Faraaz (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-5191:
--
Attachment: query_78_2780fbcb-7083-eba1-a9d5-039d32c059be.sys.drill

> OutOfMemoryException - TPC-DS query4 on SF100
> -
>
> Key: DRILL-5191
> URL: https://issues.apache.org/jira/browse/DRILL-5191
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.10.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Priority: Critical
> Attachments: 2789eba3-60f0-0b2f-eba8-82331735d5c4.sys.drill, 
> query_78_2780fbcb-7083-eba1-a9d5-039d32c059be.sys.drill
>
>
> TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS 
> cluster
> Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql
> total number of fragments : 1,125
> Stack trace from drillbit.log
> {noformat}
> 2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State to report: RUNNING
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> 
> FAILED
> 2017-01-11 11:17:57,009 [BitServer-6] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:31012 <--> /10.10.100.201:44712 (data server).  
> Closing connection.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate 
> buffer of size 16384 due to memory limit. Current allocation: 16777216
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested 
> AWAITING_ALLOCATION --> FAILED
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State to report: RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> 
> FINISHED
> 2017-01-11 11:17:57,010 [BitServer-6] INFO  
> o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding 
> remaining 3240924 byte(s) in buffer.
> 2017-01-11 11:17:57,011 [BitServer-10] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> C

[jira] [Updated] (DRILL-5191) OutOfMemoryException - TPC-DS query4 on SF100

2017-01-11 Thread Khurram Faraaz (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-5191:
--
Summary: OutOfMemoryException - TPC-DS query4 on SF100  (was: 
OutOfMemoryException - TPCDS query4 )

> OutOfMemoryException - TPC-DS query4 on SF100
> -
>
> Key: DRILL-5191
> URL: https://issues.apache.org/jira/browse/DRILL-5191
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.10.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Priority: Critical
> Attachments: 2789eba3-60f0-0b2f-eba8-82331735d5c4.sys.drill
>
>
> TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS 
> cluster
> Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql
> total number of fragments : 1,125
> Stack trace from drillbit.log
> {noformat}
> 2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State to report: RUNNING
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING --> 
> FAILED
> 2017-01-11 11:17:57,009 [BitServer-6] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:31012 <--> /10.10.100.201:44712 (data server).  
> Closing connection.
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate 
> buffer of size 16384 due to memory limit. Current allocation: 16777216
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191) 
> ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
>  ~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
> at 
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
>  ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested 
> AWAITING_ALLOCATION --> FAILED
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State to report: RUNNING
> 2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED --> 
> FINISHED
> 2017-01-11 11:17:57,010 [BitServer-6] INFO  
> o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding 
> remaining 3240924 byte(s) in buffer.
> 2017-01-11 11:17:57,011 [BitServer-10] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.202:3