[ https://issues.apache.org/jira/browse/DRILL-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Westin updated DRILL-2750: -------------------------------- Fix Version/s: (was: 0.9.0) 1.0.0 > Running 1 or more queries against Drillbits having insufficient DirectMem > renders the Drillbits in an unusable state > -------------------------------------------------------------------------------------------------------------------- > > Key: DRILL-2750 > URL: https://issues.apache.org/jira/browse/DRILL-2750 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow > Affects Versions: 0.9.0 > Environment: RHEL 6.4 > Reporter: Kunal Khatua > Assignee: Chris Westin > Priority: Critical > Fix For: 1.0.0 > > > When running queries against a Drill cluster with limited DirectMem; if one > or more queries fail due to insufficient memory, then even queries that > should easily run within the allocated memory fail. > The initial failure when queries with large memory requirements fail: > 2015-04-10 09:57:55 [pip0] ERROR PipSQuawkling fetchRows - [ 1 / 16_par1000 ] > Failure while executing query. > java.sql.SQLException: Failure while executing query. > at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144) > at > net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187) > at org.apache.drill.jdbc.DrillResultSet.next(DrillResultSet.java:85) > at PipSQuawkling.fetchRows(PipSQuawkling.java:319) > at PipSQuawkling.executeTest(PipSQuawkling.java:154) > at PipSQuawkling.run(PipSQuawkling.java:76) > Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: > Failure while running fragment.[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on > ucs-node5.perf.lab:31010 ] > [ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at java.lang.Thread.run(Thread.java:744) > After that, subsequent queries that should run, fail with the following: > 2015-04-10 09:59:29 [pip0] ERROR PipSQuawkling executeQuery - [ 2 / > rerun_06_par1000 ] exception while executing query: Failure while executing > query. > java.sql.SQLException: exception while executing query: Failure while > executing query. > at net.hydromatic.avatica.Helper.createException(Helper.java:40) > at > net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406) > at > net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351) > at > net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78) > at PipSQuawkling.executeQuery(PipSQuawkling.java:284) > at PipSQuawkling.executeTest(PipSQuawkling.java:144) > at PipSQuawkling.run(PipSQuawkling.java:76) > Caused by: java.sql.SQLException: Failure while executing query. > at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144) > at > org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105) > at > org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44) > at > net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404) > ... 5 more > Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: > Failure while trying to start remote fragment, You attempted to create a new > child allocator with initial reservation 6000000 but only 110395 bytes of > memory were available. [ 689006cb-d703-42c3-860d > -bfecc0a66312 on ucs-node10.perf.lab:31010 ] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)