Khurram Faraaz created DRILL-6229: ------------------------------------- Summary: memory leak when output_batch_size is set to 4096 Key: DRILL-6229 URL: https://issues.apache.org/jira/browse/DRILL-6229 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Reporter: Khurram Faraaz
When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we continuously see in each test run, the memory leak when tests are run from test framework. Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01 alter system set `drill.exec.memory.operator.output_batch_size` = 4096; To run flatten tests from drill test framework bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40 The below two tests time out and we see the memory leak Timeouts: /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter24.q Timeouts: /root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter19.q oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection ... (user client) closed unexpectedly. Drillbit down? Stack trace printed in drill test framework log {noformat} [Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ] at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610) at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553) at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602) at oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602) at oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618) at oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35) at oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607) at oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at java.lang.Thread.run(Thread.java:748) Channel is closed, discarding remaining 908 byte(s) in buffer. Exception while closing client. java.lang.RuntimeException: Exception while closing at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459) at org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810) at org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823) at org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36) at oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205) at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102) at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562) at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100) Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (4198400) Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit) at oadd.org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520) at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) ... 8 more {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)