[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908827#comment-14908827 ] Victoria Markman commented on DRILL-3616: - This is fixed in 1.2.0 #Thu Sep 24 00:27:47 UTC 2015 git.commit.id.abbrev=d2caa62 Ran the whole window functions test suite with cancellations and planner.memory.max_query_memory_per_node = 1048576 causing sort to spill to disk. > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Victoria Markman > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.d
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709425#comment-14709425 ] ASF GitHub Bot commented on DRILL-3616: --- Github user adeneche closed the pull request at: https://github.com/apache/drill/pull/112 > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Deneche A. Hakim > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ...
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707758#comment-14707758 ] Aman Sinha commented on DRILL-3616: --- I have committed the patch. commit id: 52f82130e However, I did not do it by merging the pull request. [~adeneche] could you close the pull request ? > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) > ~[drill-common-1.2
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706007#comment-14706007 ] ASF GitHub Bot commented on DRILL-3616: --- Github user adeneche commented on the pull request: https://github.com/apache/drill/pull/112#issuecomment-133222575 addressed the comment left by @amansinha100 by adding a comment to ExternalSortBatch.cleanupBatchGroups() > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.com
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705930#comment-14705930 ] Aman Sinha commented on DRILL-3616: --- I made one minor comment in the pull request. Overall, LGTM. +1 > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 10 common frames omitted > 2015-08-05 22:
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702145#comment-14702145 ] Deneche A. Hakim commented on DRILL-3616: - [~amansinha100] can you please review the pull request ? > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 10 common frames omitted > 2015-08-05 22
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697434#comment-14697434 ] ASF GitHub Bot commented on DRILL-3616: --- GitHub user adeneche opened a pull request: https://github.com/apache/drill/pull/112 DRILL-3616: Memory leak in a cleanup code after canceling queries wit… …h window functions spilling to disk You can merge this pull request into a Git repository by running: $ git pull https://github.com/adeneche/incubator-drill DRILL-3616 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit 4ddcfbb04cadd0e6fe777593d865e4ae1be82ea8 Author: adeneche Date: 2015-07-13T16:09:40Z DRILL-3616: Memory leak in a cleanup code after canceling queries with window functions spilling to disk > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Deneche A. Hakim > Fix For: 1.2.0 > > Attachments: DRILL-3616.1.patch.txt > > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec
[jira] [Commented] (DRILL-3616) Memory leak in a cleanup code after canceling queries with window functions spilling to disk
[ https://issues.apache.org/jira/browse/DRILL-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661118#comment-14661118 ] Chris Westin commented on DRILL-3616: - This bug was found with the new allocator branch, so there's no point in chasing it until that is merged. > Memory leak in a cleanup code after canceling queries with window functions > spilling to disk > > > Key: DRILL-3616 > URL: https://issues.apache.org/jira/browse/DRILL-3616 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Flow >Affects Versions: 1.2.0 > Environment: private-locking-allocator-branch >Reporter: Victoria Markman >Assignee: Victoria Markman > > Bunch of concurrent queries with window functions were cancelled. > Got an error in drillbit.log that might indicate that we have a memory leak > in in cleanup code after cancellation. > Assigning to myself for creation of a reproducible test case. > {code} > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from > CANCELLATION_REQUESTED --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FAILED > 2015-08-05 22:43:56,475 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:0:0: State change requested from FAILED > --> FINISHED > 2015-08-05 22:43:56,476 [2a3d6e54-12c2-2519-3ea1-736cb1e39e2a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Unaccounted for outstanding allocation (902492) > Fragment 0:0 > [Error Id: 1b9714b9-5a39-48ec-80e7-c49c79825cda on atsqa4-133.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.lang.RuntimeException: Exception while closing > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:139) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:439) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:352) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:173) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 5 common frames omitted > Caused by: java.lang.IllegalStateException: Unaccounted for outstanding > allocation (902492) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:1278) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 10 common frames omitted > 2015-08-05 22:43:56,477 [2a3d6e54-f2c1-4682-