[ https://issues.apache.org/jira/browse/DRILL-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maksym Rymar updated DRILL-8480: -------------------------------- Description: Drill fails to execute a query with the following exception: {code:java} org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished Fragment: 1:0 Please, refer to logs for more information. [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on compute7.vmcluster.com:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657) at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished at org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71) at org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) at org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567) at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240) ... 5 common frames omitted Suppressed: java.lang.IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished ... 15 common frames omitted Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (32768) Allocator(op:1:0:8:UnorderedReceiver) 1000000/32768/32768/10000000000 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) at org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159) at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571) ... 7 common frames omitted Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1016640) Allocator(frag:1:0) 30000000/1016640/30016640/90715827882 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574) ... 7 common frames omitted {code} Steps to reproduce: # Enable unequal join: {code:java} alter session set `planner.enable_nljoin_for_scalar_only`=false; {code} # Disable join optimization to prevent Drill from flipping sides of join that may break the query execution because the NestedLoopJoin operator that executes unequal joins supports only the left join. {code:java} alter session set `planner.enable_join_optimization`=false; {code} # Execute join one side which is UNION ALL with DISTINCT: {code:java} SELECT * FROM ( ( SELECT DISTINCT log_number FROM dfs.tmp.`tableWithNumber2.parquet` ) UNION ALL ( SELECT DISTINCT log_number FROM dfs.tmp.`tableWithNumber2.parquet` ) ) t1 LEFT JOIN ( SELECT log_number AS server_number FROM dfs.tmp.`tableWithNumber2.parquet` ) t3 ON ( t3.server_number >= t1.log_number ) {code} {{Parquet file for the reproduce: [^tableWithNumber2.parquet]. It contains 10 000 rows, with a single column of random double values. The file was generated by Drill:}} {code:java} apache drill> select *, sqlTypeOf(log_number) as log_number_column_type from dfs.tmp.`tableWithNumber2.parquet` limit 10; +------------+------------------------+ | log_number | log_number_column_type | +------------+------------------------+ | 4.0 | DOUBLE | | 5.0 | DOUBLE | | 4.0 | DOUBLE | | 3.0 | DOUBLE | | 4.0 | DOUBLE | | 0.0 | DOUBLE | | 0.0 | DOUBLE | | 8.0 | DOUBLE | | 3.0 | DOUBLE | | 4.0 | DOUBLE | +------------+------------------------+ 10 rows selected (0.175 seconds) {code} Also attaching the profile of the failed query. was: Drill fails to execute a query with the following exception: ``` org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished Fragment: 1:0 Please, refer to logs for more information. [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on compute7.vmcluster.com:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657) at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished at org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71) at org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121) at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) at org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567) at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240) ... 5 common frames omitted Suppressed: java.lang.IllegalStateException: Cleanup before finished. 0 out of 1 streams have finished ... 15 common frames omitted Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (32768) Allocator(op:1:0:8:UnorderedReceiver) 1000000/32768/32768/10000000000 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) at org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159) at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571) ... 7 common frames omitted Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1016640) Allocator(frag:1:0) 30000000/1016640/30016640/90715827882 (res/actual/peak/limit) at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) at org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) at org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574) ... 7 common frames omitted ``` Steps to reproduce: # Enable unequal join: {{alter session set `planner.enable_nljoin_for_scalar_only`=false;}} # Disable join optimization to prevent Drill from flipping sides of join that may break the query execution because the NestedLoopJoin operator that executes unequal joins supports only the left join. {{alter session set `planner.enable_join_optimization`=false;}} # Execute join one side which is UNION ALL with DISTINCT: {{SELECT *}} {{FROM (}} {{ (}} {{ SELECT DISTINCT}} {{ log_number}} {{ FROM}} {{ dfs.tmp.`tableWithNumber2.parquet`}} {{ )}} {{ UNION ALL}} {{ (}} {{ SELECT DISTINCT}} {{ log_number}} {{ FROM}} {{ dfs.tmp.`tableWithNumber2.parquet` }} {{ )}} {{) t1}} {{LEFT JOIN (}} {{ SELECT}} {{ log_number AS server_number}} {{ FROM}} {{ dfs.tmp.`tableWithNumber2.parquet` }} {{) t3}} {{ON (}} {{ t3.server_number >= t1.log_number}} {{)}} {{Parquet file for the reproduce: }}{{[^tableWithNumber2.parquet]. It contains 10 000 rows, with a single column of random double values:}} {{```}} {{apache drill> select *, sqlTypeOf(log_number) as log_number_column_type from dfs.tmp.`tableWithNumber2.parquet` limit 10; +------------+------------------------+ | log_number | log_number_column_type | +------------+------------------------+ | 4.0 | DOUBLE | | 5.0 | DOUBLE | | 4.0 | DOUBLE | | 3.0 | DOUBLE | | 4.0 | DOUBLE | | 0.0 | DOUBLE | | 0.0 | DOUBLE | | 8.0 | DOUBLE | | 3.0 | DOUBLE | | 4.0 | DOUBLE | +------------+------------------------+ 10 rows selected (0.175 seconds)}} {{```}} Also attaching the profile of the failed query. > Cleanup before finished. 0 out of 1 streams have finished > --------------------------------------------------------- > > Key: DRILL-8480 > URL: https://issues.apache.org/jira/browse/DRILL-8480 > Project: Apache Drill > Issue Type: Bug > Affects Versions: 1.21.1 > Reporter: Maksym Rymar > Assignee: Maksym Rymar > Priority: Major > Attachments: 1a349ff1-d1f9-62bf-ed8c-26346c548005.sys.drill, > tableWithNumber2.parquet > > > Drill fails to execute a query with the following exception: > {code:java} > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: Cleanup before finished. 0 out of 1 streams have > finished > Fragment: 1:0 > Please, refer to logs for more information. > [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on > compute7.vmcluster.com:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362) > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of > 1 streams have finished > at > org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111) > at > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) > at > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71) > at > org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121) > at > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91) > at > org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144) > at > org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) > at > org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240) > ... 5 common frames omitted > Suppressed: java.lang.IllegalStateException: Cleanup before finished. > 0 out of 1 streams have finished > ... 15 common frames omitted > Suppressed: java.lang.IllegalStateException: Memory was leaked by > query. Memory leaked: (32768) > Allocator(op:1:0:8:UnorderedReceiver) 1000000/32768/32768/10000000000 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > at > org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159) > at > org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77) > at > org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) > at > org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571) > ... 7 common frames omitted > Suppressed: java.lang.IllegalStateException: Memory was leaked by > query. Memory leaked: (1016640) > Allocator(frag:1:0) 30000000/1016640/30016640/90715827882 > (res/actual/peak/limit) > at > org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) > at > org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581) > at > org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574) > ... 7 common frames omitted {code} > Steps to reproduce: > # Enable unequal join: > {code:java} > alter session set `planner.enable_nljoin_for_scalar_only`=false; > {code} > # Disable join optimization to prevent Drill from flipping sides of join > that may break the query execution because the NestedLoopJoin operator that > executes unequal joins supports only the left join. > {code:java} > alter session set `planner.enable_join_optimization`=false; > {code} > # Execute join one side which is UNION ALL with DISTINCT: > {code:java} > SELECT * > FROM ( > ( > SELECT DISTINCT > log_number > FROM > dfs.tmp.`tableWithNumber2.parquet` > ) > UNION ALL > ( > SELECT DISTINCT > log_number > FROM > dfs.tmp.`tableWithNumber2.parquet` > ) > ) t1 > LEFT JOIN ( > SELECT > log_number AS server_number > FROM > dfs.tmp.`tableWithNumber2.parquet` > ) t3 > ON ( > t3.server_number >= t1.log_number > ) > {code} > {{Parquet file for the reproduce: [^tableWithNumber2.parquet]. It contains 10 > 000 rows, with a single column of random double values. The file was > generated by Drill:}} > {code:java} > apache drill> select *, sqlTypeOf(log_number) as log_number_column_type from > dfs.tmp.`tableWithNumber2.parquet` limit 10; > +------------+------------------------+ > | log_number | log_number_column_type | > +------------+------------------------+ > | 4.0 | DOUBLE | > | 5.0 | DOUBLE | > | 4.0 | DOUBLE | > | 3.0 | DOUBLE | > | 4.0 | DOUBLE | > | 0.0 | DOUBLE | > | 0.0 | DOUBLE | > | 8.0 | DOUBLE | > | 3.0 | DOUBLE | > | 4.0 | DOUBLE | > +------------+------------------------+ > 10 rows selected (0.175 seconds) {code} > Also attaching the profile of the failed query. -- This message was sent by Atlassian Jira (v8.20.10#820010)