[ https://issues.apache.org/jira/browse/DRILL-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venki Korukanti updated DRILL-2643: ----------------------------------- Fix Version/s: (was: 1.0.0) 1.1.0 > Exception during xsort.ExternalSortBatch.cleanup (possible memory leak ?) > ------------------------------------------------------------------------- > > Key: DRILL-2643 > URL: https://issues.apache.org/jira/browse/DRILL-2643 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators > Affects Versions: 0.8.0 > Reporter: Victoria Markman > Assignee: Venki Korukanti > Fix For: 1.1.0 > > Attachments: DRILL-2643.patch, t1.parquet, t2.parquet > > > In this case j1,j2 are views created on top of parquet files, BOTH views > have order by on multiple columns in different order with nulls first/last. > Also, table in in view j1, consists of 99 parquet files. See attached > views.txt file on how to create views (make sure to create views in a > different workspace, views have the same names as tables) > {code} > select DISTINCT > COALESCE(j1.c_varchar || j2.c_varchar || 'EMPTY') as > concatentated_string > from > j1 INNER JOIN j2 ON > (j1.d18 = j2.d18) > ; > {code} > The same can be reproduced with parquet files and subqueries: > (pay attention parquet files are named the same as views: j1, j2) > {code} > select DISTINCT > COALESCE(sq1.c_varchar || sq2.c_varchar || 'EMPTY') as > concatentated_string > from > (select c_varchar, c_integer from j1 order by j1.c_varchar desc nulls > first ) as sq1(c_varchar, c_integer) > INNER JOIN > (select c_varchar, c_integer from j2 order by j2.c_varchar nulls > last) as sq2(c_varchar, c_integer) > ON (sq1.c_integer = sq2.c_integer) > {code} > You do need to have sort in order to reproduce the problem. > This query works: > {code} > select DISTINCT > COALESCE(j1.c_varchar || j2.c_varchar || 'EMPTY') as > concatentated_string > from j1,j2 > where j1.c_integer = j2.c_integer; > {code} > {code} > 2015-04-01 00:43:42,455 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO > o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: > Executed 99 out of 99 using 16 threads. Time: 20ms total, 2.877318ms avg, 3ms > max. > 2015-04-01 00:43:42,458 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host > atsqa4-136.qa.lab. Skipping affinity to that host. > 2015-04-01 00:43:42,458 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO > o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: > Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.562620ms avg, 1ms max. > 2015-04-01 00:43:42,485 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING --> > RUNNING > 2015-04-01 00:43:45,613 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] WARN > o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 32 batch groups. > Current allocated memory: 16642330 > 2015-04-01 00:43:45,676 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO > o.a.d.exec.vector.BaseValueVector - Realloc vector null. [16384] -> [32768] > 2015-04-01 00:43:45,676 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO > o.a.d.exec.vector.BaseValueVector - Realloc vector > ``c_varchar`(VARCHAR:OPTIONAL)_bits`(UINT1:REQUIRED). [4096] -> [8192] > 2015-04-01 00:43:45,679 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO > o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] -> [65536] > 2015-04-01 00:43:45,680 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO > o.a.d.exec.vector.BaseValueVector - Realloc vector > ``c_varchar`(VARCHAR:OPTIONAL)_bits`(UINT1:REQUIRED). [8192] -> [16384] > 2015-04-01 00:43:45,709 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] WARN > o.a.d.exec.memory.AtomicRemainder - Tried to close remainder, but it has > already been closed > java.lang.Exception: null > at > org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:196) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:298) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.cleanup(ExternalSortBatch.java:162) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.cleanup(AbstractSingleRecordBatch.java:121) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.cleanup(RemovingRecordBatch.java:191) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.join.MergeJoinBatch.cleanup(MergeJoinBatch.java:260) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.cleanup(AbstractSingleRecordBatch.java:121) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.cleanup(HashAggBatch.java:281) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.internalStop(ScreenCreator.java:176) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:123) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:58) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:163) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > {code} > Explain plan: > {code} > 00-01 HashAgg(group=[{0}]) > 00-02 Project(concatentated_string=[||(||($0, $2), 'EMPTY')]) > 00-03 MergeJoin(condition=[=($1, $3)], joinType=[inner]) > 00-05 SelectionVectorRemover > 00-07 Sort(sort0=[$1], dir0=[ASC]) > 00-09 SelectionVectorRemover > 00-11 Sort(sort0=[$0], dir0=[DESC-nulls-first]) > 00-13 Project(c_varchar=[$1], c_integer=[$0]) > 00-15 Scan(groupscan=[ParquetGroupScan > [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/joins/j1]], > selectionRoot=/drill/testdata/joins/j1, numFiles=1, columns=[`c_varchar`, > `c_integer`]]]) > 00-04 Project(c_varchar0=[$0], c_integer0=[$1]) > 00-06 SelectionVectorRemover > 00-08 Sort(sort0=[$1], dir0=[ASC]) > 00-10 SelectionVectorRemover > 00-12 Sort(sort0=[$0], dir0=[ASC-nulls-last]) > 00-14 Project(c_varchar=[$1], c_integer=[$0]) > 00-16 Scan(groupscan=[ParquetGroupScan > [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/joins/j2]], > selectionRoot=/drill/testdata/joins/j2, numFiles=1, columns=[`c_va > {code} > I'm sure there is an easier way to reproduce this failure, can't think about > it right now. Please talk to me if you have trouble reproducing this failure. > Issue with the "maybe" redundant sort will be addressed in a different bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)