[ https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731088#comment-14731088 ]
Jacques Nadeau commented on DRILL-3555: --------------------------------------- My bad. I didn't see the conversation and it wasn't linked here. I'm changing my -1 to a +1. thanks for the explanation. > Changing defaults for planner.memory.max_query_memory_per_node causes queries > with window function to fail > ---------------------------------------------------------------------------------------------------------- > > Key: DRILL-3555 > URL: https://issues.apache.org/jira/browse/DRILL-3555 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Affects Versions: 1.1.0, 1.2.0 > Environment: 4 Nodes. Direct Memory= 48 GB each > Reporter: Abhishek Girish > Assignee: Deneche A. Hakim > Priority: Critical > Labels: window_function > Fix For: 1.2.0 > > > Changing the default value for planner.memory.max_query_memory_per_node from > 2 GB to anything higher causes queries with window functions to fail. > Changed system options > {code:sql} > > select * from sys.options where status like '%CHANGE%'; > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > | name | kind | type | status | > num_val | string_val | bool_val | float_val | > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED | > null | null | true | null | > | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED | > 8589934592 | null | null | null | > +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+ > 2 rows selected (0.249 seconds) > {code} > Query > {code:sql} > > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM > > store_sales ss LIMIT 20; > java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: > DrillRuntimeException: Adding this batch causes the total size to exceed max > allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. > maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73) > at > sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87) > at sqlline.TableOutputFormat.print(TableOutputFormat.java:118) > at sqlline.SqlLine.print(SqlLine.java:1583) > at sqlline.Commands.execute(Commands.java:852) > at sqlline.Commands.sql(Commands.java:751) > at sqlline.SqlLine.dispatch(SqlLine.java:738) > at sqlline.SqlLine.begin(SqlLine.java:612) > at sqlline.SqlLine.start(SqlLine.java:366) > at sqlline.SqlLine.main(SqlLine.java:259) > {code} > Log: > {code} > 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING --> > FINISHED > 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED > 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR > o.a.d.e.p.i.s.SortRecordBatchBuilder - Adding this batch causes the total > size to exceed max allowed size. Current runningBytes 1073638500, Incoming > batchBytes 127875. maxBytes 1073741824 > 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING --> > FAILED > ... > 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED --> > FINISHED > 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: DrillRuntimeException: > Adding this batch causes the total size to exceed max allowed size. Current > runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > DrillRuntimeException: Adding this batch causes the total size to exceed max > allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. > maxBytes 1073741824 > Fragment 1:0 > [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding > this batch causes the total size to exceed max allowed size. Current > runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824 > at > org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... > 2015-07-23 18:17:06,021 [BitServer-3] WARN o.a.d.e.w.b.ControlMessageHandler > - Dropping request to cancel fragment. > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist. > 2015-07-23 18:17:06,022 [BitServer-3] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State change requested RUNNING --> > CANCELLATION_REQUESTED > 2015-07-23 18:17:06,023 [BitServer-3] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State to report: > CANCELLATION_REQUESTED > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)