[ https://issues.apache.org/jira/browse/DRILL-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Hou updated DRILL-5778: ------------------------------ Attachment: 264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0.sys.drill drillbit.log > Drill seems to run out of memory but completes execution > -------------------------------------------------------- > > Key: DRILL-5778 > URL: https://issues.apache.org/jira/browse/DRILL-5778 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators > Affects Versions: 1.11.0 > Reporter: Robert Hou > Assignee: Paul Rogers > Fix For: 1.12.0 > > Attachments: 264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0.sys.drill, > drillbit.log > > > Query is: > {noformat} > ALTER SESSION SET `exec.sort.disable_managed` = false; > alter session set `planner.width.max_per_node` = 1; > alter session set `planner.disable_exchanges` = true; > alter session set `planner.width.max_per_query` = 1; > alter session set `planner.memory.max_query_memory_per_node` = 2147483648; > select count(*) from (select * from (select id, flatten(str_list) str from > dfs.`/drill/testdata/resource-manager/flatten-large-small.json`) d order by > d.str) d1 where d1.id=0; > {noformat} > Plan is: > {noformat} > | 00-00 Screen > 00-01 Project(EXPR$0=[$0]) > 00-02 StreamAgg(group=[{}], EXPR$0=[$SUM0($0)]) > 00-03 UnionExchange > 01-01 StreamAgg(group=[{}], EXPR$0=[COUNT()]) > 01-02 Project($f0=[0]) > 01-03 SelectionVectorRemover > 01-04 Filter(condition=[=($0, 0)]) > 01-05 SingleMergeExchange(sort0=[1 ASC]) > 02-01 SelectionVectorRemover > 02-02 Sort(sort0=[$1], dir0=[ASC]) > 02-03 Project(id=[$0], str=[$1]) > 02-04 HashToRandomExchange(dist0=[[$1]]) > 03-01 UnorderedMuxExchange > 04-01 Project(id=[$0], str=[$1], > E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($1, 1301011)]) > 04-02 Flatten(flattenField=[$1]) > 04-03 Project(id=[$0], str=[$1]) > 04-04 Scan(groupscan=[EasyGroupScan > [selectionRoot=maprfs:/drill/testdata/resource-manager/flatten-large-small.json, > numFiles=1, columns=[`id`, `str_list`], > files=[maprfs:///drill/testdata/resource-manager/flatten-large-small.json]]]) > {noformat} > From drillbit.log: > {noformat} > 2017-09-08 05:07:21,515 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Actual batch schema & sizes { > str(type: REQUIRED VARCHAR, count: 4096, std size: 54, actual size: 134, > data size: 548360) > id(type: OPTIONAL BIGINT, count: 4096, std size: 8, actual size: 9, data > size: 36864) > Records: 4096, Total size: 1073819648, Data size: 585224, Gross row width: > 262163, Net row width: 143, Density: 1} > 2017-09-08 05:07:21,515 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] ERROR > o.a.d.e.p.i.x.m.ExternalSortBatch - Insufficient memory to merge two batches. > Incoming batch size: 1073819648, available memory: 2147483648 > 2017-09-08 05:07:21,517 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] INFO > o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug > option: true > 2017-09-08 05:07:21,517 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.compile.JaninoClassCompiler - Compiling (source size=3.3 KiB): > ... > 2017-09-08 05:07:21,536 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.exec.compile.ClassTransformer - Compiled and merged > SingleBatchSorterGen2677: bytecode size = 3.6 KiB, time = 19 ms. > 2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.t.g.SingleBatchSorterGen2677 - Took 5608 us to sort 4096 records > 2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Input Batch Estimates: record size = 143 > bytes; net = 1073819648 bytes, gross = 1610729472, records = 4096 > 2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Spill batch size: net = 1048476 bytes, > gross = 1572714 bytes, records = 7332; spill file = 268435456 bytes > 2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Output batch size: net = 9371505 bytes, > gross = 14057257 bytes, records = 65535 > 2017-09-08 05:07:21,566 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Available memory: 2147483648, buffer > memory = 2143289744, merge memory = 2128740638 > 2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.t.g.SingleBatchSorterGen2677 - Took 4303 us to sort 4096 records > 2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Input Batch Estimates: record size = 266 > bytes; net = 1073819648 bytes, gross = 1610729472, records = 4096 > 2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Spill batch size: net = 1048572 bytes, > gross = 1572858 bytes, records = 3942; spill file = 268435456 bytes > 2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Output batch size: net = 16777152 bytes, > gross = 25165728 bytes, records = 63072 > 2017-09-08 05:07:21,571 [264d780f-41ac-2c4f-6bc8-bdbb5eeb3df0:frag:0:0] DEBUG > o.a.d.e.p.i.x.m.ExternalSortBatch - Available memory: 2147483648, buffer > memory = 2143289360, merge memory = 2113929344 > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)