[ https://issues.apache.org/jira/browse/DRILL-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
salim achouche updated DRILL-6626: ---------------------------------- Labels: pull-request-available (was: ) > Hash Aggregate: Index out of bounds with small output batch size and spilling > ----------------------------------------------------------------------------- > > Key: DRILL-6626 > URL: https://issues.apache.org/jira/browse/DRILL-6626 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators > Affects Versions: 1.14.0 > Reporter: Boaz Ben-Zvi > Assignee: salim achouche > Priority: Major > Labels: pull-request-available > > This new IOOB failure was seen while trying to recreate the NPE failure in > DRILL-6622 (over TPC-DS SF1). The proposed fix for the latter (PR #1391) does > not seem to make a difference. > This IOOB can easily be created with other large Hash-Agg queries that need > to spill. > The IOOB was caused after restricting the output batch size (to force many), > and the Hash Aggr memory (to force a spill): > {code} > 0: jdbc:drill:zk=local> alter system set > `drill.exec.memory.operator.output_batch_size` = 262144; > +-------+--------------------------------------------------------+ > | ok | summary | > +-------+--------------------------------------------------------+ > | true | drill.exec.memory.operator.output_batch_size updated. | > +-------+--------------------------------------------------------+ > 1 row selected (0.106 seconds) > 0: jdbc:drill:zk=local> > 0: jdbc:drill:zk=local> alter session set `exec.errors.verbose` = true; > +-------+-------------------------------+ > | ok | summary | > +-------+-------------------------------+ > | true | exec.errors.verbose updated. | > +-------+-------------------------------+ > 1 row selected (0.081 seconds) > 0: jdbc:drill:zk=local> > 0: jdbc:drill:zk=local> alter session set `exec.hashagg.mem_limit` = 16777216; > +-------+----------------------------------+ > | ok | summary | > +-------+----------------------------------+ > | true | exec.hashagg.mem_limit updated. | > +-------+----------------------------------+ > 1 row selected (0.089 seconds) > 0: jdbc:drill:zk=local> > 0: jdbc:drill:zk=local> SELECT c_customer_id FROM > dfs.`/data/tpcds/sf1/parquet/customer` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT ca_address_id FROM > dfs.`/data/tpcds/sf1/parquet/customer_address` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT cd_credit_rating FROM > dfs.`/data/tpcds/sf1/parquet/customer_demographics` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT hd_buy_potential FROM > dfs.`/data/tpcds/sf1/parquet/household_demographics` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT i_item_id FROM > dfs.`/data/tpcds/sf1/parquet/item` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT p_promo_id FROM > dfs.`/data/tpcds/sf1/parquet/promotion` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT t_time_id FROM > dfs.`/data/tpcds/sf1/parquet/time_dim` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT d_date_id FROM > dfs.`/data/tpcds/sf1/parquet/date_dim` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT s_store_id FROM > dfs.`/data/tpcds/sf1/parquet/store` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT w_warehouse_id FROM > dfs.`/data/tpcds/sf1/parquet/warehouse` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT sm_ship_mode_id FROM > dfs.`/data/tpcds/sf1/parquet/ship_mode` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT r_reason_id FROM > dfs.`/data/tpcds/sf1/parquet/reason` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT cc_call_center_id FROM > dfs.`/data/tpcds/sf1/parquet/call_center` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT web_site_id FROM > dfs.`/data/tpcds/sf1/parquet/web_site` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT wp_web_page_id FROM > dfs.`/data/tpcds/sf1/parquet/web_page` > . . . . . . . . . . . > UNION > . . . . . . . . . . . > SELECT cp_catalog_page_id FROM > dfs.`/data/tpcds/sf1/parquet/catalog_page`; > Error: SYSTEM ERROR: IndexOutOfBoundsException: Index: 26474, Size: 7 > Fragment 4:0 > [Error Id: d44e64ea-f474-436e-94b0-61c61eec2227 on 172.30.8.176:31020] > (java.lang.IndexOutOfBoundsException) Index: 26474, Size: 7 > java.util.ArrayList.rangeCheck():653 > java.util.ArrayList.get():429 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.rehash():293 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1300():120 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate.resizeAndRehashIfNeeded():805 > org.apache.drill.exec.physical.impl.common.HashTableTemplate.put():682 > > org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.checkGroupAndAggrValues():1379 > org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.doWork():604 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():273 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229 > org.apache.drill.exec.physical.impl.BaseRootExec.next():103 > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93 > org.apache.drill.exec.physical.impl.BaseRootExec.next():93 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 > org.apache.hadoop.security.UserGroupInformation.doAs():1657 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():281 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1142 > java.util.concurrent.ThreadPoolExecutor$Worker.run():617 > java.lang.Thread.run():745 (state=,code=0) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)