[ https://issues.apache.org/jira/browse/HIVE-26128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-26128 started by Ádám Szita. ----------------------------------------- > Enabling dynamic runtime filtering in iceberg tables throws exception at > runtime > -------------------------------------------------------------------------------- > > Key: HIVE-26128 > URL: https://issues.apache.org/jira/browse/HIVE-26128 > Project: Hive > Issue Type: Bug > Reporter: Rajesh Balamohan > Assignee: Ádám Szita > Priority: Major > Labels: stability > > E.g TPCDS Q2 at 10 TB scale throws the following error when run with > "hive.disable.unsafe.external.table.operations=false". Iceberg tables were > created as external tables and setting > "hive.disable.unsafe.external.table.operations=false" will enable it to have > dynamic runtime filtering; but throws the following error at runtime > {noformat} > ]Vertex failed, vertexName=Map 6, vertexId=vertex_1649658279052_0000_1_03, > diagnostics=[Vertex vertex_1649658279052_0000_1_03 [Map 6] killed/failed due > to:ROOT_INPUT_INIT_FAILURE, Vertex Input: date_dim initializer failed, > vertex=vertex_1649658279052_0000_1_03 [Map 6], > java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 > at java.util.ArrayList.rangeCheck(ArrayList.java:659) > at java.util.ArrayList.get(ArrayList.java:435) > at > org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translateLeaf(HiveIcebergFilterFactory.java:114) > at > org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:86) > at > org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:80) > at > org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.generateFilterExpression(HiveIcebergFilterFactory.java:59) > at > org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getSplits(HiveIcebergInputFormat.java:92) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:592) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:900) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:274) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:192) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:173) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:167) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:750) > ]Vertex killed, vertexName=Map 13, vertexId=vertex_1649658279052_0000_1_07, > diagnostics=[Vertex received Kill in INITED state., Vertex > vertex_1649658279052_0000_1_07 [Map 13] killed/failed due > to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 10, > vertexId=vertex_1649658279052_0000_1_06, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_06 [Map 10] killed/failed > due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 5, > vertexId=vertex_1649658279052_0000_1_04, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_04 [Map 5] killed/failed > due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 4, > vertexId=vertex_1649658279052_0000_1_11, diagnostics=[Vertex received Kill in > NEW state., Vertex vertex_1649658279052_0000_1_11 [Reducer 4] killed/failed > due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3, > vertexId=vertex_1649658279052_0000_1_10, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_10 [Reducer 3] > killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer > 12, vertexId=vertex_1649658279052_0000_1_09, diagnostics=[Vertex received > Kill in INITED state., Vertex vertex_1649658279052_0000_1_09 [Reducer 12] > killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 1, > vertexId=vertex_1649658279052_0000_1_08, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_08 [Map 1] killed/failed > due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 8, > vertexId=vertex_1649658279052_0000_1_02, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_02 [Reducer 8] > killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer > 9, vertexId=vertex_1649658279052_0000_1_01, diagnostics=[Vertex received Kill > in INITED state., Vertex vertex_1649658279052_0000_1_01 [Reducer 9] > killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 7, > vertexId=vertex_1649658279052_0000_1_00, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1649658279052_0000_1_00 [Map 7] killed/failed > due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. > failedVertices:2 killedVertices:10 (state=08S01,code=2) > {noformat} -- This message was sent by Atlassian Jira (v8.20.7#820007)