[ https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-2914: --------------------------------- Labels: pull-request-available (was: ) > Certain Hive queries started to fail on generating splits > --------------------------------------------------------- > > Key: HDDS-2914 > URL: https://issues.apache.org/jira/browse/HDDS-2914 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Reporter: Istvan Fajth > Assignee: Aravindan Vijayan > Priority: Critical > Labels: pull-request-available > > After updating a cluster where I am running TPCDS queries, some queries > started to fail. > The update happened from an early dec state to the jan 10 state of master. > Most likely the addition of HDDS-2188 is related to the problem, but it is > still under investigation. > The exception I see in the queries: > {code} > [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: > inventory initializer failed, vertex=vertex_XXXX [Map 1] > org.apache.tez.dag.app.dag.impl.AMUserCodeException: > java.lang.RuntimeException: ORC split generation failed with exception: > java.io.IOException: File > o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 > should have had overlap on block starting at 0 > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456) > at > com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817) > at > com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: ORC split generation failed with > exception: java.io.IOException: File > o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 > should have had overlap on block starting at 0 > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > ... 5 more > Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File > o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 > should have had overlap on block starting at 0 > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909) > ... 17 more > Caused by: java.io.IOException: File > o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 > should have had overlap on block starting at 0 > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ... 3 more > {code} > I haven't found any more specific stack traces so far. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org