[jira] [Resolved] (HDDS-2914) Certain Hive queries started to fail on generating splits
[
https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan resolved HDDS-2914.
-
Resolution: Fixed
Thank you for fixing the issue [~pifta]. I have merged the PR.
> Certain Hive queries started to fail on generating splits
> -
>
> Key: HDDS-2914
> URL: https://issues.apache.org/jira/browse/HDDS-2914
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After updating a cluster where I am running TPCDS queries, some queries
> started to fail.
> The update happened from an early dec state to the jan 10 state of master.
> Most likely the addition of HDDS-2188 is related to the problem, but it is
> still under investigation.
> The exception I see in the queries:
> {code}
> [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input:
> inventory initializer failed, vertex=vertex_ [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> ... 5 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(
[jira] [Resolved] (HDDS-2914) Certain Hive queries started to fail on generating splits
[
https://issues.apache.org/jira/browse/HDDS-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham resolved HDDS-2914.
--
Fix Version/s: 0.5.0
Resolution: Fixed
> Certain Hive queries started to fail on generating splits
> -
>
> Key: HDDS-2914
> URL: https://issues.apache.org/jira/browse/HDDS-2914
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Aravindan Vijayan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After updating a cluster where I am running TPCDS queries, some queries
> started to fail.
> The update happened from an early dec state to the jan 10 state of master.
> Most likely the addition of HDDS-2188 is related to the problem, but it is
> still under investigation.
> The exception I see in the queries:
> {code}
> [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input:
> inventory initializer failed, vertex=vertex_ [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> ... 5 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File
> o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_001_001_/bucket_0
> should have had overlap on block starting at 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> a
