[jira] [Commented] (HIVE-26902) Failed to close AbstractFileMergeOperator
[ https://issues.apache.org/jira/browse/HIVE-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17657099#comment-17657099 ] MK commented on HIVE-26902: --- same problem, any suggestion to solve this issue? [~zhenkuan_zhang] > Failed to close AbstractFileMergeOperator > - > > Key: HIVE-26902 > URL: https://issues.apache.org/jira/browse/HIVE-26902 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 3.1.2 > Environment: hadoop：3.2.1 > hive：3.1.2 > spark：2.4.6 > hive on spark >Reporter: zhenkuan_zhang >Priority: Major > > when i set hive.merge.sparkfiles to true.Sometimes an error is reported when > SQL is running。The error log is as follows： > org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close > AbstractFileMergeOperator > at > org.apache.hadoop.hive.ql.exec.spark.SparkMergeFileRecordHandler.close(SparkMergeFileRecordHandler.java:115) > at > org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58) > at > org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:96) > at > scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at > org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127) > at > org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127) > at > org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:2212) > at > org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:2212) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > at org.apache.spark.scheduler.Task.run(Task.scala:123) > at > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close > AbstractFileMergeOperator > at > org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315) > at > org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265) > at > org.apache.hadoop.hive.ql.exec.spark.SparkMergeFileRecordHandler.close(SparkMergeFileRecordHandler.java:113) > ... 17 more > Caused by: java.io.IOException: Unable to rename > hdfs://olapCluster/user/hive/warehouse/bi_dw.db/kpy_sfc_fyd_parts_d74_hour_temp/.hive-staging_hive_2023-01-03_13-15-16_144_4347904191947316325-50073/_task_tmp.-ext-1/_tmp.03_0 > to > hdfs://olapCluster/user/hive/warehouse/bi_dw.db/sfc__temp/.hive-staging_hive_2023-01-03_13-15-16_144_4347904191947316325-50073/_tmp.-ext-1/03_0 > at > org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254) > ... 19 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26902) Failed to close AbstractFileMergeOperator
[ https://issues.apache.org/jira/browse/HIVE-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653882#comment-17653882 ] Rajesh Balamohan commented on HIVE-26902: - Hive on spark is no longer supported. Highly unlikely this ticket may get fixed. > Failed to close AbstractFileMergeOperator > - > > Key: HIVE-26902 > URL: https://issues.apache.org/jira/browse/HIVE-26902 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 3.1.2 > Environment: hadoop：3.2.1 > hive：3.1.2 > spark：2.4.6 > hive on spark >Reporter: zhenkuan_zhang >Priority: Major > > when i set hive.merge.sparkfiles to true.Sometimes an error is reported when > SQL is running。The error log is as follows： > org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close > AbstractFileMergeOperator > at > org.apache.hadoop.hive.ql.exec.spark.SparkMergeFileRecordHandler.close(SparkMergeFileRecordHandler.java:115) > at > org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58) > at > org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:96) > at > scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at > org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127) > at > org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$12.apply(AsyncRDDActions.scala:127) > at > org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:2212) > at > org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:2212) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > at org.apache.spark.scheduler.Task.run(Task.scala:123) > at > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close > AbstractFileMergeOperator > at > org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315) > at > org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265) > at > org.apache.hadoop.hive.ql.exec.spark.SparkMergeFileRecordHandler.close(SparkMergeFileRecordHandler.java:113) > ... 17 more > Caused by: java.io.IOException: Unable to rename > hdfs://olapCluster/user/hive/warehouse/bi_dw.db/kpy_sfc_fyd_parts_d74_hour_temp/.hive-staging_hive_2023-01-03_13-15-16_144_4347904191947316325-50073/_task_tmp.-ext-1/_tmp.03_0 > to > hdfs://olapCluster/user/hive/warehouse/bi_dw.db/sfc__temp/.hive-staging_hive_2023-01-03_13-15-16_144_4347904191947316325-50073/_tmp.-ext-1/03_0 > at > org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254) > ... 19 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
