Hello, We are using some external tables backed by aws S3. And we are intermittently getting this error, most likely at the last stage of the reduce, I see some similar posts in net but could not find any solution, Is there any way yo solve it:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: s3a://xxx/pn/.hive-staging_hive_2020-01-27_08-05-54_195_618381349208890136-176126/_task_tmp.-ext-10000/year=2019/product=xxx/abc=2019-12-16/xxx=BCD/yz=x/pattern=xxx/_tmp.000001_0 to: s3a://xxx/tables/xxx/xxxx/xxx/xx/xxxx/xx/.hive-staging_hive_2020-01-27_08-05-54_195_6 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:236) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:133) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1014) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) thanks Souvik
