[ https://issues.apache.org/jira/browse/HIVE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103046#comment-17103046 ]
philipse commented on HIVE-23279: --------------------------------- is it reproduced? Have u checked the src and destination priviledge, > Unable to move source hdfs://xxx:8020/warehouse/.../.hive-staging_hive_... | > 无法移动 hivestage 文件 > ---------------------------------------------------------------------------------------------- > > Key: HIVE-23279 > URL: https://issues.apache.org/jira/browse/HIVE-23279 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Affects Versions: 2.1.1 > Environment: Centos 7 > > Cloudera Manager Agent 6.2.1 > Cloudera Manager Management Daemon 6.2.1 > Hadoop 3.0.0 cdh6.2.1 > HDFS 3.0.0 cdh6.2.1 > HttpFS 3.0.0 cdh6.2.1 > hadoop-kms 3.0.0 cdh6.2.1 > MapReduce 2 3.0.0 cdh6.2.1 > YARN 3.0.0 cdh6.2.1 > Hive 2.1.1 cdh6.2.1 > HCatalog 2.1.1 cdh6.2.1 > Hue 4.3.0 cdh6.2.1 > Impala 3.2.0 cdh6.2.1 > Java 8 java version "1.8.0_181" Java (TM) SE Runtime Environment (build > 1.8.0_181-b13) Java HotSpot (TM) 64-Bit Server VM (build 25.181-b13, mixed > mode) > Parquet 1.9.0 cdh6.2.1 > spark 2.4.0 cdh6.2.1 > Reporter: geekyouth > Assignee: geekyouth > Priority: Blocker > Labels: HiveServer2, cdh6 > Attachments: chrome_k3C4YK0O5V.png, chrome_qy2E1vTR3j.png > > > {color:#de350b}*error log from Cloudera Manager 6.2.1:;) | cdh 日志:*{color} > > [HiveServer2-Background-Pool: Thread-58]: Error running hive query: > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask. {color:#de350b}Unable to move > source{color} > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000 > to destination > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01 > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329) > ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:258) > ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92) > ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345) > [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181] > at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_181] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > [hadoop-common-3.0.0-cdh6.2.1.jar:?] > at > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357) > [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_181] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_181] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_181] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > {color:#de350b}Unable to move source{color} > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000 > to destination > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01 > at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3445) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3401) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3396) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3693) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1614) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1525) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1489) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:501) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256) > ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > ... 11 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: {color:#de350b}Unable to > move source{color} > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 > to destination > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0 > at > org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:3418) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3363) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3693) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1614) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1525) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1489) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:501) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256) > ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > ... 11 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > {color:#de350b}Unable to move source{color} > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 > to destination > hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0 > at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3445) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3401) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive.access$200(Hive.java:175) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3346) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3331) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > ... 4 more > {color:#de350b}Caused by: java.io.IOException: rename for src path: > hdfs{color}://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 > to dest > path:hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3343) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3331) > ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1] > ... 4 more > > Additional notes: > 补充说明: > > My account for logging in to Cloudera Manager hue is hive, and the / > warehouse directory has been authorized to hive, and I tried to set it up as > follows: > 我登录 Cloudera Manager hue 的账户是 hive ,/warehouse 目录已授权给 hive ,并且尝试设置如下: > ------------------------------------------------------------------------------------hive.warehouse.subdir.inherit.perms=false > hive.exec.scratchdir=/warehouse/.hive-staging > ------------------------------------------------------------------------------------ > or > 或者 > ------------------------------------------------------------------------------------hive.warehouse.subdir.inherit.perms=true > ------------------------------------------------------------------------------------ > None of the above configurations can solve the problem > I have been troubled by this problem for several weeks, > I found that google and stackoverflow.com can't solve the problem > > 以上配置全都无法解决问题 > 我已经被这个问题困扰几周了,我查找了 google 和 stackoverflow.com 都无法解决问题 > > My software version: > 我的软件版本: > Cloudera Manager Agent 6.2.1Cloudera Manager Management Daemon 6.2.1Hadoop > 3.0.0 cdh6.2.1HDFS 3.0.0 cdh6.2.1HttpFS 3.0.0 cdh6.2.1hadoop-kms 3.0.0 > cdh6.2.1MapReduce 2 3.0.0 cdh6.2.1YARN 3.0.0 cdh6.2.1Hive 2.1.1 > cdh6.2.1HCatalog 2.1.1 cdh6.2.1Hue 4.3.0 cdh6.2.1Impala 3.2.0 cdh6.2.1Java 8 > java version "1.8.0_181" Java(TM) SE Runtime Environment (build > 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed > mode)Parquet 1.9.0 cdh6.2.1spark 2.4.0 cdh6.2.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)