[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283589#comment-15283589 ] Sergio Peña commented on HIVE-13743: [~thejas] is right. The destination path that does not exist might be encrypted (when created) by its parent directory if the parent is in an encryption zone. May we try to check if the destination parent directory in case the destination path does not exist? > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.1.0 >Reporter: Rajesh Balamohan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13743.2.patch, HIVE-13743.patch > > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283308#comment-15283308 ] Thejas M Nair commented on HIVE-13743: -- +1 to this patch. [~spena] what are your thoughts on the overall existing logic of this function and follow up jira ? > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.1.0 >Reporter: Rajesh Balamohan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13743.2.patch, HIVE-13743.patch > > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283050#comment-15283050 ] Thejas M Nair commented on HIVE-13743: -- I haven't gone very deep into the logic, but based on a quick look- The current logic seems broken, if the destination file/directory does not exist, it does not mean that the destination file's zone is not encrypted. What are your thoughts [~spena] ? The new patch from [~ashutoshc] doesn't fix that issue, but it should get it working with Hadoop 2.8. We could get this in and follow up with another patch to address what seems to be an issue with current logic. Comments regarding the patch - # Can you also include the exception in the log message ? # The indentation of return false; seems to be off > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13743.patch > > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282397#comment-15282397 ] Ashutosh Chauhan commented on HIVE-13743: - Thanks [~rajesh.balamohan] for verification. [~spena] can you take a quick look at the patch? > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13743.patch > > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282299#comment-15282299 ] Rajesh Balamohan commented on HIVE-13743: - [~ashutoshc] - Checked the patch in Hadoop 2.8 cluster and patch works as expected. No longer seeing this issue. > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13743.patch > > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281238#comment-15281238 ] Rajesh Balamohan commented on HIVE-13743: - Tried a dirty hack by removing "HdfsEncryptionShim" check in needToCopy (which would get rid of the FileNotFoundException) and it went through without issues. > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281237#comment-15281237 ] Rajesh Balamohan commented on HIVE-13743: - hive - 2.1.0-snapshot tez - 0.8.4 hadoop - 2.8.0-snapshot (built on may-10) {noformat} hive> create table test(id int) stored as orc; OK Time taken: 0.091 seconds hive> insert into test values(10); Query ID = rajesh_20160512060245_a6f56633-fd78-4c8b-81a5-863f80a161e2 Total jobs = 1 Launching Job 1 out of 1 -- VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -- Map 1 .. container SUCCEEDED 1 100 0 0 -- VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 0.08 s -- Status: DAG finished successfully in 0.08 seconds ... Loading data to table rajesh.test Table rajesh.test stats: [numFiles=1, numRows=1, totalSize=182, rawDataSize=4] OK Time taken: 0.839 seconds ... ... .. hive> create table test_1 as select * from test; Query ID = rajesh_20160512060301_9233d9d1-e964-4e94-935a-ac778392d031 Total jobs = 1 Launching Job 1 out of 1 -- VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -- Map 1 .. container SUCCEEDED 1 100 0 0 -- VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 0.05 s -- Status: DAG finished successfully in 0.05 seconds Moving data to directory hdfs://xyz:8020/apps/hive/warehouse/rajesh.db/test_1 Failed with exception Unable to move source hdfs://xyz:8020/apps/hive/warehouse/rajesh.db/.hive-staging_hive_2016-05-12_06-03-01_281_5180494512279326104-1/-ext-10002 to destination hdfs://xyz:8020/apps/hive/warehouse/rajesh.db/test_1 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://xyz:8020/apps/hive/warehouse/rajesh.db/.hive-staging_hive_2016-05-12_06-03-01_281_5180494512279326104-1/-ext-10002 to destination hdfs://xyz:8020/apps/hive/warehouse/rajesh.db/test_1 {noformat} Notice "create table test_1 as select * from test;" fails in the move task. > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) >
[jira] [Commented] (HIVE-13743) Data move codepath is broken with hive (2.1.0-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HIVE-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281126#comment-15281126 ] Ashutosh Chauhan commented on HIVE-13743: - [~rajesh.balamohan] Do you have small repro for this? > Data move codepath is broken with hive (2.1.0-SNAPSHOT) > --- > > Key: HIVE-13743 > URL: https://issues.apache.org/jira/browse/HIVE-13743 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan > > Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop > 2.8.0-snapshot. > {noformat} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path > not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > at > org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) > at org.apache.hadoop.ipc.Client.call(Client.java:1448) > at org.apache.hadoop.ipc.Client.call(Client.java:1385) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy30.getEZForPath(Unknown > Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ > ... > ... > ... > 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to > move source > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 > to destination > hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1.db/date_dim1 > {noformat} > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2836 > hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing > FileNotFoundException as the destf is not present yet. This causes moveFile > to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)