[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332848#comment-16332848 ] Wei-Chiu Chuang commented on HDFS-13040: Attach my initial patch [^HDFS-13040.001.patch] . Also an inotify example client made by [~r1pp3rj4ck] and [~pifta] [^TransactionReader.java] Usage: (build) javac -cp `hadoop classpath`:. TransactionReader.java (run) java -cp `hadoop classpath`:. TransactionReader h...@example.com hdfs.keytab > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > Credit:
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334397#comment-16334397 ] Daryn Sharp commented on HDFS-13040: I don’t think this looks right. At a minimum, the short user should not be used or it may cause issues in a multi-realm environment. Using a proxy user also means you are allowing the client privs to directly access the external service used by the NN which may not be intended. The client's credentials are clearly valid or it couldn’t get a RemoteException from the NN. The client doesn’t transmit credentials, so the server-side doAs context cannot authenticate to any external service. For it ever to work, which it shouldn’t, there is a bug causing implicit fallback to the login user’s credentials. Whether the client principal matches the NN’s principal should be irrevelant. The underlying problem needs to be more concisely stated. Let's start with the exception the NN receives that triggers the long-winded RemoteException to the client. Presumably the edit log is a REST based? Probably using the buggy AuthenticatedURL? BTW, you can actually use minikdc to login 2 distinct principals for a test. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos pri
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335359#comment-16335359 ] Istvan Fajth commented on HDFS-13040: - Hi [~jojochuang], let me add as well a quick comment on the patch, it seems that you have added the proxyUser usage to the getCurrentEditLogTxid() method. In the DFSInotifyEventInputStream there are two RPC calls, one is to the getCurrentEditLogTxid(), and an other to getEditsFromTxid(), I believe the problem can arise with any one of those, however the current call sequence is implying that if the getCurrentEditLogTxid succeeds, then getEditsFromTxid can succeed unless the ticket experies exactly between the two calls. [~daryn], sorry for the long comment, but let me add what I know, and think important at this point, I am happy to share more if there are any questions. I have tried to set up a test as well with MiniKDC, MiniJournalCluster, and MiniDFSCluster, I had struggles when I wanted to login with a new keytab, and call doAs with the new UGI, as the UGI contained as well traces of the principal that was used by the cluster. What I see: When I do a UserGroupInformation.reset() after cluster startup, this code in the JUnit test: {code:java} UserGroupInformation ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hdfs", generalHDFSKeytabFile.getAbsolutePath()); LOG.info("global: "+UserGroupInformation.getCurrentUser() + ", login as:" + UserGroupInformation.getLoginUser()); LOG.info("ugi: "+ugi.getUserName() + " authenticated via " + ugi.getAuthenticationMethod()); ugi.doAs(new PrivilegedExceptionAction() { @Override public Void run() throws Exception { LOG.info("Current user in doAs: "+UserGroupInformation.getCurrentUser()+" loginuser is:"+UserGroupInformation.getLoginUser()); fs.mkdirs(new Path("/user/test")); fs.listFiles(new Path("/"), true); LOG.info("ListFiles success"); return null; } }); {code} produces the following log: {code:java} 018-01-23 04:08:45,689 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2346)) - Cluster is active 2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream (TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(534)) - global: pifta (auth:KERBEROS), login as:pifta (auth:KERBEROS) 2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream (TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(535)) - ugi: h...@example.com authenticated via KERBEROS 2018-01-23 04:08:45,715 INFO hdfs.TestDFSInotifyEventInputStream (TestDFSInotifyEventInputStream.java:run(539)) - Current user in doAs: h...@example.com (auth:KERBEROS) loginuser is:pifta (auth:KERBEROS) 2018-01-23 04:08:45,722 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1923)) - PriviledgedActionException as:pifta (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-01-23 04:08:45,723 WARN ipc.Client (Client.java:run(713)) - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-01-23 04:08:45,723 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1923)) - PriviledgedActionException as:pifta (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2018-01-23 04:08:45,728 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1923)) - PriviledgedActionException as:h...@example.com (auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "host.local/192.168.0.122"; destination host is: "localhost":58506; 2018-01-23 04:08:45,728 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster {code} So this time I do have a loginUser called pifta that does not have any authentication or tgt it is the system user I use. When I don't reset the ugi, the same code provides the following log: {code:java} 2018-01-23 04:17:49,936 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2346)) - Cluster is active 2018-01-23 04:17:49,953 INFO hdfs.TestDFSInotifyEventInputStream (TestDFSInotifyEventInputStream.java:testWithKerberizedCluster(534)) - global: hdfs/localh...@example.com (auth:KERBEROS), login as:hdfs/localh...@examp
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335965#comment-16335965 ] Wei-Chiu Chuang commented on HDFS-13040: Thanks Daryn your opinion is valuable. The multi-realm is a good point. I haven't thought about that. On the other hand, I feel the concern over privilege is a little exaggerated. inotify requests are authenticated and authorized by NameNode FWIW. Maybe I don't quite understand your comments. Please allow me a little more time to go through AuthenticatedURL and other implementation to get back to you. Thanks. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359236#comment-16359236 ] Xiao Chen commented on HDFS-13040: -- Hey [~daryn], Wei-Chiu is out and I worked with Istvan on this one. (Thanks again for having a reproducing env [~pifta]!) The stacktrace is below: {noformat} 2018-02-09 17:59:55,779 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception initializing http://IP:8480/getJournal?jid=nameservice1&segmentTxId=230876&storageInfo=-60%3A1546104427%3A0%3Acluster2 java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1944) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:477) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:471) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1716) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1778) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1944) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:338) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:206) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470) ... 30 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.K
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359238#comment-16359238 ] Xiao Chen commented on HDFS-13040: -- Side note: I never really understand the {{JDK performed authentication on our behalf.}} behavior, except for a vague concept that there is some thing happening in the jdk that if you already authenticated, the next time it wouldn't trigger a new spengo sequence. Any insights on this appreciated! > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals si
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359308#comment-16359308 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 54s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}197m 43s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSRollback | | | hadoop.hdfs.TestErasureCodingExerciseAPIs | | | hadoop.hdfs.TestSetrepIncreasing | | | hadoop.hdfs.TestHDFSFileSystemContract | | | hadoop.hdfs.security.TestDelegationTokenForProxyUser | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot | | | hadoop.hdfs.TestDecommission | | | hadoop.hdfs.TestSeekBug | | | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData | | | hadoop.hdfs.TestWriteReadStripedFile | | | hadoop.hdfs.TestDFSShell | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 | | | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | | | hadoop.hdfs.TestFsShellPermission | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestBalancerBandwidth | | | hadoop.hdfs.TestDFSClientRetries | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/129100
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361857#comment-16361857 ] Xiao Chen commented on HDFS-13040: -- Worked on the test based on [~pifta]'s original. I think one thing that worths clarifying with Istvan is that, in the test the UGI is just the setting up the client, for doing an NN RPC. So if I understand correctly, what you observed from tests are all RPC behavior. The bug here is after the client authenticates with NN, when the NN is doing its part to get the edits see below After some debugging, it seems this needs extra efforts to make a real test that lets NN read the edits via {{EditLogFileInputStream$URLLog}}. Currently {{DFSInotifyEventInputStream#poll}} would end up making the NN go the {{EditLogFileInputStream$FileLog}}, hence not triggering the authentication path reported. On the other hand, we can really unit test the stream, since we do need the context of a NN RPC, and the context of NN loginuser. Will continue working on this when I can find cycles [~daryn] / [~kihwal], Does the fix itself make sense to you? > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin wi
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362045#comment-16362045 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 48s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 18m 56s{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 14s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 14s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 37s{color} | {color:red} hadoop-auth in trunk failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 40s{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 35s{color} | {color:red} hadoop-hdfs in trunk failed. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 21s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 49s{color} | {color:red} root generated 1235 new + 0 unchanged - 0 fixed = 1235 total (was 0) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 5s{color} | {color:orange} root: The patch generated 7 new + 141 unchanged - 1 fixed = 148 total (was 142) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 82 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 2s{color} | {color:red} The patch 768 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 10s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 42s{color} | {color:red} hadoop-common-project/hadoop-auth generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 4s{color} | {color:green} hadoop-auth in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 18s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}221m 21s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-common-project/hadoo
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363516#comment-16363516 ] Xiao Chen commented on HDFS-13040: -- Patch 3 ready for review. Cleaned up Istvan's test case, and verified the test fails-before, passes-after. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A possible workaround is for the inotify client to use the active NameNode's > server pr
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363687#comment-16363687 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 55s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 6s{color} | {color:red} root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was 1234) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 1s{color} | {color:green} root: The patch generated 0 new + 150 unchanged - 2 fixed = 150 total (was 152) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 35s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 4s{color} | {color:green} hadoop-auth in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}234m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.log.TestLogLevel | | | hadoop.http.TestHttpServerWithSpengo | | | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | | | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | h
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363771#comment-16363771 ] Istvan Fajth commented on HDFS-13040: - Hi [~xiaochen], I have checked into the proposal, a few questions/suggestions on the test code if I may: - in the test code's initKerberizedCluster method you set the log level for KerberosAuthenticator, and UGI to DEBUG, I am not sure if it is necessary, and if you decide to remove that you don't need to make the LOG public in those classes. - also in the initKerberizedCluster method, the basedir is set based on the TestDFSInotifyEventInputStream class, which might remain there due to my mistake, when I put the test into a separate class to upload just this test to this JIRA. - After the Thread.sleep in the test code's testWithKerberizedCluster method there is a log.error that I believe as well remained there from my code, and I believe it should not be error level - Class doc says: Class for Kerberized test cases for \{@link TestDFSInotifyEventInputStream} should this link to just DFSInotifyEventInputStream instead of the test class? > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing w
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364764#comment-16364764 ] Daryn Sharp commented on HDFS-13040: Let me catch up. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A possible workaround is for the inotify client to use the active NameNode's > server principal. However, that's not going to work when there's a namenode > failover, becaus
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364893#comment-16364893 ] Daryn Sharp commented on HDFS-13040: So many points to cover. You're on the right path. *SPNEGO* "JDK performed authentication on our behalf". The JDK will always transparently do spnego by internally handling the 401 and reissuing the request with a TGS. The client never sees a 401. It may see 403 if authentication failed. If a client does see 401, spnego wasn't possible because of no tgt, no tgs available, etc. The KerberosAuthenticator is going to try spnego again, and virtually guaranteed to fail for the same reason. Pretty much what you are seeing. *Sample Code: TransactionReader* I cringe when I see "secure" code like this because it makes people think security is too hard. All of the ugi conf/relogin/doas is unnecessary. Seriously, just rip it out. Pretend like you are writing what you think is insecure code. Enable security in core-site, kinit before you run it... That's it. Or if you prefer, you can leave in just this one line: UserGroupInformation.loginUserFromKeytab(princ, keytab); *Issues* {quote}At NameNode startup, the NameNode acquires a tgt, and it is saved in its ticket cache {quote} I suspected this might be the case. Please don't start your NN from a ticket cache, let alone share that ticket cache with user tools. Set the confs for keytab and principal. Forget all the expired ticket cache stuff. It's a distraction and self-inflicted pain. Let's remove some confusion and describe your NN as running as "hdfs/nn1@REALM", and you are using "superuser@REALM" to run inotify. I know you are using hdfs but it'll be clear soon. As a client, you have credentials for "superuser@REALM". After authentication to the NN's RPC server it creates a "superuser" ugi, but it has no credentials, just an identity. The inotify method calls an edit log method that makes an external rest call which needs credentials. But "superuser" has no creds, boom, big ugly gssapi stack trace. With the proposed patch, an explict doAs the login user flips you back to "hdfs/nn1@REALM" which does have credentials. It's what you want, but not correctly implemented. The root issue is a client must have an immutable identity determined when created. The URLLog ctor should save off the current user and use that for the doAs. Since the login user creates the edit log, it means any use of it, regardless of whether from rpc, will retain that identity. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.Name
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367694#comment-16367694 ] Xiao Chen commented on HDFS-13040: -- Thanks Daryn for the comments. It's helpful and ... rhythmic... Probably the fist time I laughed when seeing the gssapi stacktrace. :) {quote}It's what you want, but not correctly implemented. {quote} I agree the proposed approach of caching the creator UGI would be optimal. But from current code base, we actually need to cache this at a way upper level. Because from the stack trace above, every {{getEditsFromTxid}} creates a new {{EditLogFileInputStream}} object during {{[selectInputStreams|https://github.com/apache/hadoop/blob/8d5ea7470a3225319e3bef5626b837572c2e0d3c/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L509]}}. Since we want to cache the NN login user UGI, it seems we'll need to cache this somewhere outside, like in FSEditLog or the JournalSet. I fear that's more complexity than benefit to us. So this patch still kept the current logic of using the login user - thinking from NN's perspective, this should always be the logged in NN user regardless of where the call is from. Let me know if there is any gaps in my understanding. Addressed [~pifta]'s test comments, thanks for the review[^HDFS-13040.04.patch]. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.se
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367695#comment-16367695 ] Xiao Chen commented on HDFS-13040: -- ... for some reason Jira doesn't let me reference to it. "the stack trace above" means https://issues.apache.org/jira/browse/HDFS-13040?focusedCommentId=16359236&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16359236 > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367865#comment-16367865 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 7s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 30 unchanged - 1 fixed = 30 total (was 31) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 18s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 52s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}108m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized | | | hadoop.hdfs.TestErasureCodingPolicies | | | hadoop.hdfs.TestDatanodeReport | | | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication | | | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | | | hadoop.hdfs.TestPipelines | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910961/HDFS-13040.04.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 5343cd1a29b1 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4c2119f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDF
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369326#comment-16369326 ] Daryn Sharp commented on HDFS-13040: {quote}So this patch still kept the current logic of using the login user - thinking from NN's perspective, this should always be the logged in NN user regardless of where the call is from. Let me know if there is any gaps in my understanding. {quote} Agreed. Yes, it _should_ effectively be the login user, but the current user is more correct in the event a different identity is ever needed to communicate with the remote edit log services. I don't feel strongly though. You might consider floating the doAs login user up to {{getEditsFromTxid}} when it calls {{log.selectInputStreams}}. Advantage is it will "just work" if the underlying implementation ever needs make to additional REST calls, else you'll need to wrap those calls too. Not a big deal, I don't use/care about QJM, so it's just advice. –– The test isn't actually testing what it purports, and it's expecting the wrong behavior. Ie. "// verify we can poll after client tgt expired". The test uses already created client within a different doAs context with the presumption that it executes as the doAs user. It doesn't. The client retains the identity upon creation so it's testing the default/login user can make calls. It's also using an already open rpc connection, else you would see an authentication failure (correct behavior, not null). Set the rpc timeout to a low value to force the connection to close and the poll will fail. Could the unit test just explicitly set the conf keys, rather than create a new specialized builder method in mini QJM cluster? > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apac
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370999#comment-16370999 ] Xiao Chen commented on HDFS-13040: -- Thanks for the review Daryn. Patch 5 attached to address the comments except the 'current user' one. I agree it's the most correct thing to do, but maybe we can leave it out to a future jira. {quote} floating the doAs login user up to {{getEditsFromTxid}} {quote} Good idea, done this way and left the stream class untouched. {quote}Could the unit test just explicitly set the conf keys {quote} Not really, because the journal part of the QJMHA cluster needs to be started first for us to know the correct journal URI, so we can't know the uri beforehand. {{initHAConf}} currently sets the shared edits dir key, presumably for the same reason. {quote}the test {quote} Good catch, and helpful explanations. Addressed by using the correct UGIs. hdfs@ is the client, and hdfs/localhost@ is the NN user. Verified I can see the big beautiful gssapi stack trace without the fix. 1 odd thing I found in the test though is I had to set the proxy users for it to work, otherwise the mkdirs after the relogin would throw {quote}AuthorizationException): User: hdfs/localh...@example.com is not allowed to impersonate h...@example.com {quote} at me. Debugging this, it seems to be a designed rpc server auth behavior from this [code|https://github.com/apache/hadoop/blob/121e1e1280c7b019f6d2cc3ba9eae1ead0dd8408/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L2260]. Though my debugging shows the {{protocolUser}} is {{hdfs@ (auth:SIMPLE)}}, while the {{realUser}} is {{hdfs/localhost@ (auth:KERBEROS)}}, still weird > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) >
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371121#comment-16371121 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 11s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 36s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}171m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911340/HDFS-13040.05.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux df4251397187 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 121e1e1 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/23138/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/23138/testReport/ | | Max. process+thread count | 2920 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDF
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373627#comment-16373627 ] Xiao Chen commented on HDFS-13040: -- Verified patch 5 works in Istvan's cluster too. [~daryn] could you please take another look? Thanks much. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A possible workaround is for the inotify cli
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378191#comment-16378191 ] Wei-Chiu Chuang commented on HDFS-13040: Code and test both look good to me except the following: {quote}1 odd thing I found in the test though is I had to set the proxy users for it to work, otherwise the mkdirs after the relogin would throw {quote} This does not make sense to me. I commented out the following code and it does not fail for me: {code:java} // the mkdir after relogin would fail if these are not set conf.set("hadoop.proxyuser.hdfs.users", "hdfs"); conf.set("hadoop.proxyuser.hdfs.hosts", "*");{code} > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383934#comment-16383934 ] Xiao Chen commented on HDFS-13040: -- Thanks for checking [~jojochuang]. I rerun this and it appear to be passing in all 100 runs Guess that was due to some local partial build and ugi myth... Patch 6 to remove those lines. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos prin
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384033#comment-16384033 ] Wei-Chiu Chuang commented on HDFS-13040: Thanks for updating the patch. A few more follow-up cosmetic comments: 1. the code in NameNodeRpcServer could be cleaner if you move them into a new method, and make getEditsFromTxid() a wrapper around the new method. 2. Test: It seems to me you can annotate shutdownCluster() with \{{@After}} and then you don't need the try .. finally logic in testWithKerberizedCluster(). 3. Suggest adding comments to explain behind setting client connection timeout. 4. \{{initKerberizedCluster()}}: I suspect similar utility methods exist in the codebase already. I couldn't find them though. If there are not, would it make sense to make this method a uility method that can be shared? > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses th
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384172#comment-16384172 ] genericqa commented on HDFS-13040: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 6s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}113m 23s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}167m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912807/HDFS-13040.06.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b7c97009a7a5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 83798f1 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/23272/testReport/ | | Max. process+thread count | 3201 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/23272/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Kerberized inotify client fails despite kinit properly > --
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384185#comment-16384185 ] Xiao Chen commented on HDFS-13040: -- Thanks for the review Wei-Chiu. Patch 7 addressed 1-3. I searched for {{// Windows will not reverse name lookup "127.0.0.1" to "localhost".}} in trunk and found 6 occurrences. They look similar but I have not scrutinized all. I think a util of creating kdc with a provided configuration should be doable, but would like to limit the scope here and split it out to another Jira if you're ok. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to w
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384278#comment-16384278 ] Wei-Chiu Chuang commented on HDFS-13040: LGTM +1 pending Jenkins. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A possible workaround is for the inotify client to use the active NameNode's
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384371#comment-16384371 ] genericqa commented on HDFS-13040: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 1s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 18s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 39s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}179m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.qjournal.server.TestJournalNode | | | hadoop.hdfs.server.namenode.ha.TestHAAppend | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | | | hadoop.hdfs.TestRollingUpgrade | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f | | JIRA Issue | HDFS-13040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912828/HDFS-13040.07.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e9755abee354 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 60080fb | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/23276/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/jo
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384375#comment-16384375 ] Xiao Chen commented on HDFS-13040: -- Failed tests look unrelated (mostly oom), committing this. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A possible workaround is for the inotify client to use t
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384380#comment-16384380 ] Xiao Chen commented on HDFS-13040: -- Committed to trunk, branch-3.1 and branch-3.0. Branch-2 patch attached for precommit. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.branch-2.01.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneous
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384387#comment-16384387 ] Hudson commented on HDFS-13040: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13762 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13762/]) HDFS-13040. Kerberized inotify client fails despite kinit properly. (xiao: rev c75105f07b4cdbc2773435fc1125446233113c15) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStreamKerberized.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniQJMHACluster.java > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.half.test.patch, > TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, Name
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384391#comment-16384391 ] Xiao Chen commented on HDFS-13040: -- Besides the backport conflict and the java 7/8 conflict, branch-2's minikdc is apacheds based. branch-3 uses kerby. So without a way to specify min lifetime to be lower, the branch-2 patch does not have the test, similar to HADOOP-12559 before. > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.branch-2.01.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit t
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385946#comment-16385946 ] Wei-Chiu Chuang commented on HDFS-13040: +1 branch-2 patch. Thanks! > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.branch-2.01.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has been verified to work for a > CDH5.10.2 cluster, however it seems impossible to craft a unit test for this > fix because the way Hadoop UGI handles Kerberos credentials (I can't have a > single process that logins as two Kerberos principals simultaneously and let > them establish connection) > A pos
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386425#comment-16386425 ] Xiao Chen commented on HDFS-13040: -- I have triggered a few pre-commits but none can complete within timeout. Latest one is [https://builds.apache.org/job/PreCommit-HDFS-Build/23290https://builds.apache.org/job/PreCommit-HDFS-Build/23290] Manually did a {{mvn clean test \-Dtest=TestDFSInotifyEventInputStream,TestNNWithQJM}}, passed. Given Wei-Chiu's +1, I committed this to branch-2. Thanks for the reviews [~jojochuang] [~daryn], and [~pifta] for the reproduction and tests! > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.branch-2.01.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that Name
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[ https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400979#comment-16400979 ] Xiao Chen commented on HDFS-13040: -- For reference [the test error with UGI mentioned above|https://issues.apache.org/jira/browse/HDFS-13040?focusedCommentId=16378191&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16378191] is HADOOP-14699, fixed by HADOOP-9747. Not clear why branch-2 didn't fail. Guessing UGI behavior changed :) > Kerberized inotify client fails despite kinit properly > -- > > Key: HDFS-13040 > URL: https://issues.apache.org/jira/browse/HDFS-13040 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 > Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2 >Reporter: Wei-Chiu Chuang >Assignee: Xiao Chen >Priority: Major > Fix For: 3.1.0, 2.10.0, 3.0.2 > > Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch, > HDFS-13040.03.patch, HDFS-13040.04.patch, HDFS-13040.05.patch, > HDFS-13040.06.patch, HDFS-13040.07.patch, HDFS-13040.branch-2.01.patch, > HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java, > TransactionReader.java > > > This issue is similar to HDFS-10799. > HDFS-10799 turned out to be a client side issue where client is responsible > for renewing kerberos ticket actively. > However we found in a slightly setup even if client has valid Kerberos > credentials, inotify still fails. > Suppose client uses principal h...@example.com, > namenode 1 uses server principal hdfs/nn1.example@example.com > namenode 2 uses server principal hdfs/nn2.example@example.com > *After Namenodes starts for longer than kerberos ticket lifetime*, the client > fails with the following error: > {noformat} > 18/01/19 11:23:02 WARN security.UserGroupInformation: > PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS) > cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We > encountered an error reading > https://nn2.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3, > > https://nn1.example.com:8481/getJournal?jid=ns1&segmentTxId=8662&storageInfo=-60%3A353531113%3A0%3Acluster3. > During automatic edit log failover, we noticed that all of the remaining > edit log streams are shorter than the current one! The best remaining edit > log ends at transaction 8683, but we thought we could read up to transaction > 8684. If you continue, metadata will be lost forever! > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210) > {noformat} > Typically if NameNode has an expired Kerberos ticket, the error handling for > the typical edit log tailing would let NameNode to relogin with its own > Kerberos principal. However, when inotify uses the same code path to retrieve > edits, since the current user is the inotify client's principal, unless > client uses the same principal as the NameNode, NameNode can't do it on > behalf of the client. > Therefore, a more appropriate approach is to use proxy user so that NameNode > can retrieving edits on behalf of the client. > I will attach a patch to fix it. This patch has bee