Authentication Failure talking to Ranger KMS

2016-10-11 Thread Benjamin Ross 
All,
I'm trying to use httpfs to write to an encryption zone with security off.  I 
can read from an encryption zone, but I can't write to one.

Here's the applicable namenode logs.  httpfs and root both have all possible 
privileges in the KMS.  What am I missing?


2016-10-07 15:48:16,164 DEBUG ipc.Server 
(Server.java:authorizeConnection(2095)) - Successfully authorized userInfo {
  effectiveUser: "root"
  realUser: "httpfs"
}
protocol: "org.apache.hadoop.hdfs.protocol.ClientProtocol"

2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:processOneRpc(1902)) -  
got #2
2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:run(2179)) - IPC Server 
handler 9 on 8020: org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 
10.41.1.64:47622 Call#2 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
2016-10-07 15:48:16,165 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:root (auth:PROXY) via httpfs (auth:SIMPLE) 
from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(NameNodeRpcServer.java:create(699)) - *DIR* NameNode.create: file 
/tmp/cryptotest/hairyballs for DFSClient_NONMAPREDUCE_-1005188439_28 at 
10.41.1.64
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(FSNamesystem.java:startFileInt(2411)) - DIR* NameSystem.startFile: 
src=/tmp/cryptotest/hairyballs, holder=DFSClient_NONMAPREDUCE_-1005188439_28, 
clientMachine=10.41.1.64, createParent=true, replication=3, createFlag=[CREATE
, OVERWRITE], blockSize=134217728, 
supportedVersions=[CryptoProtocolVersion{description='Encryption zones', 
version=2, unknownValue=null}]
2016-10-07 15:48:16,167 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:hdfs (auth:SIMPLE) 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:484)
2016-10-07 15:48:16,171 DEBUG client.KerberosAuthenticator 
(KerberosAuthenticator.java:authenticate(205)) - Using fallback authenticator 
sequence.
2016-10-07 15:48:16,176 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:hdfs 
(auth:SIMPLE) 
cause:org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, messag
e: Forbidden
2016-10-07 15:48:16,176 DEBUG ipc.Server (ProtobufRpcEngine.java:call(631)) - 
Served: create queueTime= 2 procesingTime= 10 exception= IOException
2016-10-07 15:48:16,177 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:root 
(auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.IOException: 
java.util.concurrent.ExecutionException: java.io.IOException: org.apach
e.hadoop.security.authentication.client.AuthenticationException: Authentication 
failed, status: 403, message: Forbidden
2016-10-07 15:48:16,177 INFO  ipc.Server (Server.java:logException(2299)) - IPC 
Server handler 9 on 8020, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 
Call#2 Retry#0
java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: 
org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, message: Forbidden
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:750)
at 
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.generateEncryptedDataEncryptionKey(FSNamesystem.java:2352)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2478)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2377)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:716)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: 
org.apac

Re: Authentication Failure talking to Ranger KMS

2016-10-11 Thread Wei-Chiu Chuang 
Somes to me you encountered this bug? HDFS-10481 

If you’re using CDH, this is fixed in CDH5.5.5, CDH5.7.2 and CDH5.8.2

Wei-Chiu Chuang
A very happy Clouderan

> On Oct 11, 2016, at 8:38 AM, Benjamin Ross  wrote:
> 
> All,
> I'm trying to use httpfs to write to an encryption zone with security off.  I 
> can read from an encryption zone, but I can't write to one.
> 
> Here's the applicable namenode logs.  httpfs and root both have all possible 
> privileges in the KMS.  What am I missing?
> 
> 
> 2016-10-07 15:48:16,164 DEBUG ipc.Server 
> (Server.java:authorizeConnection(2095)) - Successfully authorized userInfo {
>   effectiveUser: "root"
>   realUser: "httpfs"
> }
> protocol: "org.apache.hadoop.hdfs.protocol.ClientProtocol"
> 
> 2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:processOneRpc(1902)) -  
> got #2
> 2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:run(2179)) - IPC Server 
> handler 9 on 8020: org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 
> 10.41.1.64:47622 Call#2 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
> 2016-10-07 15:48:16,165 DEBUG security.UserGroupInformation 
> (UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
> as:root (auth:PROXY) via httpfs (auth:SIMPLE) 
> from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
> 2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
> (NameNodeRpcServer.java:create(699)) - *DIR* NameNode.create: file 
> /tmp/cryptotest/hairyballs for DFSClient_NONMAPREDUCE_-1005188439_28 at 
> 10.41.1.64
> 2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
> (FSNamesystem.java:startFileInt(2411)) - DIR* NameSystem.startFile: 
> src=/tmp/cryptotest/hairyballs, holder=DFSClient_NONMAPREDUCE_-1005188439_28, 
> clientMachine=10.41.1.64, createParent=true, replication=3, createFlag=[CREATE
> , OVERWRITE], blockSize=134217728, 
> supportedVersions=[CryptoProtocolVersion{description='Encryption zones', 
> version=2, unknownValue=null}]
> 2016-10-07 15:48:16,167 DEBUG security.UserGroupInformation 
> (UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
> as:hdfs (auth:SIMPLE) 
> from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:484)
> 2016-10-07 15:48:16,171 DEBUG client.KerberosAuthenticator 
> (KerberosAuthenticator.java:authenticate(205)) - Using fallback authenticator 
> sequence.
> 2016-10-07 15:48:16,176 DEBUG security.UserGroupInformation 
> (UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:hdfs 
> (auth:SIMPLE) 
> cause:org.apache.hadoop.security.authentication.client.AuthenticationException:
>  Authentication failed, status: 403, messag
> e: Forbidden
> 2016-10-07 15:48:16,176 DEBUG ipc.Server (ProtobufRpcEngine.java:call(631)) - 
> Served: create queueTime= 2 procesingTime= 10 exception= IOException
> 2016-10-07 15:48:16,177 DEBUG security.UserGroupInformation 
> (UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:root 
> (auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: org.apach
> e.hadoop.security.authentication.client.AuthenticationException: 
> Authentication failed, status: 403, message: Forbidden
> 2016-10-07 15:48:16,177 INFO  ipc.Server (Server.java:logException(2299)) - 
> IPC Server handler 9 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 
> Call#2 Retry#0
> java.io.IOException: java.util.concurrent.ExecutionException: 
> java.io.IOException: 
> org.apache.hadoop.security.authentication.client.AuthenticationException: 
> Authentication failed, status: 403, message: Forbidden
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:750)
> at 
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.generateEncryptedDataEncryptionKey(FSNamesystem.java:2352)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2478)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2377)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:716)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211)
> at org.

RE: Authentication Failure talking to Ranger KMS

2016-10-11 Thread Benjamin Ross 
That seems promising.  But shouldn't I be able to work around it by just 
ensuring that httpfs has all necessary privileges in the KMS service under 
Ranger?

Thanks,
Ben



From: Wei-Chiu Chuang [weic...@cloudera.com]
Sent: Tuesday, October 11, 2016 11:57 AM
To: Benjamin Ross
Cc: user@hadoop.apache.org; u...@ranger.incubator.apache.org
Subject: Re: Authentication Failure talking to Ranger KMS

Somes to me you encountered this bug? 
HDFS-10481<https://issues.apache.org/jira/browse/HDFS-10481>
If you’re using CDH, this is fixed in CDH5.5.5, CDH5.7.2 and CDH5.8.2

Wei-Chiu Chuang
A very happy Clouderan

On Oct 11, 2016, at 8:38 AM, Benjamin Ross 
mailto:br...@lattice-engines.com>> wrote:

All,
I'm trying to use httpfs to write to an encryption zone with security off.  I 
can read from an encryption zone, but I can't write to one.

Here's the applicable namenode logs.  httpfs and root both have all possible 
privileges in the KMS.  What am I missing?


2016-10-07 15:48:16,164 DEBUG ipc.Server 
(Server.java:authorizeConnection(2095)) - Successfully authorized userInfo {
  effectiveUser: "root"
  realUser: "httpfs"
}
protocol: "org.apache.hadoop.hdfs.protocol.ClientProtocol"

2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:processOneRpc(1902)) -  
got #2
2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:run(2179)) - IPC Server 
handler 9 on 8020: org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 
10.41.1.64:47622 Call#2 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
2016-10-07 15:48:16,165 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:root (auth:PROXY) via httpfs (auth:SIMPLE) 
from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(NameNodeRpcServer.java:create(699)) - *DIR* NameNode.create: file 
/tmp/cryptotest/hairyballs for DFSClient_NONMAPREDUCE_-1005188439_28 at 
10.41.1.64
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(FSNamesystem.java:startFileInt(2411)) - DIR* NameSystem.startFile: 
src=/tmp/cryptotest/hairyballs, holder=DFSClient_NONMAPREDUCE_-1005188439_28, 
clientMachine=10.41.1.64, createParent=true, replication=3, createFlag=[CREATE
, OVERWRITE], blockSize=134217728, 
supportedVersions=[CryptoProtocolVersion{description='Encryption zones', 
version=2, unknownValue=null}]
2016-10-07 15:48:16,167 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:hdfs (auth:SIMPLE) 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:484)
2016-10-07 15:48:16,171 DEBUG client.KerberosAuthenticator 
(KerberosAuthenticator.java:authenticate(205)) - Using fallback authenticator 
sequence.
2016-10-07 15:48:16,176 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:hdfs 
(auth:SIMPLE) 
cause:org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, messag
e: Forbidden
2016-10-07 15:48:16,176 DEBUG ipc.Server (ProtobufRpcEngine.java:call(631)) - 
Served: create queueTime= 2 procesingTime= 10 exception= IOException
2016-10-07 15:48:16,177 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:root 
(auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.IOException: 
java.util.concurrent.ExecutionException: java.io.IOException: org.apach
e.hadoop.security.authentication.client.AuthenticationException: Authentication 
failed, status: 403, message: Forbidden
2016-10-07 15:48:16,177 INFO  ipc.Server (Server.java:logException(2299)) - IPC 
Server handler 9 on 8020, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 
Call#2 Retry#0
java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: 
org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, message: Forbidden
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:750)
at 
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.generateEncryptedDataEncryptionKey(FSNamesystem.java:2352)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2478)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2377)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:716)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405)
at 
org.apache.hadoop.hdfs.protocol.proto.

RE: Authentication Failure talking to Ranger KMS

2016-10-11 Thread Benjamin Ross 
Just for kicks I tried applying the patch in that ticket and it didn't have any 
effect.  It makes sense because my issue is on CREATE, and the ticket only has 
to do with OPEN.

Note that I don't have these issues using WebHDFS, only using httpfs, so it 
definitely seems like we're on the right track...

Thanks in advance,
Ben




From: Benjamin Ross
Sent: Tuesday, October 11, 2016 12:02 PM
To: Wei-Chiu Chuang
Cc: user@hadoop.apache.org; u...@ranger.incubator.apache.org
Subject: RE: Authentication Failure talking to Ranger KMS

That seems promising.  But shouldn't I be able to work around it by just 
ensuring that httpfs has all necessary privileges in the KMS service under 
Ranger?

Thanks,
Ben



From: Wei-Chiu Chuang [weic...@cloudera.com]
Sent: Tuesday, October 11, 2016 11:57 AM
To: Benjamin Ross
Cc: user@hadoop.apache.org; u...@ranger.incubator.apache.org
Subject: Re: Authentication Failure talking to Ranger KMS

Somes to me you encountered this bug? 
HDFS-10481<https://issues.apache.org/jira/browse/HDFS-10481>
If you’re using CDH, this is fixed in CDH5.5.5, CDH5.7.2 and CDH5.8.2

Wei-Chiu Chuang
A very happy Clouderan

On Oct 11, 2016, at 8:38 AM, Benjamin Ross 
mailto:br...@lattice-engines.com>> wrote:

All,
I'm trying to use httpfs to write to an encryption zone with security off.  I 
can read from an encryption zone, but I can't write to one.

Here's the applicable namenode logs.  httpfs and root both have all possible 
privileges in the KMS.  What am I missing?


2016-10-07 15:48:16,164 DEBUG ipc.Server 
(Server.java:authorizeConnection(2095)) - Successfully authorized userInfo {
  effectiveUser: "root"
  realUser: "httpfs"
}
protocol: "org.apache.hadoop.hdfs.protocol.ClientProtocol"

2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:processOneRpc(1902)) -  
got #2
2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:run(2179)) - IPC Server 
handler 9 on 8020: org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 
10.41.1.64:47622 Call#2 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER
2016-10-07 15:48:16,165 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:root (auth:PROXY) via httpfs (auth:SIMPLE) 
from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205)
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(NameNodeRpcServer.java:create(699)) - *DIR* NameNode.create: file 
/tmp/cryptotest/hairyballs for DFSClient_NONMAPREDUCE_-1005188439_28 at 
10.41.1.64
2016-10-07 15:48:16,166 DEBUG hdfs.StateChange 
(FSNamesystem.java:startFileInt(2411)) - DIR* NameSystem.startFile: 
src=/tmp/cryptotest/hairyballs, holder=DFSClient_NONMAPREDUCE_-1005188439_28, 
clientMachine=10.41.1.64, createParent=true, replication=3, createFlag=[CREATE
, OVERWRITE], blockSize=134217728, 
supportedVersions=[CryptoProtocolVersion{description='Encryption zones', 
version=2, unknownValue=null}]
2016-10-07 15:48:16,167 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction 
as:hdfs (auth:SIMPLE) 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:484)
2016-10-07 15:48:16,171 DEBUG client.KerberosAuthenticator 
(KerberosAuthenticator.java:authenticate(205)) - Using fallback authenticator 
sequence.
2016-10-07 15:48:16,176 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:hdfs 
(auth:SIMPLE) 
cause:org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, messag
e: Forbidden
2016-10-07 15:48:16,176 DEBUG ipc.Server (ProtobufRpcEngine.java:call(631)) - 
Served: create queueTime= 2 procesingTime= 10 exception= IOException
2016-10-07 15:48:16,177 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:root 
(auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.IOException: 
java.util.concurrent.ExecutionException: java.io.IOException: org.apach
e.hadoop.security.authentication.client.AuthenticationException: Authentication 
failed, status: 403, message: Forbidden
2016-10-07 15:48:16,177 INFO  ipc.Server (Server.java:logException(2299)) - IPC 
Server handler 9 on 8020, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 
Call#2 Retry#0
java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: 
org.apache.hadoop.security.authentication.client.AuthenticationException: 
Authentication failed, status: 403, message: Forbidden
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:750)
at 
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
at 
org.apache.hadoop.hdfs.server.name