[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370483#comment-14370483
]
Yi Liu commented on HDFS-7256:
--
Hi Xiaoyu, {{hadoop.security.key.provider.path}} is only used as the default
key provider when you use {{hadoop key}} command.
The failure you got is not related to this, it's because the kerberos
authentication failed, I think your local keberos ticket cache (tgt) is out of
date or you have not done a "kinit". Even you remove this config, you can
still get the same issue.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>Assignee: Yi Liu
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370241#comment-14370241
]
Xiaoyu Yao commented on HDFS-7256:
--
bq. If you don't want specify -provider every time, please configure in
core-site.xml
{code}
hadoop.security.key.provider.path
kms://http@localhost:16000/kms
{code}
Is there any purpose of *hadoop.security.key.provider.path*? I can't find any
document about this. When we have this added in a Kerberos environment.
HiveServer2 queries failed to get DelegationToken from KMS. Is there any know
issue with this?
{code}
select count(*), symbol from stocks group by symbol;
INFO : Number of reduce tasks not specified. Estimated from input data size: 7
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=
INFO : Cleaning up the staging area /user/hehe/.staging/job_1426024489715_0006
ERROR : Job Submission failed with exception
'java.io.IOException(org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt))'
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:794)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:459)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:429)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:999)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[ https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175105#comment-14175105 ] Yi Liu commented on HDFS-7256: -- For *rename*, actually only allowed in same EZ, not allowed between EZs even with same EZ-keys. > Encryption Key created in Java Key Store after Namenode start unavailable for > EZ Creation > -- > > Key: HDFS-7256 > URL: https://issues.apache.org/jira/browse/HDFS-7256 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, security >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao > > Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ > with a Key created after NN starts. > Briefly check the code and found that the KeyProivder is loaded by FSN only > at the NN start. My work around is to restart the NN which triggers the > reload of Key Provider. Is this expected? > Repro Steps: > Create a new Key after NN and KMS starts > hadoop/bin/hadoop key create ezkey1 -size 256 -provider > jceks://file/home/hadoop/kms.keystore > List Keys > hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider > jceks://file/home/hadoop/kms.keystore -metadata > Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore > ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created: > Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null > key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: > Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null > key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: > Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null > Create Encryption Zone > hadoop/bin/hdfs dfs -mkdir /Ez1 > hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1 > -path /Ez1 > RemoteException: Key ezkey1 doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175101#comment-14175101
]
Yi Liu commented on HDFS-7256:
--
Thanks [~xyao] for trying this. Responses to your comments:
*1.* I think you are using java JCE crypto codec (If openssl is not configured
or incorrect version, JCE will be used), by default, JCE only supports 128bits,
if you want to use 256bits, you need to download additional thing from Oracle.
*2.* Ideally {{hadoop.security.key.provider.path}} is better in
_CommonConfigurationKeysPublic_, it's committed early and we do not modified it
later.
*3.* You are talking about *rename* which is not allowed between EZs with
different EZ-keys or from EZ to non-EZ directly, but {{cp}} is allowed.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[ https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174896#comment-14174896 ] Xiaoyu Yao commented on HDFS-7256: -- Thanks [~hitliuyi] again for the clarification. Three more follow up questions: 1. KMS and Hadoop Key Shell allows creating keys of length > 128. But HDFS seems to have a hard limitation of AES-CTS 128 only. Is this expected? hadoop@hadoopdev:~/deploy$ hadoop/bin/hadoop key list -metadata Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/] key2 : cipher: AES/CTR/NoPadding, length: 256, description: null, created: Thu Oct 16 22:42:20 PDT 2014, version: 1, attributes: [key.acl.name=key2] key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: Thu Oct 16 14:28:53 PDT 2014, version: 1, attributes: null hadoop@hadoopdev:~/deploy$ hadoop/bin/hdfs crypto -createZone -path /ez2 -keyName key2 RemoteException: java.util.concurrent.ExecutionException: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: java.security.InvalidKeyException: Illegal key size 2. Thanks for pointing me the 'hadoop.security.key.provider.path'. That's exactly what I'm looking for. However, I did not find it as it is hard coded in KeyProviderFactory.java, which is different from other security configuration keys in CommonConfigurationKeysPublic.java. If this key is targeted for public usage, I would suggest to put it in CommonConfigurationKeysPublic.java and also include in the hadoop key shell help message. 3. The document mentioned that copy file between EZs with different EZ-keys or copy file form EZ to non-EZ directory are not allowed. But my test shows it works completely fine. Is this explicitly blocked or just not recommended? > Encryption Key created in Java Key Store after Namenode start unavailable for > EZ Creation > -- > > Key: HDFS-7256 > URL: https://issues.apache.org/jira/browse/HDFS-7256 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, security >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao > > Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ > with a Key created after NN starts. > Briefly check the code and found that the KeyProivder is loaded by FSN only > at the NN start. My work around is to restart the NN which triggers the > reload of Key Provider. Is this expected? > Repro Steps: > Create a new Key after NN and KMS starts > hadoop/bin/hadoop key create ezkey1 -size 256 -provider > jceks://file/home/hadoop/kms.keystore > List Keys > hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider > jceks://file/home/hadoop/kms.keystore -metadata > Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore > ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created: > Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null > key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: > Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null > key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created: > Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null > Create Encryption Zone > hadoop/bin/hdfs dfs -mkdir /Ez1 > hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1 > -path /Ez1 > RemoteException: Key ezkey1 doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174859#comment-14174859
]
Yi Liu commented on HDFS-7256:
--
For the warning, I found KMS client or HttpFS client always try
{{KerberosAuthenticator}} first, then will trigger a fallback if server is not
security enabled. So we can ignore that warning and it appears in non-secured
mode.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174818#comment-14174818
]
Yi Liu commented on HDFS-7256:
--
{quote}
Can you point me the link to fs-encryption/KMS user doc if there is a different
one
{quote}
HDFS encryption is not included in 2.5.1 or before, so there is no on-line
document, I mean you could compile the user doc using:
{{mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site}}
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174813#comment-14174813
]
Yi Liu commented on HDFS-7256:
--
[~xyao], ideally the hdfs encryption is recommended used in secured
environment(keberos enabled), and in this case, there is no this warning.
Furthermore
{quote}
The client runs with user 'hadoop'. The proxyuser and delegation token(use
default) are set up in kms-site.xml.
hadoop.kms.proxyuser.hadoop.users
*
{quote}
Your use case is not the proxyuser, the reason is as I said in above comment.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174804#comment-14174804
]
Yi Liu commented on HDFS-7256:
--
Thanks [~xyao].
*For your question 1:*
Please don't specify {{hadoop.security.crypto.jce.provider}}, it's a jce
provider used for jce cryptocodec. Not for key provider uri.
So please configure in hdfs-site.xml
{code}
dfs.encryption.key.provider.uri
kms://http@localhost:16000/kms
{code}
And in kms-site.xml
{code}
hadoop.kms.key.provider.uri
jceks://file@/home/hadoop/kms.keystore
{code}
When you use hadoop key shell, please specify
{code}
-provider kms://http@localhost:16000/kms
{code}
If you don't want specify {{-provider}} every time, please configure in
core-site.xml
{code}
hadoop.security.key.provider.path
kms://http@localhost:16000/kms
{code}
*For your question 2:*
For the warning, you see it from kms log?
If so, It's a warning and doesn't affect functionality, if kerberos is *not*
enabled, the request sent to kms is without an user for the first time, but it
will fail and trigger authenticatation again with the user name, then it
successes.
There was ever a bug (HADOOP-11151) to let request having an user name for the
first time in non-secured mode, let me check in latest trunk whether it's
fixed, if not, I can fix that.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174744#comment-14174744
]
Xiaoyu Yao commented on HDFS-7256:
--
Thanks [~hitliuyi] for the detail explanation. I configured my test
environment based on HDFS-6134 proposal:
https://issues.apache.org/jira/secure/attachment/12660368/HDFSDataatRestEncryption.pdf.
Can you point me the link to fs-encryption/KMS user doc if there is a different
one?
I do have a KMS setup with JavaKeyStoreProvider pointing to the same java key
store file.
Based on your suggestion, I just switch to use 'kms://http@localhost:16000/kms'
instead of the java key store file
'jceks://file/home/hadoop/kms.keystore' directly for the
'dfs.encryption.key.provider.uri' in hdfs-site.xml and
'hadoop.security.crypto.jce.provider' in core-site.xml.
Below I have two follow up questions when executing the the 'hadoop key'
command after the change. Can you confirm if these are expected or not?
1. Have to specify -provider explicitly even though
hadoop.security.crypto.jce.provider='kms://http@localhost:16000/kms' is
configured in core-site.xml.
hadoop@hadoopdev:~/deploy$ hadoop/bin/hadoop key list
There are no non-transient KeyProviders configured.
Use the -provider option to specify a provider. If you
want to list a transient provider then you must use the
-provider argument.
2. Keys are returned with -provider specified but WARN message is logged in
kms.log on Anonymous request. My understanding is that KMS should proxy user
'hadoop' based the proxy user setting below. Do I miss anything?
hadoop@hadoopdev:~/deploy$ hadoop/bin/hadoop key list -provider
kms://http@localhost:16000/kms
Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/]
key1
{code}
2014-10-16 22:08:38,386 WARN AuthenticationFilter - Authentication exception:
Anonymous requests are disallowed
org.apache.hadoop.security.authentication.client.AuthenticationException:
Anonymous requests are disallowed
at
org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:184)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:330)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)
{/code}
The client runs with user 'hadoop'. The proxyuser and delegation token(use
default) are set up in kms-site.xml.
hadoop.kms.proxyuser.hadoop.users
*
...
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvi
[jira] [Commented] (HDFS-7256) Encryption Key created in Java Key Store after Namenode start unavailable for EZ Creation
[
https://issues.apache.org/jira/browse/HDFS-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174669#comment-14174669
]
Yi Liu commented on HDFS-7256:
--
Thanks [~xyao] for testing this, this should be not an issue. Let me explain
below.
HDFS encryption at rest requires user to configure a KMS, and the backing
KeyProvider of KMS can be a {{JavaKeyStoreProvider}} or a third-party keystore
which implements Hadoop {{KeyProvider}} interface.
In your case, {{JavaKeyStoreProvider}} is used directly, actually both FSN and
DFSClient will have KeyProvider instance (different), FSN uses KeyProvider
instance to get EncryptionZone key and get Encrypted data encryption keys, and
DFSClient uses KeyProvider instance to decrypt the data encryption keys.
JavaKeyStoreProvider uses local java keystore file, it can't satisfy multiple
nodes accessing.
"hadoop key create ..." command constructs its KeyProvider instance in client
side, and create/flush key to java keystore file, and FSN will not reload the
java keystore file. That's the reason why you see the exception.
So please configure a KMS and the backing KeyProvider could be a
{{JavaKeyStoreProvider}}, for more information, please refer to the
fs-encryption/KMS user doc.
> Encryption Key created in Java Key Store after Namenode start unavailable for
> EZ Creation
> --
>
> Key: HDFS-7256
> URL: https://issues.apache.org/jira/browse/HDFS-7256
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, security
>Affects Versions: 2.6.0
>Reporter: Xiaoyu Yao
>
> Hit an error on "RemoteException: Key ezkey1 doesn't exist." when creating EZ
> with a Key created after NN starts.
> Briefly check the code and found that the KeyProivder is loaded by FSN only
> at the NN start. My work around is to restart the NN which triggers the
> reload of Key Provider. Is this expected?
> Repro Steps:
> Create a new Key after NN and KMS starts
> hadoop/bin/hadoop key create ezkey1 -size 256 -provider
> jceks://file/home/hadoop/kms.keystore
> List Keys
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hadoop key list -provider
> jceks://file/home/hadoop/kms.keystore -metadata
> Listing keys for KeyProvider: jceks://file/home/hadoop/kms.keystore
> ezkey1 : cipher: AES/CTR/NoPadding, length: 256, description: null, created:
> Thu Oct 16 18:51:30 EDT 2014, version: 1, attributes: null
> key2 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 19:44:09 EDT 2014, version: 1, attributes: null
> key1 : cipher: AES/CTR/NoPadding, length: 128, description: null, created:
> Tue Oct 14 17:52:36 EDT 2014, version: 1, attributes: null
> Create Encryption Zone
> hadoop/bin/hdfs dfs -mkdir /Ez1
> hadoop@SaturnVm:~/deploy$ hadoop/bin/hdfs crypto -createZone -keyName ezkey1
> -path /Ez1
> RemoteException: Key ezkey1 doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
