[jira] [Commented] (HADOOP-10670) Allow AuthenticationFilters to load secret from signature secret files
[ https://issues.apache.org/jira/browse/HADOOP-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380510#comment-14380510 ] Charles Lamb commented on HADOOP-10670: --- [~wheat9],[~drankye], Is it possible that something didn't get committed with this patch? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-auth: Compilation failure: Compilation failure: [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java:[287,29] cannot find symbol [ERROR] symbol: class FileSignerSecretProvider [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java:[301,29] cannot find symbol [ERROR] symbol: class FileSignerSecretProvider [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter > Allow AuthenticationFilters to load secret from signature secret files > -- > > Key: HADOOP-10670 > URL: https://issues.apache.org/jira/browse/HADOOP-10670 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng >Priority: Minor > Fix For: 2.7.0 > > Attachments: HADOOP-10670-v4.patch, HADOOP-10670-v5.patch, > HADOOP-10670-v6.patch, hadoop-10670-v2.patch, hadoop-10670-v3.patch, > hadoop-10670.patch > > > In Hadoop web console, by using AuthenticationFilterInitializer, it's allowed > to configure AuthenticationFilter for the required signature secret by > specifying signature.secret.file property. This improvement would also allow > this when AuthenticationFilterInitializer isn't used in situations like > webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317993#comment-14317993 ] Charles Lamb commented on HADOOP-9477: -- Hi [~dapengsun], As near as I can tell, the .004 patch is the same as the .003 patch that I posted, right? If so, then I'm happy with it. +1 (non-binding). Thanks. Charles > posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Dapeng Sun > Fix For: 2.7.0 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, > HADOOP-9477.patch, HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317531#comment-14317531 ] Charles Lamb commented on HADOOP-9477: -- That would be great. I'm happy to help. > posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.7.0 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316904#comment-14316904 ] Charles Lamb commented on HADOOP-9477: -- [~drankye], I was informed offline that it may have been a little uncool for me to post a patch. Sorry about that. My intent was only to give you my comments as changed code rather than to take over the patch and posting diffs seemed like the most expeditious way to do that. Apologies. Charles > posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.7.0 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316744#comment-14316744 ] Charles Lamb commented on HADOOP-9477: -- TestHttpServerLifecycle passes on my local machine with the patch applied. > posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.7.0 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9477) posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-9477: - Attachment: HADOOP-9477.003.patch [~drankye], I have posted a rebased version of your patch with a few minor cleanups. Good catch on checking groupResults for null. I also put in a serial number for the patch number (I started with .003 since you already posted 2 of them). > posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.7.0 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11478) HttpFSServer does not properly impersonate a real user when executing "open" operation in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293894#comment-14293894 ] Charles Lamb commented on HADOOP-11478: --- [~ranadip], I assume that when you configured your kms acls per the instructions in HADOOP-11479 that this problem went away. Feel free to reopen if that's not the case. Charles > HttpFSServer does not properly impersonate a real user when executing "open" > operation in a kerberised environment > -- > > Key: HADOOP-11478 > URL: https://issues.apache.org/jira/browse/HADOOP-11478 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip >Priority: Blocker > > Setup: > - Kerberos enabled in the cluster, including Hue SSO > - Encryption enabled using KMS. Encryption key and encryption zone created. > KMS key level ACL created to allow only real user to have all access to the > key and no one else. > Manifestation: > Using Hue, real user logged in using Kerberos credentials. For direct access, > user does kinit and then uses curl calls. > New file creation inside encryption zone goes ahead fine as expected. > But attempts to view the contents of the file fails with exception: > "User [httpfs] is not authorized to perform [DECRYPT_EEK] on key with ACL > name [mykeyname]!!" > Perhaps, this is linked to bug #HDFS-6849. In the file HttpFSServer.java, the > OPEN handler calls command.execute(fs) directly (and this fails). In CREATE, > that call is wrapped within fsExecute(user, command). Apparently, this seems > to cause the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11478) HttpFSServer does not properly impersonate a real user when executing "open" operation in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HADOOP-11478. --- Resolution: Not a Problem > HttpFSServer does not properly impersonate a real user when executing "open" > operation in a kerberised environment > -- > > Key: HADOOP-11478 > URL: https://issues.apache.org/jira/browse/HADOOP-11478 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip >Priority: Blocker > > Setup: > - Kerberos enabled in the cluster, including Hue SSO > - Encryption enabled using KMS. Encryption key and encryption zone created. > KMS key level ACL created to allow only real user to have all access to the > key and no one else. > Manifestation: > Using Hue, real user logged in using Kerberos credentials. For direct access, > user does kinit and then uses curl calls. > New file creation inside encryption zone goes ahead fine as expected. > But attempts to view the contents of the file fails with exception: > "User [httpfs] is not authorized to perform [DECRYPT_EEK] on key with ACL > name [mykeyname]!!" > Perhaps, this is linked to bug #HDFS-6849. In the file HttpFSServer.java, the > OPEN handler calls command.execute(fs) directly (and this fails). In CREATE, > that call is wrapped within fsExecute(user, command). Apparently, this seems > to cause the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11510) Expose truncate API via FileContext
[
https://issues.apache.org/jira/browse/HADOOP-11510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293534#comment-14293534
]
Charles Lamb commented on HADOOP-11510:
---
[~hitliuyi],
Looks good. One small nit. To be consistent in testTruncateThroughFileContext
you could add a few more finals to the decls.
Just out of curiosity, why 3 in newLength = fileLength/3?
> Expose truncate API via FileContext
> ---
>
> Key: HADOOP-11510
> URL: https://issues.apache.org/jira/browse/HADOOP-11510
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HADOOP-11510.001.patch
>
>
> We also need to expose truncate API via {{org.apache.hadoop.fs.FileContext}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11469) KMS should skip default.key.acl and whitelist.key.acl when loading key acl
[ https://issues.apache.org/jira/browse/HADOOP-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291791#comment-14291791 ] Charles Lamb commented on HADOOP-11469: --- I wish I could, but I'm not a real committer. I only play one on TV. (That's why my +1 was non-binding). [~hitliuyi], would you be able to help out here? > KMS should skip default.key.acl and whitelist.key.acl when loading key acl > -- > > Key: HADOOP-11469 > URL: https://issues.apache.org/jira/browse/HADOOP-11469 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Minor > Attachments: HADOOP-11469.001.patch, HADOOP-11469.002.patch, > HADOOP-11469.003.patch > > > KMSACLs#setKeyACLs, loads key ACLs from the configuration by checking if the > key name contains "key.acl". However, this also matches "default.key.acl" and > "whitelist.key.acl" which is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11493: -- Attachment: HADOOP-11493.002.patch > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch, HADOOP-11493.001.patch, > HADOOP-11493.002.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287732#comment-14287732 ] Charles Lamb commented on HADOOP-11493: --- Oh, good catch. The .002 patch fixes those other two. Thanks. > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch, HADOOP-11493.001.patch, > HADOOP-11493.002.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11493: -- Attachment: HADOOP-11493.001.patch > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch, HADOOP-11493.001.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286931#comment-14286931 ] Charles Lamb commented on HADOOP-11493: --- Thanks for the review [~ajisakaa]. The new patch adds the change to the doc that you suggested. Charles > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch, HADOOP-11493.001.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HADOOP-11479. --- Resolution: Not a Problem > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285884#comment-14285884 ] Charles Lamb commented on HADOOP-11479: --- [~ranadip], I haven't heard back about whether the suggested KMS ACLs got you past this problem. I'm going to close this for now, but feel free to reopen it. > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11493: -- Status: Patch Available (was: Open) > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
[ https://issues.apache.org/jira/browse/HADOOP-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11493: -- Attachment: HADOOP-11493.000.patch > Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER > > > Key: HADOOP-11493 > URL: https://issues.apache.org/jira/browse/HADOOP-11493 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11493.000.patch > > > "does is": > > hadoop.kms.acl.ROLLOVER > * > > ACL for rollover-key operations. > If the user does is not in the GET ACL, the key material is not returned > as part of the response. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11493) Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER
Charles Lamb created HADOOP-11493: - Summary: Typo in kms-acls.xml description for hadoop.kms.acl.ROLLOVER Key: HADOOP-11493 URL: https://issues.apache.org/jira/browse/HADOOP-11493 Project: Hadoop Common Issue Type: Bug Components: kms Affects Versions: 2.7.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial "does is": hadoop.kms.acl.ROLLOVER * ACL for rollover-key operations. If the user does is not in the GET ACL, the key material is not returned as part of the response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280870#comment-14280870 ] Charles Lamb commented on HADOOP-11479: --- [~ranadip], I am pretty sure that the READ exception is being thrown when the NN is doing getMetadata. Here's the explanation: There are KMS operation ACLs (hadoop.kms...) and KMS key ACLs (default.key.acl...). The KMS key ACLs are more coarse-grained (MANAGEMENT, GENERATE_EEK, DECRYPT_EEK, READ, ALL) than the KMS operation ACLs (which cover each public KMS method call individually). So, by default, the HDFS user has READ permission on all keys (default.key.acl.READ=*). This gives that user access to the getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata and getCurrentKey methods. But, then, to lock down HDFS user access to key material, you also need to add the HDFS user to the blacklist for the following KMS operation ACLs: CREATE, DELETE, ROLLOVER, GET, GET_KEYS, SET_KEY_MATERIAL, DECRYPT_EEK (This is what setting the KMS Blacklist property in CM does: it is a shortcut to setting these seven KMS operation ACLs) There is also a specific KMS operation ACL for GET_METADATA, but you don't want to set that in this case. > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11469) KMS should skip default.key.acl and whitelist.key.acl when loading key acl
[ https://issues.apache.org/jira/browse/HADOOP-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279722#comment-14279722 ] Charles Lamb commented on HADOOP-11469: --- LGTM. +1 (non-binding) > KMS should skip default.key.acl and whitelist.key.acl when loading key acl > -- > > Key: HADOOP-11469 > URL: https://issues.apache.org/jira/browse/HADOOP-11469 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Minor > Attachments: HADOOP-11469.001.patch, HADOOP-11469.002.patch, > HADOOP-11469.003.patch > > > KMSACLs#setKeyACLs, loads key ACLs from the configuration by checking if the > key name contains "key.acl". However, this also matches "default.key.acl" and > "whitelist.key.acl" which is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279284#comment-14279284 ] Charles Lamb commented on HADOOP-11479: --- [~ranadip], Just to confirm, you are running the hdfs crypto command as user 'ranadip' (assuming your cli prompt is showing user@host) and you are running your namenode as hdfs (as indicated by the KMS message "User [hdfs] is not..."). Assuming the NN is running as user hdfs, then user 'hdfs' is the hdfs superuser and it must have (as you have noted) GENERATE_EEK and READ access to ranskey1. As [~hitliuyi] said, the hdfs crypto -createZone command can only be run by the superuser. Charles > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11469) KMS should skip default.key.acl and whitelist.key.acl when loading key acl
[ https://issues.apache.org/jira/browse/HADOOP-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278839#comment-14278839 ] Charles Lamb commented on HADOOP-11469: --- That's a nice simple test solution. The only nit I offer is that you could use a final for the conf and kmsACLs, but that's my own hang-up. +1 (non-binding). > KMS should skip default.key.acl and whitelist.key.acl when loading key acl > -- > > Key: HADOOP-11469 > URL: https://issues.apache.org/jira/browse/HADOOP-11469 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Minor > Attachments: HADOOP-11469.001.patch, HADOOP-11469.002.patch > > > KMSACLs#setKeyACLs, loads key ACLs from the configuration by checking if the > key name contains "key.acl". However, this also matches "default.key.acl" and > "whitelist.key.acl" which is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11469) KMS should skip default.key.acl and whitelist.key.acl when loading key acl
[ https://issues.apache.org/jira/browse/HADOOP-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11469: -- Description: KMSACLs#setKeyACLs, loads key ACLs from the configuration by checking if the key name contains "key.acl". However, this also matches "default.key.acl" and "whitelist.key.acl" which is incorrect. (was: In KMSACLs#setKeyACLs, it loads key ACL from configuration by judging if the key name contains "key.acl". This will also matching "default.key.acl" and "whitelist.key.acl".) Summary: KMS should skip default.key.acl and whitelist.key.acl when loading key acl (was: KMS should skip default.key.acl and whiltelist.key.acl when loading key acl) > KMS should skip default.key.acl and whitelist.key.acl when loading key acl > -- > > Key: HADOOP-11469 > URL: https://issues.apache.org/jira/browse/HADOOP-11469 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Minor > Attachments: HADOOP-11469.001.patch > > > KMSACLs#setKeyACLs, loads key ACLs from the configuration by checking if the > key name contains "key.acl". However, this also matches "default.key.acl" and > "whitelist.key.acl" which is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11469) KMS should skip default.key.acl and whiltelist.key.acl when loading key acl
[ https://issues.apache.org/jira/browse/HADOOP-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276937#comment-14276937 ] Charles Lamb commented on HADOOP-11469: --- Hi [~dian.fu], In general this looks good, but I think it would be good to add a unit test for this. Charles > KMS should skip default.key.acl and whiltelist.key.acl when loading key acl > --- > > Key: HADOOP-11469 > URL: https://issues.apache.org/jira/browse/HADOOP-11469 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Minor > Attachments: HADOOP-11469.001.patch > > > In KMSACLs#setKeyACLs, it loads key ACL from configuration by judging if the > key name contains "key.acl". This will also matching "default.key.acl" and > "whitelist.key.acl". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
[ https://issues.apache.org/jira/browse/HADOOP-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263994#comment-14263994 ] Charles Lamb commented on HADOOP-11455: --- Hi [~hitliuyi], Thanks for the review. Yes, that's a good idea. The attached .001 patch addresses this. Charles > KMS and Credential CLI should request confirmation for deletion by default > -- > > Key: HADOOP-11455 > URL: https://issues.apache.org/jira/browse/HADOOP-11455 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11455.000.patch, HADOOP-11455.001.patch > > > The hadoop key delete and hadoop credential delete currently only ask for > confirmation of the delete if -i is specified. Asking for confirmation should > be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
[ https://issues.apache.org/jira/browse/HADOOP-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11455: -- Attachment: HADOOP-11455.001.patch > KMS and Credential CLI should request confirmation for deletion by default > -- > > Key: HADOOP-11455 > URL: https://issues.apache.org/jira/browse/HADOOP-11455 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11455.000.patch, HADOOP-11455.001.patch > > > The hadoop key delete and hadoop credential delete currently only ask for > confirmation of the delete if -i is specified. Asking for confirmation should > be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
[ https://issues.apache.org/jira/browse/HADOOP-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261614#comment-14261614 ] Charles Lamb commented on HADOOP-11455: --- The FB warnings are unrelated. > KMS and Credential CLI should request confirmation for deletion by default > -- > > Key: HADOOP-11455 > URL: https://issues.apache.org/jira/browse/HADOOP-11455 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11455.000.patch > > > The hadoop key delete and hadoop credential delete currently only ask for > confirmation of the delete if -i is specified. Asking for confirmation should > be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
[ https://issues.apache.org/jira/browse/HADOOP-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11455: -- Attachment: HADOOP-11455.000.patch The attached patch relates to the hadoop credential delete and hadoop key delete commands. It changes -i/-interactive to -f/-force and changes the polarity of that option. Asking for confirmation of the delete is now the default. If a user/script wants to force the delete without confirmation, the -f flag can be used. The patch also cleans up some typos. > KMS and Credential CLI should request confirmation for deletion by default > -- > > Key: HADOOP-11455 > URL: https://issues.apache.org/jira/browse/HADOOP-11455 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11455.000.patch > > > The hadoop key delete and hadoop credential delete currently only ask for > confirmation of the delete if -i is specified. Asking for confirmation should > be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
[ https://issues.apache.org/jira/browse/HADOOP-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11455: -- Status: Patch Available (was: Open) > KMS and Credential CLI should request confirmation for deletion by default > -- > > Key: HADOOP-11455 > URL: https://issues.apache.org/jira/browse/HADOOP-11455 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11455.000.patch > > > The hadoop key delete and hadoop credential delete currently only ask for > confirmation of the delete if -i is specified. Asking for confirmation should > be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11455) KMS and Credential CLI should request confirmation for deletion by default
Charles Lamb created HADOOP-11455: - Summary: KMS and Credential CLI should request confirmation for deletion by default Key: HADOOP-11455 URL: https://issues.apache.org/jira/browse/HADOOP-11455 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.7.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor The hadoop key delete and hadoop credential delete currently only ask for confirmation of the delete if -i is specified. Asking for confirmation should be the default action for both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11394) hadoop-aws documentation missing.
[ https://issues.apache.org/jira/browse/HADOOP-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244164#comment-14244164 ] Charles Lamb commented on HADOOP-11394: --- non-binding +1 from me. > hadoop-aws documentation missing. > - > > Key: HADOOP-11394 > URL: https://issues.apache.org/jira/browse/HADOOP-11394 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-11394.001.patch, HADOOP-11394.002.patch > > > In HADOOP-10714, the documentation source files for hadoop-aws were moved > from src/site to src/main/site. The build is no longer actually generating > the HTML site from these source files, because src/site is the expected path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11289) Fix typo in RpcUtil log message
[
https://issues.apache.org/jira/browse/HADOOP-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205180#comment-14205180
]
Charles Lamb commented on HADOOP-11289:
---
Thank you [~wheat9] for the quick review and commit. I should mention FTR that
the patch doesn't need any tests since it is a log message typo fix.
> Fix typo in RpcUtil log message
> ---
>
> Key: HADOOP-11289
> URL: https://issues.apache.org/jira/browse/HADOOP-11289
> Project: Hadoop Common
> Issue Type: Bug
> Components: net
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Trivial
> Fix For: 2.7.0
>
> Attachments: HADOOP-11289.001.patch
>
>
> From RpcUtil.java:
> LOG.info("Malfromed RPC request from " + e.getRemoteAddress());
> s/Malfromed/malformed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11289) Fix typo in RpcUtil log message
[
https://issues.apache.org/jira/browse/HADOOP-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HADOOP-11289:
--
Status: Patch Available (was: Open)
> Fix typo in RpcUtil log message
> ---
>
> Key: HADOOP-11289
> URL: https://issues.apache.org/jira/browse/HADOOP-11289
> Project: Hadoop Common
> Issue Type: Bug
> Components: net
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Trivial
> Attachments: HADOOP-11289.001.patch
>
>
> From RpcUtil.java:
> LOG.info("Malfromed RPC request from " + e.getRemoteAddress());
> s/Malfromed/malformed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11289) Fix typo in RpcUtil log message
[
https://issues.apache.org/jira/browse/HADOOP-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HADOOP-11289:
--
Attachment: HADOOP-11289.001.patch
Patch fixes the typo.
> Fix typo in RpcUtil log message
> ---
>
> Key: HADOOP-11289
> URL: https://issues.apache.org/jira/browse/HADOOP-11289
> Project: Hadoop Common
> Issue Type: Bug
> Components: net
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Trivial
> Attachments: HADOOP-11289.001.patch
>
>
> From RpcUtil.java:
> LOG.info("Malfromed RPC request from " + e.getRemoteAddress());
> s/Malfromed/malformed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11289) Fix typo in RpcUtil log message
[
https://issues.apache.org/jira/browse/HADOOP-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HADOOP-11289:
--
Summary: Fix typo in RpcUtil log message (was: Fix typo in RpcInfo log
message)
> Fix typo in RpcUtil log message
> ---
>
> Key: HADOOP-11289
> URL: https://issues.apache.org/jira/browse/HADOOP-11289
> Project: Hadoop Common
> Issue Type: Bug
> Components: net
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Trivial
>
> From RpcUtil.java:
> LOG.info("Malfromed RPC request from " + e.getRemoteAddress());
> s/Malfromed/malformed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11289) Fix typo in RpcInfo log message
Charles Lamb created HADOOP-11289:
-
Summary: Fix typo in RpcInfo log message
Key: HADOOP-11289
URL: https://issues.apache.org/jira/browse/HADOOP-11289
Project: Hadoop Common
Issue Type: Bug
Components: net
Affects Versions: 2.7.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
>From RpcUtil.java:
LOG.info("Malfromed RPC request from " + e.getRemoteAddress());
s/Malfromed/malformed/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
[ https://issues.apache.org/jira/browse/HADOOP-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HADOOP-11026. --- Resolution: Duplicate Fix Version/s: 2.7.0 The doc changes were included in HDFS-6843. > add FileSystem contract specification for FSDataInputStream and > FSDataOutputStream#isEncrypted > -- > > Key: HADOOP-11026 > URL: https://issues.apache.org/jira/browse/HADOOP-11026 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, test >Affects Versions: 3.0.0, 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Fix For: 2.7.0 > > Attachments: HADOOP-11026-prelim.001.patch, HADOOP-11026.001.patch > > > Following on to HDFS-6843, the contract specification for FSDataInputStream > and FSDataOutputStream needs to be updated to reflect the addition of > isEncrypted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11176) KMSClientProvider authentication fails when both currentUgi and loginUgi are a proxied user
[ https://issues.apache.org/jira/browse/HADOOP-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11176: -- Summary: KMSClientProvider authentication fails when both currentUgi and loginUgi are a proxied user (was: KMSClientProvider authentication fails when when both currentUgi and loginUgi is a proxied user) > KMSClientProvider authentication fails when both currentUgi and loginUgi are > a proxied user > --- > > Key: HADOOP-11176 > URL: https://issues.apache.org/jira/browse/HADOOP-11176 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh > Attachments: HADOOP-11176.1.patch > > > In a secure environment, with kerberos, when the KMSClientProvider instance > is created in the context of a proxied user, The initial SPNEGO handshake is > made with the currentUser (the proxied user) as the Principal.. this will > fail, since the proxied user is not logged in. > The handshake must be done using the real user. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11176) KMSClientProvider authentication fails when when both currentUgi and loginUgi is a proxied user
[
https://issues.apache.org/jira/browse/HADOOP-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163811#comment-14163811
]
Charles Lamb commented on HADOOP-11176:
---
[~asuresh],
Just two nits.
This change:
{noformat}
final String doAsUser = (currentUgi.getAuthenticationMethod() ==
UserGroupInformation.AuthenticationMethod.PROXY)
? currentUgi.getShortUserName() : null;
{noformat}
seems to be a whitespace only change.
This change:
{noformat}
return authUrl.openConnection(
url,
authToken, doAsUser);
{noformat}
seems to be an extraneous reformatting.
> KMSClientProvider authentication fails when when both currentUgi and loginUgi
> is a proxied user
> ---
>
> Key: HADOOP-11176
> URL: https://issues.apache.org/jira/browse/HADOOP-11176
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Arun Suresh
> Attachments: HADOOP-11176.1.patch
>
>
> In a secure environment, with kerberos, when the KMSClientProvider instance
> is created in the context of a proxied user, The initial SPNEGO handshake is
> made with the currentUser (the proxied user) as the Principal.. this will
> fail, since the proxied user is not logged in.
> The handshake must be done using the real user.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11160) Fix typo in nfs3 server duplicate entry reporting
[ https://issues.apache.org/jira/browse/HADOOP-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157253#comment-14157253 ] Charles Lamb commented on HADOOP-11160: --- And thank you [~wheat9] for the quick review and commit. > Fix typo in nfs3 server duplicate entry reporting > -- > > Key: HADOOP-11160 > URL: https://issues.apache.org/jira/browse/HADOOP-11160 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Fix For: 2.6.0 > > Attachments: HDFS-7183.001.patch > > > Fix typo: "systms". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[ https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153394#comment-14153394 ] Charles Lamb commented on HADOOP-10714: --- bq. charles —you happy with this? Yes, +1 (non-binding) from me. > AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call > -- > > Key: HADOOP-10714 > URL: https://issues.apache.org/jira/browse/HADOOP-10714 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.5.0 >Reporter: David S. Wang >Assignee: Juan Yu >Priority: Critical > Labels: s3 > Attachments: HADOOP-10714-007.patch, HADOOP-10714-1.patch, > HADOOP-10714.001.patch, HADOOP-10714.002.patch, HADOOP-10714.003.patch, > HADOOP-10714.004.patch, HADOOP-10714.005.patch, HADOOP-10714.006.patch > > > In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need > to have the number of entries at 1000 or below. Otherwise we get a Malformed > XML error similar to: > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS > Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: > MalformedXML, AWS Error Message: The XML you provided was not well-formed or > did not validate against our published schema, S3 Extended Request ID: > DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480) > at > com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878) > Note that this is mentioned in the AWS documentation: > http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html > "The Multi-Object Delete request contains a list of up to 1000 keys that you > want to delete. In the XML, you provide the object key names, and optionally, > version IDs if you want to delete a specific version of the object from a > versioning-enabled bucket. For each key, Amazon S3….” > Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the > problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[
https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151919#comment-14151919
]
Charles Lamb commented on HADOOP-10714:
---
[~j...@cloudera.com],
The tests all worked like a champ:
{noformat}
---
T E S T S
---
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.68 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.28 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.416 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.451 sec -
in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.289 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 35.512 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.683 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Results :
Tests run: 46, Failures: 0, Errors: 0, Skipped: 3
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.871 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.332 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 47.507 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.011 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.172 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.234 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.172 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Results :
Tests run: 46, Failures: 0, Errors: 0, Skipped: 3
{noformat}
> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
> --
>
> Key: HADOOP-10714
> URL: https://issues.apache.org/jira/browse/HADOOP-10714
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.5.0
>Reporter: David S. Wang
>Assignee: Juan Yu
>Priority: Critical
> Labels: s3
> Attachments: HADOOP-10714-007.patch, HADOOP-10714-1.patch,
> HADOOP-10714.001.patch, HADOOP-10714.002.patch, HADOOP-10714.003.patch,
> HADOOP-10714.004.patch, HADOOP-10714.005.patch, HADOOP-10714.006.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need
> to have the number of entries at 1000 or below. Otherwise we get a Malformed
> XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
> Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code:
> MalformedXML, AWS Error Message: The XML you provided was not well-formed or
> did not validate against our published schema, S3 Extended Request ID:
> DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[ https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151740#comment-14151740 ] Charles Lamb commented on HADOOP-10714: --- [~j...@cloudera.com], I'm having trouble running the tests with the latest patch. I'll contact you offline to see if we can figure out what's going on. > AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call > -- > > Key: HADOOP-10714 > URL: https://issues.apache.org/jira/browse/HADOOP-10714 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.5.0 >Reporter: David S. Wang >Assignee: Juan Yu >Priority: Critical > Labels: s3 > Attachments: HADOOP-10714-007.patch, HADOOP-10714-1.patch, > HADOOP-10714.001.patch, HADOOP-10714.002.patch, HADOOP-10714.003.patch, > HADOOP-10714.004.patch, HADOOP-10714.005.patch, HADOOP-10714.006.patch > > > In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need > to have the number of entries at 1000 or below. Otherwise we get a Malformed > XML error similar to: > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS > Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: > MalformedXML, AWS Error Message: The XML you provided was not well-formed or > did not validate against our published schema, S3 Extended Request ID: > DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480) > at > com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878) > Note that this is mentioned in the AWS documentation: > http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html > "The Multi-Object Delete request contains a list of up to 1000 keys that you > want to delete. In the XML, you provide the object key names, and optionally, > version IDs if you want to delete a specific version of the object from a > versioning-enabled bucket. For each key, Amazon S3….” > Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the > problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11113) Namenode not able to reconnect to KMS after KMS restart
[ https://issues.apache.org/jira/browse/HADOOP-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-3: -- Assignee: Arun Suresh (was: Charles Lamb) > Namenode not able to reconnect to KMS after KMS restart > --- > > Key: HADOOP-3 > URL: https://issues.apache.org/jira/browse/HADOOP-3 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Arun Suresh >Assignee: Arun Suresh > > It is observed that if KMS is restarted without the Namenode being restarted, > NN will not be able to reconnect with KMS. > It seems that the KMS auth cookie goes stale and it does not get flushed, so > the KMSClient in the NN cannot reconnect with the new KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-11113) Namenode not able to reconnect to KMS after KMS restart
[ https://issues.apache.org/jira/browse/HADOOP-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HADOOP-3: - Assignee: Charles Lamb (was: Arun Suresh) > Namenode not able to reconnect to KMS after KMS restart > --- > > Key: HADOOP-3 > URL: https://issues.apache.org/jira/browse/HADOOP-3 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Arun Suresh >Assignee: Charles Lamb > > It is observed that if KMS is restarted without the Namenode being restarted, > NN will not be able to reconnect with KMS. > It seems that the KMS auth cookie goes stale and it does not get flushed, so > the KMSClient in the NN cannot reconnect with the new KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[
https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141944#comment-14141944
]
Charles Lamb commented on HADOOP-10714:
---
Hi Juan,
The Hadoop Coding Standards are here:
https://wiki.apache.org/hadoop/CodeReviewChecklist
As mentioned at the top of that file, it's basically the Sun Java Coding
standards with an indentation of 2, not 4. But to be clear (I learned this
lesson the hard way), it's 2 for new lines and 4 for continuations of lines).
BTW (sorry, I can't help myself), the LOG.debug line that ATM mentioned
probably does not need to be broken after the '('.
> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
> --
>
> Key: HADOOP-10714
> URL: https://issues.apache.org/jira/browse/HADOOP-10714
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.5.0
>Reporter: David S. Wang
>Assignee: Juan Yu
>Priority: Critical
> Labels: s3
> Attachments: HADOOP-10714-1.patch, HADOOP-10714.001.patch,
> HADOOP-10714.002.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need
> to have the number of entries at 1000 or below. Otherwise we get a Malformed
> XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
> Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code:
> MalformedXML, AWS Error Message: The XML you provided was not well-formed or
> did not validate against our published schema, S3 Extended Request ID:
> DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878)
> Note that this is mentioned in the AWS documentation:
> http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
> "The Multi-Object Delete request contains a list of up to 1000 keys that you
> want to delete. In the XML, you provide the object key names, and optionally,
> version IDs if you want to delete a specific version of the object from a
> versioning-enabled bucket. For each key, Amazon S3….”
> Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[ https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141692#comment-14141692 ] Charles Lamb commented on HADOOP-10714: --- BTW Juan, I apologize for my terse comments above. I had written them up in an editor buffer and then in a rush I just cut and pasted them into the comment frame without putting a greeting or any preamble in there. In general, it looks like the patch does what it is supposed to do so nice work on this. I also realized that my 2nd to last comment about the if (!renameSupported()) got badly formatted. The idea I was trying to convey is that there should be braces around the return; for those two if's. Thanks Juan. > AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call > -- > > Key: HADOOP-10714 > URL: https://issues.apache.org/jira/browse/HADOOP-10714 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.5.0 >Reporter: David S. Wang >Assignee: Juan Yu >Priority: Critical > Labels: s3 > Attachments: HADOOP-10714-1.patch, HADOOP-10714.001.patch > > > In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need > to have the number of entries at 1000 or below. Otherwise we get a Malformed > XML error similar to: > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS > Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: > MalformedXML, AWS Error Message: The XML you provided was not well-formed or > did not validate against our published schema, S3 Extended Request ID: > DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480) > at > com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878) > Note that this is mentioned in the AWS documentation: > http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html > "The Multi-Object Delete request contains a list of up to 1000 keys that you > want to delete. In the XML, you provide the object key names, and optionally, > version IDs if you want to delete a specific version of the object from a > versioning-enabled bucket. For each key, Amazon S3….” > Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the > problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[
https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141506#comment-14141506
]
Charles Lamb commented on HADOOP-10714:
---
General:
* Several lines bust the 80 char limit.
* There are several places where there are extra newlines. There should only
ever be one blank line.
* Remove unused imports in your new files.
S3AFileSystem
* // deleteUnnecessaryFakeDirectories
This looks like the name of a method rather than a comment.
TestS3AContractRename
* s/s3a don't/s3a doesn't/
core-site.xml
{quote}
{quote}
can be changed to:
{quote}
{quote}
* You can remove the extra newline after .
S3AFileSystemBaseTest
* s/If you keys/If your keys/, s/as passed/as passed./
S3AScaleTestBase
* remove the extra newline after the class comment.
S3ATestUtils
* You can move "implements S3ATestConstants" to the line above.
* testReceivedData decl has incorrect indentation. Extra newline at the end of
this method.
* generateTestData has the same indent problem in the decl.
* I'd prefer that you do import statics of various org.junit.Assert methods in
TestS3AFileSYstemBasicOps rather than extending S3AFileSystemBaseTest with it.
TestS3AFileSystemBasicOps
* remove the extra newline after the class comment.
I'm unclear about what the test renaming is about. Could you please comment on
that in the Jira?
TestS3AFileSystemContract
{quote}
if (!renameSupported()) return;
{quote}
should be changed to:
{quote}
if (!renameSupported()) {
return;
}
{quote}
* remove the blank line before the closing } of
testRenameDirectoryAsExistingDirectory(), ditto blank line before last } in the
file.
> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
> --
>
> Key: HADOOP-10714
> URL: https://issues.apache.org/jira/browse/HADOOP-10714
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.5.0
>Reporter: David S. Wang
>Assignee: Juan Yu
>Priority: Critical
> Labels: s3
> Attachments: HADOOP-10714-1.patch, HADOOP-10714.001.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need
> to have the number of entries at 1000 or below. Otherwise we get a Malformed
> XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
> Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code:
> MalformedXML, AWS Error Message: The XML you provided was not well-formed or
> did not validate against our published schema, S3 Extended Request ID:
> DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874)
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878)
> Note that this is mentioned in the AWS documentation:
> http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
> "The Multi-Object Delete request contains a list of up to 1000 keys that you
> want to delete. In the XML, you provide the object key names, and optionally,
> version IDs if you want to delete a specific version of the object from a
> versioning-enabled bucket. For each key, Amazon S3….”
> Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params
[ https://issues.apache.org/jira/browse/HADOOP-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136347#comment-14136347 ] Charles Lamb commented on HADOOP-11097: --- No tests are required -- it's a simple doc change. > kms docs say proxyusers, not proxyuser for config params > > > Key: HADOOP-11097 > URL: https://issues.apache.org/jira/browse/HADOOP-11097 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11097.001.patch > > > The KMS docs have the proxy parameters named proxyusers, not proxyuser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params
[ https://issues.apache.org/jira/browse/HADOOP-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-11097 started by Charles Lamb. - > kms docs say proxyusers, not proxyuser for config params > > > Key: HADOOP-11097 > URL: https://issues.apache.org/jira/browse/HADOOP-11097 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11097.001.patch > > > The KMS docs have the proxy parameters named proxyusers, not proxyuser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params
[ https://issues.apache.org/jira/browse/HADOOP-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11097: -- Attachment: HADOOP-11097.001.patch Here's a patch that corrects the parameters. > kms docs say proxyusers, not proxyuser for config params > > > Key: HADOOP-11097 > URL: https://issues.apache.org/jira/browse/HADOOP-11097 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Trivial > Attachments: HADOOP-11097.001.patch > > > The KMS docs have the proxy parameters named proxyusers, not proxyuser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params
Charles Lamb created HADOOP-11097: - Summary: kms docs say proxyusers, not proxyuser for config params Key: HADOOP-11097 URL: https://issues.apache.org/jira/browse/HADOOP-11097 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial The KMS docs have the proxy parameters named proxyusers, not proxyuser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[ https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127123#comment-14127123 ] Charles Lamb commented on HADOOP-11062: --- I forgot to ask why there's a surefire version change. non-binding +1 from me. > CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used > -- > > Key: HADOOP-11062 > URL: https://issues.apache.org/jira/browse/HADOOP-11062 > Project: Hadoop Common > Issue Type: Bug > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch, > HADOOP-11062.2.patch, HADOOP-11062.3.patch > > > there are a few testcases, cryptocodec related that require Hadoop native > code and OpenSSL. > These tests should be skipped if -Pnative is not used when running the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[ https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127048#comment-14127048 ] Charles Lamb commented on HADOOP-11062: --- Hi [~asuresh], I wonder if you should use equalsIgnoreCase instead in order to idiot-proof against "true" getting changed to "True". I ran with and without -Pnative and it def'ly log.warns if there's no -Pnative. It is silent if -Pnative is specified but there's no openssl present. > CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used > -- > > Key: HADOOP-11062 > URL: https://issues.apache.org/jira/browse/HADOOP-11062 > Project: Hadoop Common > Issue Type: Bug > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch, > HADOOP-11062.2.patch > > > there are a few testcases, cryptocodec related that require Hadoop native > code and OpenSSL. > These tests should be skipped if -Pnative is not used when running the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[
https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122883#comment-14122883
]
Charles Lamb commented on HADOOP-11062:
---
Hi [~asuresh],
I was under the impression from [~tucu00] that if -Pnative is not specified,
then some sort of helpful warning should be written to the log. I applied your
patch and ran TestCryptoCodec without the native profile using
{quote}
mvn test -Dtest=TestCryptoCodec
{quote}
The surefire report only shows that the tests in TCC were skipped and there's
no corresponding output.
{quote}
---
Test set: org.apache.hadoop.crypto.TestCryptoCodec
---
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.268 sec - in
org.apache.hadoop.crypto.TestCryptoCodec
{quote}
Actually, I may be doing something wrong, but running with the native profile
{quote}
mvn -Pnative test -Dtest=TestCryptoCodec
{quote}
produces the same surefire output (i.e. it skips both tests).
If the only requirement is that we skip the tests cleanly, then this patch is
fine.
Other tests like TestCryptoStreamsWithOpensslAesCtrCryptoCodec,
TestOpensslSecureRandom, and TestCryptoStreamsForLocalFS generate this warning
{quote}
2014-09-05 08:11:56,484 WARN util.NativeCodeLoader
(NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
{quote}
I don't know if that's a sufficient warning or not.
> CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
> --
>
> Key: HADOOP-11062
> URL: https://issues.apache.org/jira/browse/HADOOP-11062
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 2.6.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
> Attachments: HADOOP-11062.1.patch, HADOOP-11062.1.patch
>
>
> there are a few testcases, cryptocodec related that require Hadoop native
> code and OpenSSL.
> These tests should be skipped if -Pnative is not used when running the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[ https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11062: -- Assignee: Arun Suresh (was: Charles Lamb) > CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used > -- > > Key: HADOOP-11062 > URL: https://issues.apache.org/jira/browse/HADOOP-11062 > Project: Hadoop Common > Issue Type: Bug > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > > there are a few testcases, cryptocodec related that require Hadoop native > code and OpenSSL. > These tests should be skipped if -Pnative is not used when running the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-11062) CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used
[ https://issues.apache.org/jira/browse/HADOOP-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HADOOP-11062: - Assignee: Charles Lamb (was: Andrew Wang) > CryptoCodec testcases requiring OpenSSL should be run only if -Pnative is used > -- > > Key: HADOOP-11062 > URL: https://issues.apache.org/jira/browse/HADOOP-11062 > Project: Hadoop Common > Issue Type: Bug > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > > there are a few testcases, cryptocodec related that require Hadoop native > code and OpenSSL. > These tests should be skipped if -Pnative is not used when running the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
[ https://issues.apache.org/jira/browse/HADOOP-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-11026 stopped by Charles Lamb. > add FileSystem contract specification for FSDataInputStream and > FSDataOutputStream#isEncrypted > -- > > Key: HADOOP-11026 > URL: https://issues.apache.org/jira/browse/HADOOP-11026 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, test >Affects Versions: 3.0.0, 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11026-prelim.001.patch, HADOOP-11026.001.patch > > > Following on to HDFS-6843, the contract specification for FSDataInputStream > and FSDataOutputStream needs to be updated to reflect the addition of > isEncrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
[ https://issues.apache.org/jira/browse/HADOOP-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11026: -- Attachment: HADOOP-11026.001.patch This patch adds the test case to AbstractConcreteOpenTest, and sections on FSDataInput/OutputStream.isEncrypted() to filesystem.md and fsdatainputstream.md. > add FileSystem contract specification for FSDataInputStream and > FSDataOutputStream#isEncrypted > -- > > Key: HADOOP-11026 > URL: https://issues.apache.org/jira/browse/HADOOP-11026 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, test >Affects Versions: 3.0.0, 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11026-prelim.001.patch, HADOOP-11026.001.patch > > > Following on to HDFS-6843, the contract specification for FSDataInputStream > and FSDataOutputStream needs to be updated to reflect the addition of > isEncrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
[ https://issues.apache.org/jira/browse/HADOOP-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-11026 started by Charles Lamb. > add FileSystem contract specification for FSDataInputStream and > FSDataOutputStream#isEncrypted > -- > > Key: HADOOP-11026 > URL: https://issues.apache.org/jira/browse/HADOOP-11026 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, test >Affects Versions: 3.0.0, 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11026-prelim.001.patch > > > Following on to HDFS-6843, the contract specification for FSDataInputStream > and FSDataOutputStream needs to be updated to reflect the addition of > isEncrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
[ https://issues.apache.org/jira/browse/HADOOP-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-11026: -- Attachment: HADOOP-11026-prelim.001.patch Here are some preliminary diffs for the FSDataInputStream#isEncrypted doc and the AbstractConcreteOpenTest update. There is no equivalent to fsdatainputstream.md for the output side. The closest is filesystem.md. Let me know where you think the FSDataOutputStream doc should go. > add FileSystem contract specification for FSDataInputStream and > FSDataOutputStream#isEncrypted > -- > > Key: HADOOP-11026 > URL: https://issues.apache.org/jira/browse/HADOOP-11026 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, test >Affects Versions: 3.0.0, 2.6.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-11026-prelim.001.patch > > > Following on to HDFS-6843, the contract specification for FSDataInputStream > and FSDataOutputStream needs to be updated to reflect the addition of > isEncrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-11026) add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted
Charles Lamb created HADOOP-11026: - Summary: add FileSystem contract specification for FSDataInputStream and FSDataOutputStream#isEncrypted Key: HADOOP-11026 URL: https://issues.apache.org/jira/browse/HADOOP-11026 Project: Hadoop Common Issue Type: Bug Components: documentation, test Affects Versions: 3.0.0, 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Following on to HDFS-6843, the contract specification for FSDataInputStream and FSDataOutputStream needs to be updated to reflect the addition of isEncrypted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-11006) cp should automatically use /.reserved/raw when run by the superuser
Charles Lamb created HADOOP-11006: - Summary: cp should automatically use /.reserved/raw when run by the superuser Key: HADOOP-11006 URL: https://issues.apache.org/jira/browse/HADOOP-11006 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb On HDFS-6134, Sanjay Radia asked for cp to automatically prepend /.reserved/raw if the cp is being performed by the superuser and /.reserved/raw is supported by both the source and destination filesystems. This behavior only occurs if none of the src and target pathnames are /.reserved/raw. The -disablereservedraw flag can be used to disable this option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096358#comment-14096358 ] Charles Lamb commented on HADOOP-10919: --- bq. Q. when you say "distcp /r/r/src /r/r/dest" are the keys read from src and preserved in the dest? Does the act of copying the keys from a /r/r/src into a /r/r/dest automatically set up a matching EZ in the destination? Yes to the first question and no to the second. Copying the keys occurs and that is almost good enough to set up a matching EZ. However, what doesn't happen is a call to createEncryptionZone so there is not an actual EZ in place on the dst. The admin is expected to have done that before the distcp. If the admin wants a parallel EZ (i.e. with the same keys, ez-key, etc.) -- and presumably they do because they're copying from /.r/r to /.r/r and preserving the keys along the way (this is my case "(a)" above) -- then it is also expected that if the dest NN is not the same as the src (likely) that the NN and the clients accessing that NN will have equal access to the KMS (presumably the same KMS is shared across src and dst). Does this make sense? > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[
https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096037#comment-14096037
]
Charles Lamb commented on HADOOP-10919:
---
I'll update the HDFS-6509 doc to reflect the bit about trashing.
{quote}
1.src subtree and dst subtree do not have EZ - easy, same as today
{quote}
Agreed.
{quote}
2.src subtree has no EZ but dest does have EZ in a portion of its subtree.
Possible outcomes
1. if user performing operation has permissions in dest EZ then the files
within the dest EZ subtree are encrypted
{quote}
Agreed.
{quote}
2.src subtree has no EZ but dest does have EZ in a portion of its subtree.
Possible outcomes
...
2. if user does not (say Admin) what do we expect to happen?
{quote}
The behavior should be the same as what happens today: user (the admin) gets a
permission violation because the admin does not have access to the target.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
1. files copied as encrypted but cannot be decryptied at the dest since
it does not have an EZ zone- useful as a backup
{quote}
/.r/r: raw files are copied to dest so dest contains encrypted (and unreadable)
files
!/.r/r: files are decrypted by distcp and copied to dst (decrypted). Files are
readable because they have been decrypted during the copy.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
...
2. files copied as encrypted and a matching EZ is created automatically.
Can an admin do this operation since he does not have access to the keys?
{quote}
I don't think that distcp can, or should, create a matching EZ automatically.
It is too hard for it to know what the intent of the copy is. Should the new ez
have the same ez-key as the src ez or a different one? Sure, we could have an
option to let the user specify that, but for the first crack I wanted to keep
it fairly simple. So, the theory is that the admin creates the empty EZ before
performing the distcp. The admin can either set up the EZ with the same ez-key
as the src ez (call this "(a)" below, or the dest can have a different ez-key
than the src (call this "(b)" below. After the ez is created, then distcp will
try to maintain the files as encrypted. In either of those scenarios, there are
a couple of cases:
distcp with /.r/r: (a) works ok because the EDEKs for each file are copied from
src to dst. (b) does not work because when the files are opened in the dest
hierarchy, the EDEKs will be decrypted with the new ez-key(dst) and that won't
work. This could be made to work by having the KMS decrypt the EDEKs and
re-encrypt them with the new ez-key(dst), but it would assume that the distcp
invoker had proper credentials with the KMS for the keys. So in general this
scenario is only useful when the src-ez and the dst-ez have been setup with the
same ez-key. There are other issues with this that are discussed under
HDFS-6134, such as different key lengths, etc.
distcp with no /.r/r: Both of (a) and (b) work ok as long as the invoker has
access to the files that are being copied. distcp decrypts the files on read
and they get re-encrypted on write. This is pretty much the same as today.
{quote}
3.src subtree has EZ but dest does not. Possible outcomes
...
3. throw an error which can be overidden by a flag in which case the
files are decryoted and copied to in dest are left decrypted . This only works
if the user has permissions for decryption; admin cannot do this.
{quote}
/.r/r: The files aren't decrypted so this scenario is perfectly acceptable.
!/.r/r: As you say, the admin can't do this because they presumably don't have
access to the files on the src (and probably not on the target either). So this
scenario is about some random user doing a distcp of some subset of the tree on
their own. I think that what you're suggesting is a way of trying to keep the
user from shooting themselves in the foot by ensuring that they don't leave
unencrypted data hanging around in the dest. I can see this both ways. On the
one hand, someone has given the user access to the files and keys. They are
expected to "do the right thing" with the decrypted file contents, including
not putting it somewhere "unsafe". It is "transparent encryption" after all.
And they might actually want to leave it hanging around in unencrypted form
because (e.g.) maybe dst is on a cluster inside a SCIF and it's ok to leave the
files unencrypted.
But I think I like your suggestion that we throw an exception in this case
(user not using /.r/r, any of the source paths are in an ez, dest is not in an
ez) unless a flag is set.
{quote}
4.both src and dest have EZ at exactly the same part of the subtree.
Possible outcomes
1. If user has permission to decrypt and encrypt, then the data is copied
and encryption is redone with new keys,
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095158#comment-14095158 ] Charles Lamb commented on HADOOP-10919: --- Hi Sanjay, The trashing would be due to non-admin users having access to the raw.* xattrs via /.r/r. If they were able to corrupt the xattrs, then that would effectively trash the file. It's assumed that an hdfs admin would not (intentionally) do that. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094661#comment-14094661 ] Charles Lamb commented on HADOOP-10919: --- Hi [~sanjay.radia], bq. Is this mentioned in the distcp doc and I missed it? Yes, third para of the second page: "Only HDFS admins have access to the raw hierarchy as this will prevent regular users from trashing files in an EZ." > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093620#comment-14093620 ] Charles Lamb commented on HADOOP-10919: --- I should clarify case (1). If you are distcp'ing from the ez root or higher, then you don't need to pre-create the EZ because all of the raw.* xattrs will be preserved. Given that, I'm wondering what would the purpose be for checking that the target is an EZ? > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093611#comment-14093611 ] Charles Lamb commented on HADOOP-10919: --- Sanjay, There are three scenarios. (1) An administrator who does not have access to the keys in the KMS would use the /.reserved/raw prefix on src and dest: distcp /.reserved/raw/src /.reserved/raw/dest The /.reserved/raw is the only interface that exposes the raw.* xattrs holding the encryption metadata. This allows the raw.* xattrs to be preserved on the dest as well as to copy the files without decrypting them. This scenario assumes that an ez has been set up on dest. As you suggested, it would be a good idea to check that the dest is actually an ez. (2) A non-admin user who has access to some subset of files in an ez could use the non-/.reserved/raw prefix and copy a hierarchy from one ez to another. In that case, the raw.* xattrs from the src ez would not be preserved. This scenario assumes that the dest ez is already set up. Of course the dest files will have new keys associated with them since they'll be new copies. (3) Neither src or dst has /.reserved/raw and one or the other of src/dest is not an ez. It is not necessary to have the target also be an ez. The use case would be that the user wants to copy a subset of the ez into/out-of a non-encrypted file system. distcp without the /.reserved/raw prefix could be used for this. Does this all make sense? > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093565#comment-14093565 ] Charles Lamb commented on HADOOP-10919: --- Sanjay, I just re-read your comment and I realized that I mis-spoke. Yes, I think it would make sense. I'll open a jira for that. Thanks. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093557#comment-14093557 ] Charles Lamb commented on HADOOP-10919: --- bq. Charles, you list disadvantage for the .raw scheme where the target of a distcp is not an encrypted zone. Would it make sense for distcp to check for that and to fail the distcp? Hi Sanjay, Presently distcp requires both src and target to be either both in /.reserved/raw or neither in /.reserved/raw. I'll update the HDFS-6509 document and comments. Thanks for catching that. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HADOOP-10919. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Thanks for the review [~andrew.wang]. I've committed this to the fs-encryption branch. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10919: -- Description: Refer to the doc attached to HDFS-6509 for background. Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve extended attributes in the raw.* namespace by default whenever the src and target are in /.reserved/raw. To not preserve raw xattrs, don't specify /.reserved/raw in either the src or target. was: Refer to the doc attached to HDFS-6509 for background. Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve extended attributes in the raw.* namespace by default whenever the src and target are in /.reserved/raw. A new option to -p (preserve) which explicitly disables this copy will be added. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. To not preserve raw xattrs, don't specify > /.reserved/raw in either the src or target. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10919: -- Attachment: HADOOP-10919.002.patch Thanks for the review [~andrew.wang]. I believe the .002 patch addresses your comments above. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. A new option to -p (preserve) which explicitly > disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10919: -- Attachment: HADOOP-10919.001.patch This patch more closely matches the proposed behavior of distcp wrt raw xattrs: . There is no -pd option. . Determination of whether or not to preserve raw.* xattrs is based only on whether all of the source and target pathnames are in the /.reserved/raw hierarchy. . If any of the sources or target paths are different wrt polarity of /.r/r then an exception is thrown. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HADOOP-10919.001.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. A new option to -p (preserve) which explicitly > disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10919: -- Attachment: (was: HADOOP-10919.001.patch) > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. A new option to -p (preserve) which explicitly > disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10919: -- Attachment: HADOOP-10919.001.patch Here's a patch to implement the -pd option for cp, similar to distcp -pd. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HADOOP-10919.001.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. A new option to -p (preserve) which explicitly > disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
[ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-10919 started by Charles Lamb. > Copy command should preserve raw.* namespace extended attributes > > > Key: HADOOP-10919 > URL: https://issues.apache.org/jira/browse/HADOOP-10919 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HADOOP-10919.001.patch > > > Refer to the doc attached to HDFS-6509 for background. > Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve > extended attributes in the raw.* namespace by default whenever the src and > target are in /.reserved/raw. A new option to -p (preserve) which explicitly > disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10936) Change default KeyProvider bitlength to 128
[ https://issues.apache.org/jira/browse/HADOOP-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085333#comment-14085333 ] Charles Lamb commented on HADOOP-10936: --- Seems like the right thing. > Change default KeyProvider bitlength to 128 > --- > > Key: HADOOP-10936 > URL: https://issues.apache.org/jira/browse/HADOOP-10936 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hadoop-10936.001.patch > > > You need to download unlimited strength JCE to work with 256-bit keys. It'd > be good to change the default to 128 to avoid needing the unlimited strength > JCE, and print out the bitlength being used in places. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10886) CryptoCodec#getCodecclasses throws NPE when configurations not loaded.
[
https://issues.apache.org/jira/browse/HADOOP-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084458#comment-14084458
]
Charles Lamb commented on HADOOP-10886:
---
+1. Thanks Uma.
> CryptoCodec#getCodecclasses throws NPE when configurations not loaded.
> --
>
> Key: HADOOP-10886
> URL: https://issues.apache.org/jira/browse/HADOOP-10886
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 3.0.0
>
> Attachments: HADOOP-10886.patch
>
>
> There are some test cases which will not load the xml defaults. In this case,
> CryptoCodec#getCodecclasses will fail with NPE.
> {noformat}
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Splitter.split(Splitter.java:371)
> at
> org.apache.hadoop.crypto.CryptoCodec.getCodecClasses(CryptoCodec.java:100)
> at
> org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:54)
> at
> org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:91)
> at
> org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS.init(TestCryptoStreamsForLocalFS.java:53)
> {noformat}
> https://builds.apache.org/job/Hadoop-fs-encryption-nightly/71/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
Charles Lamb created HADOOP-10919: - Summary: Copy command should preserve raw.* namespace extended attributes Key: HADOOP-10919 URL: https://issues.apache.org/jira/browse/HADOOP-10919 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Refer to the doc attached to HDFS-6509 for background. Like distcp -p (see MAPREDUCE-6007), the copy command also needs to rpeserve extended attributes in the raw.* namespace by default whenever the src and target are in /.reserved/raw. A new option to -p (preserve) which explicitly disables this copy will be added. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10870) Failed to load OpenSSL cipher error logs on systems with old openssl versions
[
https://issues.apache.org/jira/browse/HADOOP-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069605#comment-14069605
]
Charles Lamb commented on HADOOP-10870:
---
Thanks Colin. This is an improvement.
> Failed to load OpenSSL cipher error logs on systems with old openssl versions
> -
>
> Key: HADOOP-10870
> URL: https://issues.apache.org/jira/browse/HADOOP-10870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Stephen Chu
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10870-fs-enc.001.patch
>
>
> I built Hadoop from fs-encryption branch and deployed Hadoop (without
> enabling any security confs) on a Centos 6.4 VM with an old version of
> openssl.
> {code}
> [root@schu-enc hadoop-common]# rpm -qa | grep openssl
> openssl-1.0.0-27.el6_4.2.x86_64
> openssl-devel-1.0.0-27.el6_4.2.x86_64
> {code}
> When I try to do a simple "hadoop fs -ls", I get
> {code}
> [hdfs@schu-enc hadoop-common]$ hadoop fs -ls
> 2014-07-21 19:35:14,486 ERROR [main] crypto.OpensslCipher
> (OpensslCipher.java:(87)) - Failed to load OpenSSL Cipher.
> java.lang.UnsatisfiedLinkError: Cannot find AES-CTR support, is your version
> of Openssl new enough?
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:84)
> at
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.(OpensslAesCtrCryptoCodec.java:50)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
> at org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:55)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:591)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:561)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:139)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2624)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2606)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:228)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:211)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:194)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:155)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> 2014-07-21 19:35:14,495 WARN [main] crypto.CryptoCodec
> (CryptoCodec.java:getInstance(66)) - Crypto codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
> {code}
> It would be an improvment to clean up/shorten this error log.
> hadoop checknative shows the error as well
> {code}
> [hdfs@schu-enc ~]$ hadoop checknative
> 2014-07-21 19:38:38,376 INFO [main] bzip2.Bzip2Factory
> (Bzip2Factory.java:isNativeBzip2Loaded(70)) - Successfully loaded &
> initialized native-bzip2 library system-native
> 2014-07-21 19:38:38,395 INFO [main] zlib.ZlibFactory
> (ZlibFactory.java:(49)) - Successfully loaded & initialized
> native-zlib library
> 2014-07-21 19:38:38,411 ERROR [main] crypto.OpensslCipher
> (OpensslCipher.java:(87)) - Failed to load OpenSSL Cipher.
> java.lang.UnsatisfiedLinkError: Cannot find AES-CTR support, is your version
> of Openssl new enough?
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:84)
> at
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:82)
> Native library checking:
> hadoop: true /home/hdfs/hadoop-3.0.0-SNA
[jira] [Commented] (HADOOP-10736) Add key attributes to the key shell
[ https://issues.apache.org/jira/browse/HADOOP-10736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046285#comment-14046285 ] Charles Lamb commented on HADOOP-10736: --- bq. Attributes must be in attribute=value form, or quoted like "attribute = value" IMHO, It's overkill. attribute=value is all they need to know. They only need to know one way that works. The fact that you're trimming is icing on the cake in case they botch it. > Add key attributes to the key shell > --- > > Key: HADOOP-10736 > URL: https://issues.apache.org/jira/browse/HADOOP-10736 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder >Assignee: Mike Yoder > Fix For: 3.0.0 > > Attachments: HADOOP-10736.patch > > > The recent work in HADOOP-10696 added attribute-value pairs to keys in the > KMS. Now the key shell needs to be updated to set/get these attributes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10736) Add key attributes to the key shell
[ https://issues.apache.org/jira/browse/HADOOP-10736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045318#comment-14045318 ] Charles Lamb commented on HADOOP-10736: --- KeyProvider.java: Use final on StringBuilder decl. s/append( MessageFormat/append(MessageFormat/ s/) )/))/ s/)-1)/) - 1)/ If I'm reading toString() correctly, it looks like it will be [foo=bar],[baz=quux],[k=v]. IWBNI there was a space after the , so ], [. KeyShell.java: init(): Map<> attributes could use a final decl. Ditto attrval, attr, and val. s/( /(/ and s/ )/)/ If I'm reading the code correctly, I think the usage message should be: "\nAttributes must be in attributes=value form\n", since I don't think you can really have 'attr = value' in the shell unless you are expecting three different args, right? You have this correct further down in the USAGE string. Nit: I might change the message to be: "\nAttributes and values must be specified as attr=val\n" (if you make this change, then also update the USAGE and DESC strings below). TestKeyShell.java: In the comment, should jceks provider be JCEKS provider? Yeah, I know that's a real nit... final on tmpDir decl. Ditto delArgs, listArgs, listArgsM, etc. s/( /(/ and s/ )/)/ I guess while you're in the neighborhood, you might as well convert the import static org.junit.Assert.* to a set of explicit import statics. Consider using GenericTestUtils for matching the output. > Add key attributes to the key shell > --- > > Key: HADOOP-10736 > URL: https://issues.apache.org/jira/browse/HADOOP-10736 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder >Assignee: Mike Yoder > Fix For: 3.0.0 > > Attachments: HADOOP-10736.patch > > > The recent work in HADOOP-10696 added attribute-value pairs to keys in the > KMS. Now the key shell needs to be updated to set/get these attributes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10695) KMSClientProvider should respect a configurable timeout.
[ https://issues.apache.org/jira/browse/HADOOP-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038999#comment-14038999 ] Charles Lamb commented on HADOOP-10695: --- The 1000 is an arg to the setFooTimeout() method in your patch. > KMSClientProvider should respect a configurable timeout. > > > Key: HADOOP-10695 > URL: https://issues.apache.org/jira/browse/HADOOP-10695 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Mike Yoder > Attachments: HADOOP-10695.patch > > > It'd be good if KMSClientProvider used a timeout, so it doesn't hang forever > if the KMServer is down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10713) Refactor CryptoCodec#generateSecureRandom to take a byte[]
[ https://issues.apache.org/jira/browse/HADOOP-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038261#comment-14038261 ] Charles Lamb commented on HADOOP-10713: --- +1 > Refactor CryptoCodec#generateSecureRandom to take a byte[] > -- > > Key: HADOOP-10713 > URL: https://issues.apache.org/jira/browse/HADOOP-10713 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HADOOP-10713.001.patch, HADOOP-10713.002.patch > > > Following suit with the Java Random implementations, it'd be better if we > switched CryptoCodec#generateSecureRandom to take a byte[] for parity. > Also, let's document that this method needs to be thread-safe, which is an > important consideration for CryptoCodec implementations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10695) KMSClientProvider should respect a configurable timeout.
[ https://issues.apache.org/jira/browse/HADOOP-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038105#comment-14038105 ] Charles Lamb commented on HADOOP-10695: --- I would think you would want separate params for the connect timeout and read timeout. Nits: private static final String TIMEOUT_ATTR = CONFIG_PREFIX + "timeout"; // secs private static final int defaultTimeout = 60; // seconds I'd prefer if the // comments were /* */ comments on the line above. Also, perhaps rename "defaultTimeout" to DEFAULT_TIMEOUT. Should there be constants for the 1000 (seconds/ms) conversion factors? I wouldn't be surprised if there already were some of those in the HADOOP codebase that you could use, but someone else can comment on that. > KMSClientProvider should respect a configurable timeout. > > > Key: HADOOP-10695 > URL: https://issues.apache.org/jira/browse/HADOOP-10695 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Mike Yoder > Attachments: HADOOP-10695.patch > > > It'd be good if KMSClientProvider used a timeout, so it doesn't hang forever > if the KMServer is down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10719) Add generateEncryptedKey and decryptEncryptedKey methods to KeyProvider
[ https://issues.apache.org/jira/browse/HADOOP-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036444#comment-14036444 ] Charles Lamb commented on HADOOP-10719: --- [~tucu00] In general, LGTM. When encrypting the key, what is the reason for calling SHA1PRNG.nextBytes on it? Is that adding entropy to the SecureRNG? Javadoc: is it ok to use "key material" as a noun with the indefinite article, as in "a key material". Maybe "new key material" instead of "a key material"? or "generates a byte[] of key material"? Ditto here: bq. Decrypts an encrypted key material using the bq. The generated key material is of the same length as the KeyVersion material. The generated key material is of the same length as the KeyVersion material and is encrypted using the same cipher. > Add generateEncryptedKey and decryptEncryptedKey methods to KeyProvider > --- > > Key: HADOOP-10719 > URL: https://issues.apache.org/jira/browse/HADOOP-10719 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10719.patch > > > This is a follow up on > [HDFS-6134|https://issues.apache.org/jira/browse/HDFS-6134?focusedCommentId=14036044&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14036044] > KeyProvider API should have 2 new methods: > * KeyVersion generateEncryptedKey(String keyVersionName, byte[] iv) > * KeyVersion decryptEncryptedKey(String keyVersionName, byte[] iv, KeyVersion > encryptedKey) > The implementation would do a known transformation on the IV (i.e.: xor with > 0xff the original IV). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10696) Add optional tags to KeyProvider Options and Metadata
[
https://issues.apache.org/jira/browse/HADOOP-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033246#comment-14033246
]
Charles Lamb commented on HADOOP-10696:
---
Hi Tucu,
Just a few nits.
As has been previously mentioned, there are whitespace issues. There are a
bunch of places where there are trailing whitespace in the lines that should be
removed.
Is there any reason that KeyProvider.TAGS_FIELD can't be shared with
KMSRESTConstants.TAGS_FIELD?
TestKeyProvider.java: s/String[]{"a"/String[] {"a/
> Add optional tags to KeyProvider Options and Metadata
> -
>
> Key: HADOOP-10696
> URL: https://issues.apache.org/jira/browse/HADOOP-10696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10696.patch, HADOOP-10696.patch,
> HADOOP-10696.patch
>
>
> In addition to having an optional description, KeyProvider Options and
> Metadata should support optional tags to help categorize keys.
> This would be useful for visualization purposes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[
https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016746#comment-14016746
]
Charles Lamb commented on HADOOP-10561:
---
Yi,
This looks pretty good. A few more minor nits:
protected static enum FileAttribute{
Please put a space before the {
The indentation of this:
private EnumSet preserveStatus =
+ EnumSet.noneOf(FileAttribute.class);
should be 2, not 4.
+ "(topx) (timestamps, ownership, permission, XAttr).\n" +
How about "[topx] (Timestamps, Ownership, Permission, XAttrs).\n"
and ditto here of course:
+ ^( |\t)*\(topx\) \(timestamps, ownership,
permission, XAttr\).( )*
> Copy command with preserve option should handle Xattrs
> --
>
> Key: HADOOP-10561
> URL: https://issues.apache.org/jira/browse/HADOOP-10561
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch,
> HADOOP-10561.patch
>
>
> The design docs for Xattrs stated that we handle preserve options with copy
> commands
> From doc:
> Preserve option of commands like “cp -p” shell command and “distcp -p” should
> work on XAttrs.
> In the case of source fs supports XAttrs but target fs does not support,
> XAttrs will be ignored
> with warning message
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016484#comment-14016484 ] Charles Lamb commented on HADOOP-10561: --- Hi Yi, I thought we had standardized on "XAttr" as opposed to XATTR or xattr. It doesn't matter to me which one we use, but I do want us to be consistent. In the code we've been using XAttr as a general rule, modulo any specific camel-casing that needs to be done. > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HADOOP-10561.1.patch, HADOOP-10561.1.patch, > HADOOP-10561.patch > > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > XAttrs will be ignored > with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015438#comment-14015438 ] Charles Lamb commented on HADOOP-10561: --- Hi Yi, The usage message is a little bit confusing. At first it was unclear to me that -ptopx was really -p with options for t, o, p, and x. I would if we could make this a little clearer by saying something like this instead? public static final String USAGE = "[-f] [-p | -p[t|o|p|x]] ... "; or maybe public static final String USAGE = "[-f] [-p | -p[topx]] ... "; and then in the text: + "must be a directory. Passing -p preserves status\n" + + "([topx]) (timestamps, ownership, permission, XATTR). \n" + + "If -p is specified with no , then preserves " + + "timestamps, ownership, permission. Passing -f\n" + Also, s/XATTR/XAttr/. > Copy command with preserve option should handle Xattrs > -- > > Key: HADOOP-10561 > URL: https://issues.apache.org/jira/browse/HADOOP-10561 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HADOOP-10561.patch > > > The design docs for Xattrs stated that we handle preserve options with copy > commands > From doc: > Preserve option of commands like “cp -p” shell command and “distcp -p” should > work on XAttrs. > In the case of source fs supports XAttrs but target fs does not support, > XAttrs will be ignored > with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10628) Javadoc and few code style improvement for Crypto input and output streams
[ https://issues.apache.org/jira/browse/HADOOP-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HADOOP-10628. --- Resolution: Fixed Thanks Yi, I committed this to fs-encryption. > Javadoc and few code style improvement for Crypto input and output streams > -- > > Key: HADOOP-10628 > URL: https://issues.apache.org/jira/browse/HADOOP-10628 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10628.patch > > > There are some additional comments from [~clamb] related to javadoc and few > code style on HADOOP-10603, let's fix them in this follow-on JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10635) Add a method to CryptoCodec to generate SRNs for IV
[ https://issues.apache.org/jira/browse/HADOOP-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012441#comment-14012441 ] Charles Lamb commented on HADOOP-10635: --- In general this looks good. I have a few little nits. For ternary constructs, I believe the standard is to put the ? and : at the end of the line rather than the start of the line below. So, thing ? yes : no; s/jce/JCE/ In generateSecureRandom, byte[] data could benefit from a final. It's a shame that HadoopIllegalArgumentException can't be used to wrap the GeneralSecurityException. Do you want to take the exception message from the GSE and include it in the HIAE message text to give the user some idea of what may be going on? > Add a method to CryptoCodec to generate SRNs for IV > --- > > Key: HADOOP-10635 > URL: https://issues.apache.org/jira/browse/HADOOP-10635 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Alejandro Abdelnur >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HADOOP-10635.patch > > > SRN generators are provided by crypto libraries. the CryptoCodec gives access > to a crypto library, thus it makes sense to expose the SRN generator on the > CryptoCodec API. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10628) Javadoc and few code style improvement for Crypto input and output streams
[ https://issues.apache.org/jira/browse/HADOOP-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011440#comment-14011440 ] Charles Lamb commented on HADOOP-10628: --- +1. Thanks Yi. > Javadoc and few code style improvement for Crypto input and output streams > -- > > Key: HADOOP-10628 > URL: https://issues.apache.org/jira/browse/HADOOP-10628 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10628.patch > > > There are some additional comments from [~clamb] related to javadoc and few > code style on HADOOP-10603, let's fix them in this follow-on JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[ https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007829#comment-14007829 ] Charles Lamb commented on HADOOP-10603: --- +1 > Crypto input and output streams implementing Hadoop stream interfaces > - > > Key: HADOOP-10603 > URL: https://issues.apache.org/jira/browse/HADOOP-10603 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Alejandro Abdelnur >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: CryptoInputStream.java, CryptoOutputStream.java, > HADOOP-10603.1.patch, HADOOP-10603.10.patch, HADOOP-10603.2.patch, > HADOOP-10603.3.patch, HADOOP-10603.4.patch, HADOOP-10603.5.patch, > HADOOP-10603.6.patch, HADOOP-10603.7.patch, HADOOP-10603.8.patch, > HADOOP-10603.9.patch, HADOOP-10603.patch > > > A common set of Crypto Input/Output streams. They would be used by > CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills. > Note we cannot use the JDK Cipher Input/Output streams directly because we > need to support the additional interfaces that the Hadoop FileSystem streams > implement (Seekable, PositionedReadable, ByteBufferReadable, > HasFileDescriptor, CanSetDropBehind, CanSetReadahead, > HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[ https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007827#comment-14007827 ] Charles Lamb commented on HADOOP-10603: --- Sounds good. +1 > Crypto input and output streams implementing Hadoop stream interfaces > - > > Key: HADOOP-10603 > URL: https://issues.apache.org/jira/browse/HADOOP-10603 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Alejandro Abdelnur >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: CryptoInputStream.java, CryptoOutputStream.java, > HADOOP-10603.1.patch, HADOOP-10603.10.patch, HADOOP-10603.2.patch, > HADOOP-10603.3.patch, HADOOP-10603.4.patch, HADOOP-10603.5.patch, > HADOOP-10603.6.patch, HADOOP-10603.7.patch, HADOOP-10603.8.patch, > HADOOP-10603.9.patch, HADOOP-10603.patch > > > A common set of Crypto Input/Output streams. They would be used by > CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills. > Note we cannot use the JDK Cipher Input/Output streams directly because we > need to support the additional interfaces that the Hadoop FileSystem streams > implement (Seekable, PositionedReadable, ByteBufferReadable, > HasFileDescriptor, CanSetDropBehind, CanSetReadahead, > HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[
https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007457#comment-14007457
]
Charles Lamb commented on HADOOP-10603:
---
Yi,
I'm sorry I didn't get these comments to you sooner.
In general, please add blank lines before all block comments.
AESCTRCryptoCodec.java
+public abstract class AESCTRCryptoCodec extends CryptoCodec {
+ /**
+ * For AES, the algorithm block is fixed size of 128 bits.
+ * @see http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+ */
+ private static final int AES_BLOCK_SIZE = 16;
+ /**
+ * IV is produced by combining initial IV and the counter using addition.
+ * IV length should be the same as {@link #AES_BLOCK_SIZE}
+ */
The IV is produced by adding the initial IV to the counter. IV length
should be the same as {@link #AES_BLOCK_SIZE}.
+ @Override
+ public void calculateIV(byte[] initIV, long counter, byte[] IV) {
...
+ByteBuffer buf = ByteBuffer.wrap(IV);
add a final decl.
CryptoCodec.java
+ /**
+ * Get block size of a block cipher.
Get the block size of a block cipher.
+ * For different algorithms, the block size may be different.
+ * @return int block size
@return the block size
+ * Get a {@link #org.apache.hadoop.crypto.Encryptor}.
s/Get a/Get an/
+ * @return Encryptor
@return the Encryptor
+ * @return Decryptor
@return the Decryptor
+ * This interface is only for Counter (CTR) mode. Typically calculating
+ * IV(Initialization Vector) is up to Encryptor or Decryptor, for
+ * example {@link #javax.crypto.Cipher} will maintain encryption context
+ * internally when do encryption/decryption continuously using its
+ * Cipher#update interface.
This interface is only needed for AES-CTR mode. The IV is generally
calculated by the Encryptor or Decryptor and maintained as internal
state. For example, a {@link #javax.crypto.Cipher} will maintain its
encryption context internally using the Cipher#update interface.
+ * In Hadoop, multiple nodes may read splits of a file, so decrypting of
+ * file is not continuous, even for encrypting may be not continuous. For
+ * each part, we need to calculate the counter through file position.
Encryption/Decryption is not always on the entire file. For example,
in Hadoop, a node may only decrypt a portion of a file (i.e. a
split). In these situations, the counter is derived from the file
position.
+ *
+ * Typically IV for a file position is produced by combining initial IV and
+ * the counter using any lossless operation (concatenation, addition, or
XOR).
+ * @see
http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
The IV can be calculated based on the file position by combining the
initial IV and the counter with a lossless operation (concatenation, addition,
or XOR).
@see
http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
CryptoInputStream.java
+public class CryptoInputStream extends FilterInputStream implements
+Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor,
+CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess {
Add a newline here please.
+ /**
+ * Whether underlying stream supports
s/Whether underlying/Whether the underlying/
+ /**
+ * Padding = pos%(algorithm blocksize); Padding is put into {@link
#inBuffer}
+ * before any other data goes in. The purpose of padding is to put input data
+ * at proper position.
s/put input data/put the input data/
+ @Override
+ public int read(byte[] b, int off, int len) throws IOException {
+int remaining = outBuffer.remaining();
final int remaining...
+ if (usingByteBufferRead == null) {
+if (in instanceof ByteBufferReadable) {
+ try {
+n = ((ByteBufferReadable) in).read(inBuffer);
+usingByteBufferRead = Boolean.TRUE;
+ } catch (UnsupportedOperationException e) {
+usingByteBufferRead = Boolean.FALSE;
+ }
+}
+if (!usingByteBufferRead.booleanValue()) {
+ n = readFromUnderlyingStream();
+}
+ } else {
+if (usingByteBufferRead.booleanValue()) {
+ n = ((ByteBufferReadable) in).read(inBuffer);
+} else {
+ n = readFromUnderlyingStream();
+}
+ }
For the code above, I wonder if we shouldn't maintain a reference to
the actual ByteBuffer once it is known to be ByteBufferReadable. If
the caller switches BBs, then it is possible that this could throw a
UnsupportedOperationException. So the check would be to see if the BB
was the same one that was already known to be BBReadable, and if not,
then check it again.
+ // Read data from underlying stream.
+ private int readFromUnderlyingStream() throws IOException {
+int toRead = inBuffer.remaining();
+byte[] tmp = getTmpBuf();
+int n = in.read(tmp, 0, to
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[ https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006628#comment-14006628 ] Charles Lamb commented on HADOOP-10603: --- CryptoInputStream.java: Shouldn't usingByteBufferRead be a class variable so that we don't keep checking in instanceof ByteBufferReadable everytime we call read()? > Crypto input and output streams implementing Hadoop stream interfaces > - > > Key: HADOOP-10603 > URL: https://issues.apache.org/jira/browse/HADOOP-10603 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Alejandro Abdelnur >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: CryptoInputStream.java, CryptoOutputStream.java, > HADOOP-10603.1.patch, HADOOP-10603.2.patch, HADOOP-10603.3.patch, > HADOOP-10603.4.patch, HADOOP-10603.5.patch, HADOOP-10603.6.patch, > HADOOP-10603.7.patch, HADOOP-10603.8.patch, HADOOP-10603.9.patch, > HADOOP-10603.patch > > > A common set of Crypto Input/Output streams. They would be used by > CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills. > Note we cannot use the JDK Cipher Input/Output streams directly because we > need to support the additional interfaces that the Hadoop FileSystem streams > implement (Seekable, PositionedReadable, ByteBufferReadable, > HasFileDescriptor, CanSetDropBehind, CanSetReadahead, > HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind). -- This message was sent by Atlassian JIRA (v6.2#6252)
