[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045596#comment-14045596 ] Andrew Wang commented on HADOOP-10720: -- Great, sounds good. Looking forward to the patch! > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045580#comment-14045580 ] Alejandro Abdelnur commented on HADOOP-10720: - On generateEDEK API being batch? I wouldn't make it batch, that is a client/server implementation detail. being batch would complicate things for clients as clients would have to hold on to N values and then use them as needed. I'd say let the KMS client do that holding entirely, and the NN asks one at the time. On how watermarks are determined? First, I would start simple, fixed by configuration, then i would make them variable based on demand. On punting the caching and batch to a follow up jira? sure, we could have the KMS client/server batch REST API for now and the client having a queue, which gets refilled when depleted. later we add the refill daemons on both ends and the queue on the server side. by doing this we don't have to change the REST API for the optimizations. > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045573#comment-14045573 ] Andrew Wang commented on HADOOP-10720: -- This sounds really good Tucu. We're still going to have some complexity on the NN since there's still the possibility of a synchronous call, but having the pre-populated cache on the KMSClientProvider will save us having to do it in the NN. Few q's and comments: * Are you planning to make the generateEDEK API batch? Would be more efficient at filling the queue. * How are the watermarks determined? The rate of generation will differ greatly between EZs, ideally we have enough to handle bursts of write traffic (i.e. MR output) * This client caching and batch stuff could also be punted to a later JIRA, I'm interested in seeing this committed soon so we can start using it on the fs-encryption branch. > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[
https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045560#comment-14045560
]
Alejandro Abdelnur commented on HADOOP-10720:
-
further details on how we could optimizing the extension operations in the KMS
(server/client) to reduce/eliminate calls from the NN on file create():
The {{KMSClientProvider}} would implement the {{CryptoExtensions}} interface,
thus the {{KeyProviderExtensions}} would delegate to it for generating and
decrypting EDEKs.
The KMS client will have a queue with IV&EDEK per key. If the client queue is
new/empty a call to the KMS asking for N IV&EDEKs for the key will be send, the
KMS response will have at least one IV&EDEK. the returned IV&EDEKs are put int
he queue and the first entry from the queue is returned to the KMS client
caller. The KMS client will have a daemon thread that keeps requesting IV&EDEKs
to the KMS for a key using a low and a high marks to start and stop requesting
more IV&EDEKs to the KMS. This means that most of the time the KMS client will
have IV&EDEKs locally cached for the NN to use.
On the KMS side a the same queue and replenisher daemon thread will have
another cache of IV&EDEK. On KMS client requests, typically, IV&EDEKS will not
be generated synchronously but simply drained from the queue. if the queue is
new/empty a small number of IV&EDEKs will be generated synchronously and
returned to the caller. if the queue is not empty and the client request N but
the queue has less than N, the available elements will be returned and the
daemon thread will start producing more up to the high mark.
This dual cache will significantly reduce NN->KMS calls and reduce/eliminate
the calls that happen synchronously with NN create() call.
Suggestion: look at the JDK ThreadPoolExecutor uses a similar logic using
wait/notify thus not sleeping/looping-around.
> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---
>
> Key: HADOOP-10720
> URL: https://issues.apache.org/jira/browse/HADOOP-10720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
> Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
> COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch,
> HADOOP-10720.patch, HADOOP-10720.patch
>
>
> KMS client/server should implement support for generating encrypted keys and
> decrypting them via the REST API being introduced by HADOOP-10719.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[
https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045467#comment-14045467
]
Hadoop QA commented on HADOOP-10759:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652744/HADOOP-10759.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4183//console
This message is automatically generated.
> Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
> --
>
> Key: HADOOP-10759
> URL: https://issues.apache.org/jira/browse/HADOOP-10759
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
>Affects Versions: 2.4.0
> Environment: Linux64
>Reporter: sam liu
>Priority: Minor
> Attachments: HADOOP-10759.patch
>
>
> In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there
> is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be
> removed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HADOOP-10759: - Status: Patch Available (was: Open) > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > Attachments: HADOOP-10759.patch > > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HADOOP-10759: - Status: Open (was: Patch Available) > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > Attachments: HADOOP-10759.patch > > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HADOOP-10759: - Attachment: HADOOP-10759.patch > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > Attachments: HADOOP-10759.patch > > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HADOOP-10759: - Status: Patch Available (was: Open) > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > Attachments: HADOOP-10759.patch > > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HADOOP-10759: - Priority: Minor (was: Trivial) > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10649) Override default ACL for service authorization
[
https://issues.apache.org/jira/browse/HADOOP-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045442#comment-14045442
]
Hadoop QA commented on HADOOP-10649:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652732/HADOOP-10649.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4182//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4182//console
This message is automatically generated.
> Override default ACL for service authorization
> ---
>
> Key: HADOOP-10649
> URL: https://issues.apache.org/jira/browse/HADOOP-10649
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10649.patch, HADOOP-10649.patch
>
>
> The default service authorization for a protocol is “*” and this authorizes
> everyone for the specific protocol.
> It should be possible to override the default ACL and specify a different acl
> as the default ACL value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10753) Could be better to move CommonConfigurationKeysPublic to org.apache.hadoop.common package
[ https://issues.apache.org/jira/browse/HADOOP-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045432#comment-14045432 ] Kai Zheng commented on HADOOP-10753: Jason, I can't agree with you any more. I will do that. Thanks for your suggestion! Do you think it's better to change the references in current code base to use the new API of org.apache.hadoop.conf.CommonConfigurationKeysPublic? > Could be better to move CommonConfigurationKeysPublic to > org.apache.hadoop.common package > - > > Key: HADOOP-10753 > URL: https://issues.apache.org/jira/browse/HADOOP-10753 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Kai Zheng >Priority: Minor > > As discussed in HADOOP-8943, it would be better to place > CommonConfigurationKeysPublic into a more neutral place like > org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would > help common facilities like security related codes avoid coupling with fs > stuffs. > Could anyone post your thought about this, and if confirmed, I would provide > a patch for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[
https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10693:
Attachment: HADOOP-10693.5.patch
[~cmccabe], I updated the patch for your comments.
{quote}
I ran {{TestCryptoCodec}} and it succeeded whether or not {{openssl.so}} was
installed. Can you make the test fail when {{-Drequire.openssl}} is specified
and {{OpenSSLAESCTRCryptoCodec}} is not available? You should be able to check
{{System#getProperty}} or something like that. That way we could be sure our
test was covering this on upstream Jenkins.
{quote}
I found it’s a bit inconvenient to do this. *1.* {{-Drequire.openssl}} is only
for build, we can’t get this in hadoop code. *2.* If user specify
{{-Drequire.openssl}} but there is no openssl available, then build will fail.
Thoughts? Maybe existing logic is acceptable?: if openssl native available,
then we test it; if not, we ignore it. Just like we do for {{TestCodec}}.
> Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
> --
>
> Key: HADOOP-10693
> URL: https://issues.apache.org/jira/browse/HADOOP-10693
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
> Attachments: HADOOP-10693.1.patch, HADOOP-10693.2.patch,
> HADOOP-10693.3.patch, HADOOP-10693.4.patch, HADOOP-10693.5.patch,
> HADOOP-10693.patch
>
>
> In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java
> JCE provider.
> To get high performance, the configured JCE provider should utilize native
> code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it.
>
> Considering not all hadoop user will use the provider like Diceros or able to
> get signed certificate from oracle to develop a custom provider, so this JIRA
> will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL
> directly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10649) Override default ACL for service authorization
[ https://issues.apache.org/jira/browse/HADOOP-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10649: -- Attachment: HADOOP-10649.patch Added information on the ability to change default ACL value to the Service Level Authorization document. New patch is attached. > Override default ACL for service authorization > --- > > Key: HADOOP-10649 > URL: https://issues.apache.org/jira/browse/HADOOP-10649 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10649.patch, HADOOP-10649.patch > > > The default service authorization for a protocol is “*” and this authorizes > everyone for the specific protocol. > It should be possible to override the default ACL and specify a different acl > as the default ACL value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
sam liu created HADOOP-10759: Summary: Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh Key: HADOOP-10759 URL: https://issues.apache.org/jira/browse/HADOOP-10759 Project: Hadoop Common Issue Type: Bug Components: bin Affects Versions: 2.4.0 Environment: Linux64 Reporter: sam liu Priority: Trivial In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[
https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045387#comment-14045387
]
Yi Liu commented on HADOOP-10693:
-
Thanks [~cmccabe] for review :-). I will update the patch later.
{quote}
There are a bunch of warnings that this code generates that we should fix:
{quote}
Right, I also noticed them, actually it was caused by the fix of one previous
comment:
_{quote}
{code}
+ if (!dlsym_EVP_CipherUpdate(context, (unsigned char *)output_bytes, \
+ &output_len, (unsigned char *)input_bytes, input_len)) {
{code}
These casts could be avoided if you just made the type of {{output_bytes}} and
{{input_bytes}} unsigned.
{quote}_
OK, I will change it back.
{quote}
I ran {{TestCryptoCodec}} and it succeeded whether or not {{openssl.so}} was
installed. Can you make the test fail when {{-Drequire.openssl}} is specified
and {{OpenSSLAESCTRCryptoCodec}} is not available? You should be able to check
{{System#getProperty}} or something like that. That way we could be sure our
test was covering this on upstream Jenkins.
{quote}
Right, {{TestCryptoCodec}} checks {{OpenSSLCipher.isNativeCodeLoaded()}} to
decide whether to test OpenSSLAESCTRCryptoCodec, this is the same as in
{{TestCodec}}
Your suggestion is good, let’s assert the exception.
{quote}
A comment about naming. I find names like {{testJCEAESCTRCryptoCodec}} awfully
hard to read. Once you get 10 capital letters in a row, it just kind of mashes
together. I would prefer something like {{testJceAesCtrCryptoCodec}}, where we
capitalize only the first letter of each acronym. I think it would make sense
to rename some of this stuff in that way... what do you think?
{quote}
Agree, let’s change the name.
{quote}
Think about what happens if we manage to get {{jKey}}, but getting {{jlv}}
fails. Then we'll throw, but never call {{ReleaseByteArrayElements}} on
{{jKey}}.
{quote}
Yes, there is potential issue, let’s check them separately.
> Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
> --
>
> Key: HADOOP-10693
> URL: https://issues.apache.org/jira/browse/HADOOP-10693
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
> Attachments: HADOOP-10693.1.patch, HADOOP-10693.2.patch,
> HADOOP-10693.3.patch, HADOOP-10693.4.patch, HADOOP-10693.patch
>
>
> In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java
> JCE provider.
> To get high performance, the configured JCE provider should utilize native
> code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it.
>
> Considering not all hadoop user will use the provider like Diceros or able to
> get signed certificate from oracle to develop a custom provider, so this JIRA
> will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL
> directly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045380#comment-14045380 ] Alejandro Abdelnur commented on HADOOP-10720: - The KMS should have separate ACLs for the extension operations. > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10758) KMS: add ACLs on per key basis.
[
https://issues.apache.org/jira/browse/HADOOP-10758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045379#comment-14045379
]
Alejandro Abdelnur commented on HADOOP-10758:
-
Keys when created could have a special attribute 'key.acl.id'.
The KMS, when a user requests a key, it would assert that the user has access
to it by checking the specified ACL id. For this the KMS would have hot
reloadable ACLs from a key-acls.xml file and using Hadoop {{ACL}} class would
assert access to the key.
This could be implemented as a {{KeyProvider}} proxy class and it would
leverage HADOOP-10750 & HADOOP-10757.
> KMS: add ACLs on per key basis.
> ---
>
> Key: HADOOP-10758
> URL: https://issues.apache.org/jira/browse/HADOOP-10758
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
>
> The KMS server should enforce ACLs on per key basis.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10758) KMS: add ACLs on per key basis.
Alejandro Abdelnur created HADOOP-10758: --- Summary: KMS: add ACLs on per key basis. Key: HADOOP-10758 URL: https://issues.apache.org/jira/browse/HADOOP-10758 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh The KMS server should enforce ACLs on per key basis. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10757) KeyProvider KeyVersion should provide the key name
[
https://issues.apache.org/jira/browse/HADOOP-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045370#comment-14045370
]
Alejandro Abdelnur commented on HADOOP-10757:
-
The {{KMSClientProvider}} and the {{KMS}} REST API must handle the name
explicitly because the {{KeyProvider}} configured in the {{KMS}} may handle it
explicitly as well.
> KeyProvider KeyVersion should provide the key name
> --
>
> Key: HADOOP-10757
> URL: https://issues.apache.org/jira/browse/HADOOP-10757
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
>
> Currently the {{KeyVersion}} does not provide a way to get the key name to do
> a reverse lookup to get the metadata of the key.
> For the {{JavaKeyStoreProvider}} and the {{UserProvider}} this is not an
> issue because the key name is encoded in the key version name.
> This encoding of the key name in the key version name cannot be expected in
> all KeyProvider implementations. It is common for key management systems to
> use UUID to refer to specific key materials (KeyVersions in Hadoop parlance).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10750) KMSKeyProviderCache should be in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045372#comment-14045372
]
Alejandro Abdelnur commented on HADOOP-10750:
-
BTW, the cache should also cache key metadata entries.
> KMSKeyProviderCache should be in hadoop-common
> --
>
> Key: HADOOP-10750
> URL: https://issues.apache.org/jira/browse/HADOOP-10750
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
>
> KMS has {{KMSCacheKeyProvider}}, this class should be available in
> hadoop-common for users of {{KeyProvider}} instances to wrap them and avoid
> several, potentially expensive, key retrievals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10757) KeyProvider KeyVersion should provide the key name
[
https://issues.apache.org/jira/browse/HADOOP-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045368#comment-14045368
]
Alejandro Abdelnur commented on HADOOP-10757:
-
We could add a {{String getKeyName()}} method to the {{KeyVersion}}.
For the {{JavaKeyStoreProvider}} implementation we don't need to store the key
name, we simply decode it from the key version name on read.
> KeyProvider KeyVersion should provide the key name
> --
>
> Key: HADOOP-10757
> URL: https://issues.apache.org/jira/browse/HADOOP-10757
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
>
> Currently the {{KeyVersion}} does not provide a way to get the key name to do
> a reverse lookup to get the metadata of the key.
> For the {{JavaKeyStoreProvider}} and the {{UserProvider}} this is not an
> issue because the key name is encoded in the key version name.
> This encoding of the key name in the key version name cannot be expected in
> all KeyProvider implementations. It is common for key management systems to
> use UUID to refer to specific key materials (KeyVersions in Hadoop parlance).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10757) KeyProvider KeyVersion should provide the key name
Alejandro Abdelnur created HADOOP-10757:
---
Summary: KeyProvider KeyVersion should provide the key name
Key: HADOOP-10757
URL: https://issues.apache.org/jira/browse/HADOOP-10757
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Currently the {{KeyVersion}} does not provide a way to get the key name to do a
reverse lookup to get the metadata of the key.
For the {{JavaKeyStoreProvider}} and the {{UserProvider}} this is not an issue
because the key name is encoded in the key version name.
This encoding of the key name in the key version name cannot be expected in all
KeyProvider implementations. It is common for key management systems to use
UUID to refer to specific key materials (KeyVersions in Hadoop parlance).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10736) Add key attributes to the key shell
[ https://issues.apache.org/jira/browse/HADOOP-10736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045318#comment-14045318 ] Charles Lamb commented on HADOOP-10736: --- KeyProvider.java: Use final on StringBuilder decl. s/append( MessageFormat/append(MessageFormat/ s/) )/))/ s/)-1)/) - 1)/ If I'm reading toString() correctly, it looks like it will be [foo=bar],[baz=quux],[k=v]. IWBNI there was a space after the , so ], [. KeyShell.java: init(): Map<> attributes could use a final decl. Ditto attrval, attr, and val. s/( /(/ and s/ )/)/ If I'm reading the code correctly, I think the usage message should be: "\nAttributes must be in attributes=value form\n", since I don't think you can really have 'attr = value' in the shell unless you are expecting three different args, right? You have this correct further down in the USAGE string. Nit: I might change the message to be: "\nAttributes and values must be specified as attr=val\n" (if you make this change, then also update the USAGE and DESC strings below). TestKeyShell.java: In the comment, should jceks provider be JCEKS provider? Yeah, I know that's a real nit... final on tmpDir decl. Ditto delArgs, listArgs, listArgsM, etc. s/( /(/ and s/ )/)/ I guess while you're in the neighborhood, you might as well convert the import static org.junit.Assert.* to a set of explicit import statics. Consider using GenericTestUtils for matching the output. > Add key attributes to the key shell > --- > > Key: HADOOP-10736 > URL: https://issues.apache.org/jira/browse/HADOOP-10736 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder >Assignee: Mike Yoder > Fix For: 3.0.0 > > Attachments: HADOOP-10736.patch > > > The recent work in HADOOP-10696 added attribute-value pairs to keys in the > KMS. Now the key shell needs to be updated to set/get these attributes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045306#comment-14045306 ] Hudson commented on HADOOP-10754: - SUCCESS: Integrated in Hadoop-trunk-Commit #5785 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5785/]) HADOOP-10754. Reenable several HA ZooKeeper-related tests on Windows. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605924) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElectorRealZK.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverControllerStress.java > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10754.1.patch > > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10754: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Arpit, thank you for the code review. I committed this to trunk and branch-2. > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10754.1.patch > > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10719) Add generateEncryptedKey and decryptEncryptedKey methods to KeyProvider
[ https://issues.apache.org/jira/browse/HADOOP-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned HADOOP-10719: --- Assignee: Arun Suresh (was: Alejandro Abdelnur) > Add generateEncryptedKey and decryptEncryptedKey methods to KeyProvider > --- > > Key: HADOOP-10719 > URL: https://issues.apache.org/jira/browse/HADOOP-10719 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: HADOOP-10719.patch, HADOOP-10719.patch, > HADOOP-10719.patch, HADOOP-10719.patch, HADOOP-10719.patch > > > This is a follow up on > [HDFS-6134|https://issues.apache.org/jira/browse/HDFS-6134?focusedCommentId=14036044&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14036044] > KeyProvider API should have 2 new methods: > * KeyVersion generateEncryptedKey(String keyVersionName, byte[] iv) > * KeyVersion decryptEncryptedKey(String keyVersionName, byte[] iv, KeyVersion > encryptedKey) > The implementation would do a known transformation on the IV (i.e.: xor with > 0xff the original IV). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10750) KMSKeyProviderCache should be in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur reassigned HADOOP-10750:
---
Assignee: Arun Suresh (was: Alejandro Abdelnur)
> KMSKeyProviderCache should be in hadoop-common
> --
>
> Key: HADOOP-10750
> URL: https://issues.apache.org/jira/browse/HADOOP-10750
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Arun Suresh
>
> KMS has {{KMSCacheKeyProvider}}, this class should be available in
> hadoop-common for users of {{KeyProvider}} instances to wrap them and avoid
> several, potentially expensive, key retrievals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned HADOOP-10756: --- Assignee: Arun Suresh (was: Alejandro Abdelnur) > KMS audit log should consolidate successful similar requests > > > Key: HADOOP-10756 > URL: https://issues.apache.org/jira/browse/HADOOP-10756 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > > Every rejected access should be audited, but successful accesses should be > consolidated within a given amount of time if the request is from the same > user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
[ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned HADOOP-10720: --- Assignee: Arun Suresh (was: Alejandro Abdelnur) > KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API > --- > > Key: HADOOP-10720 > URL: https://issues.apache.org/jira/browse/HADOOP-10720 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Arun Suresh > Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, > COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, > HADOOP-10720.patch, HADOOP-10720.patch > > > KMS client/server should implement support for generating encrypted keys and > decrypting them via the REST API being introduced by HADOOP-10719. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045271#comment-14045271
]
Akira AJISAKA commented on HADOOP-9705:
---
Thanks Chris for reviewing!
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045264#comment-14045264 ] Alejandro Abdelnur commented on HADOOP-10756: - This could be done with a wrapper log4j appender which discards successful audit log for the same same user:key if the last audit log for such user:key was done less than X seconds/minutes ago. > KMS audit log should consolidate successful similar requests > > > Key: HADOOP-10756 > URL: https://issues.apache.org/jira/browse/HADOOP-10756 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > > Every rejected access should be audited, but successful accesses should be > consolidated within a given amount of time if the request is from the same > user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10756) KMS audit log should consolidate successful similar requests
Alejandro Abdelnur created HADOOP-10756: --- Summary: KMS audit log should consolidate successful similar requests Key: HADOOP-10756 URL: https://issues.apache.org/jira/browse/HADOOP-10756 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Every rejected access should be audited, but successful accesses should be consolidated within a given amount of time if the request is from the same user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10736) Add key attributes to the key shell
[ https://issues.apache.org/jira/browse/HADOOP-10736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045245#comment-14045245 ] Mike Yoder commented on HADOOP-10736: - [~clamb] - could you have a look at this for me before I bother [~tucu00]? Thanks, -Mike > Add key attributes to the key shell > --- > > Key: HADOOP-10736 > URL: https://issues.apache.org/jira/browse/HADOOP-10736 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Mike Yoder >Assignee: Mike Yoder > Fix For: 3.0.0 > > Attachments: HADOOP-10736.patch > > > The recent work in HADOOP-10696 added attribute-value pairs to keys in the > KMS. Now the key shell needs to be updated to set/get these attributes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10750) KMSKeyProviderCache should be in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044786#comment-14044786
]
Alejandro Abdelnur edited comment on HADOOP-10750 at 6/26/14 8:45 PM:
--
On #1, correct. Plus both expiration times should be configurable.
On #2, KMS is an obvious candidate. KeyProvider implementations that want to
manage their own caching as well. And any service using KeyProviders, i.e.
Oozie, NN, httpfs, Knox, etc. Even AMs of Yarn Jobs could benefit from it.
was (Author: tucu00):
On #1, correct. Plus both expiration times should be configurable.
On #2, KMS is an obvious candidate. KeyProvider implementations that want to
manage their own caching as well. And any service using KeyProviders, i.e.
Oozie, NN, httpfs, Knox, etc. Even AMs of Yarn Jobs could benefit for it.
> KMSKeyProviderCache should be in hadoop-common
> --
>
> Key: HADOOP-10750
> URL: https://issues.apache.org/jira/browse/HADOOP-10750
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>
> KMS has {{KMSCacheKeyProvider}}, this class should be available in
> hadoop-common for users of {{KeyProvider}} instances to wrap them and avoid
> several, potentially expensive, key retrievals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10725) Implement listStatus and getFileInfo in the native client
[ https://issues.apache.org/jira/browse/HADOOP-10725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10725: -- Attachment: HADOOP-10725-pnative.002.patch rebase on previous changes, clean up URI stuff a bit > Implement listStatus and getFileInfo in the native client > - > > Key: HADOOP-10725 > URL: https://issues.apache.org/jira/browse/HADOOP-10725 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10725-pnative.001.patch, > HADOOP-10725-pnative.002.patch > > > Implement listStatus and getFileInfo in the native client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-8943: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045046#comment-14045046 ] Brandon Li commented on HADOOP-8943: I've committed the patch. Thank you, Kai, for the contribution! > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10755) Support negative caching of user-group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045037#comment-14045037
]
Hadoop QA commented on HADOOP-10755:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652660/HADOOP-10755.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4181//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4181//console
This message is automatically generated.
> Support negative caching of user-group mapping
> --
>
> Key: HADOOP-10755
> URL: https://issues.apache.org/jira/browse/HADOOP-10755
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch,
> HADOOP-10755.002.patch, HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from
> an unknown user leads to a high-rate of group mapping calls. In one case,
> this was happening at a rate of 450 calls per second with the shell-based
> group mapping, enough to severely impact overall namenode performance and
> also leading to large amounts of log spam (prints a stack trace each time).
> Let's consider negative caching of group mapping, as well as quashing the
> rate of this log message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045016#comment-14045016 ] Arpit Agarwal commented on HADOOP-10754: +1 Verified tests pass on Windows with Zookeeper 3.4.6. I got ZK test timeouts which went away after I deleted .m2 and let maven download dependencies again. Ran the tests a few times in a loop and can no longer repro the timeouts. It could have been an unrelated issue. > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Attachments: HADOOP-10754.1.patch > > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10755) Support negative caching of user-group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044988#comment-14044988
]
Hadoop QA commented on HADOOP-10755:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652650/HADOOP-10755.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.security.TestGroupsCaching
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4180//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4180//console
This message is automatically generated.
> Support negative caching of user-group mapping
> --
>
> Key: HADOOP-10755
> URL: https://issues.apache.org/jira/browse/HADOOP-10755
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch,
> HADOOP-10755.002.patch, HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from
> an unknown user leads to a high-rate of group mapping calls. In one case,
> this was happening at a rate of 450 calls per second with the shell-based
> group mapping, enough to severely impact overall namenode performance and
> also leading to large amounts of log spam (prints a stack trace each time).
> Let's consider negative caching of group mapping, as well as quashing the
> rate of this log message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10755) Support negative caching of user-group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-10755:
---
Attachment: HADOOP-10755.002.patch
Disable negative cache in {{TestGroupsCaching#testGroupsCaching}}, as its
original logic do.
Whether we should turn this negative cache on by default is a discussable
problem here.
> Support negative caching of user-group mapping
> --
>
> Key: HADOOP-10755
> URL: https://issues.apache.org/jira/browse/HADOOP-10755
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch,
> HADOOP-10755.002.patch, HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from
> an unknown user leads to a high-rate of group mapping calls. In one case,
> this was happening at a rate of 450 calls per second with the shell-based
> group mapping, enough to severely impact overall namenode performance and
> also leading to large amounts of log spam (prints a stack trace each time).
> Let's consider negative caching of group mapping, as well as quashing the
> rate of this log message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10755) Support negative caching of user-group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044968#comment-14044968
]
Hadoop QA commented on HADOOP-10755:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652649/HADOOP-10755.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.security.TestGroupsCaching
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4179//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4179//console
This message is automatically generated.
> Support negative caching of user-group mapping
> --
>
> Key: HADOOP-10755
> URL: https://issues.apache.org/jira/browse/HADOOP-10755
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch,
> HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from
> an unknown user leads to a high-rate of group mapping calls. In one case,
> this was happening at a rate of 450 calls per second with the shell-based
> group mapping, enough to severely impact overall namenode performance and
> also leading to large amounts of log spam (prints a stack trace each time).
> Let's consider negative caching of group mapping, as well as quashing the
> rate of this log message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-10667) implement TCP connection reuse for native client
[ https://issues.apache.org/jira/browse/HADOOP-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10667. --- Resolution: Fixed Fix Version/s: HADOOP-10388 > implement TCP connection reuse for native client > > > Key: HADOOP-10667 > URL: https://issues.apache.org/jira/browse/HADOOP-10667 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: HADOOP-10388 > > Attachments: HADOOP-10667-pnative.001.patch, > HADOOP-10667-pnative.002.patch, HADOOP-10667-pnative.003.patch, > HADOOP-10667-pnative.004.patch > > > The HDFS / YARN native clients should re-use TCP connections to avoid the > overhead of the three-way handshake, similar to how the Java code does. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10755) Support negative caching of user-group mapping
[
https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044944#comment-14044944
]
Hadoop QA commented on HADOOP-10755:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652521/HDFS-5369.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4178//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4178//console
This message is automatically generated.
> Support negative caching of user-group mapping
> --
>
> Key: HADOOP-10755
> URL: https://issues.apache.org/jira/browse/HADOOP-10755
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch,
> HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from
> an unknown user leads to a high-rate of group mapping calls. In one case,
> this was happening at a rate of 450 calls per second with the shell-based
> group mapping, enough to severely impact overall namenode performance and
> also leading to large amounts of log spam (prints a stack trace each time).
> Let's consider negative caching of group mapping, as well as quashing the
> rate of this log message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10667) implement TCP connection reuse for native client
[ https://issues.apache.org/jira/browse/HADOOP-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10667: -- Attachment: HADOOP-10667-pnative.004.patch > implement TCP connection reuse for native client > > > Key: HADOOP-10667 > URL: https://issues.apache.org/jira/browse/HADOOP-10667 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10667-pnative.001.patch, > HADOOP-10667-pnative.002.patch, HADOOP-10667-pnative.003.patch, > HADOOP-10667-pnative.004.patch > > > The HDFS / YARN native clients should re-use TCP connections to avoid the > overhead of the three-way handshake, similar to how the Java code does. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[
https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044915#comment-14044915
]
Hadoop QA commented on HADOOP-10754:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652639/HADOOP-10754.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4177//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4177//console
This message is automatically generated.
> Reenable several HA ZooKeeper-related tests on Windows.
> ---
>
> Key: HADOOP-10754
> URL: https://issues.apache.org/jira/browse/HADOOP-10754
> Project: Hadoop Common
> Issue Type: Test
> Components: ha, test
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HADOOP-10754.1.patch
>
>
> Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can
> reenable several tests that had been broken on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10755) Support negative caching of user-group mapping
[ https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-10755: --- Attachment: HADOOP-10755.001.patch Fixed some typos in the comments for the unit tests. > Support negative caching of user-group mapping > -- > > Key: HADOOP-10755 > URL: https://issues.apache.org/jira/browse/HADOOP-10755 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Lei (Eddy) Xu > Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch, > HDFS-5369.000.patch > > > We've seen a situation at a couple of our customers where interactions from > an unknown user leads to a high-rate of group mapping calls. In one case, > this was happening at a rate of 450 calls per second with the shell-based > group mapping, enough to severely impact overall namenode performance and > also leading to large amounts of log spam (prints a stack trace each time). > Let's consider negative caching of group mapping, as well as quashing the > rate of this log message. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044902#comment-14044902 ] Hudson commented on HADOOP-8943: SUCCESS: Integrated in Hadoop-trunk-Commit #5783 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5783/]) HADOOP-8943. Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10667) implement TCP connection reuse for native client
[ https://issues.apache.org/jira/browse/HADOOP-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044898#comment-14044898 ] Colin Patrick McCabe commented on HADOOP-10667: --- bq. What is 'conns_created' for? Future logging purposes? Analytics? I was going to add that to the connection name, so that multiple connections to the same node could be disambiguated when debugging. Let me add that. bq. I noticed that proxies give 'calls' 'socketaddr_in' structs. The proxy struct seems to be allocated on the stack in several places. ie in ndfc.c. Seems a bit dangerous unless we know for sure call structs will be deallocated before proxies? The call is allocated inside the proxy, so it will last for as long as the proxy. > implement TCP connection reuse for native client > > > Key: HADOOP-10667 > URL: https://issues.apache.org/jira/browse/HADOOP-10667 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10667-pnative.001.patch, > HADOOP-10667-pnative.002.patch, HADOOP-10667-pnative.003.patch > > > The HDFS / YARN native clients should re-use TCP connections to avoid the > overhead of the three-way handshake, similar to how the Java code does. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10755) Support negative caching of user-group mapping
[ https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-10755: --- Attachment: HADOOP-10755.000.patch This patch changes default value of negative cache timeout to 30 seconds. Also it changes the configuration key of negative cache timeout as "negative-cache.secs". [~vinayrpet] Thank you for the comments! I have one more question: should negative cache be enabled by default? > Support negative caching of user-group mapping > -- > > Key: HADOOP-10755 > URL: https://issues.apache.org/jira/browse/HADOOP-10755 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Lei (Eddy) Xu > Attachments: HADOOP-10755.000.patch, HDFS-5369.000.patch > > > We've seen a situation at a couple of our customers where interactions from > an unknown user leads to a high-rate of group mapping calls. In one case, > this was happening at a rate of 450 calls per second with the shell-based > group mapping, enough to severely impact overall namenode performance and > also leading to large amounts of log spam (prints a stack trace each time). > Let's consider negative caching of group mapping, as well as quashing the > rate of this log message. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HADOOP-10755) Support negative caching of user-group mapping
[ https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu moved HDFS-5369 to HADOOP-10755: -- Component/s: (was: namenode) security Affects Version/s: (was: 2.2.0) 2.2.0 Key: HADOOP-10755 (was: HDFS-5369) Project: Hadoop Common (was: Hadoop HDFS) > Support negative caching of user-group mapping > -- > > Key: HADOOP-10755 > URL: https://issues.apache.org/jira/browse/HADOOP-10755 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Andrew Wang >Assignee: Lei (Eddy) Xu > Attachments: HDFS-5369.000.patch > > > We've seen a situation at a couple of our customers where interactions from > an unknown user leads to a high-rate of group mapping calls. In one case, > this was happening at a rate of 450 calls per second with the shell-based > group mapping, enough to severely impact overall namenode performance and > also leading to large amounts of log spam (prints a stack trace each time). > Let's consider negative caching of group mapping, as well as quashing the > rate of this log message. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[
https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044853#comment-14044853
]
Hadoop QA commented on HADOOP-10565:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652631/HADOOP-10565.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4176//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4176//console
This message is automatically generated.
> Support IP ranges (CIDR) in proxyuser.hosts
>
>
> Key: HADOOP-10565
> URL: https://issues.apache.org/jira/browse/HADOOP-10565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10565.patch, HADOOP-10565.patch,
> HADOOP-10565.patch, HADOOP-10565.patch, HADOOP-10565.patch
>
>
> In some use cases, there will be many hosts from which the user can
> impersonate.
> This requires specifying many ips in the XML configuration.
> It is cumbersome to specify and maintain long list of ips in proxyuser.hosts
> The problem can be solved if we enable proxyuser.hosts to accept ip ranges in
> CIDR format.
> In addition, the current ip authorization involve a liner scan of the ips and
> an attempt to do InetAddress.getByName() for each ip/host.
> It may be beneficial to group this functionality of ip authorization by
> looking up "ip addresses/host names/ip-ranges" into a separate class. This
> could be reused in other usecases which require similar functionality
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044847#comment-14044847 ] Brandon Li commented on HADOOP-8943: +1. The patch looks good to me. I agree that CommonConfigurationKeysPublic should be moved to the right package. > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[
https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044841#comment-14044841
]
Hadoop QA commented on HADOOP-10739:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652628/HADOOP-10739.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4175//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4175//console
This message is automatically generated.
> Renaming a file into a directory containing the same filename results in a
> confusing I/O error
> --
>
> Key: HADOOP-10739
> URL: https://issues.apache.org/jira/browse/HADOOP-10739
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: chang li
> Attachments: HADOOP-10739.patch
>
>
> Renaming a file to another existing filename says "File
> exists" but colliding with a file in a directory results in the cryptic
> "Input/output error".
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10754: --- Status: Patch Available (was: Open) > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 2.4.0, 3.0.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Attachments: HADOOP-10754.1.patch > > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10754: --- Attachment: HADOOP-10754.1.patch Here is the patch. I should have done this earlier as part of my HADOOP-9555 patch. (Sorry!) [~arpitagarwal], could you please code review? > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Attachments: HADOOP-10754.1.patch > > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
[ https://issues.apache.org/jira/browse/HADOOP-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10754: --- Priority: Trivial (was: Major) > Reenable several HA ZooKeeper-related tests on Windows. > --- > > Key: HADOOP-10754 > URL: https://issues.apache.org/jira/browse/HADOOP-10754 > Project: Hadoop Common > Issue Type: Test > Components: ha, test >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > > Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can > reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10754) Reenable several HA ZooKeeper-related tests on Windows.
Chris Nauroth created HADOOP-10754: -- Summary: Reenable several HA ZooKeeper-related tests on Windows. Key: HADOOP-10754 URL: https://issues.apache.org/jira/browse/HADOOP-10754 Project: Hadoop Common Issue Type: Test Components: ha, test Affects Versions: 2.4.0, 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Now that our version of ZooKeeper has been upgraded in HADOOP-9555, we can reenable several tests that had been broken on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[
https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044823#comment-14044823
]
Colin Patrick McCabe commented on HADOOP-10693:
---
bq. Actually this is already included in the latest patch HADOOP-10693.2.patch.
Yeah, you're right. I was looking at v1 of the patch when I made that comment.
bq. I get your point, I will handle the fallback in HADOOP-10735 I’m
thinking we can do it more simpler: If openssl native is not available, in
OpenSSLAESCTRCryptoCodec, we use javax.crypto.Cipher instead of
org.apache.hadoop.crypto.OpenSSLCipher if fallback is allowed by configuration.
JCE is always available, we don’t need a comma list, because fallback codec
should be the same algorithm and padding, it’s better not allowed user to
configure. Otherwise if user configure several codecs, different codec may be
chose in different environments, decryption may fail.
OK. Let's deal with this in HADOOP-10735, then.
There are a bunch of warnings that this code generates that we should fix:
{code}
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:
In function ‘Java_org_apache_hadoop_cry
pto_OpenSSLCipher_update’:
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:271:3:
warning: pointer targets in passi
ng argument 2 of ‘dlsym_EVP_CipherUpdate’ differ in signedness [-Wpointer-sign]
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:271:3:
note: expected ‘unsigned char *’
but argument is of type ‘char *’
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:271:3:
warning: pointer targets in passi
ng argument 4 of ‘dlsym_EVP_CipherUpdate’ differ in signedness [-Wpointer-sign]
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:271:3:
note: expected ‘const unsigned ch
ar *’ but argument is of type ‘char *’
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:
In function ‘Java_org_apache_hadoop_cry
pto_OpenSSLCipher_doFinal’:
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:314:3:
warning: pointer targets in passi
ng argument 2 of ‘dlsym_EVP_CipherFinal_ex’ differ in signedness
[-Wpointer-sign]
/home/cmccabe/hadoop3/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpenSSLCipher.c:314:3:
note: expected ‘unsigned char *’
but argument is of type ‘char *’
{code}
Seems like they are related to using {{char*}} instead of {{unsigned char*}}
bq. Add test cases TestCryptoCodec#testJCEAESCTRCryptoCodec and
TestCryptoCodec#testOpenSSLAESCTRCryptoCodec. Add TestOpenSSLCipher. We
originally had TestCryptoStreamsWithOpenSSLAESCTRCryptoCodec
Thanks.
I ran {{TestCryptoCodec}} and it succeeded whether or not {{openssl.so}} was
installed. Can you make the test fail when {{\-Drequire.openssl}} is specified
and {{OpenSSLAESCTRCryptoCodec}} is not available? You should be able to check
{{System#getProperty}} or something like that. That way we could be sure our
test was covering this on upstream Jenkins.
A comment about naming. I find names like {{testJCEAESCTRCryptoCodec}} awfully
hard to read. Once you get 10 capital letters in a row, it just kind of mashes
together. I would prefer something like {{testJceAesCtrCryptoCodec}}, where we
capitalize only the first letter of each acronym. I think it would make sense
to rename some of this stuff in that way... what do you think?
{code}
jbyte *jKey = (*env)->GetByteArrayElements(env, key, NULL);
jbyte *jIv = (*env)->GetByteArrayElements(env, iv, NULL);
if (jKey == NULL || jIv == NULL) {
THROW(env, "java/lang/InternalError", "Cannot get bytes array.");
return (jlong)0;
}
{code}
Think about what happens if we manage to get {{jKey}}, but getting {{jlv}}
fails. Then we'll throw, but never call {{ReleaseByteArrayElements}} on
{{jKey}}.
> Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
> --
>
> Key: HADOOP-10693
> URL: https://issues.apache.org/jira/browse/HADOOP-10693
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
>
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Status: Patch Available (was: In Progress) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Attachment: (was: HADOOP-10739.patch) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10565: -- Attachment: HADOOP-10565.patch Fixing a few more spacing issues. > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch, HADOOP-10565.patch, > HADOOP-10565.patch, HADOOP-10565.patch, HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if we enable proxyuser.hosts to accept ip ranges in > CIDR format. > In addition, the current ip authorization involve a liner scan of the ips and > an attempt to do InetAddress.getByName() for each ip/host. > It may be beneficial to group this functionality of ip authorization by > looking up "ip addresses/host names/ip-ranges" into a separate class. This > could be reused in other usecases which require similar functionality -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10750) KMSKeyProviderCache should be in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044786#comment-14044786
]
Alejandro Abdelnur commented on HADOOP-10750:
-
On #1, correct. Plus both expiration times should be configurable.
On #2, KMS is an obvious candidate. KeyProvider implementations that want to
manage their own caching as well. And any service using KeyProviders, i.e.
Oozie, NN, httpfs, Knox, etc. Even AMs of Yarn Jobs could benefit for it.
> KMSKeyProviderCache should be in hadoop-common
> --
>
> Key: HADOOP-10750
> URL: https://issues.apache.org/jira/browse/HADOOP-10750
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>
> KMS has {{KMSCacheKeyProvider}}, this class should be available in
> hadoop-common for users of {{KeyProvider}} instances to wrap them and avoid
> several, potentially expensive, key retrievals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Status: Patch Available (was: In Progress) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Status: In Progress (was: Patch Available) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Status: In Progress (was: Patch Available) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Attachment: HADOOP-10739.patch > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Attachment: (was: HADOOP-10739.patch) > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10739) Renaming a file into a directory containing the same filename results in a confusing I/O error
[ https://issues.apache.org/jira/browse/HADOOP-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HADOOP-10739: -- Attachment: HADOOP-10739.patch > Renaming a file into a directory containing the same filename results in a > confusing I/O error > -- > > Key: HADOOP-10739 > URL: https://issues.apache.org/jira/browse/HADOOP-10739 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.4.0 >Reporter: Jason Lowe >Assignee: chang li > Attachments: HADOOP-10739.patch > > > Renaming a file to another existing filename says "File > exists" but colliding with a file in a directory results in the cryptic > "Input/output error". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10750) KMSKeyProviderCache should be in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044776#comment-14044776
]
Larry McCay commented on HADOOP-10750:
--
Interesting...
I have a couple high-level comments/questions:
1. if this is a generic cache provider then it would make sense to remove the
KMS part of the name and any dependencies within - if there are any.
2. I'm not sure that I can envision the typical usage pattern for it. In order
for the cache to be meaningful it would have to be used across many access
attempts. This means that it would need to be an instance of the cache provider
(or a backend store of some sort) that lives across multiple usages. I can
easily see how it can be used in a server type environment such as KMS but for
general use it isn't quite as clear. Can you describe the general use that you
have in mind?
> KMSKeyProviderCache should be in hadoop-common
> --
>
> Key: HADOOP-10750
> URL: https://issues.apache.org/jira/browse/HADOOP-10750
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>
> KMS has {{KMSCacheKeyProvider}}, this class should be available in
> hadoop-common for users of {{KeyProvider}} instances to wrap them and avoid
> several, potentially expensive, key retrievals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10753) Could be better to move CommonConfigurationKeysPublic to org.apache.hadoop.common package
[
https://issues.apache.org/jira/browse/HADOOP-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044751#comment-14044751
]
Jason Lowe commented on HADOOP-10753:
-
org.apache.hadoop.conf may be a more appropriate package.
org.apache.hadoop.fs.CommonConfigurationKeysPublic is marked Public, therefore
people are free to depend upon it. We can't simply move it because that break
backwards compatibility. We may be able to get away with moving it then
leaving a trivial derivation in org.apache.hadoop.fs via something like this:
{code}
package org.apache.hadoop.fs;
@InterfaceAudience.Public
@Deprecated
public class CommonConfigurationKeysPublic extends
org.apache.hadoop.conf.CommonConfigurationKeysPublic {
}
{code}
> Could be better to move CommonConfigurationKeysPublic to
> org.apache.hadoop.common package
> -
>
> Key: HADOOP-10753
> URL: https://issues.apache.org/jira/browse/HADOOP-10753
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
>Priority: Minor
>
> As discussed in HADOOP-8943, it would be better to place
> CommonConfigurationKeysPublic into a more neutral place like
> org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would
> help common facilities like security related codes avoid coupling with fs
> stuffs.
> Could anyone post your thought about this, and if confirmed, I would provide
> a patch for it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044745#comment-14044745
]
Hudson commented on HADOOP-9705:
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/])
HADOOP-9705. FsShell cp -p does not preserve directory attibutes. Contributed
by Akira AJISAKA. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605672)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
>
[jira] [Commented] (HADOOP-10715) Remove public GraphiteSink#setWriter()
[ https://issues.apache.org/jira/browse/HADOOP-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044747#comment-14044747 ] Hudson commented on HADOOP-10715: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/]) HADOOP-10715. Remove public GraphiteSink#setWriter() (raviprak: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605646) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGraphiteMetrics.java > Remove public GraphiteSink#setWriter() > -- > > Key: HADOOP-10715 > URL: https://issues.apache.org/jira/browse/HADOOP-10715 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 2.5.0 >Reporter: Ted Yu >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10715.patch, hadoop-10715-v2.patch > > > During review of HADOOP-10660, Ravi brought up the notion of making > GraphiteSink#setWriter() private. > This JIRA is to address Ravi's comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044701#comment-14044701
]
Hudson commented on HADOOP-9705:
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/])
HADOOP-9705. FsShell cp -p does not preserve directory attibutes. Contributed
by Akira AJISAKA. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605672)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> S
[jira] [Commented] (HADOOP-10715) Remove public GraphiteSink#setWriter()
[ https://issues.apache.org/jira/browse/HADOOP-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044703#comment-14044703 ] Hudson commented on HADOOP-10715: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/]) HADOOP-10715. Remove public GraphiteSink#setWriter() (raviprak: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605646) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGraphiteMetrics.java > Remove public GraphiteSink#setWriter() > -- > > Key: HADOOP-10715 > URL: https://issues.apache.org/jira/browse/HADOOP-10715 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 2.5.0 >Reporter: Ted Yu >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10715.patch, hadoop-10715-v2.patch > > > During review of HADOOP-10660, Ravi brought up the notion of making > GraphiteSink#setWriter() private. > This JIRA is to address Ravi's comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9867) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[ https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044691#comment-14044691 ] Vinayakumar B commented on HADOOP-9867: --- I feel this case is related to thia jira also. Refer the example given by by jason in one of the above comments. > org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record > delimiters well > -- > > Key: HADOOP-9867 > URL: https://issues.apache.org/jira/browse/HADOOP-9867 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 0.20.2, 0.23.9, 2.2.0 > Environment: CDH3U2 Redhat linux 5.7 >Reporter: Kris Geusebroek >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, > HADOOP-9867.patch > > > Having defined a recorddelimiter of multiple bytes in a new InputFileFormat > sometimes has the effect of skipping records from the input. > This happens when the input splits are split off just after a > recordseparator. Starting point for the next split would be non zero and > skipFirstLine would be true. A seek into the file is done to start - 1 and > the text until the first recorddelimiter is ignored (due to the presumption > that this record is already handled by the previous maptask). Since the re > ord delimiter is multibyte the seek only got the last byte of the delimiter > into scope and its not recognized as a full delimiter. So the text is skipped > until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9867) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[ https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044681#comment-14044681 ] Rushabh S Shah commented on HADOOP-9867: Hey Vinayakaumar, Thanks for checking out the patch and providing valuable feedback. I did ran into this test case while solving this jira. I am going to file another jira for this specific test case (and a couple of more which I came across) since the test case you mentioned is not in the scope of this jira. > org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record > delimiters well > -- > > Key: HADOOP-9867 > URL: https://issues.apache.org/jira/browse/HADOOP-9867 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 0.20.2, 0.23.9, 2.2.0 > Environment: CDH3U2 Redhat linux 5.7 >Reporter: Kris Geusebroek >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, > HADOOP-9867.patch > > > Having defined a recorddelimiter of multiple bytes in a new InputFileFormat > sometimes has the effect of skipping records from the input. > This happens when the input splits are split off just after a > recordseparator. Starting point for the next split would be non zero and > skipFirstLine would be true. A seek into the file is done to start - 1 and > the text until the first recorddelimiter is ignored (due to the presumption > that this record is already handled by the previous maptask). Since the re > ord delimiter is multibyte the seek only got the last byte of the delimiter > into scope and its not recognized as a full delimiter. So the text is skipped > until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[ https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10693: Attachment: HADOOP-10693.4.patch > Implementation of AES-CTR CryptoCodec using JNI to OpenSSL > -- > > Key: HADOOP-10693 > URL: https://issues.apache.org/jira/browse/HADOOP-10693 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10693.1.patch, HADOOP-10693.2.patch, > HADOOP-10693.3.patch, HADOOP-10693.4.patch, HADOOP-10693.patch > > > In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java > JCE provider. > To get high performance, the configured JCE provider should utilize native > code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it. > > Considering not all hadoop user will use the provider like Diceros or able to > get signed certificate from oracle to develop a custom provider, so this JIRA > will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL > directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10741) A lightweight WebHDFS client library
[ https://issues.apache.org/jira/browse/HADOOP-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044609#comment-14044609 ] Larry McCay commented on HADOOP-10741: -- I'd like to add that it would be desirable for such a library to be usable through Apache Knox as well as directly to webhdfs. Incidentally, Knox has a ClientDSL that includes support for webhdfs and does not depend on common. It currently only supports HTTP Basic authentication for presenting credentials to Knox but that will need to be supplemented with SPNEGO given support for it by the gateway on the way. > A lightweight WebHDFS client library > > > Key: HADOOP-10741 > URL: https://issues.apache.org/jira/browse/HADOOP-10741 > Project: Hadoop Common > Issue Type: New Feature > Components: tools >Reporter: Tsz Wo Nicholas Sze >Assignee: Mohammad Kamrul Islam > > One of the motivations for creating WebHDFS is for applications connecting to > HDFS from outside the cluster. In order to do so, users have to either > # install Hadoop and use WebHdfsFileSsytem, or > # develop their own client using the WebHDFS REST API. > For #1, it is very difficult to manage and unnecessarily complicated for > other applications since Hadoop is not a lightweight library. For #2, it is > not easy to deal with security and handle transient errors. > Therefore, we propose adding a lightweight WebHDFS client as a separated > library which does not depend on Common and HDFS. The client can be packaged > as a standalone jar. Other applications simply add the jar to their > classpath for using it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044564#comment-14044564
]
Hudson commented on HADOOP-9705:
FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/595/])
HADOOP-9705. FsShell cp -p does not preserve directory attibutes. Contributed
by Akira AJISAKA. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605672)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCopyPreserveFlag.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> Sys
[jira] [Commented] (HADOOP-10715) Remove public GraphiteSink#setWriter()
[ https://issues.apache.org/jira/browse/HADOOP-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044566#comment-14044566 ] Hudson commented on HADOOP-10715: - FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/595/]) HADOOP-10715. Remove public GraphiteSink#setWriter() (raviprak: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605646) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGraphiteMetrics.java > Remove public GraphiteSink#setWriter() > -- > > Key: HADOOP-10715 > URL: https://issues.apache.org/jira/browse/HADOOP-10715 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 2.5.0 >Reporter: Ted Yu >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10715.patch, hadoop-10715-v2.patch > > > During review of HADOOP-10660, Ravi brought up the notion of making > GraphiteSink#setWriter() private. > This JIRA is to address Ravi's comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[
https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044554#comment-14044554
]
Hadoop QA commented on HADOOP-8943:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652593/hadoop-8943-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//console
This message is automatically generated.
> Support multiple group mapping providers
>
>
> Key: HADOOP-8943
> URL: https://issues.apache.org/jira/browse/HADOOP-8943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 2.5.0
>
> Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch,
> hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> Discussed with Natty about LdapGroupMapping, we need to improve it so that:
> 1. It's possible to do different group mapping for different
> users/principals. For example, AD user should go to LdapGroupMapping service
> for group, but service principals such as hdfs, mapred can still use the
> default one ShellBasedUnixGroupsMapping;
> 2. Multiple ADs can be supported to do LdapGroupMapping;
> 3. It's possible to configure what kind of users/principals (regarding
> domain/realm is an option) should use which group mapping service/mechanism.
> 4. It's possible to configure and combine multiple existing mapping providers
> without writing codes implementing new one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044531#comment-14044531 ] Kai Zheng commented on HADOOP-8943: --- Opened HADOOP-10753 to discuss the concern. > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10753) Could be better to move CommonConfigurationKeysPublic to org.apache.hadoop.common package
[ https://issues.apache.org/jira/browse/HADOOP-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10753: --- Description: As discussed in HADOOP-8943, it would be better to place CommonConfigurationKeysPublic into a more neutral place like org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would help common facilities like security related codes avoid coupling with fs stuffs. Could anyone post your thought about this, and if confirmed, I would provide a patch for it. was: As discussed in HADOOP-8943, it would be better to place CommonConfigurationKeysPublic into a more neutral place like org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would help common facilities like security related codes coupling with fs stuffs. Could anyone post your thought about this, and if confirmed, I would provide a patch for it. > Could be better to move CommonConfigurationKeysPublic to > org.apache.hadoop.common package > - > > Key: HADOOP-10753 > URL: https://issues.apache.org/jira/browse/HADOOP-10753 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Kai Zheng >Priority: Minor > > As discussed in HADOOP-8943, it would be better to place > CommonConfigurationKeysPublic into a more neutral place like > org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would > help common facilities like security related codes avoid coupling with fs > stuffs. > Could anyone post your thought about this, and if confirmed, I would provide > a patch for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10753) Could be better to move CommonConfigurationKeysPublic to org.apache.hadoop.common package
Kai Zheng created HADOOP-10753: -- Summary: Could be better to move CommonConfigurationKeysPublic to org.apache.hadoop.common package Key: HADOOP-10753 URL: https://issues.apache.org/jira/browse/HADOOP-10753 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor As discussed in HADOOP-8943, it would be better to place CommonConfigurationKeysPublic into a more neutral place like org.apache.hadoop.common instead of org.apache.hadoop.fs package. This would help common facilities like security related codes coupling with fs stuffs. Could anyone post your thought about this, and if confirmed, I would provide a patch for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044523#comment-14044523 ] Kai Zheng commented on HADOOP-8943: --- Brandon, I slightly updated the patch to reference the def in CommonConfigurationKeysPublic. I agree we should use the API but I would think it could be better that CommonConfigurationKeysPublic should be in the right package, like org.apache.hadoop.common instead of org.apache.hadoop.fs, which would help common facility codes like security related avoid coupling with fs stuff. For this I will open a new JIRA to document. Thanks. > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-8943: -- Attachment: hadoop-8943-v4.patch Updated the patch As Brandon suggested. > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch, > hadoop-8943-v2.patch, hadoop-8943-v3.patch, hadoop-8943-v4.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9867) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[
https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044499#comment-14044499
]
Vinayakumar B commented on HADOOP-9867:
---
Thanks [~shahrs87] for trying out the patch.
I got test failure when the input string specified in your test is as follows
with separator as "xxx" with split length as 46.
{code}String inputData = "abcxxxdefxxxghixxx"
+ "jklxxxmnoxxxpqrxxxstuxxxvw yz";{code}
Can you check again?
> org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record
> delimiters well
> --
>
> Key: HADOOP-9867
> URL: https://issues.apache.org/jira/browse/HADOOP-9867
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.2, 0.23.9, 2.2.0
> Environment: CDH3U2 Redhat linux 5.7
>Reporter: Kris Geusebroek
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch,
> HADOOP-9867.patch
>
>
> Having defined a recorddelimiter of multiple bytes in a new InputFileFormat
> sometimes has the effect of skipping records from the input.
> This happens when the input splits are split off just after a
> recordseparator. Starting point for the next split would be non zero and
> skipFirstLine would be true. A seek into the file is done to start - 1 and
> the text until the first recorddelimiter is ignored (due to the presumption
> that this record is already handled by the previous maptask). Since the re
> ord delimiter is multibyte the seek only got the last byte of the delimiter
> into scope and its not recognized as a full delimiter. So the text is skipped
> until the next delimiter (ignoring a full record!!)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10752) Add support for hardware crc on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Nevill updated HADOOP-10752: --- Attachment: jira-10752.patch > Add support for hardware crc on ARM aarch64 architecture > > > Key: HADOOP-10752 > URL: https://issues.apache.org/jira/browse/HADOOP-10752 > Project: Hadoop Common > Issue Type: Improvement > Components: native, performance >Affects Versions: trunk-win > Environment: ARM aarch64 Juno development platform >Reporter: Edward Nevill >Priority: Minor > Labels: performance > Fix For: trunk-win > > Attachments: jira-10752.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > This patch adds support for hardware crc for ARM's new 64 bit architecture. > The patch is completely conditionalized on __arch64__ > For the moment I have only done the non pipelined version as the hw I have > only has 1 crc execute unit. > Some initial benchmarks on terasort give > sw crc: 107 sec > hw crc: 103 sec > The performance improvement is quite small, but this is limited by the fact > that I am using early stage hw which is not performant. > I have also built it on x86 and I think the change is fairly safe for other > architectures because post conditionalization the src is identical on other > architectures. > This is the first patch I have submitted for Hadoop so I would welcome any > feedback and help. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10752) Add support for hardware crc on ARM aarch64 architecture
Edward Nevill created HADOOP-10752: -- Summary: Add support for hardware crc on ARM aarch64 architecture Key: HADOOP-10752 URL: https://issues.apache.org/jira/browse/HADOOP-10752 Project: Hadoop Common Issue Type: Improvement Components: native, performance Affects Versions: trunk-win Environment: ARM aarch64 Juno development platform Reporter: Edward Nevill Priority: Minor Fix For: trunk-win This patch adds support for hardware crc for ARM's new 64 bit architecture. The patch is completely conditionalized on __arch64__ For the moment I have only done the non pipelined version as the hw I have only has 1 crc execute unit. Some initial benchmarks on terasort give sw crc: 107 sec hw crc: 103 sec The performance improvement is quite small, but this is limited by the fact that I am using early stage hw which is not performant. I have also built it on x86 and I think the change is fairly safe for other architectures because post conditionalization the src is identical on other architectures. This is the first patch I have submitted for Hadoop so I would welcome any feedback and help. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[
https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1409#comment-1409
]
Hadoop QA commented on HADOOP-10565:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652568/HADOOP-10565.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4173//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4173//console
This message is automatically generated.
> Support IP ranges (CIDR) in proxyuser.hosts
>
>
> Key: HADOOP-10565
> URL: https://issues.apache.org/jira/browse/HADOOP-10565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10565.patch, HADOOP-10565.patch,
> HADOOP-10565.patch, HADOOP-10565.patch
>
>
> In some use cases, there will be many hosts from which the user can
> impersonate.
> This requires specifying many ips in the XML configuration.
> It is cumbersome to specify and maintain long list of ips in proxyuser.hosts
> The problem can be solved if we enable proxyuser.hosts to accept ip ranges in
> CIDR format.
> In addition, the current ip authorization involve a liner scan of the ips and
> an attempt to do InetAddress.getByName() for each ip/host.
> It may be beneficial to group this functionality of ip authorization by
> looking up "ip addresses/host names/ip-ranges" into a separate class. This
> could be reused in other usecases which require similar functionality
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10565: -- Attachment: HADOOP-10565.patch Attaching the patch which addresses the above comments by [~arpitagarwal] > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch, HADOOP-10565.patch, > HADOOP-10565.patch, HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if we enable proxyuser.hosts to accept ip ranges in > CIDR format. > In addition, the current ip authorization involve a liner scan of the ips and > an attempt to do InetAddress.getByName() for each ip/host. > It may be beneficial to group this functionality of ip authorization by > looking up "ip addresses/host names/ip-ranges" into a separate class. This > could be reused in other usecases which require similar functionality -- This message was sent by Atlassian JIRA (v6.2#6252)
