[jira] [Updated] (HADOOP-10583) bin/hadoop key throws NPE with no args and assorted other fixups
[ https://issues.apache.org/jira/browse/HADOOP-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10583: -- Attachment: HADOOP-10583.5.patch Forgot to fix the unit test failures in the .4 version. > bin/hadoop key throws NPE with no args and assorted other fixups > > > Key: HADOOP-10583 > URL: https://issues.apache.org/jira/browse/HADOOP-10583 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Labels: patch > Fix For: 3.0.0 > > Attachments: HADOOP-10583.1.patch, HADOOP-10583.2.patch, > HADOOP-10583.3.patch, HADOOP-10583.4.patch, HADOOP-10583.5.patch > > > bin/hadoop key throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10594) Improve Concurrency in Groups
[ https://issues.apache.org/jira/browse/HADOOP-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10594: -- Status: Patch Available (was: Open) > Improve Concurrency in Groups > - > > Key: HADOOP-10594 > URL: https://issues.apache.org/jira/browse/HADOOP-10594 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10594.patch > > > The static field GROUPS in Groups can be accessed by holding a lock only. > This object is effectively immutable after construction and hence can safely > published using a volatile field. This enables threads to access this GROUPS > object without holding lock -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10584) ActiveStandbyElector goes down if ZK quorum become unavailable
[ https://issues.apache.org/jira/browse/HADOOP-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993080#comment-13993080 ] Karthik Kambatla commented on HADOOP-10584: --- More background: We saw this when ZK became inaccessible for a few minutes. ZKFC went down and the corresponding master was transitioned to Standby. bq. You mean instead of calling fatalError() like its doing now? Yes. Or, we should have two retry modes. The retries we have today followed by a call to becomeStandby, within an outer retry-forever loop that sleeps for a shorter time between inner-loops. > ActiveStandbyElector goes down if ZK quorum become unavailable > -- > > Key: HADOOP-10584 > URL: https://issues.apache.org/jira/browse/HADOOP-10584 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Critical > > ActiveStandbyElector retries operations for a few times. If the ZK quorum > itself is down, it goes down and the daemons will have to be brought up > again. > Instead, it should log the fact that it is unable to talk to ZK, call > becomeStandby on its client, and continue to attempt connecting to ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10467) Enable proxyuser specification to support list of users in addition to list of groups.
[ https://issues.apache.org/jira/browse/HADOOP-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992827#comment-13992827 ] Hudson commented on HADOOP-10467: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/]) HADOOP-10467. Enable proxyuser specification to support list of users in addition to list of groups. (Contributed bt Benoy Antony) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593162) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyUsers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyUsers.java > Enable proxyuser specification to support list of users in addition to list > of groups. > -- > > Key: HADOOP-10467 > URL: https://issues.apache.org/jira/browse/HADOOP-10467 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10467.patch, HADOOP-10467.patch, > HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch, > HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch, HADOOP-10467.patch > > > Today , the proxy user specification supports only list of groups. In some > cases, it is useful to specify the list of users in addition to list of > groups. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10583) bin/hadoop key throws NPE with no args and assorted other fixups
[
https://issues.apache.org/jira/browse/HADOOP-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992902#comment-13992902
]
Alejandro Abdelnur commented on HADOOP-10583:
-
a few comments on the patch:
* KeyProvider.java, has a false change.
* KeyShell.java #init(String[] args) method is all manual parsing, given that
you are fixing this, wouldn’t make sense to change this to use common-cli?
* KMSClientProvider.java, toString() should be {{"KMSClientProvider[" + kmsUrl
+ "]"}}, so it is clear that we are toStringing the provider.
> bin/hadoop key throws NPE with no args and assorted other fixups
>
>
> Key: HADOOP-10583
> URL: https://issues.apache.org/jira/browse/HADOOP-10583
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Minor
> Labels: patch
> Fix For: 3.0.0
>
> Attachments: HADOOP-10583.1.patch, HADOOP-10583.2.patch
>
>
> bin/hadoop key throws NPE.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10562) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HADOOP-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992788#comment-13992788 ] Hudson commented on HADOOP-10562: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1751 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/]) YARN-2018. TestClientRMService.testTokenRenewalWrongUser fails after HADOOP-10562. (Contributed by Ming Ma) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1592783) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > Namenode exits on exception without printing stack trace in > AbstractDelegationTokenSecretManager > > > Key: HADOOP-10562 > URL: https://issues.apache.org/jira/browse/HADOOP-10562 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.2.1, 2.4.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Critical > Fix For: 3.0.0, 1.3.0, 2.5.0 > > Attachments: HADOOP-10562.1.patch, HADOOP-10562.branch-1.1.patch, > HADOOP-10562.patch > > > Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10577) Fix some minors error and compile on macosx
[ https://issues.apache.org/jira/browse/HADOOP-10577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HADOOP-10577: --- Affects Version/s: HADOOP-10388 > Fix some minors error and compile on macosx > --- > > Key: HADOOP-10577 > URL: https://issues.apache.org/jira/browse/HADOOP-10577 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: HADOOP-10577.v1.patch, HADOOP-10577.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[ https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992898#comment-13992898 ] Alejandro Abdelnur commented on HADOOP-10586: - Charles, my bad on my offline comment on this to you, Option are settable via KeyShell, the options are set in the Configuration itself instead creating an Option and setting the values. What is missing is, as you told me, setting the description. The description cannot be set via Configuration, I would suggest refactoring this to create an Option object and sett the properties that are coming via CLI and add description to it. > KeyShell doesn't allow setting Options via CLI > -- > > Key: HADOOP-10586 > URL: https://issues.apache.org/jira/browse/HADOOP-10586 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > > You should be able to set any of the Options passed to the KeyProvider via > the CLI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10585) Retry polices ignore interrupted exceptions
Daryn Sharp created HADOOP-10585:
Summary: Retry polices ignore interrupted exceptions
Key: HADOOP-10585
URL: https://issues.apache.org/jira/browse/HADOOP-10585
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}. This
prevents {{FsShell}} commands from being aborted during retries. It also
causes orphaned webhdfs DN DFSClients to keep running after the webhdfs client
closes the connection. Jetty goes into a loop constantly sending interrupts to
the handler thread. Webhdfs retries cause multiple nodes to have these
orphaned clients. The DN cannot shutdown until orphaned clients complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10564) Add username to native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995461#comment-13995461
]
Colin Patrick McCabe commented on HADOOP-10564:
---
Abe said:
bq. In hadoop-native-core/common/user.c seems like it should be ERANGE? It
seems like it's checking an upper boundary.
OK.
Binglin said:
bq. 1. it's hard to remember which field needs free(some are stack alloc, some
are heap) which didn't, could you add comments of each field's memory ownership?
I added comments to {{hrpc_proxy_init}}.
bq. 2. in patch line: 690
Fixed
bq. 3. reactor.c 71 RB_NFIND may always find nothing, based on what the RB tree
compare method's content(only pointer equal means equal). I am not familiar
with RB_tree's semantic and the header file doesn't provide any document. And
hrpc_conn_usable may be redundant cause RB_NFIND already checks those fields.
There is some documentation in {{tree.h}}:
{code}
/* Finds the first node greater than or equal to the search key */ \
attr struct type * \
name##_RB_NFIND(struct name *head, struct type *elm) \
{code}
As the comment says, {{RB_NFIND}} finds the first node greater than or equal to
the search key. This is similar to {{java.util.TreeMap.floorKey}}.
> Add username to native RPCv9 client
> ---
>
> Key: HADOOP-10564
> URL: https://issues.apache.org/jira/browse/HADOOP-10564
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: HADOOP-10388
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10564-pnative.002.patch,
> HADOOP-10564-pnative.003.patch, HADOOP-10564-pnative.004.patch,
> HADOOP-10564.001.patch
>
>
> Add the ability for the native RPCv9 client to set a username when initiating
> a connection.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10158) SPNEGO should work with multiple interfaces/SPNs.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993111#comment-13993111 ] Hudson commented on HADOOP-10158: - SUCCESS: Integrated in Hadoop-trunk-Commit #5598 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5598/]) HADOOP-10158. SPNEGO should work with multiple interfaces/SPNs. Contributed by Daryn Sharp. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593362) * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm > SPNEGO should work with multiple interfaces/SPNs. > - > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee >Assignee: Daryn Sharp >Priority: Critical > Fix For: 2.5.0 > > Attachments: HADOOP-10158-readkeytab.patch, > HADOOP-10158-readkeytab.patch, HADOOP-10158.patch, HADOOP-10158.patch, > HADOOP-10158.patch, HADOOP-10158_multiplerealms.patch, > HADOOP-10158_multiplerealms.patch, HADOOP-10158_multiplerealms.patch > > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10585) Retry polices ignore interrupted exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-10585:
-
Status: Patch Available (was: Open)
> Retry polices ignore interrupted exceptions
> ---
>
> Key: HADOOP-10585
> URL: https://issues.apache.org/jira/browse/HADOOP-10585
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10585.patch
>
>
> Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}.
> This prevents {{FsShell}} commands from being aborted during retries. It
> also causes orphaned webhdfs DN DFSClients to keep running after the webhdfs
> client closes the connection. Jetty goes into a loop constantly sending
> interrupts to the handler thread. Webhdfs retries cause multiple nodes to
> have these orphaned clients. The DN cannot shutdown until orphaned clients
> complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
[ https://issues.apache.org/jira/browse/HADOOP-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997048#comment-13997048 ] Chris Douglas commented on HADOOP-10591: (well, it pools the Compressor/Decompressor, but the intent is to pool the direct buffers) > Compression codecs must used pooled direct buffers or deallocate direct > buffers when stream is closed > - > > Key: HADOOP-10591 > URL: https://issues.apache.org/jira/browse/HADOOP-10591 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hari Shreedharan > > Currently direct buffers allocated by compression codecs like Gzip (which > allocates 2 direct buffers per instance) are not deallocated when the stream > is closed. Eventually for long running processes which create a huge number > of files, these direct buffers are left hanging till a full gc, which may or > may not happen in a reasonable amount of time - especially if the process > does not use a whole lot of heap. > Either these buffers should be pooled or they should be deallocated when the > stream is closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10572) Example NFS mount command must pass noacl as it isn't supported by the server yet
[
https://issues.apache.org/jira/browse/HADOOP-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HADOOP-10572:
Fix Version/s: 2.5.0
> Example NFS mount command must pass noacl as it isn't supported by the server
> yet
> -
>
> Key: HADOOP-10572
> URL: https://issues.apache.org/jira/browse/HADOOP-10572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HADOOP-10572.patch
>
>
> Use of the documented default mount command results in the below server side
> log WARN event, cause the client tries to locate the ACL program (#100227):
> {code}
> 12:26:11.975 AM TRACE org.apache.hadoop.oncrpc.RpcCall
> Xid:-1114380537, messageType:RPC_CALL, rpcVersion:2, program:100227,
> version:3, procedure:0, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 12:26:11.976 AM TRACE org.apache.hadoop.oncrpc.RpcProgram
> NFS3 procedure #0
> 12:26:11.976 AM WARNorg.apache.hadoop.oncrpc.RpcProgram
> Invalid RPC call program 100227
> {code}
> The client mount command must pass {{noacl}} to avoid this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10596) HttpServer2 should apply the authentication filter to some urls instead of null
[ https://issues.apache.org/jira/browse/HADOOP-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996663#comment-13996663 ] Zhijie Shen commented on HADOOP-10596: -- bq. Introducing the secret file is a much larger change. I think a better idea is to clean it up. If you strongly object adding more configs in the scope of HttpServer2, I can definitely move it out and this is a proof-of-concept patch, though I'm not sure it's a good idea to have some configs here, others elsewhere. bq. I think there is a configuration to toggle whether NN web UI can be accessed without spnego in secure mode. Let me introduce more background about SPNEGO: 1. We can configure to use authentication or not. 2. When authentication is enabled, we can choose whether to use SPNEGO for web access. 3. When SPNEGO is enabled for web, we need to define what URLs we want to protect. The configuration you mentioned sounds like the 2nd step. The problem I found is with the 3rd step. All YARN daemons use initSpnego to initiate the authentication filter to protect web resources (not sure HDFS is using the same method or not as there's an alternative way). The problem is: 1. The authentication filter is actually protect nothing since no urls has been applied to this filter. 2. initSpnego doesn't provide enough flexibility of configuring secret file and the customized filter class programmatically. YARN daemons start the web app inside it instead of putting it into another container, such as Tomcat (It seems that web hdfs is doing this and can use web.xml to configure the filter) > HttpServer2 should apply the authentication filter to some urls instead of > null > --- > > Key: HADOOP-10596 > URL: https://issues.apache.org/jira/browse/HADOOP-10596 > Project: Hadoop Common > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: HADOOP-10596.1.patch > > > HttpServer2 should apply the authentication filter to some urls instead of > null. In addition, it should be more flexible for users to configure SPNEGO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10602) Documentation has broken "Go Back" hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996765#comment-13996765 ] Chris Nauroth commented on HADOOP-10602: I found occurrences of this in docs for multiple sub-projects, at least HDFS and MapReduce, so I'm filing the jira in Common for now. > Documentation has broken "Go Back" hyperlinks. > -- > > Key: HADOOP-10602 > URL: https://issues.apache.org/jira/browse/HADOOP-10602 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Priority: Trivial > Labels: newbie > > Multiple pages of our documentation have "Go Back" links that are broken, > because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995284#comment-13995284 ] Benoy Antony commented on HADOOP-10566: --- Doing a resend since the mail servers were down last week: [~wheat9], [~arpitagarwal] , Could one of you please review and commit this patch ? This is already reviewed by [~daryn]. His comments are addressed. It already passed Jenkins. The earlier comments related to commits seem to be some error. Looks like, it was due to confusion with HADOOP-10556. > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch, HADOOP-10566.patch, > HADOOP-10566.patch, HADOOP-10566.patch, HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10605) CryptoFileSystem decorator documentation
Alejandro Abdelnur created HADOOP-10605: --- Summary: CryptoFileSystem decorator documentation Key: HADOOP-10605 URL: https://issues.apache.org/jira/browse/HADOOP-10605 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Alejandro Abdelnur Assignee: Yi Liu Documentation explaining how the Crypto filesystem works and how it is configured. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
[ https://issues.apache.org/jira/browse/HADOOP-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997046#comment-13997046 ] Chris Douglas commented on HADOOP-10591: The [CodecPool|https://hadoop.apache.org/docs/r2.4.0/api/org/apache/hadoop/io/compress/CodecPool.html] class implements pooling of direct buffers for compression codecs. > Compression codecs must used pooled direct buffers or deallocate direct > buffers when stream is closed > - > > Key: HADOOP-10591 > URL: https://issues.apache.org/jira/browse/HADOOP-10591 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hari Shreedharan > > Currently direct buffers allocated by compression codecs like Gzip (which > allocates 2 direct buffers per instance) are not deallocated when the stream > is closed. Eventually for long running processes which create a huge number > of files, these direct buffers are left hanging till a full gc, which may or > may not happen in a reasonable amount of time - especially if the process > does not use a whole lot of heap. > Either these buffers should be pooled or they should be deallocated when the > stream is closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10155) Hadoop-crypto which includes native cipher implementation.
[ https://issues.apache.org/jira/browse/HADOOP-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10155: Issue Type: Task (was: Sub-task) Parent: (was: HADOOP-10150) > Hadoop-crypto which includes native cipher implementation. > --- > > Key: HADOOP-10155 > URL: https://issues.apache.org/jira/browse/HADOOP-10155 > Project: Hadoop Common > Issue Type: Task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > > Native cipher is used to improve performance, when using OpenSSL and with > AES-NI enabled, Native cipher is 20x faster than Java cipher, for example > CBC/CTR mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10523) Hadoop services (such as RM, NN and JHS) throw confusing exception during token auto-cancelation
[
https://issues.apache.org/jira/browse/HADOOP-10523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997554#comment-13997554
]
Daryn Sharp commented on HADOOP-10523:
--
I see. In your scenarios, I'd say the user shouldn't be canceling tokens that
have been submitted with a job unless they are trying to pre-maturely abort the
job. I know that oozie tokens aren't cancelled which is unfortunate. I think
last year I posted a patch that would cancel after all jobs using the tokens
completed but it ran into roadblocks. I need to lookup and revisit that jira.
In the two suggested approach, I'm not sure how they would be implemented if I
understand them correctly. For #1, the RM can't really test the
validity/existence of a token w/o issuing a renew or cancel and catching the
exception. For #2, the RM still won't know that the token was externally
cancelled, and the issuing service like the NN must cache cancelled tokens and
periodically clean the cache. Due to the complexity, I'd be reluctant to
endorse the approach. I'd also be reluctant to not return errors to a client -
instead returning a token already cancelled instead of token doesn't exist
exception.
I think the better solution is for users to not cancel tokens. Tokens are
supposed to be an "invisible" implementation detail of job submission and thus
not require user manipulation. I'd suggest modifying the RM to either swallow
the cancel error on job completion, or to simply emit a single line in the log
instead of a backtrace.
> Hadoop services (such as RM, NN and JHS) throw confusing exception during
> token auto-cancelation
> -
>
> Key: HADOOP-10523
> URL: https://issues.apache.org/jira/browse/HADOOP-10523
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.3.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: 2.5.0
>
> Attachments: HADOOP-10523.1.patch
>
>
> When a user explicitly cancels the token, the system (such as RM, NN and JHS)
> also periodically tries to cancel the same token. During the second cancel
> (originated by RM/NN/JHS), Hadoop processes throw the following
> error/exception in the log file. Although the exception is harmless, it
> creates a lot of confusions and causes the dev to spend a lot of time to
> investigate.
> This JIRA is to make sure if the token is available/not cancelled before
> attempting to cancel the token and finally replace this exception with
> proper warning message.
> {noformat}
> 2014-04-15 01:41:14,686 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Token cancelation requested for identifier::
> owner=.linkedin.com@REALM, renewer=yarn, realUser=,
> issueDate=1397525405921, maxDate=1398130205921, sequenceNumber=1,
> masterKeyId=2
> 2014-04-15 01:41:14,688 WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:yarn/HOST@ (auth:KERBEROS)
> cause:org.apache.hadoop.security.token.SecretManager$InvalidToken: Token not
> found
> 2014-04-15 01:41:14,689 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 7 on 10020, call
> org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB.cancelDelegationToken
> from 172.20.128.42:2783 Call#37759 Retry#0: error:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Token not found
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Token not found
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.cancelToken(AbstractDelegationTokenSecretManager.java:436)
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.cancelDelegationToken(HistoryClientService.java:400)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.cancelDelegationToken(MRClientProtocolPBServiceImpl.java:286)
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:301)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
> {noformat}
--
This mess
[jira] [Resolved] (HADOOP-10153) Define Crypto policy interfaces and provide its default implementation.
[ https://issues.apache.org/jira/browse/HADOOP-10153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HADOOP-10153. - Resolution: Won't Fix > Define Crypto policy interfaces and provide its default implementation. > --- > > Key: HADOOP-10153 > URL: https://issues.apache.org/jira/browse/HADOOP-10153 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > > The JIRA defines crypto policy interface, developers/users can implement > their own crypto policy to decide how files/directories are encrypted. This > JIRA also includes a default implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8076) hadoop (1.x) ant build fetches ivy JAR every time
[ https://issues.apache.org/jira/browse/HADOOP-8076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved HADOOP-8076. Resolution: Won't Fix [doing self-clean up of JIRAs] seems there is no interest in this. > hadoop (1.x) ant build fetches ivy JAR every time > - > > Key: HADOOP-8076 > URL: https://issues.apache.org/jira/browse/HADOOP-8076 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 1.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-8076.patch > > > the task does a timestamp check, as ivy JAR is final release it > should use the skip if already downloaded check. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10401) ShellBasedUnixGroupsMapping#getGroups does not always return primary group first
[
https://issues.apache.org/jira/browse/HADOOP-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996792#comment-13996792
]
Colin Patrick McCabe commented on HADOOP-10401:
---
+1. thanks, Akira
> ShellBasedUnixGroupsMapping#getGroups does not always return primary group
> first
>
>
> Key: HADOOP-10401
> URL: https://issues.apache.org/jira/browse/HADOOP-10401
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.4.0
>Reporter: Colin Patrick McCabe
>Assignee: Akira AJISAKA
> Attachments: HADOOP-10401.2.patch, HADOOP-10401.patch
>
>
> {{ShellBasedUnixGroupsMapping#getGroups}} does not always return the primary
> group first. It should do this so that clients who expect it don't get the
> wrong result. We should also document that the primary group is returned
> first in the API. Note that {{JniBasedUnixGroupsMapping}} does return the
> primary group first.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10448) Support pluggable mechanism to specify proxy user settings
[ https://issues.apache.org/jira/browse/HADOOP-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998141#comment-13998141 ] Daryn Sharp commented on HADOOP-10448: -- +1 Quickly scanned it and my prior comments appear to have been addressed. Nice work! > Support pluggable mechanism to specify proxy user settings > -- > > Key: HADOOP-10448 > URL: https://issues.apache.org/jira/browse/HADOOP-10448 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.3.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch, > HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch, HADOOP-10448.patch > > > We have a requirement to support large number of superusers. (users who > impersonate as another user) > (http://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html) > Currently each superuser needs to be defined in the core-site.xml via > proxyuser settings. This will be cumbersome when there are 1000 entries. > It seems useful to have a pluggable mechanism to specify proxy user settings > with the current approach as the default. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997891#comment-13997891 ] Alejandro Abdelnur commented on HADOOP-10607: - Larry, I was referring to JDK KeyStore not Hadoop KeyProvider. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998448#comment-13998448 ] Alejandro Abdelnur commented on HADOOP-10607: - Larry, Doing a KeyStore doesn't mean you have to store things in a jks file. The KeyStore implementation will decide where to store them. After a quick look the KeyStore API (http://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html), all the methods the patch is proposing are there. And the CredentialEntry would extend the KeyStore.Key. The actual implementation would be a KeyStoreSpi and it would be bootstrapped by a JCE provider that only provides the keystore (thus it doesn't need to be a sign JAR). This is similar to what commercial products like SafeNet and RSA do for their HSM integration. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998452#comment-13998452 ] Larry McCay commented on HADOOP-10607: -- Yes, this is true - though the KeyStore API contains a lot of stuff unrelated to what we actually need. It is a perfectly valid implementation to plug in as a provider type but forcing the API on all stores seems unnecessary. SafeNet and RSA do not limit their offerings to the KeyStore API - they do provide it as a way to plugin for those that would like to use that as the integration and would be able to plugin with the JavaKeystoreProvider in this API. Others however offer REST APIs for acquiring secrets and having to wrap that access in a KeyStore implementation just doesn't feel right. Especially when you would have to stub out the unnecessary methods. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998455#comment-13998455 ] Binglin Chang commented on HADOOP-10389: bq. The point of the generated functions is to provide type safety, so you can't pass the wrong request and response types to the functions. It also makes remote procedure calls look like a local function call, which is one of the main ideas in RPC. We can keep functions, but the repeated code in these functions can be eliminated using abstraction, so as to reduce the binary code size. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10606) NodeManager cannot launch container when using RawLocalFileSystem for fs.file.impl
BoYang created HADOOP-10606:
---
Summary: NodeManager cannot launch container when using
RawLocalFileSystem for fs.file.impl
Key: HADOOP-10606
URL: https://issues.apache.org/jira/browse/HADOOP-10606
Project: Hadoop Common
Issue Type: Bug
Components: fs, io, util
Affects Versions: 2.4.0
Environment: The environment does not matter with this issue. But I
use Windows 8 64bits, Open JDK 7.0.
Reporter: BoYang
Priority: Critical
Fix For: 2.4.0
Node manager failed to launch container when I set fs.file.impl to
org.apache.hadoop.fs.RawLocalFileSystem in core-site.xml.
The log is:
WARN ContainersLauncher #11
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch
- Failed to launch container.
java.lang.ClassCastException: org.apache.hadoop.fs.RawLocalFileSystem cannot be
cast to org.apache.hadoop.fs.LocalFileSystem
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:270)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLogPathForWrite(LocalDirsHandlerService.java:307)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:185)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
The issue is in
hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\fs\LocalDirAllocator.java.
It invokes FileSystem.getLocal(), which tries to convert the FileSystem to
LocalFileSystem and will fail when FileSystem object is RawLocalFileSystem
(RawLocalFileSystem is not a sub class of LocalFileSystem).
public static LocalFileSystem getLocal(Configuration conf)
throws IOException {
return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
}
The fix for LocalDirAllocator.java seems to be invoking LocalFileSystem.get()
instead of LocalFileSystem.getLocal()?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[
https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998428#comment-13998428
]
Yi Liu commented on HADOOP-10603:
-
In HADOOP-10047, @Gopal V add a directbuffer Decompressor API to hadoop:
{quote}
With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform
all I/O operations without copying data into byte[] buffers or other buffers
which wrap over them.
This is a proposal for adding a DirectDecompressor interface to the
io.compress, to indicate codecs which want to surface the direct buffer layer
upwards.
The implementation should work with direct heap/mmap buffers and cannot assume
.array() availability.
{quote}
So we can have similar API for encryption/decryption.
In HADOOP-10591, direct buffer allocating/deallocating is discussed. We should
have same consideration.
BTW, crypto streams test in HDFS is addressed in HDFS-6405. We will also have
common test for this JIRA.
> Crypto input and output streams implementing Hadoop stream interfaces
> -
>
> Key: HADOOP-10603
> URL: https://issues.apache.org/jira/browse/HADOOP-10603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Yi Liu
> Fix For: 3.0.0
>
> Attachments: HADOOP-10603.patch
>
>
> A common set of Crypto Input/Output streams. They would be used by
> CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills.
> Note we cannot use the JDK Cipher Input/Output streams directly because we
> need to support the additional interfaces that the Hadoop FileSystem streams
> implement (Seekable, PositionedReadable, ByteBufferReadable,
> HasFileDescriptor, CanSetDropBehind, CanSetReadahead,
> HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10608) Support incremental data copy in DistCp
[ https://issues.apache.org/jira/browse/HADOOP-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-10608: --- Summary: Support incremental data copy in DistCp (was: Support appending data in DistCp) > Support incremental data copy in DistCp > --- > > Key: HADOOP-10608 > URL: https://issues.apache.org/jira/browse/HADOOP-10608 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HADOOP-10608.000.patch > > > Currently when doing distcp with -update option, for two files with the same > file names but with different file length or checksum, we overwrite the whole > file. It will be good if we can detect the case where (sourceFile = > targetFile + appended_data), and only transfer the appended data segment to > the target. This will be very useful if we're doing incremental distcp. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10153) Define Crypto policy interfaces and provide its default implementation.
[ https://issues.apache.org/jira/browse/HADOOP-10153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10153: Issue Type: Task (was: Sub-task) Parent: (was: HADOOP-10150) > Define Crypto policy interfaces and provide its default implementation. > --- > > Key: HADOOP-10153 > URL: https://issues.apache.org/jira/browse/HADOOP-10153 > Project: Hadoop Common > Issue Type: Task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > > The JIRA defines crypto policy interface, developers/users can implement > their own crypto policy to decide how files/directories are encrypted. This > JIRA also includes a default implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997356#comment-13997356 ] Binglin Chang commented on HADOOP-10389: bq. I think the performance is actually going to be pretty good I am not worried about performance, it just may cause more redundant code, wait to see some code then:) Speak of redundant code, there are too many redundant codes in xxx.call.c, is it possible to do this using functions rather than generating repeat code? bq. The rationale behind call id in general is that in some future version of the Java RPC system, we may want to allow multiple calls to be "in flight" at once I looked closely to the rpc code, looks like concurrent rpc is supported, unit test in TestRPC.testSlowRpc test this. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[
https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997906#comment-13997906
]
Alejandro Abdelnur commented on HADOOP-10586:
-
LGTM, one minor NIT, we don't need the {{description = null}} in the Option
constructor, no? If so, mind updating the patch and setting 'submit patch'.
After test-patch report we are good to go.
> KeyShell doesn't allow setting Options via CLI
> --
>
> Key: HADOOP-10586
> URL: https://issues.apache.org/jira/browse/HADOOP-10586
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
>Affects Versions: 3.0.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Minor
> Attachments: HADOOP-10586.1.patch, HADOOP-10586.1.patch,
> HADOOP-10586.3.patch
>
>
> You should be able to set any of the Options passed to the KeyProvider via
> the CLI.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998406#comment-13998406 ] Binglin Chang commented on HADOOP-10389: I think TestRPC.testSlowRpc should only use one socket based on socket reuse logic, so the responses also go through one socket. The test basically do the following: client call (id=0) -> server client call (id=1) -> server server response(id = 1) -> client client call (id=2) -> server server response(id = 2) -> client server response(id = 0) -> client So client should recognize callid in response handling logic, otherwise the response is mismatched(it will return response 1 for call 0) bq. But last time I investigated it, each TCP socket could only do one request at once. Do you mean the current native code? or java code? > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
[
https://issues.apache.org/jira/browse/HADOOP-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998011#comment-13998011
]
Colin Patrick McCabe commented on HADOOP-10591:
---
Hmm. The JIRA talks about "direct buffers allocated by compression codecs like
Gzip (which allocates 2 direct buffers per instance)."
I assume this is a reference to {{ZlibDecompressor#compressedDirectBuf}} and
{{ZlibDecompressor#uncompressedDirectBuf}}. Those are buffers inside
{{Decompressor}} objects, not buffers inside {{Codec}} objects.
*However*... {{CodecPool}} has a cache for {{Compressor}} and {{Decompressor}}
objects, but it seems to be optional, not mandatory. For example, this code in
{{SequenceFile}} is careful to use the {{Decompressor}} cache:
{code}
keyLenDecompressor = CodecPool.getDecompressor(codec);
keyLenInFilter = codec.createInputStream(keyLenBuffer,
keyLenDecompressor);
{code}
On the other hand, there are also one-argument versions of the
{{createInputStream}} functions that always create a new {{Decompressor}} (and
similar one-argument versions for {{createOutputStream}}).
What's the right resolution here? Is it just to mark the one-argument versions
as deprecated and audit HDFS and Hadoop client programs to remove usages? That
certainly seems like a good idea, if we want to cache these {{ByteBuffers}}
without adding more caching mechanisms.
> Compression codecs must used pooled direct buffers or deallocate direct
> buffers when stream is closed
> -
>
> Key: HADOOP-10591
> URL: https://issues.apache.org/jira/browse/HADOOP-10591
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Colin Patrick McCabe
>
> Currently direct buffers allocated by compression codecs like Gzip (which
> allocates 2 direct buffers per instance) are not deallocated when the stream
> is closed. Eventually for long running processes which create a huge number
> of files, these direct buffers are left hanging till a full gc, which may or
> may not happen in a reasonable amount of time - especially if the process
> does not use a whole lot of heap.
> Either these buffers should be pooled or they should be deallocated when the
> stream is closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10584) ActiveStandbyElector goes down if ZK quorum become unavailable
[ https://issues.apache.org/jira/browse/HADOOP-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-10584: -- Attachment: hadoop-10584-prelim.patch Preliminary patch that articulates what I have in mind. > ActiveStandbyElector goes down if ZK quorum become unavailable > -- > > Key: HADOOP-10584 > URL: https://issues.apache.org/jira/browse/HADOOP-10584 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Critical > Attachments: hadoop-10584-prelim.patch > > > ActiveStandbyElector retries operations for a few times. If the ZK quorum > itself is down, it goes down and the daemons will have to be brought up > again. > Instead, it should log the fact that it is unable to talk to ZK, call > becomeStandby on its client, and continue to attempt connecting to ZK. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997756#comment-13997756 ] Alejandro Abdelnur commented on HADOOP-10607: - Larry, any reason why not using the KeyStore API directly? > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[ https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10586: -- Attachment: HADOOP-10586.3.patch [~tucu00] I believe this addresses your previous comments. > KeyShell doesn't allow setting Options via CLI > -- > > Key: HADOOP-10586 > URL: https://issues.apache.org/jira/browse/HADOOP-10586 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-10586.1.patch, HADOOP-10586.1.patch, > HADOOP-10586.3.patch > > > You should be able to set any of the Options passed to the KeyProvider via > the CLI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997861#comment-13997861 ] Larry McCay commented on HADOOP-10607: -- Hi [~tucu00] - I considered this for some time and came to the following conclusions: 1. they serve similar but different purposes and consumers 2. there is no need for versioning for credentials 3. they need to be able to evolve separately 4. they should be able to converge on some shared code for the pluggable providers 5. not all KeyProviders can be used as credential providers 6. credential providers need not add the baggage of the metadata associated with keys 7. we do need to make sure that KeyProviders can be plugged in as CredentialProviders for when they can serve both purposes The biggest driver for reusing the KeyProvider API in my mind was #7 and we can address that with an adapter for when a particular KeyProvider would fit well as a credential provider as well. What do you think? > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10554) Performance: Scan metrics for 2.4 are down compared to 0.23.9
[ https://issues.apache.org/jira/browse/HADOOP-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997846#comment-13997846 ] patrick white commented on HADOOP-10554: Test results update to Description: Issue does reproduce on a small (10 node) cluster, with consistent performance difference across release lines. Numbers from the 10 node cluster actually show a larger delta than the 350 node cluster's results (>10% regression), between 2.4.0 and 0.23.9, both in runtimes and throughput. > Performance: Scan metrics for 2.4 are down compared to 0.23.9 > - > > Key: HADOOP-10554 > URL: https://issues.apache.org/jira/browse/HADOOP-10554 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: patrick white > > Performance comparison benchmarks for Scan test's runtime and throughput > metrics are slightly out of 5% tolerance in 2.x compared against 0.23. The > trend is consistent across later releases in both lines, latest release > numbers are; > Runtime: > 2.4.0.0 ->73.6 seconds (avg 5 passes) > 0.23.9.12 ->69.4 seconds (avg 5 passes) > Diff: -5.7% > Throughput: > 2.4.0.0 -> 28.67 GB/s (avg 5 passes) > 0.23.9.12 -> 30.41 GB/s (avg 5 passes) > Diff: -6.1% > Scan test is specifically measuring the average map's input read performance. > The diff is consistent when run on a larger (350 node) perf environment, we > are in process of seeing if this reproduces in a smaller cluster, using > appropriately scaled inputs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[
https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HADOOP-10603:
-
Comment: was deleted
(was: This patch is split from HADOOP-10150 with a bit modification, it’s *not*
a final patch and I’m improving it.
The {Encryptor} and {Decryptor} interfaces in the patch are similar with
{Compressor} and {Decompressor} interfaces.
There is also a {DirectDecompressor} interface having:
{code} public void decompress(ByteBuffer src, ByteBuffer dst) throws
IOException;{code}
This can avoid some bytes copy, should we define {Encryptor} and {Decryptor} in
this way instead of definition in the patch?
{CryptoFSDataOutputStream} extends and wrap {FSDataOutputStream}, and
{CryptoFSDataInputStream} extends and wrap {FSDataInputStream}. They can be
used in Hadoop FileSystem to supply encryption/decryption functionalities.
The test cases will be updated after we finalize. I have another JIRA to cover
testing crypto streams in HDFS.
Thoughts?)
> Crypto input and output streams implementing Hadoop stream interfaces
> -
>
> Key: HADOOP-10603
> URL: https://issues.apache.org/jira/browse/HADOOP-10603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Yi Liu
> Fix For: 3.0.0
>
> Attachments: HADOOP-10603.patch
>
>
> A common set of Crypto Input/Output streams. They would be used by
> CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills.
> Note we cannot use the JDK Cipher Input/Output streams directly because we
> need to support the additional interfaces that the Hadoop FileSystem streams
> implement (Seekable, PositionedReadable, ByteBufferReadable,
> HasFileDescriptor, CanSetDropBehind, CanSetReadahead,
> HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10581) TestUserGroupInformation#testGetServerSideGroups fails because groups stored in Set and ArrayList are compared
[
https://issues.apache.org/jira/browse/HADOOP-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993112#comment-13993112
]
Hudson commented on HADOOP-10581:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #5598 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5598/])
Correcting the check-in mistake for HADOOP-10581. (kihwal:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593360)
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
HADOOP-10581. TestUserGroupInformation#testGetServerSideGroups fails.
Contributed by Mit Desai. (kihwal:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593357)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
> TestUserGroupInformation#testGetServerSideGroups fails because groups stored
> in Set and ArrayList are compared
> --
>
> Key: HADOOP-10581
> URL: https://issues.apache.org/jira/browse/HADOOP-10581
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Fix For: 2.5.0
>
> Attachments: HADOOP-10581.patch, HADOOP-10581.patch,
> HADOOP-10581.patch
>
>
> The test fails on some machines that has variety of user groups.
> Initially the groups are extracted and stored in a set
> {{Set groups = new LinkedHashSet ();}}
> when the user groups are collected by calling the {{login.getGroupNames()}},
> they are stored in an array list
> {{String[] gi = login.getGroupNames();}}
> Because these groups are stored in different structure, there will be
> inconsistency in the group count. Sets have unique list of keys while array
> list emits everything they have.
> {{assertEquals(groups.size(), gi.length);}} fails when there are more than
> one groups with same name as the count in sets will be less than the
> arraylist.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HADOOP-10592) Add unit test case for net in hadoop native client
[ https://issues.apache.org/jira/browse/HADOOP-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-10592 started by Wenwu Peng. > Add unit test case for net in hadoop native client > --- > > Key: HADOOP-10592 > URL: https://issues.apache.org/jira/browse/HADOOP-10592 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Attachments: HADOOP-10592-pnative.001.patch, > HADOOP-10592-pnative.002.patch > > > Add unit test case for net.c in hadoop native client -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992828#comment-13992828 ] Hudson commented on HADOOP-10566: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/]) HADOOP-10566. Add toLowerCase support to auth_to_local rules for service name. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593105) * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch, HADOOP-10566.patch, > HADOOP-10566.patch, HADOOP-10566.patch, HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10572) Example NFS mount command must pass noacl as it isn't supported by the server yet
[
https://issues.apache.org/jira/browse/HADOOP-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998032#comment-13998032
]
Hudson commented on HADOOP-10572:
-
FAILURE: Integrated in Hadoop-Yarn-trunk #561 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/561/])
HADOOP-10572. Example NFS mount command must pass noacl as it isn't supported
by the server yet. Contributed by Harsh J. (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594289)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
> Example NFS mount command must pass noacl as it isn't supported by the server
> yet
> -
>
> Key: HADOOP-10572
> URL: https://issues.apache.org/jira/browse/HADOOP-10572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HADOOP-10572.patch
>
>
> Use of the documented default mount command results in the below server side
> log WARN event, cause the client tries to locate the ACL program (#100227):
> {code}
> 12:26:11.975 AM TRACE org.apache.hadoop.oncrpc.RpcCall
> Xid:-1114380537, messageType:RPC_CALL, rpcVersion:2, program:100227,
> version:3, procedure:0, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 12:26:11.976 AM TRACE org.apache.hadoop.oncrpc.RpcProgram
> NFS3 procedure #0
> 12:26:11.976 AM WARNorg.apache.hadoop.oncrpc.RpcProgram
> Invalid RPC call program 100227
> {code}
> The client mount command must pass {{noacl}} to avoid this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10607) Create an API to separate Credentials/Password Storage from Applications
Larry McCay created HADOOP-10607: Summary: Create an API to separate Credentials/Password Storage from Applications Key: HADOOP-10607 URL: https://issues.apache.org/jira/browse/HADOOP-10607 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Owen O'Malley Fix For: 3.0.0 As with the filesystem API, we need to provide a generic mechanism to support multiple key storage mechanisms that are potentially from third parties. An additional requirement for long term data lakes is to keep multiple versions of each key so that keys can be rolled periodically without requiring the entire data set to be re-written. Rolling keys provides containment in the event of keys being leaked. Toward that end, I propose an API that is configured using a list of URLs of KeyProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries. Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997963#comment-13997963 ] Larry McCay commented on HADOOP-10607: -- [~tucu00] - Oh, man - sorry about that. There are enterprise credential/secret servers as HSMs and other forms. Don't you believe that there would be desire for credential providers other than jks? > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10609) .gitignore should ignore .orig and .rej files
[ https://issues.apache.org/jira/browse/HADOOP-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-10609: -- Attachment: hadoop-10609.patch > .gitignore should ignore .orig and .rej files > - > > Key: HADOOP-10609 > URL: https://issues.apache.org/jira/browse/HADOOP-10609 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: hadoop-10609.patch > > > .gitignore file should ignore .orig and .rej files -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10392) Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem)
[ https://issues.apache.org/jira/browse/HADOOP-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997329#comment-13997329 ] Akira AJISAKA commented on HADOOP-10392: The test failures was reported by MAPREDUCE-5810. Attaching the same patch to re-run Jenkins. > Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) > > > Key: HADOOP-10392 > URL: https://issues.apache.org/jira/browse/HADOOP-10392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.3.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: newbie > Attachments: HADOOP-10392.2.patch, HADOOP-10392.3.patch, > HADOOP-10392.4.patch, HADOOP-10392.4.patch, HADOOP-10392.patch > > > There're some methods calling Path.makeQualified(FileSystem), which causes > javac warning. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10577) Fix some minors error and compile on macosx
[ https://issues.apache.org/jira/browse/HADOOP-10577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved HADOOP-10577. Resolution: Fixed > Fix some minors error and compile on macosx > --- > > Key: HADOOP-10577 > URL: https://issues.apache.org/jira/browse/HADOOP-10577 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: HADOOP-10577.v1.patch, HADOOP-10577.v2.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10587) Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer objects
[ https://issues.apache.org/jira/browse/HADOOP-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10587: -- Status: Patch Available (was: Open) > Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many > DataOutputBuffer objects > > > Key: HADOOP-10587 > URL: https://issues.apache.org/jira/browse/HADOOP-10587 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-10587.001.patch > > > We can use a thread-local cache in TokenIdentifier#getBytes to avoid creating > many DataOutputBuffer objects. This will reduce our memory usage (for > example, when loading edit logs), and help prevent OOMs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
[ https://issues.apache.org/jira/browse/HADOOP-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998319#comment-13998319 ] Gopal V commented on HADOOP-10591: -- [~andrew.wang]: HADOOP-10047 was a change which avoided the need to allocate direct buffers by the decompressors implementing the DirectDecompressor interface. DirectDecompressor::decompress(ByteBuffer src, ByteBuffer dst) was meant to avoid allocating objects in the decompressor object's control. That does a decompress from src into dst without an intermediate allocation or copy. Before that ORC couldn't use own the buffer pools for src/dst. The issue in this bug pre-dates HADOOP-10047. > Compression codecs must used pooled direct buffers or deallocate direct > buffers when stream is closed > - > > Key: HADOOP-10591 > URL: https://issues.apache.org/jira/browse/HADOOP-10591 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Colin Patrick McCabe > > Currently direct buffers allocated by compression codecs like Gzip (which > allocates 2 direct buffers per instance) are not deallocated when the stream > is closed. Eventually for long running processes which create a huge number > of files, these direct buffers are left hanging till a full gc, which may or > may not happen in a reasonable amount of time - especially if the process > does not use a whole lot of heap. > Either these buffers should be pooled or they should be deallocated when the > stream is closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998315#comment-13998315
]
Colin Patrick McCabe commented on HADOOP-10389:
---
bq. I am not worried about performance, it just may cause more redundant code,
wait to see some code then. Speak of redundant code, there are too many
redundant codes in xxx.call.c, is it possible to do this using functions rather
than generating repeat code?
The point of the generated functions is to provide type safety, so you can't
pass the wrong request and response types to the functions. It also makes
remote procedure calls look like a local function call, which is one of the
main ideas in RPC.
I want to post the rest of the native client code, but I am waiting for a +1 on
HADOOP-10564.
bq. I looked closely to the rpc code, looks like concurrent rpc is supported,
unit test in TestRPC.testSlowRpc test this.
I think we are talking about two different things. What I am talking about is
sending two different responses at once on the same TCP socket. What you are
talking about (and what {{TestRPC.testSlowRpc}} tests) is making two RPCs at
once from a client.) Clearly it is possible to make more than one RPC at once
from a client. Otherwise, our performance would be pretty poor. But last time
I investigated it, each TCP socket could only do one request at once.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[
https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998295#comment-13998295
]
Yi Liu commented on HADOOP-10603:
-
This patch is split from HADOOP-10150 with a bit modification, it’s *not* a
final patch and I’m improving it.
The {{Encryptor}} and {{Decryptor}} interfaces in the patch are similar with
{{Compressor}} and {{Decompressor}} interfaces.
There is also a {{DirectDecompressor}} interface having:
{code}
public void decompress(ByteBuffer src, ByteBuffer dst) throws IOException;
{code}
This can avoid some bytes copy, should we define {{Encryptor}} and
{{Decryptor}} in this way instead of definition in the patch?
{{CryptoFSDataOutputStream}} extends and wrap {{FSDataOutputStream}}, and
{{CryptoFSDataInputStream}} extends and wrap {{FSDataInputStream}}. They can be
used in Hadoop FileSystem to supply encryption/decryption functionalities.
The test cases will be updated after we finalize. I have another JIRA to cover
testing crypto streams in HDFS.
Thoughts?
> Crypto input and output streams implementing Hadoop stream interfaces
> -
>
> Key: HADOOP-10603
> URL: https://issues.apache.org/jira/browse/HADOOP-10603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Yi Liu
> Fix For: 3.0.0
>
> Attachments: HADOOP-10603.patch
>
>
> A common set of Crypto Input/Output streams. They would be used by
> CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills.
> Note we cannot use the JDK Cipher Input/Output streams directly because we
> need to support the additional interfaces that the Hadoop FileSystem streams
> implement (Seekable, PositionedReadable, ByteBufferReadable,
> HasFileDescriptor, CanSetDropBehind, CanSetReadahead,
> HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10603) Crypto input and output streams implementing Hadoop stream interfaces
[
https://issues.apache.org/jira/browse/HADOOP-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10603:
Attachment: HADOOP-10603.patch
This patch is split from HADOOP-10150 with a bit modification, it’s *not* a
final patch and I’m improving it.
The {Encryptor} and {Decryptor} interfaces in the patch are similar with
{Compressor} and {Decompressor} interfaces.
There is also a {DirectDecompressor} interface having:
{code} public void decompress(ByteBuffer src, ByteBuffer dst) throws
IOException;{code}
This can avoid some bytes copy, should we define {Encryptor} and {Decryptor} in
this way instead of definition in the patch?
{CryptoFSDataOutputStream} extends and wrap {FSDataOutputStream}, and
{CryptoFSDataInputStream} extends and wrap {FSDataInputStream}. They can be
used in Hadoop FileSystem to supply encryption/decryption functionalities.
The test cases will be updated after we finalize. I have another JIRA to cover
testing crypto streams in HDFS.
Thoughts?
> Crypto input and output streams implementing Hadoop stream interfaces
> -
>
> Key: HADOOP-10603
> URL: https://issues.apache.org/jira/browse/HADOOP-10603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Yi Liu
> Fix For: 3.0.0
>
> Attachments: HADOOP-10603.patch
>
>
> A common set of Crypto Input/Output streams. They would be used by
> CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills.
> Note we cannot use the JDK Cipher Input/Output streams directly because we
> need to support the additional interfaces that the Hadoop FileSystem streams
> implement (Seekable, PositionedReadable, ByteBufferReadable,
> HasFileDescriptor, CanSetDropBehind, CanSetReadahead,
> HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10401) ShellBasedUnixGroupsMapping#getGroups does not always return primary group first
[
https://issues.apache.org/jira/browse/HADOOP-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-10401:
--
Resolution: Fixed
Fix Version/s: 2.5.0
Target Version/s: 2.5.0
Status: Resolved (was: Patch Available)
> ShellBasedUnixGroupsMapping#getGroups does not always return primary group
> first
>
>
> Key: HADOOP-10401
> URL: https://issues.apache.org/jira/browse/HADOOP-10401
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.4.0
>Reporter: Colin Patrick McCabe
>Assignee: Akira AJISAKA
> Fix For: 2.5.0
>
> Attachments: HADOOP-10401.2.patch, HADOOP-10401.patch
>
>
> {{ShellBasedUnixGroupsMapping#getGroups}} does not always return the primary
> group first. It should do this so that clients who expect it don't get the
> wrong result. We should also document that the primary group is returned
> first in the API. Note that {{JniBasedUnixGroupsMapping}} does return the
> primary group first.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998185#comment-13998185 ] Hudson commented on HADOOP-10566: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1779 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1779/]) HADOOP-10566. Adding files missed in previous commit 1594280 (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594282) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyServers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyServers.java HADOOP-10566. Refactor proxyservers out of ProxyUsers. Contributed by Benoy Antony. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594280) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyUsers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyUsers.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch, HADOOP-10566.patch, > HADOOP-10566.patch, HADOOP-10566.patch, HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10572) Example NFS mount command must pass noacl as it isn't supported by the server yet
[
https://issues.apache.org/jira/browse/HADOOP-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998166#comment-13998166
]
Hudson commented on HADOOP-10572:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1779 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1779/])
HADOOP-10572. Example NFS mount command must pass noacl as it isn't supported
by the server yet. Contributed by Harsh J. (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594289)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
> Example NFS mount command must pass noacl as it isn't supported by the server
> yet
> -
>
> Key: HADOOP-10572
> URL: https://issues.apache.org/jira/browse/HADOOP-10572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HADOOP-10572.patch
>
>
> Use of the documented default mount command results in the below server side
> log WARN event, cause the client tries to locate the ACL program (#100227):
> {code}
> 12:26:11.975 AM TRACE org.apache.hadoop.oncrpc.RpcCall
> Xid:-1114380537, messageType:RPC_CALL, rpcVersion:2, program:100227,
> version:3, procedure:0, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 12:26:11.976 AM TRACE org.apache.hadoop.oncrpc.RpcProgram
> NFS3 procedure #0
> 12:26:11.976 AM WARNorg.apache.hadoop.oncrpc.RpcProgram
> Invalid RPC call program 100227
> {code}
> The client mount command must pass {{noacl}} to avoid this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[ https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10586: Status: Patch Available (was: Open) > KeyShell doesn't allow setting Options via CLI > -- > > Key: HADOOP-10586 > URL: https://issues.apache.org/jira/browse/HADOOP-10586 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-10586.1.patch, HADOOP-10586.1.patch, > HADOOP-10586.3.patch > > > You should be able to set any of the Options passed to the KeyProvider via > the CLI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10572) Example NFS mount command must pass noacl as it isn't supported by the server yet
[
https://issues.apache.org/jira/browse/HADOOP-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HADOOP-10572:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Example NFS mount command must pass noacl as it isn't supported by the server
> yet
> -
>
> Key: HADOOP-10572
> URL: https://issues.apache.org/jira/browse/HADOOP-10572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HADOOP-10572.patch
>
>
> Use of the documented default mount command results in the below server side
> log WARN event, cause the client tries to locate the ACL program (#100227):
> {code}
> 12:26:11.975 AM TRACE org.apache.hadoop.oncrpc.RpcCall
> Xid:-1114380537, messageType:RPC_CALL, rpcVersion:2, program:100227,
> version:3, procedure:0, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 12:26:11.976 AM TRACE org.apache.hadoop.oncrpc.RpcProgram
> NFS3 procedure #0
> 12:26:11.976 AM WARNorg.apache.hadoop.oncrpc.RpcProgram
> Invalid RPC call program 100227
> {code}
> The client mount command must pass {{noacl}} to avoid this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10152) Distributed file cipher InputStream and OutputStream which provide 1:1 mapping of plain text data and cipher data.
[ https://issues.apache.org/jira/browse/HADOOP-10152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10152: Issue Type: Task (was: Sub-task) Parent: (was: HADOOP-10150) > Distributed file cipher InputStream and OutputStream which provide 1:1 > mapping of plain text data and cipher data. > -- > > Key: HADOOP-10152 > URL: https://issues.apache.org/jira/browse/HADOOP-10152 > Project: Hadoop Common > Issue Type: Task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > > To be easily seek and positioned read distributed file, the length of > encrypted file should be the same as the length of plain file, and the > positions have 1:1 mapping. So in this JIRA we defines distributed file > cipher InputStream(FSDecryptorStream) and OutputStream(FSEncryptorStream). > The distributed file cipher InputStream is seekable and positonedReadable. > This JIRA is different from HADOOP-10151, the file may be read and written > many times and on multiple nodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10489) UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HADOOP-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10489:
---
Attachment: HADOOP-10489.patch
As suggested in the discussion in HADOOP-10475, I synchronized the methods on
the {{subject}} instead of the UGI. I also added a test that runs a thread
that repeatedly calls {{getTokens()}} while the test calls {{addToken(...)}} to
cause the CME; it fails without the fix and passes with them.
> UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to
> ConcurrentModificationException
>
>
> Key: HADOOP-10489
> URL: https://issues.apache.org/jira/browse/HADOOP-10489
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jing Zhao
>Assignee: Robert Kanter
> Attachments: HADOOP-10489.patch
>
>
> Currently UserGroupInformation#getTokens and UserGroupInformation#addToken
> uses UGI's monitor to protect the iteration and modification of
> Credentials#tokenMap. Per
> [discussion|https://issues.apache.org/jira/browse/HADOOP-10475?focusedCommentId=13965851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13965851]
> in HADOOP-10475, this can still lead to ConcurrentModificationException.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (HADOOP-10609) .gitignore should ignore .orig and .rej files
[ https://issues.apache.org/jira/browse/HADOOP-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla moved YARN-2058 to HADOOP-10609: - Target Version/s: 2.5.0 (was: 2.5.0) Affects Version/s: (was: 2.4.0) 2.4.0 Key: HADOOP-10609 (was: YARN-2058) Project: Hadoop Common (was: Hadoop YARN) > .gitignore should ignore .orig and .rej files > - > > Key: HADOOP-10609 > URL: https://issues.apache.org/jira/browse/HADOOP-10609 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > .gitignore file should ignore .orig and .rej files -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10585) Retry polices ignore interrupted exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993063#comment-13993063
]
Kihwal Lee commented on HADOOP-10585:
-
+1 The patch looks straightforward.
> Retry polices ignore interrupted exceptions
> ---
>
> Key: HADOOP-10585
> URL: https://issues.apache.org/jira/browse/HADOOP-10585
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10585.patch
>
>
> Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}.
> This prevents {{FsShell}} commands from being aborted during retries. It
> also causes orphaned webhdfs DN DFSClients to keep running after the webhdfs
> client closes the connection. Jetty goes into a loop constantly sending
> interrupts to the handler thread. Webhdfs retries cause multiple nodes to
> have these orphaned clients. The DN cannot shutdown until orphaned clients
> complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10572) Example NFS mount command must pass noacl as it isn't supported by the server yet
[
https://issues.apache.org/jira/browse/HADOOP-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998204#comment-13998204
]
Hudson commented on HADOOP-10572:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk #1753 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/])
HADOOP-10572. Example NFS mount command must pass noacl as it isn't supported
by the server yet. Contributed by Harsh J. (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594289)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
> Example NFS mount command must pass noacl as it isn't supported by the server
> yet
> -
>
> Key: HADOOP-10572
> URL: https://issues.apache.org/jira/browse/HADOOP-10572
> Project: Hadoop Common
> Issue Type: Improvement
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HADOOP-10572.patch
>
>
> Use of the documented default mount command results in the below server side
> log WARN event, cause the client tries to locate the ACL program (#100227):
> {code}
> 12:26:11.975 AM TRACE org.apache.hadoop.oncrpc.RpcCall
> Xid:-1114380537, messageType:RPC_CALL, rpcVersion:2, program:100227,
> version:3, procedure:0, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 12:26:11.976 AM TRACE org.apache.hadoop.oncrpc.RpcProgram
> NFS3 procedure #0
> 12:26:11.976 AM WARNorg.apache.hadoop.oncrpc.RpcProgram
> Invalid RPC call program 100227
> {code}
> The client mount command must pass {{noacl}} to avoid this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10585) Retry polices ignore interrupted exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-10585:
-
Attachment: HADOOP-10585.patch
> Retry polices ignore interrupted exceptions
> ---
>
> Key: HADOOP-10585
> URL: https://issues.apache.org/jira/browse/HADOOP-10585
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10585.patch
>
>
> Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}.
> This prevents {{FsShell}} commands from being aborted during retries. It
> also causes orphaned webhdfs DN DFSClients to keep running after the webhdfs
> client closes the connection. Jetty goes into a loop constantly sending
> interrupts to the handler thread. Webhdfs retries cause multiple nodes to
> have these orphaned clients. The DN cannot shutdown until orphaned clients
> complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10609) .gitignore should ignore .orig and .rej files
[ https://issues.apache.org/jira/browse/HADOOP-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-10609: -- Status: Patch Available (was: Open) > .gitignore should ignore .orig and .rej files > - > > Key: HADOOP-10609 > URL: https://issues.apache.org/jira/browse/HADOOP-10609 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: hadoop-10609.patch > > > .gitignore file should ignore .orig and .rej files -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10154) Provide cryptographic filesystem implementation and it's data IO.
[ https://issues.apache.org/jira/browse/HADOOP-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10154: Issue Type: Task (was: Sub-task) Parent: (was: HADOOP-10150) > Provide cryptographic filesystem implementation and it's data IO. > - > > Key: HADOOP-10154 > URL: https://issues.apache.org/jira/browse/HADOOP-10154 > Project: Hadoop Common > Issue Type: Task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > > The JIRA includes Cryptographic filesystem data InputStream which extends > FSDataInputStream and OutputStream which extends FSDataOutputStream. > Implantation of Cryptographic file system is also included in this JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998431#comment-13998431 ] Yi Liu commented on HADOOP-10150: - Thanks [~tucu00] for creating these sub-tasks, let's use them. > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system-V2.docx, HADOOP cryptographic file system.pdf, > HDFSDataAtRestEncryptionAlternatives.pdf, > HDFSDataatRestEncryptionAttackVectors.pdf, > HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based > on INode feature.patch > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
[ https://issues.apache.org/jira/browse/HADOOP-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reassigned HADOOP-10591: - Assignee: Colin Patrick McCabe > Compression codecs must used pooled direct buffers or deallocate direct > buffers when stream is closed > - > > Key: HADOOP-10591 > URL: https://issues.apache.org/jira/browse/HADOOP-10591 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Colin Patrick McCabe > > Currently direct buffers allocated by compression codecs like Gzip (which > allocates 2 direct buffers per instance) are not deallocated when the stream > is closed. Eventually for long running processes which create a huge number > of files, these direct buffers are left hanging till a full gc, which may or > may not happen in a reasonable amount of time - especially if the process > does not use a whole lot of heap. > Either these buffers should be pooled or they should be deallocated when the > stream is closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10585) Retry polices ignore interrupted exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993065#comment-13993065
]
Kihwal Lee commented on HADOOP-10585:
-
The patch looks straightforward. +1 pending successful precommit.
> Retry polices ignore interrupted exceptions
> ---
>
> Key: HADOOP-10585
> URL: https://issues.apache.org/jira/browse/HADOOP-10585
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10585.patch
>
>
> Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}.
> This prevents {{FsShell}} commands from being aborted during retries. It
> also causes orphaned webhdfs DN DFSClients to keep running after the webhdfs
> client closes the connection. Jetty goes into a loop constantly sending
> interrupts to the handler thread. Webhdfs retries cause multiple nodes to
> have these orphaned clients. The DN cannot shutdown until orphaned clients
> complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10564) Add username to native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10564: -- Attachment: HADOOP-10564-pnative.004.patch > Add username to native RPCv9 client > --- > > Key: HADOOP-10564 > URL: https://issues.apache.org/jira/browse/HADOOP-10564 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10564-pnative.002.patch, > HADOOP-10564-pnative.003.patch, HADOOP-10564-pnative.004.patch, > HADOOP-10564.001.patch > > > Add the ability for the native RPCv9 client to set a username when initiating > a connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997884#comment-13997884 ] Larry McCay commented on HADOOP-10607: -- also [~tucu00] - I was thinking that higher level consumers like KMS could still use both and was hoping that inclusion in the KMS would make sense to you. Let me know your thoughts on that as well. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10601) We should prevent AuthenticationFilter to be installed twice
Zhijie Shen created HADOOP-10601: Summary: We should prevent AuthenticationFilter to be installed twice Key: HADOOP-10601 URL: https://issues.apache.org/jira/browse/HADOOP-10601 Project: Hadoop Common Issue Type: Bug Reporter: Zhijie Shen It seems that we have two way to install the authentication filter (at least from YARN aspect): 1. have SPNEGO configs and use withHttpSpnego when starting the web app; 2. add AuthenticationFilterInitializer into the configuration of filter initializer list. If both ways are used, it seems that two AuthenticationFilter will be instantiated, which is not expected. It's good to allow one or the other. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[ https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10586: -- Attachment: HADOOP-10586.1.patch @tucu00 Here's a small patch which lets a user set the Key description field. > KeyShell doesn't allow setting Options via CLI > -- > > Key: HADOOP-10586 > URL: https://issues.apache.org/jira/browse/HADOOP-10586 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-10586.1.patch > > > You should be able to set any of the Options passed to the KeyProvider via > the CLI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10585) Retry polices ignore interrupted exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996561#comment-13996561
]
Jonathan Eagles commented on HADOOP-10585:
--
+1. Committing to trunk and branch-2. Thanks [~daryn] and [~kihwal]
> Retry polices ignore interrupted exceptions
> ---
>
> Key: HADOOP-10585
> URL: https://issues.apache.org/jira/browse/HADOOP-10585
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10585.patch
>
>
> Retry polices should not use {{ThreadUtil.sleepAtLeastIgnoreInterrupts}}.
> This prevents {{FsShell}} commands from being aborted during retries. It
> also causes orphaned webhdfs DN DFSClients to keep running after the webhdfs
> client closes the connection. Jetty goes into a loop constantly sending
> interrupts to the handler thread. Webhdfs retries cause multiple nodes to
> have these orphaned clients. The DN cannot shutdown until orphaned clients
> complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10586) KeyShell doesn't allow setting Options via CLI
[ https://issues.apache.org/jira/browse/HADOOP-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HADOOP-10586: -- Attachment: HADOOP-10586.1.patch [~tucu00] Here's a small patch which lets a user set the Key description field. > KeyShell doesn't allow setting Options via CLI > -- > > Key: HADOOP-10586 > URL: https://issues.apache.org/jira/browse/HADOOP-10586 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HADOOP-10586.1.patch, HADOOP-10586.1.patch > > > You should be able to set any of the Options passed to the KeyProvider via > the CLI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[ https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10156: Resolution: Duplicate Status: Resolved (was: Patch Available) > Define Buffer-based Encryptor/Decryptor interfaces and provide implementation > for AES CTR. > -- > > Key: HADOOP-10156 > URL: https://issues.apache.org/jira/browse/HADOOP-10156 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10156.patch > > > Define encryptor and decryptor interfaces, and they are buffer-based to > improve performance. We use direct buffer to avoid bytes copy between JAVA > and native if there is JNI call. In this JIRA, AES CTR mode encryption and > decryption are implemented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10583) bin/hadoop key throws NPE with no args and assorted other fixups
[ https://issues.apache.org/jira/browse/HADOOP-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996741#comment-13996741 ] Alejandro Abdelnur commented on HADOOP-10583: - +1. > bin/hadoop key throws NPE with no args and assorted other fixups > > > Key: HADOOP-10583 > URL: https://issues.apache.org/jira/browse/HADOOP-10583 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Labels: patch > Fix For: 3.0.0 > > Attachments: HADOOP-10583.1.patch, HADOOP-10583.2.patch, > HADOOP-10583.3.patch, HADOOP-10583.4.patch, HADOOP-10583.5.patch > > > bin/hadoop key throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10604) CryptoFileSystem decorator using xAttrs and KeyProvider
Alejandro Abdelnur created HADOOP-10604: --- Summary: CryptoFileSystem decorator using xAttrs and KeyProvider Key: HADOOP-10604 URL: https://issues.apache.org/jira/browse/HADOOP-10604 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Alejandro Abdelnur Assignee: Yi Liu A FileSystem implementation that wraps an existing filesystem and provides encryption. It will require the underlying filesystem to support xAttrs. It will use the KeyProvider API to retrieve encryption keys. This is mostly the work in the patch HADOOP-10150 minus the crypto streams -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996861#comment-13996861 ] Alejandro Abdelnur commented on HADOOP-10150: - [~hitliuyi], I've created sub-tasks 7, 8 & 9. They are somehow repeated from existing ones, would you please do a clean up pass and leave the ones that make sense based on the current proposal? > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system-V2.docx, HADOOP cryptographic file system.pdf, > HDFSDataAtRestEncryptionAlternatives.pdf, > HDFSDataatRestEncryptionAttackVectors.pdf, > HDFSDataatRestEncryptionProposal.pdf, cfs.patch, extended information based > on INode feature.patch > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10607) Create an API to separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10607: - Description: As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties. We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code. Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries. Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system. was: As with the filesystem API, we need to provide a generic mechanism to support multiple key storage mechanisms that are potentially from third parties. An additional requirement for long term data lakes is to keep multiple versions of each key so that keys can be rolled periodically without requiring the entire data set to be re-written. Rolling keys provides containment in the event of keys being leaked. Toward that end, I propose an API that is configured using a list of URLs of KeyProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries. Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system. > Create an API to separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Larry McCay >Assignee: Owen O'Malley > Fix For: 3.0.0 > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10566) Refactor proxyservers out of ProxyUsers
[ https://issues.apache.org/jira/browse/HADOOP-10566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998223#comment-13998223 ] Hudson commented on HADOOP-10566: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1753 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/]) HADOOP-10566. Adding files missed in previous commit 1594280 (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594282) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyServers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyServers.java HADOOP-10566. Refactor proxyservers out of ProxyUsers. Contributed by Benoy Antony. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594280) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ProxyUsers.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestProxyUsers.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java > Refactor proxyservers out of ProxyUsers > --- > > Key: HADOOP-10566 > URL: https://issues.apache.org/jira/browse/HADOOP-10566 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10566.patch, HADOOP-10566.patch, > HADOOP-10566.patch, HADOOP-10566.patch, HADOOP-10566.patch > > > HADOOP-10498 added proxyservers feature in ProxyUsers. It is beneficial to > treat this as a separate feature since > 1> The ProxyUsers is per proxyuser where as proxyservers is per cluster. The > cardinality is different. > 2> The ProxyUsers.authorize() and ProxyUsers.isproxyUser() are synchronized > and hence share the same lock and impacts performance. > Since these are two separate features, it will be an improvement to keep them > separate. It also enables one to fine-tune each feature independently. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10587) Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer objects
Colin Patrick McCabe created HADOOP-10587: - Summary: Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer objects Key: HADOOP-10587 URL: https://issues.apache.org/jira/browse/HADOOP-10587 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-10587.001.patch We can use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer objects. This will reduce our memory usage (for example, when loading edit logs), and help prevent OOMs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10592) Add unit test case for net in hadoop native client
[ https://issues.apache.org/jira/browse/HADOOP-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenwu Peng updated HADOOP-10592: Attachment: HADOOP-10592-pnative.001.patch > Add unit test case for net in hadoop native client > --- > > Key: HADOOP-10592 > URL: https://issues.apache.org/jira/browse/HADOOP-10592 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Attachments: HADOOP-10592-pnative.001.patch, > HADOOP-10592-pnative.002.patch > > > Add unit test case for net.c in hadoop native client -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998451#comment-13998451 ] Alejandro Abdelnur commented on HADOOP-10607: - BTW, I'm trying to see if it makes sense to reuse existing stable APIs from the JDK instead creating a new one. > Create an API to Separate Credentials/Password Storage from Applications > > > Key: HADOOP-10607 > URL: https://issues.apache.org/jira/browse/HADOOP-10607 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 3.0.0 > > Attachments: 10607.patch > > > As with the filesystem API, we need to provide a generic mechanism to support > multiple credential storage mechanisms that are potentially from third > parties. > We need the ability to eliminate the storage of passwords and secrets in > clear text within configuration files or within code. > Toward that end, I propose an API that is configured using a list of URLs of > CredentialProviders. The implementation will look for implementations using > the ServiceLoader interface and thus support third party libraries. > Two providers will be included in this patch. One using the credentials cache > in MapReduce jobs and the other using Java KeyStores from either HDFS or > local file system. > A CredShell CLI will also be included in this patch which provides the > ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10583) bin/hadoop key throws NPE with no args and assorted other fixups
[ https://issues.apache.org/jira/browse/HADOOP-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10583: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Charles. > bin/hadoop key throws NPE with no args and assorted other fixups > > > Key: HADOOP-10583 > URL: https://issues.apache.org/jira/browse/HADOOP-10583 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Labels: patch > Fix For: 3.0.0 > > Attachments: HADOOP-10583.1.patch, HADOOP-10583.2.patch, > HADOOP-10583.3.patch, HADOOP-10583.4.patch, HADOOP-10583.5.patch > > > bin/hadoop key throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[
https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993836#comment-13993836
]
Hadoop QA commented on HADOOP-10590:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644085/HADOOP-10590.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3933//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3933//console
This message is automatically generated.
> ServiceAuthorizationManager is not threadsafe
> --
>
> Key: HADOOP-10590
> URL: https://issues.apache.org/jira/browse/HADOOP-10590
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10590.patch
>
>
> The mutators in ServiceAuthorizationManager are synchronized. The accessors
> are not synchronized.
> This results in visibility issues when ServiceAuthorizationManager's state
> is accessed from different threads.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10564) Add username to native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992599#comment-13992599 ] Binglin Chang commented on HADOOP-10564: Hi Colin, about user.h, we may need a struct to represent user(like ugi in hadoop), so in the future more things can be added to it, like auth method, tokens, something like: struct hadoop_user; hadoop_user_(alloc|get_login|free) It is better to add it when the change is small, thoughts? > Add username to native RPCv9 client > --- > > Key: HADOOP-10564 > URL: https://issues.apache.org/jira/browse/HADOOP-10564 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10564-pnative.002.patch, HADOOP-10564.001.patch > > > Add the ability for the native RPCv9 client to set a username when initiating > a connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10404) Some accesses to DomainSocketWatcher#closed are not protected by lock
[
https://issues.apache.org/jira/browse/HADOOP-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe reassigned HADOOP-10404:
-
Assignee: Colin Patrick McCabe (was: Tsuyoshi OZAWA)
> Some accesses to DomainSocketWatcher#closed are not protected by lock
> -
>
> Key: HADOOP-10404
> URL: https://issues.apache.org/jira/browse/HADOOP-10404
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HADOOP-10404.003.patch, HADOOP-10404.1.patch,
> HADOOP-10404.2.patch
>
>
> {code}
>* Lock which protects toAdd, toRemove, and closed.
>*/
> private final ReentrantLock lock = new ReentrantLock();
> {code}
> There're two places, NotificationHandler.handle() and kick(), where access to
> closed is without holding lock.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10489) UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to ConcurrentModificationException
[ https://issues.apache.org/jira/browse/HADOOP-10489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-10489: --- Status: Patch Available (was: Open) > UserGroupInformation#getTokens and UserGroupInformation#addToken can lead to > ConcurrentModificationException > > > Key: HADOOP-10489 > URL: https://issues.apache.org/jira/browse/HADOOP-10489 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jing Zhao >Assignee: Robert Kanter > Attachments: HADOOP-10489.patch > > > Currently UserGroupInformation#getTokens and UserGroupInformation#addToken > uses UGI's monitor to protect the iteration and modification of > Credentials#tokenMap. Per > [discussion|https://issues.apache.org/jira/browse/HADOOP-10475?focusedCommentId=13965851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13965851] > in HADOOP-10475, this can still lead to ConcurrentModificationException. -- This message was sent by Atlassian JIRA (v6.2#6252)
