[jira] [Created] (HADOOP-9643) org.apache.hadoop.security.SecurityUtil calls toUpperCase(Locale.getDefault()) on hadoop.security.authentication value.
Mark Miller created HADOOP-9643: --- Summary: org.apache.hadoop.security.SecurityUtil calls toUpperCase(Locale.getDefault()) on hadoop.security.authentication value. Key: HADOOP-9643 URL: https://issues.apache.org/jira/browse/HADOOP-9643 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.5-alpha Reporter: Mark Miller Priority: Minor Attachments: HADOOP-9643.patch With the wrong locale, something like hadoop.security.authentication=simple will cause an IllegalArgumentException because "simple".toUpperCase(Locale.getDefault()) may not equal SIMPLE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9643) org.apache.hadoop.security.SecurityUtil calls toUpperCase(Locale.getDefault()) on hadoop.security.authentication value.
[ https://issues.apache.org/jira/browse/HADOOP-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated HADOOP-9643: Attachment: HADOOP-9643.patch > org.apache.hadoop.security.SecurityUtil calls > toUpperCase(Locale.getDefault()) on hadoop.security.authentication value. > --- > > Key: HADOOP-9643 > URL: https://issues.apache.org/jira/browse/HADOOP-9643 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.0.5-alpha >Reporter: Mark Miller >Priority: Minor > Attachments: HADOOP-9643.patch > > > With the wrong locale, something like hadoop.security.authentication=simple > will cause an IllegalArgumentException because > "simple".toUpperCase(Locale.getDefault()) may not equal SIMPLE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9635) Fix Potential Stack Overflow in DomainSocket.c
[
https://issues.apache.org/jira/browse/HADOOP-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682892#comment-13682892
]
Suresh Srinivas commented on HADOOP-9635:
-
For some reason this is in trunk section in CHANGES.txt on trunk, in 2.0.5
section in 2.1.0-beta. I fixed the CHANGES.txt in trunk and 2.1.0-beta.
> Fix Potential Stack Overflow in DomainSocket.c
> --
>
> Key: HADOOP-9635
> URL: https://issues.apache.org/jira/browse/HADOOP-9635
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
>Affects Versions: 2.3.0
> Environment: OSX 10.8
>Reporter: V. Karthik Kumar
> Labels: patch, security
> Fix For: 2.3.0
>
> Attachments:
> 0001-HADOOP-9635-Fix-Potential-Stack-Overflow-in-DomainSo.patch
>
>
> When I was running on OSX, the DataNode was segfaulting. On investigation, it
> was tracked down to this code. A potential stack overflow was also
> identified.
> {code}
>utfLength = (*env)->GetStringUTFLength(env, jstr);
>if (utfLength > sizeof(path)) {
> jthr = newIOException(env, "path is too long! We expected a path "
> "no longer than %zd UTF-8 bytes.", sizeof(path));
> goto done;
>}
> // GetStringUTFRegion does not pad with NUL
>(*env)->GetStringUTFRegion(env, jstr, 0, utfLength, path);
> ...
> //strtok_r can set rest pointer to NULL when no tokens found.
> //Causes JVM to crash in rest[0]
>for (check[0] = '/', check[1] = '\0', rest = path, token = "";
>token && rest[0];
> token = strtok_r(rest, "/", &rest)) {
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to "hadoop fs -conf" doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682845#comment-13682845 ] Jason Lowe commented on HADOOP-9582: Seems reasonable to at least make a small improvement to FsShell to catch bad -conf parameters, and that should be lower risk than having GenericOptionsParser start blindly setting all Configurations passed to it as non-quiet. Some comments on the latest patch: * TestGenericOptionsParser change now seems moot, since we didn't modify GenericOptionsParser and it doesn't test the FsShell change. * Nit: In TestFsShell rather than wielding AssertionFailedError directly we can simply use assertTrue(th instanceof RuntimeException) > Non-existent file to "hadoop fs -conf" doesn't throw error > -- > > Key: HADOOP-9582 > URL: https://issues.apache.org/jira/browse/HADOOP-9582 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Ashwin Shankar > Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt > > > When we run : > hadoop fs -conf BAD_FILE -ls / > we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682722#comment-13682722 ] Timothy St. Clair commented on HADOOP-9623: --- Hi Tom - Thanks for the feedback, I will try to scrounge up some time to do the tests and report back the results. Cheers, Tim > Update jets3t dependency > > > Key: HADOOP-9623 > URL: https://issues.apache.org/jira/browse/HADOOP-9623 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Timothy St. Clair > Labels: maven > Attachments: HADOOP-9623.patch > > > Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 > enables mvn-rpmbuild to build against system dependencies. > http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9639) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned HADOOP-9639: -- Assignee: Sangjin Lee Sounds like a plan. Looking forward to the proposal! > truly shared cache for jars (jobjar/libjar) > --- > > Key: HADOOP-9639 > URL: https://issues.apache.org/jira/browse/HADOOP-9639 > Project: Hadoop Common > Issue Type: New Feature > Components: filecache >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copying of jobjars and libjars becomes so > prevalent that it consumes a large portion of the network bandwidth, not to > speak of defeating the purpose of "bringing compute to where data is". This > is wasteful because in most cases code doesn't change much across many jobs. > I'd like to propose and discuss feasibility of introducing a truly shared > cache so that multiple jobs from multiple users can share and cache jars. > This JIRA is to open the discussion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9639) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682657#comment-13682657 ] Sangjin Lee commented on HADOOP-9639: - [~jlowe] thanks for the comment. Your points are taken, and there may well be some quick actions we can take (and code) to take care of a chunk of our use cases. Having said that, we're still dealing with a fair amount of variety of MR jobs; pig v. scalding, etc., and job jars with all dependencies embedded v. thin job jars with libjars, and so on. Also, as we start addressing more use cases, the management issues will come to fore eventually anyway... I'd like to take a stab at some rough design to see if we can find a solution that stays simple and low risk, and leverages all the existing distributed cache API and the resource localization service. I'll share more when we have some more details. I'd appreciate additional feedback or suggestions on the idea! > truly shared cache for jars (jobjar/libjar) > --- > > Key: HADOOP-9639 > URL: https://issues.apache.org/jira/browse/HADOOP-9639 > Project: Hadoop Common > Issue Type: New Feature > Components: filecache >Affects Versions: 2.0.4-alpha >Reporter: Sangjin Lee > > Currently there is the distributed cache that enables you to cache jars and > files so that attempts from the same job can reuse them. However, sharing is > limited with the distributed cache because it is normally on a per-job basis. > On a large cluster, sometimes copying of jobjars and libjars becomes so > prevalent that it consumes a large portion of the network bandwidth, not to > speak of defeating the purpose of "bringing compute to where data is". This > is wasteful because in most cases code doesn't change much across many jobs. > I'd like to propose and discuss feasibility of introducing a truly shared > cache so that multiple jobs from multiple users can share and cache jars. > This JIRA is to open the discussion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9622) bzip2 codec can drop records when reading data in splits
[ https://issues.apache.org/jira/browse/HADOOP-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-9622: --- Attachment: HADOOP-9622-2.patch Added a discussion of the various use cases and assumption on how compressed input streams read blocks to CompressSplitLineReader. > bzip2 codec can drop records when reading data in splits > > > Key: HADOOP-9622 > URL: https://issues.apache.org/jira/browse/HADOOP-9622 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.4-alpha, 0.23.8 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: blockEndingInCRThenLF.txt.bz2, blockEndingInCR.txt.bz2, > HADOOP-9622-2.patch, HADOOP-9622.patch, HADOOP-9622-testcase.patch > > > Bzip2Codec.BZip2CompressionInputStream can cause records to be dropped when > reading them in splits based on where record delimiters occur relative to > compression block boundaries. > Thanks to [~knoguchi] for discovering this problem while working on PIG-3251. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682366#comment-13682366 ] Daryn Sharp commented on HADOOP-9421: - The changes in this jira fundamentally boils down to: * SASL protobufs * RPC header wrapped SASL protobufs * Server advertised SASL auth methods There's not much additional complexity involved. Using this model opens the door to a lot of features such as easier support of the alternate auth methods being proposed, IP failover, token support for multi-interface and multi-hostnames for servers, multi-plexing UGIs over a single connection, etc. Sanjay, I'll contact you offline to bring you up to speed on this lengthy jira. > Convert SASL to use ProtoBuf and add lengths for non-blocking processing > > > Key: HADOOP-9421 > URL: https://issues.apache.org/jira/browse/HADOOP-9421 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.0.3-alpha >Reporter: Sanjay Radia >Assignee: Daryn Sharp > Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421.patch, > HADOOP-9421.patch, HADOOP-9421-v2-demo.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682346#comment-13682346 ] Larry McCay commented on HADOOP-9392: - As was pointed out on 9533 - the summary above is merely a description of what was discussed on the call for preparing for the security session at summit. No decisions have been made and we can/should discuss what the next call agenda should be. All decisions will be made through public communication such as these Jiras or dev-common list. Sorry if that wasn't clear in the above summary post. > Token based authentication and Single Sign On > - > > Key: HADOOP-9392 > URL: https://issues.apache.org/jira/browse/HADOOP-9392 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 3.0.0 > > Attachments: token-based-authn-plus-sso.pdf > > > This is an umbrella entry for one of project Rhino’s topic, for details of > project Rhino, please refer to > https://github.com/intel-hadoop/project-rhino/. The major goal for this entry > as described in project Rhino was > > “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication > at the RPC layer, via SASL. However this does not provide valuable attributes > such as group membership, classification level, organizational identity, or > support for user defined attributes. Hadoop components must interrogate > external resources for discovering these attributes and at scale this is > problematic. There is also no consistent delegation model. HDFS has a simple > delegation capability, and only Oozie can take limited advantage of it. We > will implement a common token based authentication framework to decouple > internal user and service authentication from external mechanisms used to > support it (like Kerberos)” > > We’d like to start our work from Hadoop-Common and try to provide common > facilities by extending existing authentication framework which support: > 1.Pluggable token provider interface > 2.Pluggable token verification protocol and interface > 3.Security mechanism to distribute secrets in cluster nodes > 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682344#comment-13682344 ] Larry McCay commented on HADOOP-9533: - Absolutely, Eric. I should have been more clear about that. That summary is what was discussed and suggested to be presented to the community as a way forward. We can and should discuss further for establishing the agenda for the next call. Perhaps, we'll use the dev-common list for proposing the next agenda? > Centralized Hadoop SSO/Token Server > --- > > Key: HADOOP-9533 > URL: https://issues.apache.org/jira/browse/HADOOP-9533 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay > Attachments: HSSO-Interaction-Overview-rev-1.docx, > HSSO-Interaction-Overview-rev-1.pdf > > > This is an umbrella Jira filing to oversee a set of proposals for introducing > a new master service for Hadoop Single Sign On (HSSO). > There is an increasing need for pluggable authentication providers that > authenticate both users and services as well as validate tokens in order to > federate identities authenticated by trusted IDPs. These IDPs may be deployed > within the enterprise or third-party IDPs that are external to the enterprise. > These needs speak to a specific pain point: which is a narrow integration > path into the enterprise identity infrastructure. Kerberos is a fine solution > for those that already have it in place or are willing to adopt its use but > there remains a class of user that finds this unacceptable and needs to > integrate with a wider variety of identity management solutions. > Another specific pain point is that of rolling and distributing keys. A > related and integral part of the HSSO server is library called the Credential > Management Framework (CMF), which will be a common library for easing the > management of secrets, keys and credentials. > Initially, the existing delegation, block access and job tokens will continue > to be utilized. There may be some changes required to leverage a PKI based > signature facility rather than shared secrets. This is a means to simplify > the solution for the pain point of distributing shared secrets. > This project will primarily centralize the responsibility of authentication > and federation into a single service that is trusted across the Hadoop > cluster and optionally across multiple clusters. This greatly simplifies a > number of things in the Hadoop ecosystem: > 1.a single token format that is used across all of Hadoop regardless of > authentication method > 2.a single service to have pluggable providers instead of all services > 3.a single token authority that would be trusted across the cluster/s and > through PKI encryption be able to easily issue cryptographically verifiable > tokens > 4.automatic rolling of the token authority’s keys and publishing of the > public key for easy access by those parties that need to verify incoming > tokens > 5.use of PKI for signatures eliminates the need for securely sharing and > distributing shared secrets > In addition to serving as the internal Hadoop SSO service this service will > be leveraged by the Knox Gateway from the cluster perimeter in order to > acquire the Hadoop cluster tokens. The same token mechanism that is used for > internal services will be used to represent user identities. Providing for > interesting scenarios such as SSO across Hadoop clusters within an enterprise > and/or into the cloud. > The HSSO service will be comprised of three major components and capabilities: > 1.Federating IDP – authenticates users/services and issues the common > Hadoop token > 2.Federating SP – validates the token of trusted external IDPs and issues > the common Hadoop token > 3.Token Authority – management of the common Hadoop tokens – including: > a.Issuance > b.Renewal > c.Revocation > As this is a meta Jira for tracking this overall effort, the details of the > individual efforts will be submitted along with the child Jira filings. > Hadoop-Common would seem to be the most appropriate home for such a service > and its related common facilities. We will also leverage and extend existing > common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682339#comment-13682339 ] eric baldeschwieler commented on HADOOP-9533: - As always... This is a proposal for discussion and refinement. Decisions are made via discussions such as this Jira -- E14 - via thumbs on glass > Centralized Hadoop SSO/Token Server > --- > > Key: HADOOP-9533 > URL: https://issues.apache.org/jira/browse/HADOOP-9533 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay > Attachments: HSSO-Interaction-Overview-rev-1.docx, > HSSO-Interaction-Overview-rev-1.pdf > > > This is an umbrella Jira filing to oversee a set of proposals for introducing > a new master service for Hadoop Single Sign On (HSSO). > There is an increasing need for pluggable authentication providers that > authenticate both users and services as well as validate tokens in order to > federate identities authenticated by trusted IDPs. These IDPs may be deployed > within the enterprise or third-party IDPs that are external to the enterprise. > These needs speak to a specific pain point: which is a narrow integration > path into the enterprise identity infrastructure. Kerberos is a fine solution > for those that already have it in place or are willing to adopt its use but > there remains a class of user that finds this unacceptable and needs to > integrate with a wider variety of identity management solutions. > Another specific pain point is that of rolling and distributing keys. A > related and integral part of the HSSO server is library called the Credential > Management Framework (CMF), which will be a common library for easing the > management of secrets, keys and credentials. > Initially, the existing delegation, block access and job tokens will continue > to be utilized. There may be some changes required to leverage a PKI based > signature facility rather than shared secrets. This is a means to simplify > the solution for the pain point of distributing shared secrets. > This project will primarily centralize the responsibility of authentication > and federation into a single service that is trusted across the Hadoop > cluster and optionally across multiple clusters. This greatly simplifies a > number of things in the Hadoop ecosystem: > 1.a single token format that is used across all of Hadoop regardless of > authentication method > 2.a single service to have pluggable providers instead of all services > 3.a single token authority that would be trusted across the cluster/s and > through PKI encryption be able to easily issue cryptographically verifiable > tokens > 4.automatic rolling of the token authority’s keys and publishing of the > public key for easy access by those parties that need to verify incoming > tokens > 5.use of PKI for signatures eliminates the need for securely sharing and > distributing shared secrets > In addition to serving as the internal Hadoop SSO service this service will > be leveraged by the Knox Gateway from the cluster perimeter in order to > acquire the Hadoop cluster tokens. The same token mechanism that is used for > internal services will be used to represent user identities. Providing for > interesting scenarios such as SSO across Hadoop clusters within an enterprise > and/or into the cloud. > The HSSO service will be comprised of three major components and capabilities: > 1.Federating IDP – authenticates users/services and issues the common > Hadoop token > 2.Federating SP – validates the token of trusted external IDPs and issues > the common Hadoop token > 3.Token Authority – management of the common Hadoop tokens – including: > a.Issuance > b.Renewal > c.Revocation > As this is a meta Jira for tracking this overall effort, the details of the > individual efforts will be submitted along with the child Jira filings. > Hadoop-Common would seem to be the most appropriate home for such a service > and its related common facilities. We will also leverage and extend existing > common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682321#comment-13682321 ] Tom White commented on HADOOP-9623: --- Overall it looks fine. Do Jets3tNativeS3FileSystemContractTest and Jets3tS3FileSystemContractTest pass with this change? They are live tests run against the real S3 service and are not run by default. You'll need to put S3 credentials into hadoop-common-project/hadoop-common/src/test/resources/core-site.xml to run them. Nit: catch blocks should be on the same line as the preceding brace. > Update jets3t dependency > > > Key: HADOOP-9623 > URL: https://issues.apache.org/jira/browse/HADOOP-9623 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Timothy St. Clair > Labels: maven > Attachments: HADOOP-9623.patch > > > Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 > enables mvn-rpmbuild to build against system dependencies. > http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9630) Remove IpcSerializationType
[ https://issues.apache.org/jira/browse/HADOOP-9630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-9630: --- Component/s: ipc > Remove IpcSerializationType > --- > > Key: HADOOP-9630 > URL: https://issues.apache.org/jira/browse/HADOOP-9630 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Reporter: Luke Lu >Assignee: Junping Du > Labels: rpc > Fix For: 2.1.0-beta > > Attachments: HADOOP-9630.patch > > > IpcSerializationType is assumed to be protobuf for the forseeable future. Not > to be confused with RpcKind which still supports different RpcEngines. Let's > remove the dead code, which can be confusing to maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682280#comment-13682280 ] Larry McCay commented on HADOOP-9392: - A thank you to those that attended the prep-call yesterday for the summit security session. While not all interested parties were able to make it to this call, we were able to lay some groundwork for moving forward in being prepared. We intend to schedule another call for next week at a more globally appropriate time. In the mean time, the following is a summary of the call from yesterday and should be used to frame the agenda for the next call. Prep-call Summary Introductions Community driven collaboration examples * HDFS-HA as a successful model - break out concrete areas that can be worked on by different parties but are aligned and complimentary - HDFS-HA apparently did this between at least two contributing parties with functionality separated into things like: a. client failover/recovery b. transaction journalling to support the recovery Roadmap to prepare for summit: * Describe overall end-state goals for the Hadoop Security Model for Authentication (keep the scope focused on authn) * Canonical security concerns and threats for an authentication system that is an alternative to kerberos - add as document or subtask of https://issues.apache.org/jira/browse/HADOOP-9621 * Describe the various tasks/projects that are required for reaching our goals - reconcile existing Jiras as subtasks of others as appropriate Ideally at summit we will be able to focus on: * Identify a phased approach to reaching our goals * Identify the best form of collaboration model for the effort * Identify natural seams of separation for collaboration * Interested contributors commit to specific aspects of the effort > Token based authentication and Single Sign On > - > > Key: HADOOP-9392 > URL: https://issues.apache.org/jira/browse/HADOOP-9392 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 3.0.0 > > Attachments: token-based-authn-plus-sso.pdf > > > This is an umbrella entry for one of project Rhino’s topic, for details of > project Rhino, please refer to > https://github.com/intel-hadoop/project-rhino/. The major goal for this entry > as described in project Rhino was > > “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication > at the RPC layer, via SASL. However this does not provide valuable attributes > such as group membership, classification level, organizational identity, or > support for user defined attributes. Hadoop components must interrogate > external resources for discovering these attributes and at scale this is > problematic. There is also no consistent delegation model. HDFS has a simple > delegation capability, and only Oozie can take limited advantage of it. We > will implement a common token based authentication framework to decouple > internal user and service authentication from external mechanisms used to > support it (like Kerberos)” > > We’d like to start our work from Hadoop-Common and try to provide common > facilities by extending existing authentication framework which support: > 1.Pluggable token provider interface > 2.Pluggable token verification protocol and interface > 3.Security mechanism to distribute secrets in cluster nodes > 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682279#comment-13682279 ] Larry McCay commented on HADOOP-9533: - A thank you to those that attended the prep-call yesterday for the summit security session. While not all interested parties were able to make it to this call, we were able to lay some groundwork for moving forward in being prepared. We intend to schedule another call for next week at a more globally appropriate time. In the mean time, the following is a summary of the call from yesterday and should be used to frame the agenda for the next call. Prep-call Summary Introductions Community driven collaboration examples * HDFS-HA as a successful model - break out concrete areas that can be worked on by different parties but are aligned and complimentary - HDFS-HA apparently did this between at least two contributing parties with functionality separated into things like: a. client failover/recovery b. transaction journalling to support the recovery Roadmap to prepare for summit: * Describe overall end-state goals for the Hadoop Security Model for Authentication (keep the scope focused on authn) * Canonical security concerns and threats for an authentication system that is an alternative to kerberos - add as document or subtask of https://issues.apache.org/jira/browse/HADOOP-9621 * Describe the various tasks/projects that are required for reaching our goals - reconcile existing Jiras as subtasks of others as appropriate Ideally at summit we will be able to focus on: * Identify a phased approach to reaching our goals * Identify the best form of collaboration model for the effort * Identify natural seams of separation for collaboration * Interested contributors commit to specific aspects of the effort > Centralized Hadoop SSO/Token Server > --- > > Key: HADOOP-9533 > URL: https://issues.apache.org/jira/browse/HADOOP-9533 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Larry McCay > Attachments: HSSO-Interaction-Overview-rev-1.docx, > HSSO-Interaction-Overview-rev-1.pdf > > > This is an umbrella Jira filing to oversee a set of proposals for introducing > a new master service for Hadoop Single Sign On (HSSO). > There is an increasing need for pluggable authentication providers that > authenticate both users and services as well as validate tokens in order to > federate identities authenticated by trusted IDPs. These IDPs may be deployed > within the enterprise or third-party IDPs that are external to the enterprise. > These needs speak to a specific pain point: which is a narrow integration > path into the enterprise identity infrastructure. Kerberos is a fine solution > for those that already have it in place or are willing to adopt its use but > there remains a class of user that finds this unacceptable and needs to > integrate with a wider variety of identity management solutions. > Another specific pain point is that of rolling and distributing keys. A > related and integral part of the HSSO server is library called the Credential > Management Framework (CMF), which will be a common library for easing the > management of secrets, keys and credentials. > Initially, the existing delegation, block access and job tokens will continue > to be utilized. There may be some changes required to leverage a PKI based > signature facility rather than shared secrets. This is a means to simplify > the solution for the pain point of distributing shared secrets. > This project will primarily centralize the responsibility of authentication > and federation into a single service that is trusted across the Hadoop > cluster and optionally across multiple clusters. This greatly simplifies a > number of things in the Hadoop ecosystem: > 1.a single token format that is used across all of Hadoop regardless of > authentication method > 2.a single service to have pluggable providers instead of all services > 3.a single token authority that would be trusted across the cluster/s and > through PKI encryption be able to easily issue cryptographically verifiable > tokens > 4.automatic rolling of the token authority’s keys and publishing of the > public key for easy access by those parties that need to verify incoming > tokens > 5.use of PKI for signatures eliminates the need for securely sharing and > distributing shared secrets > In addition to serving as the internal Hadoop SSO service this service will > be leveraged by the Knox Gateway from the cluster perimeter in order to > acquire the Hadoop cluster tokens. The same token mechanism that is used for > internal services will be used
[jira] [Commented] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
[
https://issues.apache.org/jira/browse/HADOOP-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682277#comment-13682277
]
Daryn Sharp commented on HADOOP-9642:
-
Generally sounds reasonable to me. Although you'd probably want additional
syntax such as $\{ENV:default-value\} if the env isn't defined.
One concern is introduction of non-deterministic behavior. An example being
the "works for me" scenario where user A and user B are puzzled why each gets
different results from the same command because one of them forgot to set
additional envs beyond HADOOP_HOME.
Along those lines, it does introduce additional logistics if server process
configs reference envs. It becomes easier to make mistakes if there's a mix of
interactive/non-interactive shells commands to start processes and the most
appropriate rc (profile/bashrc/etc) isn't updated. Although it could be argued
the admin brought the pain upon himself...
> Configuration to resolve environment variables via ${env.VARIABLE} references
> -
>
> Key: HADOOP-9642
> URL: https://issues.apache.org/jira/browse/HADOOP-9642
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Steve Loughran
>Priority: Minor
>
> We should be able to get env variables from Configuration files, rather than
> just system properties. I propose using the traditional {{env}} prefix
> {{${env.PATH}}} to make it immediately clear to people reading a conf file
> that it's an env variable -and to avoid any confusion with system properties
> and existing configuration properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9638) parallel test changes caused invalid test path for several HDFS tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682265#comment-13682265 ] Hudson commented on HADOOP-9638: Integrated in Hadoop-Mapreduce-trunk #1456 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1456/]) HADOOP-9638. Parallel test changes caused invalid test path for several HDFS tests on Windows. Contributed by Andrey Klochkov. (Revision 1492297) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492297 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextTestHelper.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsCreateMkdir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsPermission.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestHDFSFileContextMainOperations.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsAtHdfsRoot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java > parallel test changes caused invalid test path for several HDFS tests on > Windows > > > Key: HADOOP-9638 > URL: https://issues.apache.org/jira/browse/HADOOP-9638 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Chris Nauroth >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9638.2.patch, HADOOP-9638.3.patch, > HADOOP-9638-branch-2.patch, HADOOP-9638.patch > > > HADOOP-9287 made changes to the tests to support running multiple tests in > parallel. Part of that patch accidentally reverted a prior change to use > paths of the form "/tmp/" when running tests against HDFS. On > Windows, use of the test root will contain a drive spec (i.e. C:\dir), and > the colon character is rejected as invalid by HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9638) parallel test changes caused invalid test path for several HDFS tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682216#comment-13682216 ] Hudson commented on HADOOP-9638: Integrated in Hadoop-Hdfs-trunk #1429 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1429/]) HADOOP-9638. Parallel test changes caused invalid test path for several HDFS tests on Windows. Contributed by Andrey Klochkov. (Revision 1492297) Result = FAILURE cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492297 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextTestHelper.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsCreateMkdir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsPermission.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestHDFSFileContextMainOperations.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsAtHdfsRoot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java > parallel test changes caused invalid test path for several HDFS tests on > Windows > > > Key: HADOOP-9638 > URL: https://issues.apache.org/jira/browse/HADOOP-9638 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Chris Nauroth >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9638.2.patch, HADOOP-9638.3.patch, > HADOOP-9638-branch-2.patch, HADOOP-9638.patch > > > HADOOP-9287 made changes to the tests to support running multiple tests in > parallel. Part of that patch accidentally reverted a prior change to use > paths of the form "/tmp/" when running tests against HDFS. On > Windows, use of the test root will contain a drive spec (i.e. C:\dir), and > the colon character is rejected as invalid by HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9638) parallel test changes caused invalid test path for several HDFS tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682112#comment-13682112 ] Hudson commented on HADOOP-9638: Integrated in Hadoop-Yarn-trunk #239 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/239/]) HADOOP-9638. Parallel test changes caused invalid test path for several HDFS tests on Windows. Contributed by Andrey Klochkov. (Revision 1492297) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492297 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextTestHelper.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsCreateMkdir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsPermission.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestHDFSFileContextMainOperations.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsAtHdfsRoot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java > parallel test changes caused invalid test path for several HDFS tests on > Windows > > > Key: HADOOP-9638 > URL: https://issues.apache.org/jira/browse/HADOOP-9638 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Chris Nauroth >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9638.2.patch, HADOOP-9638.3.patch, > HADOOP-9638-branch-2.patch, HADOOP-9638.patch > > > HADOOP-9287 made changes to the tests to support running multiple tests in > parallel. Part of that patch accidentally reverted a prior change to use > paths of the form "/tmp/" when running tests against HDFS. On > Windows, use of the test root will contain a drive spec (i.e. C:\dir), and > the colon character is rejected as invalid by HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9642) Configuration to resolve environment variables via ${env.VARIABLE} references
Steve Loughran created HADOOP-9642:
--
Summary: Configuration to resolve environment variables via
${env.VARIABLE} references
Key: HADOOP-9642
URL: https://issues.apache.org/jira/browse/HADOOP-9642
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Priority: Minor
We should be able to get env variables from Configuration files, rather than
just system properties. I propose using the traditional {{env}} prefix
{{${env.PATH}}} to make it immediately clear to people reading a conf file that
it's an env variable -and to avoid any confusion with system properties and
existing configuration properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
