Re: [PROPOSAL] Knox Hadoop Gateway Project
Diversity as has been discussed has generally been about contributors not mentors, but while we should be able to expect mentors to act primarily in the best interests of the ASF not their employer, having some diversity of mentors is going to be a good thing. The flaws in the initial landgrab over committership at the beginning of a podling's life have been demonstrated. I believe it is best to start with a known set of contributors (from one org is fine) and watch them demonstrate their ability to open up their project quickly by setting a low bar to committership. It does seem like this project now has a wealth of mentor options. I've seen projects with six mentors, five of which were quickly inactive. It makes sense I'd say to start a discussion amongst those who have volunteered to see who is best placed to be an active mentor. A project choosing their own mentors also makes sense, as these are people they are going to be working with. Just some reflections. Upayavira On Wed, Feb 13, 2013, at 12:27 AM, Josh Wills wrote: I thought that the need for diversity referred to the community, not to the mentors. I strongly advocate for newly incubating projects choosing their own initial members (you need only search for the epic thread that resulted from the Crunch Proposal circa May 2012), but it seems like recent data suggests that starting out with a small set of committers from a single organization working on a large and complex codebase is a recipe for diversity issues and a difficult path out of the incubator. Kevin et al., please choose whomever you want, but do consider getting some folks from other organizations to seed the community. It does so much good for the project over the long term. On Tue, Feb 12, 2013 at 3:31 PM, Devaraj Das d...@hortonworks.com wrote: Hi folks, Happy to see such a hugely positive response on the proposal. I'll put up a vote tomorrow. Now, on the mentors, I think we have enough at this point and we have also addressed the diversity concern. I had got the confirmation from Chris Douglas and Chris Mattmann yesterday afternoon that they would participate as mentors in the project. On the nominations for getting involved in the project, you are welcome to come in as contributors. Individuals in the initial list of committers all have a view point that the committers list be kept as such and the standard practice (based on contributions, after the project is accepted for incubation) for accepting new committers be followed here. Thanks Devaraj. On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox . Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox . Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder
Re: [PROPOSAL] Knox Hadoop Gateway Project
I thought about this a bit last night. If y'all are interested I too could also mentor the project. That should add some diversity to the mentors list. I see value in it and would like to see this community succeed. I'm not affiliated with any company. On Mon, Feb 11, 2013 at 9:23 PM, Eric Sammer esam...@cloudera.com wrote: Kevin: Makes complete sense. I'd like to offer to join the project, if it's accepted for incubation. I'm a committer on MRUnit and Flume, and on the PMC for both. I've helped both projects through the incubation phase, and I also know a little bit about this Hadoop thing. ;) Thanks! On Mon, Feb 11, 2013 at 9:28 AM, Kevin Minder kevin.min...@hortonworks.comwrote: Hi Eric, Let me answer your second question first. Q: Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? A: Yes we plan to progress the project to cover all existing ecosystem projects. In addition the project is based on a modular framework that allows for each extension to cover services that are either new or proprietary. Certainly there exist very high volume data ingest use cases for which using a gateway may be impractical but in general the idea is to support all required client interaction with Hadoop via the gateway. Now for your first question... Q: Can you explain a bit more about what the target use case is? A: One typical use case will be that the gateway will run in a DMW. It will as you say be integrations with various directory services and is extensible to cover those not included. The gateway will then propagate the identity into the Hadoop cluster using Hadoop specific mechanisms. The key point is that there will typically be a single port open on the client side to the gateway. The Hadoop cluster is firewalled, only providing access to the Hadoop services to the gateway instances. A: Another use case is that an organization is already using some SSO solution and the gateway would be integrated with that to verify any SSO token and then propagate the identity to the Hadoop services. I will collect this and add it to the proposal wiki once I have privs to create the page. Thanks! Kevin. On 2/11/13 12:03 PM, Eric Sammer wrote: Kevin: Interesting proposal. Can you explain a bit more about what the target use case is? It sounds like there's SSO-ish functionality (presumably a doAs() machine) with integration with directory services, but the proposal also mentions a single point for data and jobs. Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? Do you plan to target other ecosystem projects such as HBase? Sorry if I missed this in the proposal. Thanks! On Mon, Feb 11, 2013 at 6:55 AM, Kevin Minder kevin.min...@hortonworks.com**wrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy ===
Re: [PROPOSAL] Knox Hadoop Gateway Project
This looks like an interesting and useful project. I'd like to volunteer as a mentor on it. Thanks, Tom On 11 Feb 2013 14:56, Kevin Minder kevin.min...@hortonworks.com wrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships with Other Apache Products === Gateway is going to be used by the users and operators of Hadoop, and the Hadoop ecosystem in general. === A Excessive Fascination with the Apache Brand === Our interest in developing Gateway in Apache project is to follow an established development model, as well since many of the Hadoop ecosystem projects also are part of Apache, Gateway will
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi Hadrian, Currently I would like to keep the focus on Hadoop services to make sure that we can produce a useful product in a reasonable time frame. We have been careful with the initial code however to treat all Hadoop integrations as modular plugins. So we have and will consider increasing the scope to other apache projects at the right time. Also, the code is currently available at https://github.com/hortonworks/knox Kevin. On 2/11/13 10:54 PM, Hadrian Zbarcea wrote: +1 on the idea. +1 (strong) on the need to increase diversity. I would suggest to open a bit the scope of the project to target other apache projects that act as services. I know it was mentioned, but it sounded to me more like a possibility than a focus. I would also volunteer as a committer and, if you prefer, mentor. I assume it would be possible to take a peek at the code grant in the coming days. Cheers, Hadrian On 02/11/2013 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi Kevin, I'd like to offer joining the project if accepted for incubation. I'm actively working in Hadoop (PMC member) and Oozie (PMC member), helped driving Oozie through incubation and graduation. I'm also the initial developer of hadoop-auth and httpfs (standalone WebHDFS gateway). In, Oozie provides REST APIs for jobs submission (workflows, coordinators, bundles, mapreduce, streaming, pig). In addition I've been involved in efforts for providing HTTP access to Hadoop stack. Thanks. On Mon, Feb 11, 2013 at 11:54 AM, Kevin Minder kevin.min...@hortonworks.com wrote: The proposal is now hosted on the wiki. http://wiki.apache.org/**incubator/knoxhttp://wiki.apache.org/incubator/knox Currently just the initial proposal. I'll be incorporating feedback/clarifications shortly. On 2/11/13 11:35 AM, Bertrand Delacretaz wrote: Hi, On Mon, Feb 11, 2013 at 5:07 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ...Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage I think that would be good - just ask for more mentors on this list. Without such diversity this is basically an all-hortonworks podling, which is not good IMO. I don't seem to have privs to create http://wiki.apache.org/** incubator/knox http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in?... Yes, http://wiki.apache.org/**incubator/ishttp://wiki.apache.org/incubator/isthe right place - just ask for write access on this list, a [wiki] marker in the subject line will probably help. I don't think I have the rights to grant that access myself. -Bertrand --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.orggeneral-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**orggeneral-h...@incubator.apache.org --**--**- To unsubscribe, e-mail: general-unsubscribe@incubator.**apache.orggeneral-unsubscr...@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.**orggeneral-h...@incubator.apache.org -- Alejandro
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi folks, Happy to see such a hugely positive response on the proposal. I'll put up a vote tomorrow. Now, on the mentors, I think we have enough at this point and we have also addressed the diversity concern. I had got the confirmation from Chris Douglas and Chris Mattmann yesterday afternoon that they would participate as mentors in the project. On the nominations for getting involved in the project, you are welcome to come in as contributors. Individuals in the initial list of committers all have a view point that the committers list be kept as such and the standard practice (based on contributions, after the project is accepted for incubation) for accepting new committers be followed here. Thanks Devaraj. On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand
Re: [PROPOSAL] Knox Hadoop Gateway Project
I thought that the need for diversity referred to the community, not to the mentors. I strongly advocate for newly incubating projects choosing their own initial members (you need only search for the epic thread that resulted from the Crunch Proposal circa May 2012), but it seems like recent data suggests that starting out with a small set of committers from a single organization working on a large and complex codebase is a recipe for diversity issues and a difficult path out of the incubator. Kevin et al., please choose whomever you want, but do consider getting some folks from other organizations to seed the community. It does so much good for the project over the long term. On Tue, Feb 12, 2013 at 3:31 PM, Devaraj Das d...@hortonworks.com wrote: Hi folks, Happy to see such a hugely positive response on the proposal. I'll put up a vote tomorrow. Now, on the mentors, I think we have enough at this point and we have also addressed the diversity concern. I had got the confirmation from Chris Douglas and Chris Mattmann yesterday afternoon that they would participate as mentors in the project. On the nominations for getting involved in the project, you are welcome to come in as contributors. Individuals in the initial list of committers all have a view point that the committers list be kept as such and the standard practice (based on contributions, after the project is accepted for incubation) for accepting new committers be followed here. Thanks Devaraj. On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox . Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox . Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org On Mon, Feb 11, 2013 at 8:07 AM, Kevin Minder kevin.min...@hortonworks.com wrote: Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi Bertrand, Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage. One other question I have is about creating a wiki to host the proposal. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in? Kevin. On 2/11/13 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi, On Mon, Feb 11, 2013 at 5:07 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ...Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage I think that would be good - just ask for more mentors on this list. Without such diversity this is basically an all-hortonworks podling, which is not good IMO. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in?... Yes, http://wiki.apache.org/incubator/is the right place - just ask for write access on this list, a [wiki] marker in the subject line will probably help. I don't think I have the rights to grant that access myself. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Knox Hadoop Gateway Project
Kevin: Interesting proposal. Can you explain a bit more about what the target use case is? It sounds like there's SSO-ish functionality (presumably a doAs() machine) with integration with directory services, but the proposal also mentions a single point for data and jobs. Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? Do you plan to target other ecosystem projects such as HBase? Sorry if I missed this in the proposal. Thanks! On Mon, Feb 11, 2013 at 6:55 AM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi Eric, Let me answer your second question first. Q: Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? A: Yes we plan to progress the project to cover all existing ecosystem projects. In addition the project is based on a modular framework that allows for each extension to cover services that are either new or proprietary. Certainly there exist very high volume data ingest use cases for which using a gateway may be impractical but in general the idea is to support all required client interaction with Hadoop via the gateway. Now for your first question... Q: Can you explain a bit more about what the target use case is? A: One typical use case will be that the gateway will run in a DMW. It will as you say be integrations with various directory services and is extensible to cover those not included. The gateway will then propagate the identity into the Hadoop cluster using Hadoop specific mechanisms. The key point is that there will typically be a single port open on the client side to the gateway. The Hadoop cluster is firewalled, only providing access to the Hadoop services to the gateway instances. A: Another use case is that an organization is already using some SSO solution and the gateway would be integrated with that to verify any SSO token and then propagate the identity to the Hadoop services. I will collect this and add it to the proposal wiki once I have privs to create the page. Thanks! Kevin. On 2/11/13 12:03 PM, Eric Sammer wrote: Kevin: Interesting proposal. Can you explain a bit more about what the target use case is? It sounds like there's SSO-ish functionality (presumably a doAs() machine) with integration with directory services, but the proposal also mentions a single point for data and jobs. Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? Do you plan to target other ecosystem projects such as HBase? Sorry if I missed this in the proposal. Thanks! On Mon, Feb 11, 2013 at 6:55 AM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers
Re: [PROPOSAL] Knox Hadoop Gateway Project
Hi Kevin, This sounds like a much needed project. I endorse the concept but as Bertrand pointed out you need a bit more diversity. Otherwise I see no problem with moving forward. Good luck! On Mon, Feb 11, 2013 at 4:55 PM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships with Other Apache Products === Gateway is going to be used by the users and operators of Hadoop, and the Hadoop ecosystem in general. === A Excessive Fascination with the Apache Brand === Our interest in developing Gateway in Apache project is to follow an established development model,
Re: [PROPOSAL] Knox Hadoop Gateway Project
When do you expect the github to be made available? -Jakob On Mon, Feb 11, 2013 at 10:21 AM, Alex Karasulu akaras...@apache.orgwrote: Hi Kevin, This sounds like a much needed project. I endorse the concept but as Bertrand pointed out you need a bit more diversity. Otherwise I see no problem with moving forward. Good luck! On Mon, Feb 11, 2013 at 4:55 PM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships with Other Apache Products
Re: [PROPOSAL] Knox Hadoop Gateway Project
Kevin: Makes complete sense. I'd like to offer to join the project, if it's accepted for incubation. I'm a committer on MRUnit and Flume, and on the PMC for both. I've helped both projects through the incubation phase, and I also know a little bit about this Hadoop thing. ;) Thanks! On Mon, Feb 11, 2013 at 9:28 AM, Kevin Minder kevin.min...@hortonworks.comwrote: Hi Eric, Let me answer your second question first. Q: Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? A: Yes we plan to progress the project to cover all existing ecosystem projects. In addition the project is based on a modular framework that allows for each extension to cover services that are either new or proprietary. Certainly there exist very high volume data ingest use cases for which using a gateway may be impractical but in general the idea is to support all required client interaction with Hadoop via the gateway. Now for your first question... Q: Can you explain a bit more about what the target use case is? A: One typical use case will be that the gateway will run in a DMW. It will as you say be integrations with various directory services and is extensible to cover those not included. The gateway will then propagate the identity into the Hadoop cluster using Hadoop specific mechanisms. The key point is that there will typically be a single port open on the client side to the gateway. The Hadoop cluster is firewalled, only providing access to the Hadoop services to the gateway instances. A: Another use case is that an organization is already using some SSO solution and the gateway would be integrated with that to verify any SSO token and then propagate the identity to the Hadoop services. I will collect this and add it to the proposal wiki once I have privs to create the page. Thanks! Kevin. On 2/11/13 12:03 PM, Eric Sammer wrote: Kevin: Interesting proposal. Can you explain a bit more about what the target use case is? It sounds like there's SSO-ish functionality (presumably a doAs() machine) with integration with directory services, but the proposal also mentions a single point for data and jobs. Is it your intention to provide job submissions and data ingestion APIs for MR and HDFS, respectively? Do you plan to target other ecosystem projects such as HBase? Sorry if I missed this in the proposal. Thanks! On Mon, Feb 11, 2013 at 6:55 AM, Kevin Minder kevin.min...@hortonworks.com**wrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong
Re: [PROPOSAL] Knox Hadoop Gateway Project
I hope to have the repo converted from private to public today (2/11). On 2/11/13 1:44 PM, Jakob Homan wrote: When do you expect the github to be made available? -Jakob On Mon, Feb 11, 2013 at 10:21 AM, Alex Karasulu akaras...@apache.orgwrote: Hi Kevin, This sounds like a much needed project. I endorse the concept but as Bertrand pointed out you need a bit more diversity. Otherwise I see no problem with moving forward. Good luck! On Mon, Feb 11, 2013 at 4:55 PM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships with Other Apache Products === Gateway is going to be used by the users and
Re: [PROPOSAL] Knox Hadoop Gateway Project
The proposal is now hosted on the wiki. http://wiki.apache.org/incubator/knox Currently just the initial proposal. I'll be incorporating feedback/clarifications shortly. On 2/11/13 11:35 AM, Bertrand Delacretaz wrote: Hi, On Mon, Feb 11, 2013 at 5:07 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ...Currently all of the mentors are affiliated with Hortonworks as well. We can reach out to a number of other people and organizations that are interested in the project to diversify if this is an important consideration at this stage I think that would be good - just ask for more mentors on this list. Without such diversity this is basically an all-hortonworks podling, which is not good IMO. I don't seem to have privs to create http://wiki.apache.org/incubator/knox. Would that be the correct place to evolve the proposal? If so, how do I go about creating in?... Yes, http://wiki.apache.org/incubator/is the right place - just ask for write access on this list, a [wiki] marker in the subject line will probably help. I don't think I have the rights to grant that access myself. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Knox Hadoop Gateway Project
The current Knox source is now publicly readable. https://github.com/hortonworks/knox.git On 2/11/13 1:44 PM, Jakob Homan wrote: When do you expect the github to be made available? -Jakob On Mon, Feb 11, 2013 at 10:21 AM, Alex Karasulu akaras...@apache.orgwrote: Hi Kevin, This sounds like a much needed project. I endorse the concept but as Bertrand pointed out you need a bit more diversity. Otherwise I see no problem with moving forward. Good luck! On Mon, Feb 11, 2013 at 4:55 PM, Kevin Minder kevin.min...@hortonworks.comwrote: Knox Gateway Proposal == Abstract == Knox Gateway is a system that provides a single point of secure access for Apache Hadoop clusters. == Proposal == The Knox Gateway (“Gateway” or “Knox”) is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The Gateway runs as a server (or cluster of servers) that serve one or more Hadoop clusters. Provide perimeter security to make Hadoop security setup easier Support authentication and token verification security scenarios Deliver users a single cluster end-point that aggregates capabilities for data and jobs Enable integration with enterprise and cloud identity management environments == Background == An Apache Hadoop cluster is presented to consumers as a loose collection of independent services. This makes it difficult for users to interact with Hadoop since each service maintains it’s own method of access and security. As well, for operators, configuration and administration of a secure Hadoop cluster is a complex and many Hadoop clusters are insecure as a result. == Rationale == Organizations that are struggling with Hadoop cluster security result in a) running Hadoop without security or b) slowing adoption of Hadoop. The Gateway aims to provide perimeter security that integrates more easily into existing organizations’ security infrastructure. Doing so will simplify security for these organizations and benefit all Hadoop stakeholders (i.e. users and operators). Additionally, making a dedicated perimeter security project part of the Apache Hadoop ecosystem will prevent fragmentation in this area and further increase the value of Hadoop as a data platform. == Current Status == Prototype available, developed by the list of initial committers. === Meritocracy === We desire to build a diverse developer community around Gateway following the Apache Way. We want to make the project open source and will encourage contributors from multiple organizations following the Apache meritocracy model. === Community === We hope to extend the user and developer base in the future and build a solid open source community around Gateway. Apache Hadoop has a large ecosystem of open source projects, each with a strong community of contributors. All project communities in this ecosystem have an opportunity to participate in the advancement of the Gateway project because ultimately, Gateway will enable the security capabilities of their project to be more enterprise friendly. === Core Developers === Gateway is currently being developed by several engineers from Hortonworks - Kevin Minder, Larry McCay, John Speidel, Tom Beerbower and Sumit Mohanty. All the engineers have deep expertise in middleware, security identity systems and are quite familiar with the Hadoop ecosystem. === Alignment === The ASF is a natural host for Gateway given that it is already the home of Hadoop, Hive, Pig, HBase, Oozie and other emerging big data software projects. Gateway is designed to solve the security challenges familiar to the Hadoop ecosystem family of projects. == Known Risks == === Orphaned products Reliance on Salaried Developers === The core developers plan to work full time on the project. We believe that this project will be of general interest to many Hadoop users and will attract a diverse set of contributors. We intend to demonstrate this by having contributors from several organizations recognized as committers by the time Knox graduates from incubation. === Inexperience with Open Source === All of the core developers are active users and followers of open source. As well, Hortonworks has a strong heritage of success with contributions to Apache Hadoop Projects. === Homogeneous Developers === The current core developers are from Hortonworks, however, we hope to establish a developer community that includes contributors from several corporations. === Reliance on Salaried Developers === Currently, the developers are paid to do work on Gateway. However, once the project has a community built around it, we expect to get committers and developers from outside the current core developers. === Relationships with Other Apache Products === Gateway is going to be used by the
Re: [PROPOSAL] Knox Hadoop Gateway Project
+1 on the idea. +1 (strong) on the need to increase diversity. I would suggest to open a bit the scope of the project to target other apache projects that act as services. I know it was mentioned, but it sounded to me more like a possibility than a focus. I would also volunteer as a committer and, if you prefer, mentor. I assume it would be possible to take a peek at the code grant in the coming days. Cheers, Hadrian On 02/11/2013 10:10 AM, Bertrand Delacretaz wrote: On Mon, Feb 11, 2013 at 3:55 PM, Kevin Minder kevin.min...@hortonworks.com wrote: ... === Nominated Mentors === Owen O’Malley (omalley AT apache DOT org) Mahadev Konar (mahadev AT apache DOT org) Alan Gates (gates AT apache DOT org) Devaraj Das (ddas AT apache DOT org) ... Considering all initial committers are from hortonworks, I'd like to see affiliations for mentors as well - or at least indicate which mentors are *not* affiliated with hortonworks. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
