[PROPOSAL] Apache Argus Proposal
Apache Argus Proposal (http://wiki.apache.org/incubator/ArgusProposal) == Abstract == Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The name “Argus” is derived from Argus Panoptes, a 100-eyed giant in Greek mythology, endowed with a role to keep “an eye” open and be an effective watchman at all times. == Background == The vision with Argus is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. XA Secure, a Hadoop security focused startup, developed the initial technology behind Argus. XA Secure was acquired by Hortonworks, which now is contributing the technology to the open source community to extend and innovate. == Rationale == Many of the projects in the Hadoop ecosystem have their own authentication, authorization, and auditing components. There are no central administration and auditing capabilities. We are looking to address these enterprises security needs of central administration and comprehensive security through the Argus project. Our initial focus would be around authorization and auditing, the longer term vision would be to tie all aspects around data security within the Hadoop platform. == Proposal Details == The vision of Argus is to enable comprehensive data security across the Hadoop platform. The goal is provide a single user interface or API to manage security policies, monitor user access and policy changes history. The framework would work with individual components in enforcing these policies and in capturing relevant audit information. Initial Goals 1. Donate the Argus source code and documentation to the Apache Software Foundation 2. Setup and standardize the open governance of the Argus project 3. Build a user and developer community 4. Deeper Integration with Hadoop Platform a. Enable integration with Apache Storm, Apache Knox and Apache Falcon for authorization and auditing 5. Configurable centralized storage of audit data into HDFS 6. Enable framework to be run in both Linux and Windows environments 7. Rationalize install procedure, making it easier for enterprises to deploy == Longer Term Goals == In longer term, Argus should provide a comprehensive security framework for Hadoop platform components, covering the following 1. Centralized security administration to manage all security related tasks in a central UI 2. Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration tool a. Standardize authorization method across all Hadoop components b. Enhanced support for different authorization methods - Role based access control, attribute based access control etc c. Enable tag based global policies 3. Centralize auditing of user access and administrative actions (security related) within all the components of Hadoop == Current Status == Argus’ technology is currently being used by enterprises and is under active development. The key components of Argus are: • Enterprise Security Administration Portal ◦ A Java Web Application, designed for administration of security policies from a single location for the entire hadoop cluster (and even multiple hadoop clusters) • Security Agents ◦ A light-weight Java Agent, which will be embedded into the hadoop component (e.g. Hive, HBase and Hadoop) as an authorization provider to enforce the security policies and also collect access events/logs. • User/Group Synchronizer Module ◦ A standalone daemon which allows the user/group information to be synched from the enterprise user repositories like LDAP/AD to Argus local database. This user/group information in Argus local database will help the security policy administrators ▪ to define security policies by selecting users/groups from a drop-down box (instead of typing their name/group in a text-box). ▪ to delegate policy administration to other users/groups ▪ to restrict view of reports based on the users/groups • Centralized Audit Logs and Monitoring ◦ Log events to central data storage/database ◦ Interactive query of audit
Re: [PROPOSAL] Apache Argus Proposal
Interesting proposal, it seems like the core technology being described has a lot in common with Apache Shiro Has collaboration with that community been considered whether just in terms of reusing existing Shiro components wherever possible or something deeper e.g. inviting Shiro developers to help bootstrap the community? Rob On 15/07/2014 03:16, Selvamohan Neethiraj sneethi...@hortonworks.com wrote: Apache Argus Proposal (http://wiki.apache.org/incubator/ArgusProposal) == Abstract == Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The name “Argus” is derived from Argus Panoptes, a 100-eyed giant in Greek mythology, endowed with a role to keep “an eye” open and be an effective watchman at all times. == Background == The vision with Argus is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. XA Secure, a Hadoop security focused startup, developed the initial technology behind Argus. XA Secure was acquired by Hortonworks, which now is contributing the technology to the open source community to extend and innovate. == Rationale == Many of the projects in the Hadoop ecosystem have their own authentication, authorization, and auditing components. There are no central administration and auditing capabilities. We are looking to address these enterprises security needs of central administration and comprehensive security through the Argus project. Our initial focus would be around authorization and auditing, the longer term vision would be to tie all aspects around data security within the Hadoop platform. == Proposal Details == The vision of Argus is to enable comprehensive data security across the Hadoop platform. The goal is provide a single user interface or API to manage security policies, monitor user access and policy changes history. The framework would work with individual components in enforcing these policies and in capturing relevant audit information. Initial Goals 1. Donate the Argus source code and documentation to the Apache Software Foundation 2. Setup and standardize the open governance of the Argus project 3. Build a user and developer community 4. Deeper Integration with Hadoop Platform a. Enable integration with Apache Storm, Apache Knox and Apache Falcon for authorization and auditing 5. Configurable centralized storage of audit data into HDFS 6. Enable framework to be run in both Linux and Windows environments 7. Rationalize install procedure, making it easier for enterprises to deploy == Longer Term Goals == In longer term, Argus should provide a comprehensive security framework for Hadoop platform components, covering the following 1. Centralized security administration to manage all security related tasks in a central UI 2. Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration tool a. Standardize authorization method across all Hadoop components b. Enhanced support for different authorization methods - Role based access control, attribute based access control etc c. Enable tag based global policies 3. Centralize auditing of user access and administrative actions (security related) within all the components of Hadoop == Current Status == Argus’ technology is currently being used by enterprises and is under active development. The key components of Argus are: • Enterprise Security Administration Portal ◦ A Java Web Application, designed for administration of security policies from a single location for the entire hadoop cluster (and even multiple hadoop clusters) • Security Agents ◦ A light-weight Java Agent, which will be embedded into the hadoop component (e.g. Hive, HBase and Hadoop) as an authorization provider to enforce the security policies and also collect access events/logs. • User/Group Synchronizer Module ◦ A standalone daemon which allows the user/group information to be synched from the enterprise user repositories like LDAP/AD to Argus local database. This user/group information in Argus local database will help the security policy administrators ▪ to define security policies by selecting users/groups from a drop-down box (instead of typing their name/group in a
Re: [VOTE] Release Apache Falcon version 0.5-incubating
Hi Justin, Thanks for taking time to review. Falcon uses bootstrap Version: v3.0.2 which is Apache License 2.0. There is a file in the Falcon source release called bootstrap.js which contains the following header: /*! * Bootstrap v3.0.2 by @fat and @mdo * Copyright 2013 Twitter, Inc. * Licensed under http://www.apache.org/licenses/LICENSE-2.0 * * Designed and built with all the love in the world by @mdo and @fat. */ More context at: https://issues.apache.org/jira/browse/FALCON-453?focusedCommentId=14021203page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14021203 Thanks! On Mon, Jul 14, 2014 at 1:08 PM, Justin Mclean jus...@classsoftware.com wrote: Hi, -1 (binding) issues with LICENSE Looks like source is bundling MIT licensed Bootstrap, MIT licensed D3JS, MIT licensed dagre and MIT licensed dust, all of those need to be added to LICENSE. There may be others I've missed. See [1] on how to do this. I checked: - vote thread good - signature an MD5 hash good - has DISCLAIMER - LICENSE mentions subcomponents but contains no licenses and not all bundles software is Apache licensed - no binary files in source release - source headers correct I also suggest you look at these for the next release. - a few files may be missing Apache headers (.twiki and .patch files) - Is copyright year correct in NOTICE? - source release contains patch files - are these needed? Thanks, Justin 1. http://www.apache.org/dev/licensing-howto.html#permissive-deps - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
Re: [VOTE] Release Apache Falcon version 0.5-incubating
Hi, There is a file in the Falcon source release called bootstrap.js which contains the following header. Thanks for clarifying that, it does look like the older version of bootstrap is under Apache and not MIT like the current version. However the LICENSE is still probably incorrect as there's at least 3 other licences that are missing.. What licences are the bindled versions of D3JS, dagre and dust licensed under? If any BSD or MIT licensed software is bundled with the source release must be included in the LICENCE as described here. [1] Thanks, Justin 1. http://www.apache.org/dev/licensing-howto.html#permissive-deps - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[RESULT][IP CLEARANCE] Brooklyn CAMP Server
With no objections being raised, we're good to go for the import Brooklyn folks! -chip On Thu, Jul 10, 2014 at 11:25:34AM -0400, Chip Childers wrote: CloudSoft is donating the Brooklyn CAMP server project to the Apache Brooklyn podling. IP Clearance documentation is here: http://incubator.apache.org/ip-clearance/brooklyn-camp-server.html Thus starts our 72 hour waiting period for any objections... -chip - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Falcon version 0.5-incubating
All are Apache License 2.0. On Tue, Jul 15, 2014 at 7:11 PM, Justin Mclean jus...@classsoftware.com wrote: Hi, There is a file in the Falcon source release called bootstrap.js which contains the following header. Thanks for clarifying that, it does look like the older version of bootstrap is under Apache and not MIT like the current version. However the LICENSE is still probably incorrect as there's at least 3 other licences that are missing.. What licences are the bindled versions of D3JS, dagre and dust licensed under? If any BSD or MIT licensed software is bundled with the source release must be included in the LICENCE as described here. [1] Thanks, Justin 1. http://www.apache.org/dev/licensing-howto.html#permissive-deps - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
Re: [VOTE] Release Apache Falcon version 0.5-incubating
I also suggest you look at these for the next release. - a few files may be missing Apache headers (.twiki and .patch files) twiki as I have explained earlier has a limitation on comments. I did add a license.txt in the folder granting ALv2 as you had suggested in an earlier rc. - Is copyright year correct in NOTICE? Yes. - source release contains patch files - are these needed? This is a mistake and will correct it. On Tue, Jul 15, 2014 at 9:19 PM, Seetharam Venkatesh venkat...@innerzeal.com wrote: All are Apache License 2.0. On Tue, Jul 15, 2014 at 7:11 PM, Justin Mclean jus...@classsoftware.com wrote: Hi, There is a file in the Falcon source release called bootstrap.js which contains the following header. Thanks for clarifying that, it does look like the older version of bootstrap is under Apache and not MIT like the current version. However the LICENSE is still probably incorrect as there's at least 3 other licences that are missing.. What licences are the bindled versions of D3JS, dagre and dust licensed under? If any BSD or MIT licensed software is bundled with the source release must be included in the LICENCE as described here. [1] Thanks, Justin 1. http://www.apache.org/dev/licensing-howto.html#permissive-deps - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
Re: [RESULT][IP CLEARANCE] Brooklyn CAMP Server
Excellent. Thanks Chip, Richard, and all. Best Alex On 15/07/2014 11:50, Chip Childers wrote: With no objections being raised, we're good to go for the import Brooklyn folks! -chip On Thu, Jul 10, 2014 at 11:25:34AM -0400, Chip Childers wrote: CloudSoft is donating the Brooklyn CAMP server project to the Apache Brooklyn podling. IP Clearance documentation is here: http://incubator.apache.org/ip-clearance/brooklyn-camp-server.html Thus starts our 72 hour waiting period for any objections... -chip - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Apache Argus Proposal
Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. All but one PPMC members for this proposal would be from Hortonworks. Personally I think this basically like hosting your project under Hortonwork Github account. I personally would like to have it open via Github to see if could build more external contributors rather than shove it directly to incubator directly. Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? Thanks, - Henry [1] http://knox.apache.org On Mon, Jul 14, 2014 at 7:16 PM, Selvamohan Neethiraj sneethi...@hortonworks.com wrote: Apache Argus Proposal (http://wiki.apache.org/incubator/ArgusProposal) == Abstract == Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The name “Argus” is derived from Argus Panoptes, a 100-eyed giant in Greek mythology, endowed with a role to keep “an eye” open and be an effective watchman at all times. == Background == The vision with Argus is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. XA Secure, a Hadoop security focused startup, developed the initial technology behind Argus. XA Secure was acquired by Hortonworks, which now is contributing the technology to the open source community to extend and innovate. == Rationale == Many of the projects in the Hadoop ecosystem have their own authentication, authorization, and auditing components. There are no central administration and auditing capabilities. We are looking to address these enterprises security needs of central administration and comprehensive security through the Argus project. Our initial focus would be around authorization and auditing, the longer term vision would be to tie all aspects around data security within the Hadoop platform. == Proposal Details == The vision of Argus is to enable comprehensive data security across the Hadoop platform. The goal is provide a single user interface or API to manage security policies, monitor user access and policy changes history. The framework would work with individual components in enforcing these policies and in capturing relevant audit information. Initial Goals 1. Donate the Argus source code and documentation to the Apache Software Foundation 2. Setup and standardize the open governance of the Argus project 3. Build a user and developer community 4. Deeper Integration with Hadoop Platform a. Enable integration with Apache Storm, Apache Knox and Apache Falcon for authorization and auditing 5. Configurable centralized storage of audit data into HDFS 6. Enable framework to be run in both Linux and Windows environments 7. Rationalize install procedure, making it easier for enterprises to deploy == Longer Term Goals == In longer term, Argus should provide a comprehensive security framework for Hadoop platform components, covering the following 1. Centralized security administration to manage all security related tasks in a central UI 2. Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration tool a. Standardize authorization method across all Hadoop components b. Enhanced support for different authorization methods - Role based access control, attribute based access control etc c. Enable tag based global policies 3. Centralize auditing of user access and administrative actions (security related) within all the components of Hadoop == Current Status == Argus’ technology is currently being used by enterprises and is under active development. The key components of Argus are: • Enterprise Security Administration Portal ◦ A Java Web Application, designed for administration of security policies from a single location for the entire hadoop cluster (and even multiple hadoop clusters) • Security Agents ◦ A light-weight Java Agent, which will be embedded into the hadoop component (e.g. Hive, HBase and Hadoop) as an authorization provider to enforce the security policies and also collect access events/logs. • User/Group Synchronizer Module ◦ A
Re: [PROPOSAL] Apache Argus Proposal
On Tue, Jul 15, 2014 at 1:59 PM, Henry Saputra henry.sapu...@gmail.com wrote: Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. Apache incubator is a great place for new projects to grow their community and diversity. Once a project is established it is much harder to change the infrastructure and there projects started on github are more likely to stay on github. I think there is significant value in these projects being done in the Apache Way. All but one PPMC members for this proposal would be from Hortonworks. Would you like to volunteer to help mentor? Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? I think Knox and Argus complement each other. Knox provides perimeter security via proxy/gateway services, while Argus is providing fine grain integrated authorization and auditing. .. Owen
Re: [PROPOSAL] Apache Argus Proposal
I had started typing up a response to Henry's mail but will discard the beginning of it to say I agree with Owen. A new project coming into the incubator quite naturally could have the initial set of committers entirely from one organization. An organization donating an existing code base, for example. However, there has been recent discussion elsewhere that the Incubator should more closely consider if an incubating project has succeeded to grow a community beyond the initially limited group before declaring a project ready for graduation. (Refer to the discussion on the graduation of Apache Tez.) This position seems reasonable, and should naturally apply here. If a project exits graduation with the same lack of PMC/committer diversity with which it entered, this is in effect Apache-washing, in my opinion. A stacked PMC is no more open a community then one controlled by a BDFL and hosted on GitHub. On Tue, Jul 15, 2014 at 2:14 PM, Owen O'Malley omal...@apache.org wrote: On Tue, Jul 15, 2014 at 1:59 PM, Henry Saputra henry.sapu...@gmail.com wrote: Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. Apache incubator is a great place for new projects to grow their community and diversity. Once a project is established it is much harder to change the infrastructure and there projects started on github are more likely to stay on github. I think there is significant value in these projects being done in the Apache Way. All but one PPMC members for this proposal would be from Hortonworks. Would you like to volunteer to help mentor? Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? I think Knox and Argus complement each other. Knox provides perimeter security via proxy/gateway services, while Argus is providing fine grain integrated authorization and auditing. .. Owen -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [PROPOSAL] Apache Argus Proposal
Owen, thanks for the reply. If possible I would like to have mentors coming from different organizations to add some check and balance to the podling. I think most of the incubator projects that have initial members from one organization have diverse mentors from different organizations to help provide more balance opinions and directions. Unfortunately my plate is a bit full right now so I will have to pass on mentoring the proposed project. Thanks for asking though =) - Henry On Tue, Jul 15, 2014 at 2:14 PM, Owen O'Malley omal...@apache.org wrote: On Tue, Jul 15, 2014 at 1:59 PM, Henry Saputra henry.sapu...@gmail.com wrote: Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. Apache incubator is a great place for new projects to grow their community and diversity. Once a project is established it is much harder to change the infrastructure and there projects started on github are more likely to stay on github. I think there is significant value in these projects being done in the Apache Way. All but one PPMC members for this proposal would be from Hortonworks. Would you like to volunteer to help mentor? Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? I think Knox and Argus complement each other. Knox provides perimeter security via proxy/gateway services, while Argus is providing fine grain integrated authorization and auditing. .. Owen - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Apache Argus Proposal
Hi Andrew, thanks for chiming in =) I like your and Owen's opinion about initial members of an incubator project. And also agree it is responsibility of the polling to achieve better diversity during its time under Apache incubator. For this particular proposal, however, the mentors mostly coming from same organization. Hopefully with this proposal announcement, the project could solicit mentors from different organizations to help add check and balance. - Henry On Tue, Jul 15, 2014 at 2:22 PM, Andrew Purtell apurt...@apache.org wrote: I had started typing up a response to Henry's mail but will discard the beginning of it to say I agree with Owen. A new project coming into the incubator quite naturally could have the initial set of committers entirely from one organization. An organization donating an existing code base, for example. However, there has been recent discussion elsewhere that the Incubator should more closely consider if an incubating project has succeeded to grow a community beyond the initially limited group before declaring a project ready for graduation. (Refer to the discussion on the graduation of Apache Tez.) This position seems reasonable, and should naturally apply here. If a project exits graduation with the same lack of PMC/committer diversity with which it entered, this is in effect Apache-washing, in my opinion. A stacked PMC is no more open a community then one controlled by a BDFL and hosted on GitHub. On Tue, Jul 15, 2014 at 2:14 PM, Owen O'Malley omal...@apache.org wrote: On Tue, Jul 15, 2014 at 1:59 PM, Henry Saputra henry.sapu...@gmail.com wrote: Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. Apache incubator is a great place for new projects to grow their community and diversity. Once a project is established it is much harder to change the infrastructure and there projects started on github are more likely to stay on github. I think there is significant value in these projects being done in the Apache Way. All but one PPMC members for this proposal would be from Hortonworks. Would you like to volunteer to help mentor? Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? I think Knox and Argus complement each other. Knox provides perimeter security via proxy/gateway services, while Argus is providing fine grain integrated authorization and auditing. .. Owen -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Apache Argus Proposal
Rob Argus main focus is to holistically solve the security challenges within the complex ecosystem of Hadoop. Argus solution will leverage existing solutions where applicable, which might include using security from Shiro for some of the Web or REST applications authentication. Regards Bosco Interesting proposal, it seems like the core technology being described has a lot in common with Apache Shiro Has collaboration with that community been considered whether just in terms of reusing existing Shiro components wherever possible or something deeper e.g. inviting Shiro developers to help bootstrap the community? Rob On Mon, Jul 14, 2014 at 7:16 PM, Selvamohan Neethiraj sneethi...@hortonworks.com wrote: Apache Argus Proposal (http://wiki.apache.org/incubator/ArgusProposal) == Abstract == Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The name “Argus” is derived from Argus Panoptes, a 100-eyed giant in Greek mythology, endowed with a role to keep “an eye” open and be an effective watchman at all times. == Background == The vision with Argus is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. XA Secure, a Hadoop security focused startup, developed the initial technology behind Argus. XA Secure was acquired by Hortonworks, which now is contributing the technology to the open source community to extend and innovate. == Rationale == Many of the projects in the Hadoop ecosystem have their own authentication, authorization, and auditing components. There are no central administration and auditing capabilities. We are looking to address these enterprises security needs of central administration and comprehensive security through the Argus project. Our initial focus would be around authorization and auditing, the longer term vision would be to tie all aspects around data security within the Hadoop platform. == Proposal Details == The vision of Argus is to enable comprehensive data security across the Hadoop platform. The goal is provide a single user interface or API to manage security policies, monitor user access and policy changes history. The framework would work with individual components in enforcing these policies and in capturing relevant audit information. Initial Goals 1. Donate the Argus source code and documentation to the Apache Software Foundation 2. Setup and standardize the open governance of the Argus project 3. Build a user and developer community 4. Deeper Integration with Hadoop Platform a. Enable integration with Apache Storm, Apache Knox and Apache Falcon for authorization and auditing 5. Configurable centralized storage of audit data into HDFS 6. Enable framework to be run in both Linux and Windows environments 7. Rationalize install procedure, making it easier for enterprises to deploy == Longer Term Goals == In longer term, Argus should provide a comprehensive security framework for Hadoop platform components, covering the following 1. Centralized security administration to manage all security related tasks in a central UI 2. Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration tool a. Standardize authorization method across all Hadoop components b. Enhanced support for different authorization methods - Role based access control, attribute based access control etc c. Enable tag based global policies 3. Centralize auditing of user access and administrative actions (security related) within all the components of Hadoop == Current Status == Argus’ technology is currently being used by enterprises and is under active development. The key components of Argus are: • Enterprise Security Administration Portal ◦ A Java Web Application, designed for administration of security policies from a single location for the entire hadoop cluster (and even multiple hadoop clusters) • Security Agents ◦ A light-weight Java Agent, which will be embedded into the hadoop component (e.g. Hive, HBase and Hadoop) as an authorization provider to enforce the security policies and also collect access events/logs. • User/Group Synchronizer
Re: [PROPOSAL] Apache Argus Proposal
I think it would be useful to expand on how this is different from Sentry too: Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The Apache Knox Gateway is a REST API Gateway for interacting with Hadoop clusters. [for the purposes of...] Authentication (LDAP and Active Directory Authentication Provider) Federation/SSO (HTTP Header Based Identity Federation) Authorization (Service Level Authorization) Auditing Apache Sentry (incubating) is a system for enforcing fine grained role based authorization to data and metadata stored on a Hadoop cluster. The proposal just says it overlaps, and that it might be good have a different group of people reimplement some of the same things. Is there a real gap that people want to build to plug here with overlap at the edges, or is the scope of the pre-existing code driving overlap? (I honestly do not know, asking.) I don't think moving from Github is hard -- or, if a project can't be bothered, should it be an Apache project? I don't know if we're ever going to see a project enter with fully-formed diversity, although this seems like the an example of the most opposite. I understand the argument that things like diversity can be fixed later, and are even helped by parking this under the Apache banner. But you can also do Apache Way things and build a bit of community outside Apache. It feels to me like there should be a minimal evidence of community momentum first. Or else what *isn't* incubatable here? This all may actually exist already and should be highlighted in the proposal if it does. I also do not know, asking here. On Tue, Jul 15, 2014 at 9:59 PM, Henry Saputra henry.sapu...@gmail.com wrote: Maybe we should start asking incubator project to try to build some kind of momentum or community before going to ASF incubator. All but one PPMC members for this proposal would be from Hortonworks. Personally I think this basically like hosting your project under Hortonwork Github account. I personally would like to have it open via Github to see if could build more external contributors rather than shove it directly to incubator directly. Just from high level it seems like it has similar goal as Apache Knox [1], what is the differences between the 2? Thanks, - Henry [1] http://knox.apache.org On Mon, Jul 14, 2014 at 7:16 PM, Selvamohan Neethiraj sneethi...@hortonworks.com wrote: Apache Argus Proposal (http://wiki.apache.org/incubator/ArgusProposal) == Abstract == Argus is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The name “Argus” is derived from Argus Panoptes, a 100-eyed giant in Greek mythology, endowed with a role to keep “an eye” open and be an effective watchman at all times. == Background == The vision with Argus is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. XA Secure, a Hadoop security focused startup, developed the initial technology behind Argus. XA Secure was acquired by Hortonworks, which now is contributing the technology to the open source community to extend and innovate. == Rationale == Many of the projects in the Hadoop ecosystem have their own authentication, authorization, and auditing components. There are no central administration and auditing capabilities. We are looking to address these enterprises security needs of central administration and comprehensive security through the Argus project. Our initial focus would be around authorization and auditing, the longer term vision would be to tie all aspects around data security within the Hadoop platform. == Proposal Details == The vision of Argus is to enable comprehensive data security across the Hadoop platform. The goal is provide a single user interface or API to manage security policies, monitor user access and policy changes history. The framework would work with individual components in enforcing these policies and in capturing relevant audit information. Initial Goals 1. Donate the Argus source code and documentation to the Apache Software Foundation 2. Setup and standardize the open governance of the Argus project 3. Build a user and developer community 4. Deeper Integration with Hadoop Platform a. Enable integration with Apache Storm, Apache Knox and Apache Falcon for authorization and auditing 5. Configurable centralized storage of audit data into HDFS 6. Enable
Re: [PROPOSAL] Apache Argus Proposal
On Tue, Jul 15, 2014 at 2:47 PM, Sean Owen sro...@apache.org wrote: The proposal just says it overlaps, and that it might be good have a different group of people reimplement some of the same things. Is there a real gap that people want to build to plug here with overlap at the edges, or is the scope of the pre-existing code driving overlap? (I honestly do not know, asking.) There are a fair number of Apache projects that overlap each other's scope: - Thrift and Avro - Parquet, Hive's ORC, and Avro's Trevni - Accumulo, HBase, and Cassandra This is accepted because Apache isn't trying to dictate a universal platform. It is about groups of developers learning from each other about how to grow their respective open source communities. That said, Sentry and Argus have different approaches, focus, and technologies. Obviously, open source projects are largely about scratching an unanswered itch. If Sentry met everyone's needs, no one would waste their time reimplementing it. You only work on something if you think you can do it better. .. Owen
Re: [VOTE] Release Apache Falcon version 0.5-incubating
Hi, All are Apache License 2.0. Are you 100% positive? I don't know the exact version included or if these links are representative of the bundled software but none of them look to be Apache licences at a casual glance. BSD licensed: https://github.com/mbostock/d3/blob/master/LICENSE MIT licensed: https://github.com/akdubya/dustjs/blob/master/LICENSE https://github.com/cpettitt/dagre-d3/blob/master/LICENSE Thanks, Justin - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[ANNOUNCE] Apache Tez 0.4.1-incubating released
The Apache Tez team is proud to announce the latest release of Apache Tez - version 0.4.1-incubating. Apache Tez is an application framework which allows for a complex directed-acyclic-graph of tasks for processing data and is built atop Apache Hadoop YARN. More details on Apache Tez can be found at http://tez.incubator.apache.org/ The release bits are available at: http://www.apache.org/dyn/closer.cgi/incubator/tez/tez-0.4.1-incubating/ The released maven artifacts have also been made available on repository.apache.org. We would like to thank all the contributors that made this release possible. Thanks Siddharth Seth on behalf of The Apache Tez Team
Re: [DISCUSS] Incubator exit criteria
On Sat, Jul 12, 2014 at 4:15 AM, Christian Grobmeier grobme...@gmail.com wrote: On 12 Jul 2014, at 8:05, Roman Shaposhnik wrote: That's actually the part of the thread that I have a lot of interest in. Is there any reason not to use attic for hibernated podlings? I am not sure if there is a real reason, but maybe its because the attic currently contains only real ASF projects where all legal things were sorted out. With a podling coming in this might not be the case. We would need to check back with the Attic people if they can handle this. Otherwise I don't see a reason why we shouldn't use the attic, imho its the reason why its there. Quick question: who's in charge of saying 'yay' or 'nay' to Incubator projects coming to the Attic? Roman, would you mind to create an update patch and send it around here for some kind of discussing/voting? Then we would see how people feel. I think this thread is already to big to get the right attention. Will do shortly. Btw, I presume you're talking about the patch to the Incubator web pages documenting the policy, right? At least that's what I'm going to patch and send for a VOTE Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
