Re: [graduation] Maturity model-based assessment of Groovy for its graduation
On Mon, Oct 19, 2015 at 3:25 AM, Emmanuel Lécharnywrote: > ...that might just be me... I agree that each of our projects regularly evaluating their state against the maturity model would be useful. We can either make that a requirement (like once a year as part of the board reporting) or make that an optional "quality label" that projects can display on their website, like a regularly updated version of what we're doing for the Groovy podling at [1]. I prefer the latter, a voluntary public report that can be used by the project's community as well as the board as a tool for evaluating the project's health but without imposing more process on our projects. (and as others have said this would be a more a discussion for comdev, it's not incubator-specific) -Bertrand [1] https://github.com/apache/incubator-groovy/blob/master/MATURITY.adoc - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [graduation] Maturity model-based assessment of Groovy for its graduation
On Mon, Oct 19, 2015 at 2:10 AM, Sam Rubywrote: > ...In particular, this document isn't a "metric", is is more of a > checklist... Yeah that was part of the maturity model's design [1]: avoid the levels of compliance that many such models have, but rather make it modular and granular. I'm glad this seems to be working well, and the idea of focusing on the Incubator output (resolutions + maturity evaluation) rather than its process sounds very good to me. -Bertrand [1] If you're curious the comdev discussions that led to the current version of the maturity model are at http://s.apache.org/apache_maturity_model - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator
Hi, On Mon, Oct 19, 2015 at 2:35 PM, Luke Hanwrote: > ...Could you please help to double check again?... http://incubator.apache.org/projects/kylin.html looks good to me now, so +1. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator
Hi Bertrand, Could you please help to double check again? Would love to have your vote changed if there's no other issue;-) Thanks. Luke Best Regards! - Luke Han On Mon, Oct 19, 2015 at 9:55 AM, Edward J. Yoonwrote: > +1 (binding) > > On Mon, Oct 19, 2015 at 10:54 AM, Li Yang wrote: > > +1 :-) > > > > On Sat, Oct 17, 2015 at 7:17 AM, Saha, Debashis wrote: > > > >> +1 > >> > >> -Original Message- > >> From: 蒋旭 [mailto:jiangxu.ch...@qq.com] > >> Sent: Friday, October 16, 2015 1:23 AM > >> To: general > >> Subject: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator > >> > >> +1 > >> Jiang Xu > >> -- 原始邮件 -- > >> 发件人: "Henry Saputra"; ; > >> 发送时间: 2015年10月16日(星期五) 上午9:16 > >> 收件人: "general@incubator.apache.org" ; > >> > >> 主题: Re: [VOTE] Graduate Apache Kylin from the Apache Incubator > >> > >> > >> > >> +1 (binding) > >> > >> On Thursday, October 15, 2015, Luke Han wrote: > >> > >> > The Apache Kylin community and project made significant advances > >> > during the incubating (from Nov 2014) and believes it is ready to > >> > graduate as a top-level project. > >> > > >> > The Apache Kylin is very active. The PPMC doubled in size (added 6 > >> > committers and 2 mentors) and increased diversity in the past year. > >> > Released 3 version in the past 6 months. There were presentations > >> > about Apache Kylin at most of the big conferences of the world > >> > (including Strata+Hadoop World London, Hadoop Summit San Jose, > >> > ApacheCon EU, Big Data Technology China, Database Technology > >> > Conference > >> > China) and some meetups (Bay Area, > >> > Beijing and one is coming in this weekend in Shanghai), and many talks > >> > around the world. > >> > The dev mailing list is growing very month, about 500+ topics per > >> > month now. > >> > The community created 1000+ JIRA tickets, many patches from > >> > contributors/committers have been merged into code base. > >> > > >> > A vote passed unanimously on the dev@ list (27 +1 votes). Please find > >> > below references to the graduation preparation artifacts: > >> > * discussion on dev list [1] > >> > * vote thread [2] > >> > * podling name search [3] > >> > * incubation status [4] > >> > * proposed resolution below > >> > > >> > We believe Apache Kylin is ready to become a top-level project and if > >> > the IPMC agree we will move to a formal vote. > >> > There are a few more items to be updated on the project status page > >> > and others during the next couple of days. > >> > > >> > > >> > Many thanks to the mentors and the IPMC for the support, Luke Han (on > >> > behalf of the Apache Kylin PPMC) > >> > > >> > [1] http://s.apache.org/KylinDisGraduate > >> > [2] http://s.apache.org/KylinGraduateVote > >> > [3] https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-86 > >> > [4] http://incubator.apache.org/projects/kylin.html > >> > > >> > > >> > > >> > Apache Kylin top-level project resolution: > >> > === > >> > > >> >WHEREAS, the Board of Directors deems it to be in the best > >> >interests of the Foundation and consistent with the > >> >Foundation's purpose to establish a Project Management > >> >Committee charged with the creation and maintenance of > >> >open-source software, for distribution at no charge to the > >> >public, relative to distributed and scalable OLAP engine > >> > > >> >NOW, THEREFORE, BE IT RESOLVED, that a Project Management > >> >Committee (PMC), to be known as the "Apache Kylin Project", > >> >be and hereby is established pursuant to Bylaws of the > >> >Foundation; and be it further > >> > > >> >RESOLVED, that the Apache Kylin Project be and hereby is > >> >responsible for the creation and maintenance of open-source > >> >software related to distributed and scalable OLAP engine; > >> >and be it further > >> > > >> >RESOLVED, that the office of "Vice President, Kylin" be and > >> >hereby is created, the person holding such office to serve at > >> >the direction of the Board of Directors as the chair of the > >> >Apache Kylin Project, and to have primary responsibility for > >> >management of the projects within the scope of responsibility > >> >of the Apache Kylin Project; and be it further > >> > > >> >RESOLVED, that the persons listed immediately below be and > >> >hereby are appointed to serve as the initial members of the > >> >Apache Kylin Project: > >> > > >> > * Dayue Gao > >> > * Jason Zhong > >> > * Julian Hyde > >> > * Luke Han > >> > * Henry Saputra > >> > * Hongbin Ma > >> > * Hua Huang > >> > * Owen O'Malley > >> >
Re: [VOTE] Release Apache Kylin 1.1-incubating (rc1)
+1 mvn test passed hongbin ma于2015年10月19日周一 上午11:28写道: > +1 > > unit test passed on my CI > > On Mon, Oct 19, 2015 at 10:46 AM, Li Yang wrote: > > > +1 > > > > Checksum is correct. > > > > Unit test passed on java version "1.7.0_71", OpenJDK Runtime Environment > > (rhel-2.5.3.1.el6-x86_64 u71-b14) > > > > > > On Thu, Oct 15, 2015 at 10:00 AM, Luke Han wrote: > > > > > +1 > > > > > > MD5 & SHA1 checksum verified > > > License files are all there > > > > > > Unit Test passed > > > > > > > > > > > > Best Regards! > > > - > > > > > > Luke Han > > > > > > On Thu, Oct 15, 2015 at 9:42 AM, ShaoFeng Shi > > > wrote: > > > > > > > Hi all, > > > > > > > > The Apache Kylin community has voted on and approved a proposal to > > > release > > > > Apache Kylin 1.1-incubating. > > > > > > > > Proposal:http://s.apache.org/Jzu > > > > > > > > Vote result: > > > > 7 binding +1 votes > > > > 6 non-binding +1 votes > > > > No -1 voteshttp://s.apache.org/kylin-1.1-result_rc1 > > > > > > > > > > > > The commit to be voted > > > > upon: > > > > > > > > > > https://github.com/apache/incubator-kylin/commit/1955a2f9aea7b7f608f0496c00807715ea4246a5 > > > > > > > > Its hash is 1955a2f9aea7b7f608f0496c00807715ea4246a5. > > > > > > > > The artifacts to be voted on are located > > > > here: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/kylin/apache-kylin-1.1-incubating-rc1/ > > > > > > > > The hashes of the artifacts are as follows: > > > > apache-kylin-1.1-incubating-src.tar.gz.md5 > > > 18dfbb012e1eb807b1a5c1b134a537aa > > > > apache-kylin-1.1-incubating-src.tar.gz.sha1 > > > > 041ee010c67a0b5611d9dd06f7fd6f37388c5374 > > > > > > > > A staged Maven repository is available for review > > > > at: > > > > https://repository.apache.org/content/repositories/orgapachekylin-1012/ > > > > > > > > Release artifacts are signed with the following > > > > key:https://people.apache.org/keys/committer/shaofengshi.asc > > > > > > > > Pursuant to the Releases section of the Incubation Policy and with > > > > the endorsement of our mentors we would now like to request > > > > the permission of the Incubator PMC to publish the release. The vote > > > > is open for 72 hours, or until the necessary number of votes (3 +1) > > > > is reached. > > > > > > > > [ ] +1 Release this package > > > > [ ] 0 I don't feel strongly about it, but I'm okay with the release > > > > [ ] -1 Do not release this package because... > > > > > > > > > > > > Shaofeng Shi, on behalf of Apache Kylin ppmcshaofeng...@apache.org > > > > > > > > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone >
Re: Starting from the other end
Hi, On Thu, Oct 15, 2015 at 8:55 PM, Ted Dunningwrote: > ...The first step (What Apache Incubator Does) can be edited at > https://docs.google.com/document/d/1dxUVHGWcj83wIVnskYL7iM7PiiScLLNS4HjGHa6LsgA/edit?usp=sharing > ... I like that. Combined with a set of commented links that point to good examples of the various phases and transitions (proposal, acceptance, etc.) I think this could replace a large part of the Incubator documentation. I don't have any specific changes to request at this point, I think the current draft is good enough to move to our wiki or website. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Kylin 1.1-incubating (rc1)
+1 > On Oct 19, 2015, at 7:40 AM, Julian Hydewrote: > > Forwarding my vote from the dev list: > > +1 (binding) > >> On Oct 19, 2015, at 1:18 AM, 周千昊 wrote: >> >> +1 >> mvn test passed >> >> hongbin ma 于2015年10月19日周一 上午11:28写道: >> >>> +1 >>> >>> unit test passed on my CI >>> On Mon, Oct 19, 2015 at 10:46 AM, Li Yang wrote: +1 Checksum is correct. Unit test passed on java version "1.7.0_71", OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14) > On Thu, Oct 15, 2015 at 10:00 AM, Luke Han wrote: > > +1 > > MD5 & SHA1 checksum verified > License files are all there > > Unit Test passed > > > > Best Regards! > - > > Luke Han > > On Thu, Oct 15, 2015 at 9:42 AM, ShaoFeng Shi > wrote: > >> Hi all, >> >> The Apache Kylin community has voted on and approved a proposal to > release >> Apache Kylin 1.1-incubating. >> >> Proposal:http://s.apache.org/Jzu >> >> Vote result: >> 7 binding +1 votes >> 6 non-binding +1 votes >> No -1 voteshttp://s.apache.org/kylin-1.1-result_rc1 >> >> >> The commit to be voted >> upon: >>> https://github.com/apache/incubator-kylin/commit/1955a2f9aea7b7f608f0496c00807715ea4246a5 >> >> Its hash is 1955a2f9aea7b7f608f0496c00807715ea4246a5. >> >> The artifacts to be voted on are located >> here: >>> https://dist.apache.org/repos/dist/dev/incubator/kylin/apache-kylin-1.1-incubating-rc1/ >> >> The hashes of the artifacts are as follows: >> apache-kylin-1.1-incubating-src.tar.gz.md5 > 18dfbb012e1eb807b1a5c1b134a537aa >> apache-kylin-1.1-incubating-src.tar.gz.sha1 >> 041ee010c67a0b5611d9dd06f7fd6f37388c5374 >> >> A staged Maven repository is available for review >> at: >>> https://repository.apache.org/content/repositories/orgapachekylin-1012/ >> >> Release artifacts are signed with the following >> key:https://people.apache.org/keys/committer/shaofengshi.asc >> >> Pursuant to the Releases section of the Incubation Policy and with >> the endorsement of our mentors we would now like to request >> the permission of the Incubator PMC to publish the release. The vote >> is open for 72 hours, or until the necessary number of votes (3 +1) >> is reached. >> >> [ ] +1 Release this package >> [ ] 0 I don't feel strongly about it, but I'm okay with the release >> [ ] -1 Do not release this package because... >> >> >> Shaofeng Shi, on behalf of Apache Kylin ppmcshaofeng...@apache.org >>> >>> >>> >>> -- >>> Regards, >>> >>> *Bin Mahone | 马洪宾* >>> Apache Kylin: http://kylin.io >>> Github: https://github.com/binmahone > > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org >
[DISCUSS] Eagle incubator proposal
Hello Everyone, My name is Arun Manoharan. Currently a product manager in the Analytics platform team at eBay Inc. I would like to start a discussion on Eagle and its joining the ASF as an incubation project. Eagle is a Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities and take actions in real time. Eagle supports a wide variety of policies on HDFS data and Hive. Eagle also provides machine learning models for detecting anomalous user behavior in Hadoop. The proposal is available on the wiki here: https://wiki.apache.org/incubator/EagleProposal The text of the proposal is also available at the end of this email. Thanks for your time and help. Thanks, Arun Eagle Abstract Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities in hadoop and take actions. Proposal Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces policies defined on sensitive data access and alerts or blocks user’s access to that sensitive data in real time. Eagle also creates user profiles based on the typical access behaviour for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import sensitive data information classified by external classification engines to help define its policies. Overview of Eagle Eagle has 3 main parts. 1.Data collection and storage - Eagle collects data from various hadoop logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. 2.Data processing and policy engine - Eagle allows users to create policies based on various metadata properties on HDFS, Hive and HBase data. 3.Eagle services - Eagle services include policy manager, query service and the visualization component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond to real time alerts. Data Collection and Storage: Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. Data Processing and Policy Engine: Processing Engine: Eagle provides stream processing API which is an abstraction of Apache Storm. It can also be extended to other streaming engines. This abstraction allows developers to assemble data transformation, filtering, external data join etc. without physically bound to a specific streaming platform. Eagle streaming API allows developers to easily integrate business logic with Eagle policy engine and internally Eagle framework compiles business logic execution DAG into program primitives of underlying stream infrastructure e.g. Apache Storm. For example, Eagle HDFS monitoring transforms audit log from Namenode to object and joins sensitivity metadata, security zone metadata which are generated from external programs or configured by user. Eagle hive monitoring filters running jobs to get hive query string and parses query string into object and then joins sensitivity metadata. Alerting Framework: Eagle Alert Framework includes stream metadata API, scalable policy engine framework, extensible policy engine framework. Stream metadata API allows developers to declare event schema including what attributes constitute an event, what is the type for each attribute, and how to dynamically resolve attribute value in runtime when user configures policy. Scalable policy engine framework allows policies to be executed on different physical nodes in parallel. It is also used to define your own policy partitioner class. Policy engine framework together with streaming partitioning capability provided by all streaming platforms will make sure policies and events can be evaluated in a fully distributed way. Extensible policy engine framework allows developer to plugin a new policy engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy engine which Eagle supports as first-class citizen. Machine Learning module: Eagle provides capabilities to define user activity patterns or user profiles for Hadoop users based on the user behaviour in the platform. These user profiles are modeled using Machine Learning algorithms and used for detection of anomalous users activities. Eagle uses Eigen Value Decomposition, and Density Estimation algorithms for generating user profile models. The model reads data from HDFS audit logs, preprocesses and aggregates data, and generates models using Spark programming APIs. Once models are generated, Eagle uses stream
Re: [VOTE] Release Apache Kylin 1.1-incubating (rc1)
Forwarding my vote from the dev list: +1 (binding) > On Oct 19, 2015, at 1:18 AM, 周千昊wrote: > > +1 > mvn test passed > > hongbin ma 于2015年10月19日周一 上午11:28写道: > >> +1 >> >> unit test passed on my CI >> >> On Mon, Oct 19, 2015 at 10:46 AM, Li Yang wrote: >> >>> +1 >>> >>> Checksum is correct. >>> >>> Unit test passed on java version "1.7.0_71", OpenJDK Runtime Environment >>> (rhel-2.5.3.1.el6-x86_64 u71-b14) >>> >>> >>> On Thu, Oct 15, 2015 at 10:00 AM, Luke Han wrote: >>> +1 MD5 & SHA1 checksum verified License files are all there Unit Test passed Best Regards! - Luke Han On Thu, Oct 15, 2015 at 9:42 AM, ShaoFeng Shi wrote: > Hi all, > > The Apache Kylin community has voted on and approved a proposal to release > Apache Kylin 1.1-incubating. > > Proposal:http://s.apache.org/Jzu > > Vote result: > 7 binding +1 votes > 6 non-binding +1 votes > No -1 voteshttp://s.apache.org/kylin-1.1-result_rc1 > > > The commit to be voted > upon: > >>> >> https://github.com/apache/incubator-kylin/commit/1955a2f9aea7b7f608f0496c00807715ea4246a5 > > Its hash is 1955a2f9aea7b7f608f0496c00807715ea4246a5. > > The artifacts to be voted on are located > here: > >>> >> https://dist.apache.org/repos/dist/dev/incubator/kylin/apache-kylin-1.1-incubating-rc1/ > > The hashes of the artifacts are as follows: > apache-kylin-1.1-incubating-src.tar.gz.md5 18dfbb012e1eb807b1a5c1b134a537aa > apache-kylin-1.1-incubating-src.tar.gz.sha1 > 041ee010c67a0b5611d9dd06f7fd6f37388c5374 > > A staged Maven repository is available for review > at: >> https://repository.apache.org/content/repositories/orgapachekylin-1012/ > > Release artifacts are signed with the following > key:https://people.apache.org/keys/committer/shaofengshi.asc > > Pursuant to the Releases section of the Incubation Policy and with > the endorsement of our mentors we would now like to request > the permission of the Incubator PMC to publish the release. The vote > is open for 72 hours, or until the necessary number of votes (3 +1) > is reached. > > [ ] +1 Release this package > [ ] 0 I don't feel strongly about it, but I'm okay with the release > [ ] -1 Do not release this package because... > > > Shaofeng Shi, on behalf of Apache Kylin ppmcshaofeng...@apache.org > >>> >> >> >> >> -- >> Regards, >> >> *Bin Mahone | 马洪宾* >> Apache Kylin: http://kylin.io >> Github: https://github.com/binmahone >> - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
Hi Arun, very interesting proposal. I may see some possible interaction with Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a kind of Change Data Capture), etc. So, I see a different perspective in Eagle, but Eagle could also leverage Falcon somehow. Regards JB On 10/19/2015 05:33 PM, Manoharan, Arun wrote: Hello Everyone, My name is Arun Manoharan. Currently a product manager in the Analytics platform team at eBay Inc. I would like to start a discussion on Eagle and its joining the ASF as an incubation project. Eagle is a Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities and take actions in real time. Eagle supports a wide variety of policies on HDFS data and Hive. Eagle also provides machine learning models for detecting anomalous user behavior in Hadoop. The proposal is available on the wiki here: https://wiki.apache.org/incubator/EagleProposal The text of the proposal is also available at the end of this email. Thanks for your time and help. Thanks, Arun Eagle Abstract Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities in hadoop and take actions. Proposal Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces policies defined on sensitive data access and alerts or blocks user’s access to that sensitive data in real time. Eagle also creates user profiles based on the typical access behaviour for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import sensitive data information classified by external classification engines to help define its policies. Overview of Eagle Eagle has 3 main parts. 1.Data collection and storage - Eagle collects data from various hadoop logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. 2.Data processing and policy engine - Eagle allows users to create policies based on various metadata properties on HDFS, Hive and HBase data. 3.Eagle services - Eagle services include policy manager, query service and the visualization component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond to real time alerts. Data Collection and Storage: Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. Data Processing and Policy Engine: Processing Engine: Eagle provides stream processing API which is an abstraction of Apache Storm. It can also be extended to other streaming engines. This abstraction allows developers to assemble data transformation, filtering, external data join etc. without physically bound to a specific streaming platform. Eagle streaming API allows developers to easily integrate business logic with Eagle policy engine and internally Eagle framework compiles business logic execution DAG into program primitives of underlying stream infrastructure e.g. Apache Storm. For example, Eagle HDFS monitoring transforms audit log from Namenode to object and joins sensitivity metadata, security zone metadata which are generated from external programs or configured by user. Eagle hive monitoring filters running jobs to get hive query string and parses query string into object and then joins sensitivity metadata. Alerting Framework: Eagle Alert Framework includes stream metadata API, scalable policy engine framework, extensible policy engine framework. Stream metadata API allows developers to declare event schema including what attributes constitute an event, what is the type for each attribute, and how to dynamically resolve attribute value in runtime when user configures policy. Scalable policy engine framework allows policies to be executed on different physical nodes in parallel. It is also used to define your own policy partitioner class. Policy engine framework together with streaming partitioning capability provided by all streaming platforms will make sure policies and events can be evaluated in a fully distributed way. Extensible policy engine framework allows developer to plugin a new policy engine with a few lines of codes. WSO2 Siddhi CEP engine is the policy engine which Eagle supports as first-class citizen. Machine Learning module: Eagle provides capabilities to define user activity patterns or user profiles for Hadoop users based on the user behaviour in the platform. These user profiles are modeled using Machine Learning
Re: [DISCUSS] Eagle incubator proposal
I would suggest that Owen O'Malley has not had enough time to be a viable mentor recently and should not be on the list of mentors. Henry and Julian are good if their schedules permit. Henry, I know has been mentoring a number of projects lately. On Mon, Oct 19, 2015 at 8:40 AM, Jean-Baptiste Onofréwrote: > Hi Arun, > > very interesting proposal. I may see some possible interaction with > Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a > kind of Change Data Capture), etc. > > So, I see a different perspective in Eagle, but Eagle could also leverage > Falcon somehow. > > Regards > JB > > > On 10/19/2015 05:33 PM, Manoharan, Arun wrote: > >> Hello Everyone, >> >> My name is Arun Manoharan. Currently a product manager in the Analytics >> platform team at eBay Inc. >> >> I would like to start a discussion on Eagle and its joining the ASF as an >> incubation project. >> >> Eagle is a Monitoring solution for Hadoop to instantly identify access to >> sensitive data, recognize attacks, malicious activities and take actions in >> real time. Eagle supports a wide variety of policies on HDFS data and Hive. >> Eagle also provides machine learning models for detecting anomalous user >> behavior in Hadoop. >> >> The proposal is available on the wiki here: >> https://wiki.apache.org/incubator/EagleProposal >> >> The text of the proposal is also available at the end of this email. >> >> Thanks for your time and help. >> >> Thanks, >> Arun >> >> >> >> Eagle >> >> Abstract >> Eagle is an Open Source Monitoring solution for Hadoop to instantly >> identify access to sensitive data, recognize attacks, malicious activities >> in hadoop and take actions. >> >> Proposal >> Eagle audits access to HDFS files, Hive and HBase tables in real time, >> enforces policies defined on sensitive data access and alerts or blocks >> user’s access to that sensitive data in real time. Eagle also creates user >> profiles based on the typical access behaviour for HDFS and Hive and sends >> alerts when anomalous behaviour is detected. Eagle can also import >> sensitive data information classified by external classification engines to >> help define its policies. >> >> Overview of Eagle >> Eagle has 3 main parts. >> 1.Data collection and storage - Eagle collects data from various hadoop >> logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. >> 2.Data processing and policy engine - Eagle allows users to create >> policies based on various metadata properties on HDFS, Hive and HBase data. >> 3.Eagle services - Eagle services include policy manager, query service >> and the visualization component. Eagle provides intuitive user interface to >> administer Eagle and an alert dashboard to respond to real time alerts. >> >> Data Collection and Storage: >> Eagle provides programming API for extending Eagle to integrate any data >> source into Eagle policy evaluation framework. For example, Eagle hdfs >> audit monitoring collects data from Kafka which is populated from namenode >> log4j appender or from logstash agent. Eagle hive monitoring collects hive >> query logs from running job through YARN API, which is designed to be >> scalable and fault-tolerant. Eagle uses HBase as storage for storing >> metadata and metrics data, and also supports relational database through >> configuration change. >> >> Data Processing and Policy Engine: >> Processing Engine: Eagle provides stream processing API which is an >> abstraction of Apache Storm. It can also be extended to other streaming >> engines. This abstraction allows developers to assemble data >> transformation, filtering, external data join etc. without physically bound >> to a specific streaming platform. Eagle streaming API allows developers to >> easily integrate business logic with Eagle policy engine and internally >> Eagle framework compiles business logic execution DAG into program >> primitives of underlying stream infrastructure e.g. Apache Storm. For >> example, Eagle HDFS monitoring transforms audit log from Namenode to object >> and joins sensitivity metadata, security zone metadata which are generated >> from external programs or configured by user. Eagle hive monitoring filters >> running jobs to get hive query string and parses query string into object >> and then joins sensitivity metadata. >> Alerting Framework: Eagle Alert Framework includes stream metadata API, >> scalable policy engine framework, extensible policy engine framework. >> Stream metadata API allows developers to declare event schema including >> what attributes constitute an event, what is the type for each attribute, >> and how to dynamically resolve attribute value in runtime when user >> configures policy. Scalable policy engine framework allows policies to be >> executed on different physical nodes in parallel. It is also used to define >> your own policy partitioner class. Policy engine framework together with >> streaming partitioning
Re: Starting from the other end
On Mon, Oct 19, 2015 at 9:52 AM, Dennis E. Hamiltonwrote: > I did some wordsmithing but not certain how that works properly in Google > Docs in terms of how changes are made evident for review, etc. > > - Dennis > > PS: When I go back and review, I see that I needed to leave more > highlights or comments to indicate places where I made adjustments. (Maybe > a wiki would be better, so there is a modification history?) > Docs maintains a complete history of all changes you can access this by clicking on the "Last change was mad ..." language in the tool bar. Also, you can set "suggesting" instead of "editing" on the right side of the toolbar so that everybody can see your changes highlighted. I have been reviewing all changes and occasionally restating them a bit to keep a more coherent voice.
Re: When podlings don't file a report
On Thu, Oct 15, 2015 at 10:56 AM, Ross Gardlerwrote: > Mentors, in my opinion, are not responsible for their podlings. They are > responsible for guiding the podlings but not for filing. > > Mentors do have a responsibility to the IPMC to make a recommendation (e.g. > "I've looked into the failure to report and am happy with the status, it's > just busy people this month" or "Looks to me like the podling community > don't take reporting seriously. As a mentor I've tried but failed to > communicate the importance this.") When is that responsibility to the IPMC discharged? Only when a shepherd, or a random volunteer, inquires? If a podling fails to report indefinitely, do we have any expectation that its Mentors will act? I suppose that as an individual volunteer I can start threads about the podlings that are not reporting on either private@incubator or general@incubator. It seems wrong to have an independent meddler taking on that task, though, instead of the project Mentors. I think this illustrates again why the "shepherd" institution as implemented by the Incubator is pernicious. Incubator shepherds shouldn't own the task of checking up on podlings -- they are unreliable and their responsibility is ephemeral. It should instead be Mentors performing shepherding tasks, since Mentors are assigned to podlings durably and are accountable to the IPMC. Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: When podlings don't file a report
This is one aspect that I feel needs fixing. Currently there are too many people who "might" be responsible. What we need is someone who *is* responsible. It's initially the mentors, but if (as per your original question) a podling has failed to file for 4 months then the mentors are clearly dropping the ball or they need assistance in impressing upon the podling community how important this is. So who next? My answer is the IPMC, but then we hit the "too many cooks problem". I'll not go into suggestions for fixing this problem as there are plenty of threads already looking at how to improve oversight. This is just one of the many symptoms of the problem. To your "shepherd" point below the term here in the IPMC is different to the term at the board level. Board shepherds *are* responsible for following up after board meetings. That's where the buck stops. Ross -Original Message- From: Marvin Humphrey [mailto:mar...@rectangular.com] Sent: Monday, October 19, 2015 1:27 PM To: general@incubator.apache.org Subject: Re: When podlings don't file a report On Thu, Oct 15, 2015 at 10:56 AM, Ross Gardlerwrote: > Mentors, in my opinion, are not responsible for their podlings. They > are responsible for guiding the podlings but not for filing. > > Mentors do have a responsibility to the IPMC to make a recommendation (e.g. > "I've looked into the failure to report and am happy with the status, > it's just busy people this month" or "Looks to me like the podling > community don't take reporting seriously. As a mentor I've tried but > failed to communicate the importance this.") When is that responsibility to the IPMC discharged? Only when a shepherd, or a random volunteer, inquires? If a podling fails to report indefinitely, do we have any expectation that its Mentors will act? I suppose that as an individual volunteer I can start threads about the podlings that are not reporting on either private@incubator or general@incubator. It seems wrong to have an independent meddler taking on that task, though, instead of the project Mentors. I think this illustrates again why the "shepherd" institution as implemented by the Incubator is pernicious. Incubator shepherds shouldn't own the task of checking up on podlings -- they are unreliable and their responsibility is ephemeral. It should instead be Mentors performing shepherding tasks, since Mentors are assigned to podlings durably and are accountable to the IPMC. Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [DISCUSS] Eagle incubator proposal
Hi JB, That is a good Point. Good to know that Falcon feeds HDFS/Hive/HBase data changes, so this feature would complement Eagle which today mainly focuses on HDFS/Hive/HBase data access including view, change, delete etc. Eagle would benefit if Eagle can instantly capture data change from Falcon. Thanks Edward Zhang On 10/19/15, 8:40, "Jean-Baptiste Onofré"wrote: >Hi Arun, > >very interesting proposal. I may see some possible interaction with >Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with >a kind of Change Data Capture), etc. > >So, I see a different perspective in Eagle, but Eagle could also >leverage Falcon somehow. > >Regards >JB > >On 10/19/2015 05:33 PM, Manoharan, Arun wrote: >> Hello Everyone, >> >> My name is Arun Manoharan. Currently a product manager in the Analytics >>platform team at eBay Inc. >> >> I would like to start a discussion on Eagle and its joining the ASF as >>an incubation project. >> >> Eagle is a Monitoring solution for Hadoop to instantly identify access >>to sensitive data, recognize attacks, malicious activities and take >>actions in real time. Eagle supports a wide variety of policies on HDFS >>data and Hive. Eagle also provides machine learning models for detecting >>anomalous user behavior in Hadoop. >> >> The proposal is available on the wiki here: >> https://wiki.apache.org/incubator/EagleProposal >> >> The text of the proposal is also available at the end of this email. >> >> Thanks for your time and help. >> >> Thanks, >> Arun >> >> >> >> Eagle >> >> Abstract >> Eagle is an Open Source Monitoring solution for Hadoop to instantly >>identify access to sensitive data, recognize attacks, malicious >>activities in hadoop and take actions. >> >> Proposal >> Eagle audits access to HDFS files, Hive and HBase tables in real time, >>enforces policies defined on sensitive data access and alerts or blocks >>user¹s access to that sensitive data in real time. Eagle also creates >>user profiles based on the typical access behaviour for HDFS and Hive >>and sends alerts when anomalous behaviour is detected. Eagle can also >>import sensitive data information classified by external classification >>engines to help define its policies. >> >> Overview of Eagle >> Eagle has 3 main parts. >> 1.Data collection and storage - Eagle collects data from various hadoop >>logs in real time using Kafka/Yarn API and uses HDFS and HBase for >>storage. >> 2.Data processing and policy engine - Eagle allows users to create >>policies based on various metadata properties on HDFS, Hive and HBase >>data. >> 3.Eagle services - Eagle services include policy manager, query service >>and the visualization component. Eagle provides intuitive user interface >>to administer Eagle and an alert dashboard to respond to real time >>alerts. >> >> Data Collection and Storage: >> Eagle provides programming API for extending Eagle to integrate any >>data source into Eagle policy evaluation framework. For example, Eagle >>hdfs audit monitoring collects data from Kafka which is populated from >>namenode log4j appender or from logstash agent. Eagle hive monitoring >>collects hive query logs from running job through YARN API, which is >>designed to be scalable and fault-tolerant. Eagle uses HBase as storage >>for storing metadata and metrics data, and also supports relational >>database through configuration change. >> >> Data Processing and Policy Engine: >> Processing Engine: Eagle provides stream processing API which is an >>abstraction of Apache Storm. It can also be extended to other streaming >>engines. This abstraction allows developers to assemble data >>transformation, filtering, external data join etc. without physically >>bound to a specific streaming platform. Eagle streaming API allows >>developers to easily integrate business logic with Eagle policy engine >>and internally Eagle framework compiles business logic execution DAG >>into program primitives of underlying stream infrastructure e.g. Apache >>Storm. For example, Eagle HDFS monitoring transforms audit log from >>Namenode to object and joins sensitivity metadata, security zone >>metadata which are generated from external programs or configured by >>user. Eagle hive monitoring filters running jobs to get hive query >>string and parses query string into object and then joins sensitivity >>metadata. >> Alerting Framework: Eagle Alert Framework includes stream metadata API, >>scalable policy engine framework, extensible policy engine framework. >>Stream metadata API allows developers to declare event schema including >>what attributes constitute an event, what is the type for each >>attribute, and how to dynamically resolve attribute value in runtime >>when user configures policy. Scalable policy engine framework allows >>policies to be executed on different physical nodes in parallel. It is >>also used to define your own policy partitioner class. Policy engine >>framework together with streaming
[RESULT] [VOTE] Release Apache REEF 0.13.0-incubating (rc1)
Hi, Voting for Apache REEF 0.13.0-incubating (rc1) is now closed. The release has passed with three binding +1's, five non-binding +1's and no other votes. Thank you everyone who participated! 3 binding +1's: - Daniel Gruno - Ross Gardler - Chris Douglas 5 non-binding +1's: - Dongjoon Hyun - Brian Cho - Markus Weimer - Andrew Chung - Julia Wang We will now continue with the release process. Thanks, Mariia From: Mariia Mykhailova Sent: Friday, October 9, 2015 7:03 PM To: 'general@incubator.apache.org'Subject: [VOTE] Release Apache REEF 0.13.0-incubating (rc1) The Apache REEF PPMC has voted to release Apache REEF 0.13.0-incubating based on the release candidate described below. Now it is the IPMC's turn to vote. The PPMC vote passed with 5 +1 votes: http://mail-archives.apache.org/mod_mbox/incubator-reef-dev/201510.mbox/%3CBL2PR03MB2900A370B6F6894F8E08EA8D2370%40BL2PR03MB290.namprd03.prod.outlook.com%3E The source tar ball, including signatures, digests, etc can be found at: https://dist.apache.org/repos/dist/dev/incubator/reef/0.13.0-incubating-rc1/ The Git tag is release-0.13.0-incubating-rc1 The Git commit ID is e1d42e8008d97714a0f7bc303dae37239a10a029 Checksums of apache-reef-0.13.0-incubating-rc1.tar.gz: MD5: 8e0a1cd6c7e17a86abbd32366c8cccac SHA512: 8f542aeaf2dc3b241bdcd0d343c607355e1f09e1ca89bbc3431b0cc1f0908479511f60900a91a6731051ffef8af30488eb85df567c32bc2db9d3d91014c4fed7 Release artifacts are signed with a key found in the KEYS file available here: https://dist.apache.org/repos/dist/release/incubator/reef/KEYS Issues Resolved in the release https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315820=12332972 The vote will be open for 72 hours. Please download the release candidate, check the hashes/signature, build it and test it, and then please vote: [ ] +1 Release this package as Apache REEF 0.13.0-incubating [ ] +0 no opinion [ ] -1 Do not release this package because ... Thanks!
Re: Starting from the other end
Bertrand, Thanks. My content vision here is that the landing page at incubator should highlight whatever task that most people landing there want to do. My suspicion is that this task is to answer either "what is incubator?" or "how can I bring my project to Apache?". Regardless, we will need a landing page for "how can I bring ..." and one for "what is..". This document is intended ultimately to satisfy the "what is..." question and hopefully form a core of the "how can I..." question. As this migrates to web form, it should *definitely* gather links to more detailed descriptions. The maturity model document and the release process descriptions are great candidates for that. Another key mission will be education of mentors about what has gone before and how various tasks can be done. I don't expect that this document will serve that purpose. Instead, I think that much of our current documentation needs to be exposed and isolated for clarity. For instance, we currently expect a mentor to scroll way down a page for an intro and then click on a link to a page that they have to interpolate a bit in order to assist with adding a new committer to a podling. The content along the way is excellent, but I think we can expose it better. In parallel, I want to use the consensus to start updating and refining our understanding of what we can improve. My guess is that a big part of what people perceive as broken is really just difference of vision. On Mon, Oct 19, 2015 at 1:28 AM, Bertrand Delacretazwrote: > Hi, > > On Thu, Oct 15, 2015 at 8:55 PM, Ted Dunning > wrote: > > ...The first step (What Apache Incubator Does) can be edited at > > > https://docs.google.com/document/d/1dxUVHGWcj83wIVnskYL7iM7PiiScLLNS4HjGHa6LsgA/edit?usp=sharing > ... > > I like that. > > Combined with a set of commented links that point to good examples of > the various phases and transitions (proposal, acceptance, etc.) I > think this could replace a large part of the Incubator documentation. > > I don't have any specific changes to request at this point, I think > the current draft is good enough to move to our wiki or website. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: Starting from the other end
On Mon, Oct 19, 2015 at 3:38 PM, Roman Shaposhnikwrote: > I think I'm with Sam. I think these are related. Or to be more explicit: > the way a project joins a foundation (regardless of what path it takes) > is via a board passing a resolution. > > Perhaps defining exact criteria that the board uses to evaluate such > resolutions would be useful. > Perhaps it would be. My focus right now is making sure that the Incubator works well. So far, direct to top-level is an exceptional case handled by people who need neither Incubator nor documentation. And Incubator alternatives don't exist yet and wouldn't be the Incubator even if they did. So the document we have here is about the Incubator as it is.
Re: Starting from the other end
On Thu, Oct 15, 2015 at 9:55 PM, Ted Dunningwrote: > We have just had a few interminable threads regarding what is wrong with > incubator and jumping directly to implementation attempts for changes. > > That isn't the way I would code something complex so it seems to me that by > analogy we might want to start from the other end and start with what we > all agree on. > > Building on that agreement and common ground, I think it will be much, much > easier to determine where we have weaknesses, bottlenecks or limitations > and also to suggest improvements. > > To do this, I am going to start a google doc to describe WHAT the incubator > is supposed to do (easier to edit than a wiki) and once that document > stabilizes a bit will move it to the wiki. > > That document should be able to let us move to similar efforts to describe > HOW the incubator currently works. I would hope that one side effect of > this second stage would be to pull together current documentation on the > current incubator. > > At that point, we can point out specific improvements and start the real > work of making things better. > > So here we go. The first step (What Apache Incubator Does) can be edited at > > https://docs.google.com/document/d/1dxUVHGWcj83wIVnskYL7iM7PiiScLLNS4HjGHa6LsgA/edit?usp=sharing I've reviewed the document and left a couple of comments. Feel free to incorporate the suggestions in the main text. In my view this document achieves the goal of stating things that we can all agree on as to what Incubator should be doing. It works great as a doc you can put on the front page of incubator.apache.org. What it currently lacks is specifics on how are we going to make sure that all the right things that this document postulates actually happen. There's not explanation of checks-n-balances. So, for example (just to pick a random spot) when we talk about "The Apache Incubator's function at this point is..." I think we can all agree on what the function is but what is the process that guarantees that that functions succeeds is the million dollar question. Ted, was it your intent to first get an agreement on *what* we all expect from the Incubator and then follow up on *how* are these expectations going to be met by our current policies? Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Starting from the other end
On Mon, Oct 19, 2015 at 5:38 PM, Roman Shaposhnikwrote: >... > Perhaps defining exact criteria that the board uses to evaluate such > resolutions would be useful. > The Board evaluates them differently based on the path. I explained Direct else-thread. Historically, the Board has trusted the Incubator to only provide resolutions for TLPs when the code/community is ready. A number of people are claiming that trust is currently misplaced, which has spun up these discussions over the past couple weeks. The two primary paths to improve/demonstrate that trust (that I'm seeing) is to improve the Incubation process(es), and to provide the Board with additional information about the state of the community. (if there are other paths people are looking at, then I've missed them) Cheers, -g
Re: Starting from the other end
On Mon, Oct 19, 2015 at 4:12 PM, Sam Rubywrote: > And that is precisely why I have held back on your invitation that I > edit the document. > > From a board evaluation perspective, I can tell you that proposals, > mentors, and the like are not required. Suggesting that these steps > are optional would be a drastic edit to the current document. > I think that this is very good input and have incorporated it (in abbreviated form) in the preamble of the document. The existence and requirements for these other paths are an excellent motivating case for the Incubator and thus this document.
Re: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator
+1 (non-binding) Good luck, guys! -- Alex On Mon, Oct 19, 2015 at 9:47 PM, Bertrand Delacretazwrote: > Hi, > > On Mon, Oct 19, 2015 at 2:35 PM, Luke Han wrote: > > ...Could you please help to double check again?... > > http://incubator.apache.org/projects/kylin.html looks good to me now, so > +1. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- -- Kind regards, Alexander.
Re: Starting from the other end
On Sun, Oct 18, 2015 at 12:33 AM, Ted Dunningwrote: >> We already have what amounts to three paths: there is the full >> Incubation process; there is is IP clearance, and there is straight to >> TLP. We can view these as three points on a spectrum. Having a well >> defined exit criteria and only applying the parts of the process that >> were needed to address deficiencies might have helped with projects >> like Subversion. >> > > That is well and good, but what I am focussed on here is making the > Incubator process work as well as possible. If the people working on > making a consistent process for other paths would like to crib this work, > that is great, but that is not the focus of the incubator. > > But please, do make comments on the document itself. Or suggest changes to > clarify or sharpen the document. I think I'm with Sam. I think these are related. Or to be more explicit: the way a project joins a foundation (regardless of what path it takes) is via a board passing a resolution. Perhaps defining exact criteria that the board uses to evaluate such resolutions would be useful. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Starting from the other end
On Mon, Oct 19, 2015 at 7:37 PM, Greg Steinwrote: > On Mon, Oct 19, 2015 at 6:25 PM, Roman Shaposhnik > wrote: >>... > >> Huge +1 to the above. Very well said and is exactly how I now start >> thing about the problem myself: Incubator is what's needed when >> there are gaps in straight to TLP. Lets identify what those gaps >> > > There is one thing the Incubator cannot solve: ASF Members on the > direct-to-TLP PMC. That has been the primary metric the Board used for > those Resolutions (Zest and Serf). There are a few other direct-to-TLP > occurrences which was only about moving code/communities around within the > Foundation (STeVe, Whimsy, ORC, etc, lately, and when we blew up umbrellas > (eg. Jakarta, XML, Hadoop)). I agree about the metric, but believe it can be generalized. Here is how I put it: "What is required is some sort of testimonial from a diverse set of people that we trust attesting to essentially what the contents of the Maturity Model covers." I see having a large number of ASF Members as being an example where we get that input first hand. But I wouldn't go so far as to say that a large number of ASF Members being direct participants is an absolute and non-negotiable requirement for direct to TLP. In fact, some of the blown up umbrellas may be counter examples. > So... I would not look at the Incubator as "get the podling community to > look like a candidate for direct-to-TLP", as it really can't do that. > > The ultimate question, I believe, is: "does the code/community operate > according to Foundation principles?" Indeed. >>... > > Cheers, > -g - Sam Ruby - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Starting from the other end
Inline On Mon, Oct 19, 2015 at 3:34 PM, Roman Shaposhnikwrote: > On Thu, Oct 15, 2015 at 9:55 PM, Ted Dunning > wrote: > > We have just had a few interminable threads regarding what is wrong with > > incubator and jumping directly to implementation attempts for changes. > ... > > > > To do this, I am going to start a google doc to describe WHAT the > incubator > > is supposed to do (easier to edit than a wiki) and once that document > > stabilizes a bit will move it to the wiki. > > > > That document should be able to let us move to similar efforts to > describe > > HOW the incubator currently works. I would hope that one side effect of > > this second stage would be to pull together current documentation on the > > current incubator. > ... > > > > > https://docs.google.com/document/d/1dxUVHGWcj83wIVnskYL7iM7PiiScLLNS4HjGHa6LsgA/edit?usp=sharing > > I've reviewed the document and left a couple of comments. Feel free > to incorporate the suggestions in the main text. > > In my view this document achieves the goal of stating things that > we can all agree on as to what Incubator should be doing. It works > great as a doc you can put on the front page of incubator.apache.org. > Good. > What it currently lacks is specifics on how are we going to make > sure that all the right things that this document postulates actually > happen. There's not explanation of checks-n-balances. > Right. If you read my original posting above, you will see that this document was intended to express intent, but not method. > So, for example (just to pick a random spot) when we talk about "The > Apache Incubator's > function at this point is..." I think we can all agree on what the > function is > but what is the process that guarantees that that functions succeeds is the > million dollar question. > Well, starting with the easy questions is a good place to get consensus. That was the point here. > Ted, was it your intent to first get an agreement on *what* we all expect > from the Incubator and then follow up on *how* are these expectations > going to be met by our current policies? > I think that is what I said in my original posting and in the preamble of this document. Was it somehow unclear?
Re: Starting from the other end
On Mon, Oct 19, 2015 at 6:25 PM, Roman Shaposhnikwrote: >... > Huge +1 to the above. Very well said and is exactly how I now start > thing about the problem myself: Incubator is what's needed when > there are gaps in straight to TLP. Lets identify what those gaps > There is one thing the Incubator cannot solve: ASF Members on the direct-to-TLP PMC. That has been the primary metric the Board used for those Resolutions (Zest and Serf). There are a few other direct-to-TLP occurrences which was only about moving code/communities around within the Foundation (STeVe, Whimsy, ORC, etc, lately, and when we blew up umbrellas (eg. Jakarta, XML, Hadoop)). So... I would not look at the Incubator as "get the podling community to look like a candidate for direct-to-TLP", as it really can't do that. The ultimate question, I believe, is: "does the code/community operate according to Foundation principles?" >... Cheers, -g
Re: Starting from the other end
On Mon, Oct 19, 2015 at 4:27 PM, Roman Shaposhnikwrote: > >> Ted, was it your intent to first get an agreement on *what* we all > expect > >> from the Incubator and then follow up on *how* are these expectations > >> going to be met by our current policies? > >> > > > > I think that is what I said in my original posting and in the preamble of > > this document. > > > > Was it somehow unclear? > > What was unclear is the timing on when do we get to talk about those > million dollar questions and in what format. I guess now that I've > commented > on this thread, I'll wait for the follow up. There is useful work going on right now on the how. See the maturity model. See the release guidelines. See pretty much everything that Marvin has been doing, for that matter. As I stated in the subject line, I am starting on the other end of the process to build more consensus about what we are doing so that some of the more difficult problems will have that consensus as a starting point.
Re: Starting from the other end
On Tue, Oct 20, 2015 at 2:36 AM, Ted Dunningwrote: > On Mon, Oct 19, 2015 at 4:27 PM, Roman Shaposhnik > wrote: > >> >> Ted, was it your intent to first get an agreement on *what* we all >> expect >> >> from the Incubator and then follow up on *how* are these expectations >> >> going to be met by our current policies? >> >> >> > >> > I think that is what I said in my original posting and in the preamble of >> > this document. >> > >> > Was it somehow unclear? >> >> What was unclear is the timing on when do we get to talk about those >> million dollar questions and in what format. I guess now that I've >> commented >> on this thread, I'll wait for the follow up. > > > There is useful work going on right now on the how. > > See the maturity model. > > See the release guidelines. > > See pretty much everything that Marvin has been doing, for that matter. I totally agree it is super useful work, but it is like doing a POC. Once you're done with it -- what's next? > As I stated in the subject line, I am starting on the other end of the > process to build more consensus about what we are doing so that some of the > more difficult problems will have that consensus as a starting point. I don't think there's any disagreement on high-level requirements for Incubator functions. I don't think there was any ever. In that way, your document captures the status quo that everybody is already agreeing with. The real disagreement is on how are we going to address the list of the following concerns: http://wiki.apache.org/incubator/IncubatorIssues2013 All of which are well within the *agreement* on what *needs* to be done. It is just that it doesn't alway happen the way we want it to. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Starting from the other end
On Sat, Oct 17, 2015 at 10:23 AM, Sam Rubywrote: > I'm growing increasingly enamored of the approach Bertrand (and > others) are advocating: focusing on what the incubator ideally > produces, namely a regular stream of Establish resolutions each > accompanied by documentation assessing to the maturity of the proposed > community. > > If that becomes the accepted goal, then as a Director I only need to > care about the end results, and as a mentor, I only need to care about > the community or communities that I am working with. Beyond that, > different communities could be free to experiment with different > processes. > > The end result is that the documentation that you (Ted) are producing, > and for that matter the documentation exists on the Incubator web > site, amounts to current best practices. Which is a good thing, but > perhaps should be considered less binding. > > We already have what amounts to three paths: there is the full > Incubation process; there is is IP clearance, and there is straight to > TLP. We can view these as three points on a spectrum. Having a well > defined exit criteria and only applying the parts of the process that > were needed to address deficiencies might have helped with projects > like Subversion. This approach seems promising. Folks may have heard some of us describe the Incubator as a "curriculum"[0] -- but what exactly defines the "curriculum"? Here are three candidates: * The incubation checklist[1] a.k.a. each podling's "status page" * ComDev's Maturity Model[2] * The draft "Project Requirements" document[3] which gathers together pointers to all ASF policy documents. My take is that these are complementary efforts and that it is worth ruminating on what each has to offer. The Maturity Model defines evaluation criteria. It is probably the most useful for the task we're focused on now, which is how to decide when a proposed TLP is ready, regardless of the path it took towards readiness. The incubation checklist is what we use to track a podling's incremental progress through the Incubator. You could see it as an Incubator-specific concrete realization of the Maturity Model, though as it is implemented today it only covers a fraction of the Model's bullet points. The Project Requirements document encompasses only policy, and thus only covers a subset of the curriculum. However, because the Foundation's current policy documentation is unbounded, irregular, and bloated, mastering this part of the curriculum requires disproportionate effort. I believe that if we can streamline all aspects of the curriculum, incubation can take far less time and effort than it does today. And if we can reduce the average time-to-incubate, many problems will simply solve themselves. Marvin Humphrey [0] The "curriculum" originally referred to the knowledge gained by a podling release manager while guiding a release candidate through the Incubator. [1] http://incubator.apache.org/projects/incubation-status-template.html [2] https://community.apache.org/apache-way/apache-project-maturity-model.html [3] http://www.apache.org/dev/project-requirements - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Starting from the other end
On Mon, Oct 19, 2015 at 11:22 PM, Marvin Humphreywrote: >... > The Maturity Model defines evaluation criteria. It is probably the most > useful for the task we're focused on now, which is how to decide when a > proposed TLP is ready, regardless of the path it took towards readiness. > As I've mentioned else-thread, that criteria has had no impact on the Board's evaluation of incoming direct-to-TLP projects. Instead, the Board took faith that *while a TLP* the community would reform itself to the desired goal. There was no requirement for the TLP to be fully compatible on Day One, but simply that the TLP knew *what* was needed, and *how* to reach that goal. I don't know about Apache Zest, but the (now) Apache Serf project always operated itself very closely to an ASF community. But, as an example, we did not use KEYS to sign releases. As a responsible TLP, we know the various gaps, and are making the appropriate changes as we fit ourselves into the ASF and produce our first ASF-branded release. This is why I suggest to avoid comparing Incubation projects against direct-to-TLP. By most measures that I can think of, they are quite distinct. Cheers, -g ps. it could be argued that a podling can become a TLP once it was self-aware enough to know how to fix/mend the gaps, much like Serf is doing. As Sam noted else-thread, this boils down to trust of those in the PMC. Serf gets it via a 10 ASF Members on Day One; a podling has a very high bar to reach comparative trust. [from the Board's perspective, IMO] > > The incubation checklist is what we use to track a podling's incremental > progress through the Incubator. You could see it as an Incubator-specific > concrete realization of the Maturity Model, though as it is implemented > today > it only covers a fraction of the Model's bullet points. > > The Project Requirements document encompasses only policy, and thus only > covers a subset of the curriculum. However, because the Foundation's > current > policy documentation is unbounded, irregular, and bloated, mastering this > part > of the curriculum requires disproportionate effort. > > I believe that if we can streamline all aspects of the curriculum, > incubation > can take far less time and effort than it does today. And if we can reduce > the average time-to-incubate, many problems will simply solve themselves. > > Marvin Humphrey > > [0] The "curriculum" originally referred to the knowledge gained by a > podling > release manager while guiding a release candidate through the > Incubator. > [1] http://incubator.apache.org/projects/incubation-status-template.html > [2] > https://community.apache.org/apache-way/apache-project-maturity-model.html > [3] http://www.apache.org/dev/project-requirements > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [DISCUSS] Eagle incubator proposal
It makes sense. I will try to contribute on this ;) Regards JB On 10/19/2015 09:46 PM, Zhang, Edward (GDI Hadoop) wrote: Hi JB, That is a good Point. Good to know that Falcon feeds HDFS/Hive/HBase data changes, so this feature would complement Eagle which today mainly focuses on HDFS/Hive/HBase data access including view, change, delete etc. Eagle would benefit if Eagle can instantly capture data change from Falcon. Thanks Edward Zhang On 10/19/15, 8:40, "Jean-Baptiste Onofré"wrote: Hi Arun, very interesting proposal. I may see some possible interaction with Falcon. In Falcon, we have HDFS files (and Hive/HBase) monitoring (with a kind of Change Data Capture), etc. So, I see a different perspective in Eagle, but Eagle could also leverage Falcon somehow. Regards JB On 10/19/2015 05:33 PM, Manoharan, Arun wrote: Hello Everyone, My name is Arun Manoharan. Currently a product manager in the Analytics platform team at eBay Inc. I would like to start a discussion on Eagle and its joining the ASF as an incubation project. Eagle is a Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities and take actions in real time. Eagle supports a wide variety of policies on HDFS data and Hive. Eagle also provides machine learning models for detecting anomalous user behavior in Hadoop. The proposal is available on the wiki here: https://wiki.apache.org/incubator/EagleProposal The text of the proposal is also available at the end of this email. Thanks for your time and help. Thanks, Arun Eagle Abstract Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive data, recognize attacks, malicious activities in hadoop and take actions. Proposal Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces policies defined on sensitive data access and alerts or blocks user¹s access to that sensitive data in real time. Eagle also creates user profiles based on the typical access behaviour for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import sensitive data information classified by external classification engines to help define its policies. Overview of Eagle Eagle has 3 main parts. 1.Data collection and storage - Eagle collects data from various hadoop logs in real time using Kafka/Yarn API and uses HDFS and HBase for storage. 2.Data processing and policy engine - Eagle allows users to create policies based on various metadata properties on HDFS, Hive and HBase data. 3.Eagle services - Eagle services include policy manager, query service and the visualization component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond to real time alerts. Data Collection and Storage: Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. Data Processing and Policy Engine: Processing Engine: Eagle provides stream processing API which is an abstraction of Apache Storm. It can also be extended to other streaming engines. This abstraction allows developers to assemble data transformation, filtering, external data join etc. without physically bound to a specific streaming platform. Eagle streaming API allows developers to easily integrate business logic with Eagle policy engine and internally Eagle framework compiles business logic execution DAG into program primitives of underlying stream infrastructure e.g. Apache Storm. For example, Eagle HDFS monitoring transforms audit log from Namenode to object and joins sensitivity metadata, security zone metadata which are generated from external programs or configured by user. Eagle hive monitoring filters running jobs to get hive query string and parses query string into object and then joins sensitivity metadata. Alerting Framework: Eagle Alert Framework includes stream metadata API, scalable policy engine framework, extensible policy engine framework. Stream metadata API allows developers to declare event schema including what attributes constitute an event, what is the type for each attribute, and how to dynamically resolve attribute value in runtime when user configures policy. Scalable policy engine framework allows policies to be executed on different physical nodes in parallel. It is also used to define your own policy partitioner class. Policy engine framework together with streaming partitioning capability provided by all streaming platforms will make sure policies and events can be
Re: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator
Hi John, The status page is updated during these days for some new information, dates and others. Could you please check and and let's know if there's still any issue. Thanks. Best Regards! - Luke Han On Tue, Oct 20, 2015 at 9:08 AM, John D. Amentwrote: > It looks weird to me that the date for the SGA is ambiguous. No one has a > more exact date than 2015? > > On Mon, Oct 19, 2015 at 8:47 AM Bertrand Delacretaz < > bdelacre...@apache.org> > wrote: > > > Hi, > > > > On Mon, Oct 19, 2015 at 2:35 PM, Luke Han wrote: > > > ...Could you please help to double check again?... > > > > http://incubator.apache.org/projects/kylin.html looks good to me now, so > > +1. > > > > -Bertrand > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
[RESULT] [VOTE] Accept Mynewt into the Apache Incubator
On Mon, Oct 12, 2015 at 9:04 AM, Sterling Hugheswrote: > Hi All, > > As mentioned in the DISCUSS thread, all feedback has been positive on > the Mynewt proposal, so I'd like to call a VOTE to accept Mynewt as a > new ASF incubator project. > > The full text of the proposal is available on the incubator wiki at > the following URL: > > https://wiki.apache.org/incubator/MynewtProposal?action=recall=20 > > I have also included the full text below. > > Vote is open until Thurs, 16th October 2015, 23:59:00 PST. > > [ ] +1 to accept Mynewt into the Apache Incubator > [ ] +0 > [ ] -1 because... > This vote is now closed and passes with 4 binding +1 votes, 3 non-binding +1 votes and no 0 or -1 votes. Thanks to all who helped with the proposal and cast the vote! Here's a vote tally: Non-binding +1s: Jim Jagielski Marvin Humphrey Bertrand Delacretaz Binding +1s: P. Taylor Goetz Justin Mclean Greg Stein Jean Baptiste Onofré No 0 or -1 votes. Thanks, Sterling PS: I didn't realize that I could also vote on the proposal until too late, but for the record, I'm also a +1 :-) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: 回复: [VOTE] Graduate Apache Kylin from the Apache Incubator
It looks weird to me that the date for the SGA is ambiguous. No one has a more exact date than 2015? On Mon, Oct 19, 2015 at 8:47 AM Bertrand Delacretazwrote: > Hi, > > On Mon, Oct 19, 2015 at 2:35 PM, Luke Han wrote: > > ...Could you please help to double check again?... > > http://incubator.apache.org/projects/kylin.html looks good to me now, so > +1. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [VOTE] Release Apache Kylin 1.1-incubating (rc1)
Signature file look good Hash files look good LICENSE and DISCLAIMER exist No 3rd party exes found in source artifacts +1 (binding) - Henry On Wed, Oct 14, 2015 at 6:42 PM, ShaoFeng Shiwrote: > Hi all, > > The Apache Kylin community has voted on and approved a proposal to release > Apache Kylin 1.1-incubating. > > Proposal:http://s.apache.org/Jzu > > Vote result: > 7 binding +1 votes > 6 non-binding +1 votes > No -1 voteshttp://s.apache.org/kylin-1.1-result_rc1 > > > The commit to be voted > upon:https://github.com/apache/incubator-kylin/commit/1955a2f9aea7b7f608f0496c00807715ea4246a5 > > Its hash is 1955a2f9aea7b7f608f0496c00807715ea4246a5. > > The artifacts to be voted on are located > here:https://dist.apache.org/repos/dist/dev/incubator/kylin/apache-kylin-1.1-incubating-rc1/ > > The hashes of the artifacts are as follows: > apache-kylin-1.1-incubating-src.tar.gz.md5 18dfbb012e1eb807b1a5c1b134a537aa > apache-kylin-1.1-incubating-src.tar.gz.sha1 > 041ee010c67a0b5611d9dd06f7fd6f37388c5374 > > A staged Maven repository is available for review > at:https://repository.apache.org/content/repositories/orgapachekylin-1012/ > > Release artifacts are signed with the following > key:https://people.apache.org/keys/committer/shaofengshi.asc > > Pursuant to the Releases section of the Incubation Policy and with > the endorsement of our mentors we would now like to request > the permission of the Incubator PMC to publish the release. The vote > is open for 72 hours, or until the necessary number of votes (3 +1) > is reached. > > [ ] +1 Release this package > [ ] 0 I don't feel strongly about it, but I'm okay with the release > [ ] -1 Do not release this package because... > > > Shaofeng Shi, on behalf of Apache Kylin ppmcshaofeng...@apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
RE: Starting from the other end
+1 from me too. I did some wordsmithing but not certain how that works properly in Google Docs in terms of how changes are made evident for review, etc. - Dennis PS: When I go back and review, I see that I needed to leave more highlights or comments to indicate places where I made adjustments. (Maybe a wiki would be better, so there is a modification history?) > -Original Message- > From: Bertrand Delacretaz [mailto:bdelacre...@apache.org] > Sent: Monday, October 19, 2015 01:29 > To: Incubator General> Subject: Re: Starting from the other end > > Hi, > > On Thu, Oct 15, 2015 at 8:55 PM, Ted Dunning > wrote: > > ...The first step (What Apache Incubator Does) can be edited at > > > https://docs.google.com/document/d/1dxUVHGWcj83wIVnskYL7iM7PiiScLLNS4HjG > Ha6LsgA/edit?usp=sharing ... > > I like that. > > Combined with a set of commented links that point to good examples of > the various phases and transitions (proposal, acceptance, etc.) I > think this could replace a large part of the Incubator documentation. > > I don't have any specific changes to request at this point, I think > the current draft is good enough to move to our wiki or website. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org