Re: [VOTE] Accept Kylin into the Apache Incubator
On Thu, Nov 20, 2014 at 11:31 PM, Luke Han wrote: > ...I would like to call a VOTE for accepting Kylin as a new incubator > project... +1, binding -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Infra for podling setup
On Thu, Nov 20, 2014 at 5:42 PM, John D. Ament wrote: > Hi, > > Since I'm new at being a mentor, I was wondering how to handle slow infra > requests for podlings? > Hi John: An occasional ping will help on the tickets. Failing that, feel free to complain to me. And if I fail, you can talk to Rich or Ross. --David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Arvind Thank you for offering to be a mentor. The proposal has been updated. Arvind Prabhakar (StreamSets) One of the comments we have received thus far is a potential concern of whether multiple communities/projects can exist in similar problem areas. Your background in Flume which solves similar problems to NiFi is a good sign that multiple projects and communities can co-exist within the space and even provide complimentary capability. Thanks Joe On Thu, Nov 20, 2014 at 9:20 PM, Joe Witt wrote: > Josh, > > Really appreciate it and have updated the proposal. > > Thanks > Joe > > On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser wrote: > >> Very exciting stuff! >> >> Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as >> a mentor. If so, I'll submit an application to join the IPMC and we can go >> from there. >> >> - Josh >> >> >> Joe Witt wrote: >> >>> Hello, >>> >>> >>> I would like to propose NiFi as an Apache Incubator Project. >>> >>> In addition to the copy provided below the Wiki version of the >>> proposal can be found here: >>> http://wiki.apache.org/incubator/NiFiProposal >>> >>> Thanks >>> >>> Joe >>> >>> >>> = NiFi Proposal = >>> >>> == Abstract == >>> NiFi is a dataflow system based on the concepts of flow-based >>> programming. >>> >>> == Proposal == >>> NiFi supports powerful and scalable directed graphs of data routing, >>> transformation, and system mediation logic. Some of the high-level >>> capabilities and objectives of NiFi include: >>>* Web-based user interface for seamless experience between design, >>> control, feedback, and monitoring of data flows >>>* Highly configurable along several dimensions of quality of service >>> such as loss tolerant versus guaranteed delivery, low latency versus >>> high throughput, and priority based queuing >>>* Fine-grained data provenance for all data received, forked, >>> joined, cloned, modified, sent, and ultimately dropped as data reaches >>> its configured end-state >>>* Component-based extension model along well defined interfaces >>> enabling rapid development and effective testing >>> >>> == Background == >>> Reliable and effective dataflow between systems can be difficult >>> whether you're running scripts on a laptop or have a massive >>> distributed computing system operated by numerous teams and >>> organizations. As the volume and rate of data grows and as the number >>> of systems, protocols, and formats increase and evolve so too does the >>> complexity and need for greater insight and agility. These are the >>> dataflow challenges that NiFi was built to tackle. >>> >>> NiFi is designed in a manner consistent with the core concepts >>> described in flow-based programming as originally documented by J. >>> Paul Morrison in the 1970s. This model lends itself well to visual >>> diagramming, concurrency, componentization, testing, and reuse. In >>> addition to staying close to the fundamentals of flow-based >>> programming, NiFi provides integration system specific features such >>> as: guaranteed delivery; back pressure; ability to gracefully handle >>> backlogs and data surges; and an operator interface that enables >>> on-the-fly data flow generation, modification, and observation. >>> >>> == Rationale == >>> NiFi provides a reliable, scalable, manageable and accountable >>> platform for developers and technical staff to create and evolve >>> powerful data flows. Such a system is useful in many contexts >>> including large-scale enterprise integration, interaction with cloud >>> services and frameworks, business to business, intra-departmental, and >>> inter-departmental flows. NiFi fits well within the Apache Software >>> Foundation (ASF) family as it depends on numerous ASF projects and >>> integrates with several others. We also anticipate developing >>> extensions for several other ASF projects such as Cassandra, Kafka, >>> and Storm in the near future. >>> >>> == Initial Goals == >>>* Ensure all dependencies are compliant with Apache License version >>> 2.0 and all that all code and documentation artifacts have the correct >>> Apache licensing markings and notice. >>>* Establish a formal release process and schedule, allowing for >>> dependable release cycles in a manner consistent with the Apache >>> development process. >>>* Determine and establish a mechanism, possibly including a >>> sub-project construct, that allows for extensions to the core >>> application to occur at a pace that differs from the core application >>> itself. >>> >>> == Current Status == >>> === Meritocracy === >>> An integration platform is only as good as its ability to integrate >>> systems in a reliable, timely, and repeatable manner. The same can be >>> said of its ability to attract talent and a variety of perspectives as >>> integration systems by their nature are always evolving. We will >>> actively seek help and encourage promotion of influence in the project >>> through meritocracy.
dashboarding incubator
Hi, one of the things I brought from ApacheCon is the idea if providing dashboard for the podlings. Bitergia presented there their approach [1], and I think could be useful. See for instance the dashboard they provided for Apache CloudStack [2]. I already discussed it with Roman, and I think we should try to do something in this direction. I know the experiment with Back Software did not satisfy everybody, but I think we should keep trying. I can commit some time and even resources to provide an early proof of concept. What do you think? Cheers, [1] http://apacheconeu2014.sched.org/event/50a7178fa1d812e50e8c13e9bca8c7c1#.VG8vQnU3eis [2] http://projects.bitergia.com/apache-cloudstack/browser/ -- Sergio Fernández Senior Researcher Knowledge and Media Technologies Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernan...@salzburgresearch.at http://www.salzburgresearch.at - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Slider 0.60.0 incubating
+1 binding On 18 November 2014 19:43, Josh Elser wrote: > Hi, > > The Apache Slider project has voted to release 0.60.0-incubating. The team > voted to release these artifacts with 6 +1's and nothing else. 2 of the 6 > votes were from IPMC members. > > Vote thread: http://s.apache.org/a6O > Result: http://s.apache.org/e3C > > Staged artifacts: http://people.apache.org/~stevel/slider/slider-release- > 0.60.0-incubating-rc1 > > SHA1: c2c06d3532708ac2ed765ebc465edda1b0067025 > Source: https://git-wip-us.apache.org/repos/asf?p=incubator-slider. > git;a=commit;h=c2c06d3532708ac2ed765ebc465edda1b0067025 > Git tag: "release-0.60.0-incubating-rc1" > > PGP key: http://pgp.mit.edu:11371/pks/lookup?op=vindex&search= > ste...@apache.org > > Basic build/test instructions: http://slider.incubator. > apache.org/developing/building.html > > Please vote on releasing the above as Apache Slider 0.60.0-incubating. > This vote will be open for 72 hours. > > Thanks! > > - Apache Slider team > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [PROPOSAL] NiFi for Incubation
Very interesting project! I'm happy to help mentoring it. On 20/11/14 03:02, Joe Witt wrote: Hello, I would like to propose NiFi as an Apache Incubator Project. In addition to the copy provided below the Wiki version of the proposal can be found here: http://wiki.apache.org/incubator/NiFiProposal Thanks Joe = NiFi Proposal = == Abstract == NiFi is a dataflow system based on the concepts of flow-based programming. == Proposal == NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of NiFi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that NiFi was built to tackle. NiFi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, NiFi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == NiFi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. NiFi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === From its inception, NiFi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, NiFi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, HttpClient, Jakarta and Maven. We also anticipate extensions and dependencies with several more ASF projects, including Accumulo, Avro, Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others. Th
Re: [VOTE] Accept Kylin into the Apache Incubator
+1 (binding) On 20/11/14 23:31, Luke Han wrote: Following the discussion earlier in the thread: http://mail-archives.apache.org/mod_mbox/incubator-general/201411.mbox/%3ccakmqrob22+n+r++date33f3pcpyujhfoeaqrms3t-udjwk6...@mail.gmail.com%3e I would like to call a VOTE for accepting Kylin as a new incubator project. The proposal is available at: https://wiki.apache.org/incubator/KylinProposal and posted the text of the proposal below also. Vote is open until 24th November 2014, 23:59:00 UTC [ ] +1 accept Kylin in the Incubator [ ] ±0 [ ] -1 because... Thanks Luke Kylin Proposal == # Abstract Kylin is a distributed and scalable OLAP engine built on Hadoop to support extremely large datasets. # Proposal Kylin is an open source Distributed Analytics Engine that provides multi-dimensional analysis (MOLAP) on Hadoop. Kylin is designed to accelerate analytics on Hadoop by allowing the use of SQL-compatible tools. Kylin provides a SQL interface and multi-dimensional analysis (MOLAP) on Hadoop to support extremely large datasets and tightly integrate with Hadoop ecosystem. ## Overview of Kylin Kylin platform has two parts of data processing and interactive: First, Kylin will read data from source, Hive, and run a set of tasks including Map Reduce job, shell script to pre-calcuate results for a specified data model, then save the resulting OLAP cube into storage such as HBase. Once these OLAP cubes are ready, a user can submit a request from any SQL-based tool or third party applications to Kylin’s REST server. The Server calls the Query Engine to determine if the target dataset already exists. If so, the engine directly accesses the target data in the form of a predefined cube, and returns the result with sub-second latency. Otherwise, the engine is designed to route non-matching queries to whichever SQL on Hadoop tool is already available on a Hadoop cluster, such as Hive. Kylin platform includes: - Metadata Manager: Kylin is a metadata-driven application. The Kylin Metadata Manager is the key component that manages all metadata stored in Kylin including all cube metadata. All other components rely on the Metadata Manager. - Job Engine: This engine is designed to handle all of the offline jobs including shell script, Java API, and Map Reduce jobs. The Job Engine manages and coordinates all of the jobs in Kylin to make sure each job executes and handles failures. - Storage Engine: This engine manages the underlying storage – specifically, the cuboids, which are stored as key-value pairs. The Storage Engine uses HBase – the best solution from the Hadoop ecosystem for leveraging an existing K-V system. Kylin can also be extended to support other K-V systems, such as Redis. - Query Engine: Once the cube is ready, the Query Engine can receive and parse user queries. It then interacts with other components to return the results to the user. - REST Server: The REST Server is an entry point for applications to develop against Kylin. Applications can submit queries, get results, trigger cube build jobs, get metadata, get user privileges, and so on. - ODBC Driver: To support third-party tools and applications – such as Tableau – we have built and open-sourced an ODBC Driver. The goal is to make it easy for users to onboard. # Background The challenge we face at eBay is that our data volume is becoming bigger and bigger while our user base is becoming more diverse. For e.g. our business users and analysts consistently ask for minimal latency when visualizing data on Tableau and Excel. So, we worked closely with our internal analyst community and outlined the product requirements for Kylin: - Sub-second query latency on billions of rows - ANSI SQL availability for those using SQL-compatible tools - Full OLAP capability to offer advanced functionality - Support for high cardinality and very large dimensions - High concurrency for thousands of users - Distributed and scale-out architecture for analysis in the TB to PB size range Existing SQL-on-Hadoop solutions commonly need to perform partial or full table or file scans to compute the results of queries. The cost of these large data scans can make many queries very slow (more than a minute). The core idea of MOLAP (multi-dimensional OLAP) is to pre-compute data along dimensions of interest and store resulting aggregates as a "cube". MOLAP is much faster but is inflexible. We realized that no existing product met our exact requirements externally – especially in the open source Hadoop community. To meet our emerging business needs, we built a platform from scratch to support MOLAP for these business requirements and then to support more others include ROLAP. With an excellent development team and several pilot customers, we have been able to bring the Kylin platform into production as well as open source it. # Rationale When data grows to petabyte scale, the process of pre-calculation of a query takes a long time and costly and powerfu
Re: [VOTE] Release Apache Slider 0.60.0 incubating
+1 (binding) Looks good to me. On Tue, Nov 18, 2014 at 2:43 PM, Josh Elser wrote: > Hi, > > The Apache Slider project has voted to release 0.60.0-incubating. The team > voted to release these artifacts with 6 +1's and nothing else. 2 of the 6 > votes were from IPMC members. > > Vote thread: http://s.apache.org/a6O > Result: http://s.apache.org/e3C > > Staged artifacts: http://people.apache.org/~stevel/slider/slider-release- > 0.60.0-incubating-rc1 > > SHA1: c2c06d3532708ac2ed765ebc465edda1b0067025 > Source: https://git-wip-us.apache.org/repos/asf?p=incubator-slider. > git;a=commit;h=c2c06d3532708ac2ed765ebc465edda1b0067025 > Git tag: "release-0.60.0-incubating-rc1" > > PGP key: http://pgp.mit.edu:11371/pks/lookup?op=vindex&search= > ste...@apache.org > > Basic build/test instructions: http://slider.incubator. > apache.org/developing/building.html > > Please vote on releasing the above as Apache Slider 0.60.0-incubating. > This vote will be open for 72 hours. > > Thanks! > > - Apache Slider team > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [PROPOSAL] NiFi for Incubation
Sergio Excellent - really appreciate it and look forward to working with you. Have updated the proposal. Sergio Fernandez (Redlink, Salzburg Research> http://wiki.apache.org/incubator/NiFiProposal Thanks Joe On Fri, Nov 21, 2014 at 7:30 AM, Sergio Fernández < sergio.fernan...@salzburgresearch.at> wrote: > Very interesting project! > I'm happy to help mentoring it. > > > On 20/11/14 03:02, Joe Witt wrote: > >> Hello, >> >> >> I would like to propose NiFi as an Apache Incubator Project. >> >> In addition to the copy provided below the Wiki version of the >> proposal can be found here: >> http://wiki.apache.org/incubator/NiFiProposal >> >> Thanks >> >> Joe >> >> >> = NiFi Proposal = >> >> == Abstract == >> NiFi is a dataflow system based on the concepts of flow-based programming. >> >> == Proposal == >> NiFi supports powerful and scalable directed graphs of data routing, >> transformation, and system mediation logic. Some of the high-level >> capabilities and objectives of NiFi include: >>* Web-based user interface for seamless experience between design, >> control, feedback, and monitoring of data flows >>* Highly configurable along several dimensions of quality of service >> such as loss tolerant versus guaranteed delivery, low latency versus >> high throughput, and priority based queuing >>* Fine-grained data provenance for all data received, forked, >> joined, cloned, modified, sent, and ultimately dropped as data reaches >> its configured end-state >>* Component-based extension model along well defined interfaces >> enabling rapid development and effective testing >> >> == Background == >> Reliable and effective dataflow between systems can be difficult >> whether you're running scripts on a laptop or have a massive >> distributed computing system operated by numerous teams and >> organizations. As the volume and rate of data grows and as the number >> of systems, protocols, and formats increase and evolve so too does the >> complexity and need for greater insight and agility. These are the >> dataflow challenges that NiFi was built to tackle. >> >> NiFi is designed in a manner consistent with the core concepts >> described in flow-based programming as originally documented by J. >> Paul Morrison in the 1970s. This model lends itself well to visual >> diagramming, concurrency, componentization, testing, and reuse. In >> addition to staying close to the fundamentals of flow-based >> programming, NiFi provides integration system specific features such >> as: guaranteed delivery; back pressure; ability to gracefully handle >> backlogs and data surges; and an operator interface that enables >> on-the-fly data flow generation, modification, and observation. >> >> == Rationale == >> NiFi provides a reliable, scalable, manageable and accountable >> platform for developers and technical staff to create and evolve >> powerful data flows. Such a system is useful in many contexts >> including large-scale enterprise integration, interaction with cloud >> services and frameworks, business to business, intra-departmental, and >> inter-departmental flows. NiFi fits well within the Apache Software >> Foundation (ASF) family as it depends on numerous ASF projects and >> integrates with several others. We also anticipate developing >> extensions for several other ASF projects such as Cassandra, Kafka, >> and Storm in the near future. >> >> == Initial Goals == >>* Ensure all dependencies are compliant with Apache License version >> 2.0 and all that all code and documentation artifacts have the correct >> Apache licensing markings and notice. >>* Establish a formal release process and schedule, allowing for >> dependable release cycles in a manner consistent with the Apache >> development process. >>* Determine and establish a mechanism, possibly including a >> sub-project construct, that allows for extensions to the core >> application to occur at a pace that differs from the core application >> itself. >> >> == Current Status == >> === Meritocracy === >> An integration platform is only as good as its ability to integrate >> systems in a reliable, timely, and repeatable manner. The same can be >> said of its ability to attract talent and a variety of perspectives as >> integration systems by their nature are always evolving. We will >> actively seek help and encourage promotion of influence in the project >> through meritocracy. >> >> === Community === >> Over the past several years, NiFi has developed a strong community of >> both developers and operators within the U.S. government. We look >> forward to helping grow this to a broader base of industries. >> >> === Core Developers === >> The initial core developers are employed by the National Security >> Agency and defense contractors. We will work to grow the community >> among a more diverse set of developers and industries. >> >> === Alignment === >> From its inception, NiFi was developed with an open source philosophy >
Re: [PROPOSAL] NiFi for Incubation
Hi, On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt wrote: ... > http://wiki.apache.org/incubator/NiFiProposal Note that AFAICS some of the nominated mentors are not members of the Incubator PMC [1] - they cannot be "formal" incubation mentors then. ASF members can join that PMC just by asking. ASF committers can be elected to that PMC with the support of ASF or Incubator PMC members, subject to an Incubator PMC vote, that can be discussed on the incubator private list. Others can help with mentoring activities of course - and that's a great way of eventually joining this PMC if you're not an ASF member. But they won't be listed as mentors on the proposal or podling status page. -Bertrand [1] http://people.apache.org/committers-by-project.html#incubator-pmc - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Bertrand, Understood. I had noticed other proposals listing these folks as mentors but also providing their IPMC status along side. I'll do the same for now and if they join the ipmc i'll update it and if they end up not doing so will remove them. If that is not reasonable then just let me know. Thanks Joe On Fri, Nov 21, 2014 at 7:59 AM, Bertrand Delacretaz wrote: > Hi, > > On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt wrote: > ... > > http://wiki.apache.org/incubator/NiFiProposal > > Note that AFAICS some of the nominated mentors are not members of the > Incubator PMC [1] - they cannot be "formal" incubation mentors then. > > ASF members can join that PMC just by asking. > > ASF committers can be elected to that PMC with the support of ASF or > Incubator PMC members, subject to an Incubator PMC vote, that can be > discussed on the incubator private list. > > Others can help with mentoring activities of course - and that's a > great way of eventually joining this PMC if you're not an ASF member. > But they won't be listed as mentors on the proposal or podling status > page. > > -Bertrand > > [1] http://people.apache.org/committers-by-project.html#incubator-pmc > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [PROPOSAL] NiFi for Incubation
Hi, On Fri, Nov 21, 2014 at 2:23 PM, Joe Witt wrote: > ...I had noticed other proposals listing these folks as mentors > but also providing their IPMC status along side. I'll do the same for now > and if they join the ipmc i'll update it and if they end up not doing so > will remove them. If that is not reasonable then just let me know That's perfectly fine, thanks! -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
I've advised Joe to 'asterix' the would-be mentors who are not iPMC yet, so that he can proceed to a vote on the base of the ones who are sooner rather than later, and the stragglers can be formally added to the metadata once they are on the iPMC.
Re: dashboarding incubator
On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández wrote: > Hi, > > one of the things I brought from ApacheCon is the idea if providing > dashboard for the podlings. Bitergia presented there their approach [1], and > I think could be useful. See for instance the dashboard they provided for > Apache CloudStack [2]. > > I already discussed it with Roman, and I think we should try to do something > in this direction. I know the experiment with Back Software did not satisfy > everybody, but I think we should keep trying. I can commit some time and > even resources to provide an early proof of concept. > I am generally against us standing up our own service that does this. We've had a couple of these systems over the years. (pulse.a,o for instance). It takes a non-trivial amount of work to setup and maintain such a system, and invariably it falls apart. Additionally, there are free services that begin monitoring projects either on request or when they are noticed, and I am curious what we really hope to gain by doing it ourselves that a site like open hub doesn't provide. In example: https://www.openhub.net/p/npanday I do agree that things like bitergia provide much deeper sense of analytics, but wonderful if it's really necessary for our purposes. --David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
I agree with David. For example, many projects are now mirrored to github, and github has very good analytics: https://github.com/apache/incubator-calcite/pulse https://github.com/apache/incubator-calcite/graphs/contributors And, I don't have to log an INFRA JIRA to get them. Which makes both me and David happy. :) Julian On Fri, Nov 21, 2014 at 6:35 AM, David Nalley wrote: > On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández > wrote: >> Hi, >> >> one of the things I brought from ApacheCon is the idea if providing >> dashboard for the podlings. Bitergia presented there their approach [1], and >> I think could be useful. See for instance the dashboard they provided for >> Apache CloudStack [2]. >> >> I already discussed it with Roman, and I think we should try to do something >> in this direction. I know the experiment with Back Software did not satisfy >> everybody, but I think we should keep trying. I can commit some time and >> even resources to provide an early proof of concept. >> > > I am generally against us standing up our own service that does this. > We've had a couple of these systems over the years. (pulse.a,o for > instance). It takes a non-trivial amount of work to setup and maintain > such a system, and invariably it falls apart. > > Additionally, there are free services that begin monitoring projects > either on request or when they are noticed, and I am curious what we > really hope to gain by doing it ourselves that a site like open hub > doesn't provide. In example: > https://www.openhub.net/p/npanday > > I do agree that things like bitergia provide much deeper sense of > analytics, but wonderful if it's really necessary for our purposes. > > --David > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[VOTE] accept NiFi into the incubator
http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and positive conversation, so I offer this vote. Vote will be open for the usual 72 hours ... Here is my [+1]
Re: [VOTE] accept NiFi into the incubator
+1 (non-binding) On Fri, Nov 21, 2014 at 12:55 PM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] > -- Sean
Re: [VOTE] Accept Kylin into the Apache Incubator
Great stuff, +1 On Thursday, November 20, 2014, Luke Han wrote: > Following the discussion earlier in the thread: > > > http://mail-archives.apache.org/mod_mbox/incubator-general/201411.mbox/%3ccakmqrob22+n+r++date33f3pcpyujhfoeaqrms3t-udjwk6...@mail.gmail.com%3e > > I would like to call a VOTE for accepting Kylin as a new incubator project. > > The proposal is available at: > https://wiki.apache.org/incubator/KylinProposal > > and posted the text of the proposal below also. > > Vote is open until 24th November 2014, 23:59:00 UTC > > [ ] +1 accept Kylin in the Incubator > [ ] ±0 > [ ] -1 because... > > > Thanks > Luke > > > Kylin Proposal > == > > # Abstract > > Kylin is a distributed and scalable OLAP engine built on Hadoop to > support extremely large datasets. > > # Proposal > > Kylin is an open source Distributed Analytics Engine that provides > multi-dimensional analysis (MOLAP) on Hadoop. Kylin is designed to > accelerate analytics on Hadoop by allowing the use of SQL-compatible > tools. Kylin provides a SQL interface and multi-dimensional analysis > (MOLAP) on Hadoop to support extremely large datasets and tightly > integrate with Hadoop ecosystem. > > ## Overview of Kylin > > Kylin platform has two parts of data processing and interactive: > First, Kylin will read data from source, Hive, and run a set of tasks > including Map Reduce job, shell script to pre-calcuate results for a > specified data model, then save the resulting OLAP cube into storage > such as HBase. Once these OLAP cubes are ready, a user can submit a > request from any SQL-based tool or third party applications to Kylin’s > REST server. The Server calls the Query Engine to determine if the > target dataset already exists. If so, the engine directly accesses the > target data in the form of a predefined cube, and returns the result > with sub-second latency. Otherwise, the engine is designed to route > non-matching queries to whichever SQL on Hadoop tool is already > available on a Hadoop cluster, such as Hive. > > Kylin platform includes: > > - Metadata Manager: Kylin is a metadata-driven application. The Kylin > Metadata Manager is the key component that manages all metadata stored > in Kylin including all cube metadata. All other components rely on the > Metadata Manager. > > - Job Engine: This engine is designed to handle all of the offline > jobs including shell script, Java API, and Map Reduce jobs. The Job > Engine manages and coordinates all of the jobs in Kylin to make sure > each job executes and handles failures. > > - Storage Engine: This engine manages the underlying storage – > specifically, the cuboids, which are stored as key-value pairs. The > Storage Engine uses HBase – the best solution from the Hadoop > ecosystem for leveraging an existing K-V system. Kylin can also be > extended to support other K-V systems, such as Redis. > > - Query Engine: Once the cube is ready, the Query Engine can receive > and parse user queries. It then interacts with other components to > return the results to the user. > > - REST Server: The REST Server is an entry point for applications to > develop against Kylin. Applications can submit queries, get results, > trigger cube build jobs, get metadata, get user privileges, and so on. > > - ODBC Driver: To support third-party tools and applications – such as > Tableau – we have built and open-sourced an ODBC Driver. The goal is > to make it easy for users to onboard. > > # Background > > The challenge we face at eBay is that our data volume is becoming > bigger and bigger while our user base is becoming more diverse. For > e.g. our business users and analysts consistently ask for minimal > latency when visualizing data on Tableau and Excel. So, we worked > closely with our internal analyst community and outlined the product > requirements for Kylin: > > - Sub-second query latency on billions of rows > - ANSI SQL availability for those using SQL-compatible tools > - Full OLAP capability to offer advanced functionality > - Support for high cardinality and very large dimensions > - High concurrency for thousands of users > - Distributed and scale-out architecture for analysis in the TB to PB size > range > > Existing SQL-on-Hadoop solutions commonly need to perform partial or > full table or file scans to compute the results of queries. The cost > of these large data scans can make many queries very slow (more than a > minute). The core idea of MOLAP (multi-dimensional OLAP) is to > pre-compute data along dimensions of interest and store resulting > aggregates as a "cube". MOLAP is much faster but is inflexible. We > realized that no existing product met our exact requirements > externally – especially in the open source Hadoop community. To meet > our emerging business needs, we built a platform from scratch to > support MOLAP for these business requirements and then to support more > others include ROLAP. With an excellent development team and several > pilot customers, we
Re: [VOTE] accept NiFi into the incubator
Hey Benson Can you please include the proposal with the vote, the wiki page can change or be removed and the mailing list acts as the archive for the initial proposal. -Jake On Fri, Nov 21, 2014 at 1:55 PM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] >
Re: [VOTE] accept NiFi into the incubator
+1, best wishes... based on revision #25[1] --tim [1] - http://wiki.apache.org/incubator/NiFiProposal?action=recall&rev=25 On Fri, Nov 21, 2014 at 1:55 PM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
One down side to the github provided analytics is that projects may be restricted in how they can make use of those analytics in e.g. presentations. Github has previously asserted that those analytic outputs are subject to a license grant from github and requested sign off in the form of a user agreement. I don't doubt that many many people make reference to them without bothering to check on the legal status, nor that many uses are probably within the bounds of fair use. But the ASF should probably keep the matter in mind. On Fri, Nov 21, 2014 at 11:54 AM, Julian Hyde wrote: > I agree with David. > > For example, many projects are now mirrored to github, and github has > very good analytics: > > https://github.com/apache/incubator-calcite/pulse > > https://github.com/apache/incubator-calcite/graphs/contributors > > And, I don't have to log an INFRA JIRA to get them. Which makes both > me and David happy. :) > > Julian > > On Fri, Nov 21, 2014 at 6:35 AM, David Nalley wrote: > > On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández > > wrote: > >> Hi, > >> > >> one of the things I brought from ApacheCon is the idea if providing > >> dashboard for the podlings. Bitergia presented there their approach > [1], and > >> I think could be useful. See for instance the dashboard they provided > for > >> Apache CloudStack [2]. > >> > >> I already discussed it with Roman, and I think we should try to do > something > >> in this direction. I know the experiment with Back Software did not > satisfy > >> everybody, but I think we should keep trying. I can commit some time and > >> even resources to provide an early proof of concept. > >> > > > > I am generally against us standing up our own service that does this. > > We've had a couple of these systems over the years. (pulse.a,o for > > instance). It takes a non-trivial amount of work to setup and maintain > > such a system, and invariably it falls apart. > > > > Additionally, there are free services that begin monitoring projects > > either on request or when they are noticed, and I am curious what we > > really hope to gain by doing it ourselves that a site like open hub > > doesn't provide. In example: > > https://www.openhub.net/p/npanday > > > > I do agree that things like bitergia provide much deeper sense of > > analytics, but wonderful if it's really necessary for our purposes. > > > > --David > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- Sean
Re: [VOTE] accept NiFi into the incubator
+1 (binding). Good luck! ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Benson Margulies Reply-To: "general@incubator.apache.org" Date: Friday, November 21, 2014 at 7:55 PM To: "general@incubator.apache.org" Subject: [VOTE] accept NiFi into the incubator >http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and >positive conversation, so I offer this vote. > >Vote will be open for the usual 72 hours ... > >Here is my [+1] - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[VOTE] Release Apache Aurora 0.6.0 (incubating) RC2
Hi All, The Apache Aurora project has voted to release 0.6.0-incubating, our second release. The team voted in favor of this release with 7 binding +1s, 1 non-binding +1. 3 of the binding +1 votes were from IPMC members. Vote thread: http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201411.mbox/%3CCAGRA8uMdo29ZP-7Czq%3DJrzQYD4m43dn5aqzFXsrKSNBhtgJPTQ%40mail.gmail.com%3E Result thread: http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201411.mbox/%3CCAGRA8uNqbPNjiuw6_t-jhzy1E-8aEBd-j5eXEcHS0ok7ee-zDA%40mail.gmail.com%3E The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=incubator-aurora.git&f=CHANGELOG&hb=0.6.0-rc2 The branch used to create the release candidate is: https://git-wip-us.apache.org/repos/asf?p=incubator-aurora.git&hb=0.6.0-rc2 The release candidate is available at: https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz The MD5 checksum of the release candidate can be found at: https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz.md5 The signature of the release candidate can be found at: https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz.asc The GPG key used to sign the release are available at: https://dist.apache.org/repos/dist/dev/incubator/aurora/KEYS Please vote on releasing the above as Apache Aurora 0.6.0-incubating. This vote will be open for 72 hours. Cheers, Bill -- Forwarded message -- From: Bill Farner Date: Fri, Nov 21, 2014 at 11:16 AM Subject: [RESULT] [VOTE] Release Apache Aurora 0.6.0 (incubating) RC2 To: "d...@aurora.incubator.apache.org" Hi All, The vote to accept Apache Aurora 0.6.0 RC2 as the official Apache Aurora 0.6.0 release has passed. +1 (Binding): --- Maxim Khutornenko David McLaughlin Kevin Sweeney Jake Farrell Henry Saputra Bill Farner Benjamin Hindman +1 (Non-binding): --- Joshua Cohen There were no 0 or -1 votes. Thanks, everyone! -=Bill On Thu, Nov 20, 2014 at 2:44 PM, Benjamin Hindman < benjamin.hind...@gmail.com> wrote: > +1 (binding), looks good to me folks! > > On Thu, Nov 20, 2014 at 10:41 AM, Bill Farner wrote: > > > +1 overall, end-to-end tests pass, unit tests pass, signature and > checksum > > pass > > > > -=Bill > > > > On Wed, Nov 19, 2014 at 10:12 PM, Henry Saputra > > > wrote: > > > > > Hash and signatures are good. > > > DISCLAIMER is correct > > > LICENSE and NOTICE files look good > > > No 3rd party executables > > > > > > +1 (binding) > > > > > > Thanks guys! > > > > > > On Thu, Nov 13, 2014 at 2:33 PM, Bill Farner > wrote: > > > > All, > > > > > > > > I propose that we accept the following release candidate as the > > official > > > > Apache Aurora 0.6.0 release. > > > > > > > > > > > > Aurora 0.6.0-rc2 includes the following: > > > > --- > > > > The CHANGELOG for the release is available at: > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-aurora.git&f=CHANGELOG&hb=0.6.0-rc2 > > > > > > > > The branch used to create the release candidate is: > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-aurora.git&hb=0.6.0-rc2 > > > > > > > > The release candidate is available at: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz > > > > > > > > The MD5 checksum of the release candidate can be found at: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz.md5 > > > > > > > > The signature of the release candidate can be found at: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/aurora/0.6.0-rc2/apache-aurora-0.6.0-rc2-incubating.tar.gz.asc > > > > > > > > The GPG key used to sign the release are available at: > > > > https://dist.apache.org/repos/dist/dev/incubator/aurora/KEYS > > > > > > > > Please download, verify, and test. > > > > > > > > The vote will close on Tue Nov 18 14:30:25 PST 2014 > > > > > > > > [ ] +1 Release this as Apache Aurora 0.6.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache Aurora 0.6.0 because... > > > > > > > > > > > > -=Bill > > > > > >
Re: dashboarding incubator
Hi, On Fri, Nov 21, 2014 at 3:35 PM, David Nalley wrote: > ...I am generally against us standing up our own service that does this. > We've had a couple of these systems over the years. (pulse.a,o for > instance). It takes a non-trivial amount of work to setup and maintain > such a system, and invariably it falls apart I agree, OTOH if someone wants to help third parties get the data that they need to implement such services externally that might be fine. -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[RESULT] [VOTE] Release Apache Slider 0.60.0 incubating
This vote passes with 3 +1's. Billie's vote was cast on dev@slider. Spoke with her directly and she said that she did forward the vote to general@incubator but I haven't seen it (not sure if it's just me..) Summary: +1 Steve Loughran +1 Billie Rinaldi +1 John Ament Thank you to those who took the time to look at the release. - Josh Josh Elser wrote: Hi, The Apache Slider project has voted to release 0.60.0-incubating. The team voted to release these artifacts with 6 +1's and nothing else. 2 of the 6 votes were from IPMC members. Vote thread: http://s.apache.org/a6O Result: http://s.apache.org/e3C Staged artifacts: http://people.apache.org/~stevel/slider/slider-release-0.60.0-incubating-rc1 SHA1: c2c06d3532708ac2ed765ebc465edda1b0067025 Source: https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=commit;h=c2c06d3532708ac2ed765ebc465edda1b0067025 Git tag: "release-0.60.0-incubating-rc1" PGP key: http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=ste...@apache.org Basic build/test instructions: http://slider.incubator.apache.org/developing/building.html Please vote on releasing the above as Apache Slider 0.60.0-incubating. This vote will be open for 72 hours. Thanks! - Apache Slider team - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
On 21 November 2014 20:44, Bertrand Delacretaz wrote: > Hi, > > On Fri, Nov 21, 2014 at 3:35 PM, David Nalley wrote: > > ...I am generally against us standing up our own service that does this. > > We've had a couple of these systems over the years. (pulse.a,o for > > instance). It takes a non-trivial amount of work to setup and maintain > > such a system, and invariably it falls apart > > I agree, OTOH if someone wants to help third parties get the data that > they need to implement such services externally that might be fine. > Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
RE: dashboarding incubator
We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz wrote: > Hi, > > On Fri, Nov 21, 2014 at 3:35 PM, David Nalley wrote: > > ...I am generally against us standing up our own service that does this. > > We've had a couple of these systems over the years. (pulse.a,o for > > instance). It takes a non-trivial amount of work to setup and > > maintain such a system, and invariably it falls apart > > I agree, OTOH if someone wants to help third parties get the data that > they need to implement such services externally that might be fine. > Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [VOTE] accept NiFi into the incubator
+1 On Fri, Nov 21, 2014 at 10:55 AM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: dashboarding incubator
Note - I should say my comment below is in the context of a not-for-profit software foundation using their services. People in different use cases should look at their specific circumstances - obviously :-D Ross -Original Message- From: Ross Gardler (MS OPEN TECH) [mailto:ross.gard...@microsoft.com] Sent: Friday, November 21, 2014 12:27 PM To: general@incubator.apache.org Subject: RE: dashboarding incubator We already evaluated the Bitergia offering - it is expensive and does not provide sufficient benefit for the money (don't get me started on how metrics are not a good evaluator of open source code...) I fully agree with the comments below. Ross -Original Message- From: jan i [mailto:j...@apache.org] Sent: Friday, November 21, 2014 12:24 PM To: general@incubator.apache.org Subject: Re: dashboarding incubator On 21 November 2014 20:44, Bertrand Delacretaz wrote: > Hi, > > On Fri, Nov 21, 2014 at 3:35 PM, David Nalley wrote: > > ...I am generally against us standing up our own service that does this. > > We've had a couple of these systems over the years. (pulse.a,o for > > instance). It takes a non-trivial amount of work to setup and > > maintain such a system, and invariably it falls apart > > I agree, OTOH if someone wants to help third parties get the data that > they need to implement such services externally that might be fine. > Having our own service will only marginally provide us with something better, and will cost (in endeffect) contractor resources, so I agree with david. rgds jan i. > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [VOTE] accept NiFi into the incubator
+1 -C On Fri, Nov 21, 2014 at 10:55 AM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] accept NiFi into the incubator
+1 (binding) On Fri Nov 21 2014 at 1:58:34 PM Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] >
Re: dashboarding incubator
On Fri, Nov 21, 2014 at 6:35 AM, David Nalley wrote: > On Fri, Nov 21, 2014 at 7:28 AM, Sergio Fernández > wrote: >> Hi, >> >> one of the things I brought from ApacheCon is the idea if providing >> dashboard for the podlings. Bitergia presented there their approach [1], and >> I think could be useful. See for instance the dashboard they provided for >> Apache CloudStack [2]. >> >> I already discussed it with Roman, and I think we should try to do something >> in this direction. I know the experiment with Back Software did not satisfy >> everybody, but I think we should keep trying. I can commit some time and >> even resources to provide an early proof of concept. >> > > I am generally against us standing up our own service that does this. Wait! Who's talking about 'our own services' (AKA stuff that INFRA manages)? The conversation is very clearly around Sergio offering his help managing whatever needs to be managed. > I do agree that things like bitergia provide much deeper sense of > analytics, but wonderful if it's really necessary for our purposes. The only way to find out is to give it a try. Honestly, I'm still a little bit miffed by our inability to do deeper metrics. I do of course understand that I'm the first one to blame -- ain't got no cycles for that. This is precisely why Segio's offer is so useful. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: dashboarding incubator
Sergio, I think there's no need to ask for permission in this particular case, unless you'd need to have INFRA help you obtaining information that is not publicly available. But then again, knowing what this information is would be useful anyway ;-) So yeah... go ahead! I'd like to help as much as I can, but I clearly can't drive it. Thanks, Roman. On Fri, Nov 21, 2014 at 4:28 AM, Sergio Fernández wrote: > Hi, > > one of the things I brought from ApacheCon is the idea if providing > dashboard for the podlings. Bitergia presented there their approach [1], and > I think could be useful. See for instance the dashboard they provided for > Apache CloudStack [2]. > > I already discussed it with Roman, and I think we should try to do something > in this direction. I know the experiment with Back Software did not satisfy > everybody, but I think we should keep trying. I can commit some time and > even resources to provide an early proof of concept. > > What do you think? > > Cheers, > > [1] > http://apacheconeu2014.sched.org/event/50a7178fa1d812e50e8c13e9bca8c7c1#.VG8vQnU3eis > [2] http://projects.bitergia.com/apache-cloudstack/browser/ > > -- > Sergio Fernández > Senior Researcher > Knowledge and Media Technologies > Salzburg Research Forschungsgesellschaft mbH > Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria > T: +43 662 2288 318 | M: +43 660 2747 925 > sergio.fernan...@salzburgresearch.at > http://www.salzburgresearch.at > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] accept NiFi into the incubator
What, the whole text? OK. On Fri, Nov 21, 2014 at 2:03 PM, Jake Farrell wrote: > Hey Benson > Can you please include the proposal with the vote, the wiki page can change > or be removed and the mailing list acts as the archive for the initial > proposal. > > -Jake > > On Fri, Nov 21, 2014 at 1:55 PM, Benson Margulies > wrote: > > > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful > and > > positive conversation, so I offer this vote. > > > > Vote will be open for the usual 72 hours ... > > > > Here is my [+1] > > >
Re: [VOTE] accept NiFi into the incubator
= NiFi Proposal = == Abstract == Ni``Fi is a dataflow system based on the concepts of flow-based programming. == Proposal == Ni``Fi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Ni``Fi include: * Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows * Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing * Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state * Component-based extension model along well defined interfaces enabling rapid development and effective testing == Background == Reliable and effective dataflow between systems can be difficult whether you're running scripts on a laptop or have a massive distributed computing system operated by numerous teams and organizations. As the volume and rate of data grows and as the number of systems, protocols, and formats increase and evolve so too does the complexity and need for greater insight and agility. These are the dataflow challenges that Ni``Fi was built to tackle. Ni``Fi is designed in a manner consistent with the core concepts described in flow-based programming as originally documented by J. Paul Morrison in the 1970s. http://www.jpaulmorrison.com/fbp/ This model lends itself well to visual diagramming, concurrency, componentization, testing, and reuse. In addition to staying close to the fundamentals of flow-based programming, Ni``Fi provides integration system specific features such as: guaranteed delivery; back pressure; ability to gracefully handle backlogs and data surges; and an operator interface that enables on-the-fly data flow generation, modification, and observation. == Rationale == Ni``Fi provides a reliable, scalable, manageable and accountable platform for developers and technical staff to create and evolve powerful data flows. Such a system is useful in many contexts including large-scale enterprise integration, interaction with cloud services and frameworks, business to business, intra-departmental, and inter-departmental flows. Ni``Fi fits well within the Apache Software Foundation (ASF) family as it depends on numerous ASF projects and integrates with several others. We also anticipate developing extensions for several other ASF projects such as Cassandra, Kafka, and Storm in the near future. == Initial Goals == * Ensure all dependencies are compliant with Apache License version 2.0 and all that all code and documentation artifacts have the correct Apache licensing markings and notice. * Establish a formal release process and schedule, allowing for dependable release cycles in a manner consistent with the Apache development process. * Establish a process which allows different release cycles for the core framework and extensions. * Grow the community to establish diversity of background and expertise. == Current Status == === Meritocracy === An integration platform is only as good as its ability to integrate systems in a reliable, timely, and repeatable manner. The same can be said of its ability to attract talent and a variety of perspectives as integration systems by their nature are always evolving. We will actively seek help and encourage promotion of influence in the project through meritocracy. === Community === Over the past several years, Ni``Fi has developed a strong community of both developers and operators within the U.S. government. In open sourcing Ni``Fi we plan to grow the community to a broader base of industries and will work to align the interaction of our existing community. === Core Developers === The initial core developers are employed by the National Security Agency and defense contractors. We will work to grow the community among a more diverse set of developers and industries. === Alignment === >From its inception, Ni``Fi was developed with an open source philosophy in mind and with the hopes of eventually being truly open sourced. The Apache way is consistent with the approach we have taken to date. The ASF clearly provides a mature and effective environment for successful development as is evident across the spectrum of well-known projects. Further, Ni``Fi depends on numerous ASF libraries and projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, Http``Client, Jakarta and Maven. We also anticipate extensions and dependencies with several more ASF projects, including Accumulo, Avro, Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others. This existing alignment with Apache and the desired community makes the Apache Incubator a good fit for Ni``Fi. == Known Risks == === Orphaned Products === Risk of orphaning is limited though it is important t
Re: [VOTE] accept NiFi into the incubator
+1 On Fri, Nov 21, 2014 at 1:00 PM, John D. Ament wrote: > +1 (binding) > > On Fri Nov 21 2014 at 1:58:34 PM Benson Margulies > wrote: > > > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful > and > > positive conversation, so I offer this vote. > > > > Vote will be open for the usual 72 hours ... > > > > Here is my [+1] > > >
Re: [VOTE] accept NiFi into the incubator
+1 (binding) -Jake On Fri, Nov 21, 2014 at 1:55 PM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] >
Re: [PROPOSAL] NiFi for Incubation
>From what I understand, that Accumulo may or may not suffer from "bifurcation" is debatable despite this blanket claim otherwise. Regardless, Sean's point is valid, even if only a theoretical concern for NiFi at this point. If the podling is interested in growing outside community I'd be happy to sign on as a mentor to help that happen. On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies wrote: > Sean, > > The precedent of Accumulo is that the govt people and agencies involved are > ready and able to have their staff collaborate openly in an Apache > community. There's no need to contemplate bifurcation; we have this > proposal because the management recognizes that this collaboration produces > better stuff that solves more problems than the 'inside the tent' > alternative. > > --benson > > > On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey wrote: > > > I'm really excited to see NiFi come to the incubator; it'd be a great > > addition to the ASF. > > > > A few points in the proposal: > > > > > >== Initial Goals == > > > > One of these should be to grow the community outside of the current > niche, > > IMHO. > > > > More on this below under "orphaned projects" > > > > > > * Determine and establish a mechanism, possibly including a > > > >sub-project construct, that allows for extensions to the core > > > >application to occur at a pace that differs from the core application > > > >itself. > > > > I don't think the proposal needs to include the "e.g. with sub-projects" > > part. Just noting > > that your goals in the incubator are to address the need to have > different > > release cycles > > for core and extensions is sufficient. > > > > > > > >=== Community === > > > >Over the past several years, NiFi has developed a strong community of > > > >both developers and operators within the U.S. government. We look > > > >forward to helping grow this to a broader base of industries. > > > > > > > > How much, if any, of this community do you expect to engage via the > > customary project > > lists once NiFi is established within the ASF? Will the project be able > to > > leverage this > > established group? > > > > > > > >=== Orphaned Products === > > > >Risk of orphaning is minimal. The project user and developer base is > > > >substantial, growing, and there is already extensive operational use > > > >of NiFi. > > > > Given that the established base is internal to the U.S. government, I'd > > encourage the > > podling to consider the risk of a bifurcated project should a substantial > > outside > > community fail to emerge or if those internal users should fail to engage > > with the > > outside community. > > > > You cover a related issue in your "Homogenous Developers" section. But I > > think > > building on the "Community" section of the current state to call this out > > as an > > independent issue is worthwhile. > > > > > > > >possible. This environment includes widely accessible source code > > > >repositories, published artifacts, ticket tracking, and extensive > > > >documentation. We also encourage contributions and frequent debate and > > > >hold regular, collaborative discussions through e-mail, chat rooms, > > > >and in-person meet-ups. > > > > Do you anticipate any difficulties moving these established communication > > mechanisms to ASF public lists? > > > > > >=== Documentation === > > > >At this time there is no NiFi documentation on the web. However, we > > > >have extensive documentation included within the application that > > > >details usage of the many functions. We will be rapidly expanding the > > > >available documentation to cover things like installation, developer > > > >guide, frequently asked questions, best practices, and more. This > > > >documentation will be posted to the NiFi wiki at apache.org. > > > > I love projects that start with documentation. :) > > > > I don't think the proposal needs to include that the documentation will > be > > posted > > to the NiFi wiki, since that's an implementation detail. Just say "this > > documentation > > will be made available via the NiFi project's use of incubator infra." > > > > (I'll save detail for the eventual dev@ list, but you should strongly > > consider not > > using the wiki to host this documentation.) > > > > -Sean > > > > On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland > wrote: > > > > > Hi Joe, > > > > > > I know you've done a tremendous amount of work to make this happen so I > > am > > > extremely happy this is *finally* making it's way to the incubator! > > > > > > I look forward to helping in anyway I can. > > > > > > Cheers! > > > Brock > > > > > > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) < > > > chris.a.mattm...@jpl.nasa.gov> wrote: > > > > > > > This is *fan freakin¹ tastic* Sounds like an awesome project and > > > > glad to hear a relationship to Tika! Awesome to see more government > > > > projects coming into the ASF! > > > > > > > > you already have a great set of mentors and I don¹t rea
Re: [PROPOSAL] NiFi for Incubation
I would tend to disagree with your assertion about Accumulo, but this is neither the time nor the place to discuss Accumulo's strengths and weaknesses. I'll make the assumption that request to enter incubation is a strong intent to grow outside of the project's current home. +1 (non-binding) for entry Andrew Purtell wrote: From what I understand, that Accumulo may or may not suffer from "bifurcation" is debatable despite this blanket claim otherwise. Regardless, Sean's point is valid, even if only a theoretical concern for NiFi at this point. If the podling is interested in growing outside community I'd be happy to sign on as a mentor to help that happen. On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies wrote: Sean, The precedent of Accumulo is that the govt people and agencies involved are ready and able to have their staff collaborate openly in an Apache community. There's no need to contemplate bifurcation; we have this proposal because the management recognizes that this collaboration produces better stuff that solves more problems than the 'inside the tent' alternative. --benson On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey wrote: I'm really excited to see NiFi come to the incubator; it'd be a great addition to the ASF. A few points in the proposal: == Initial Goals == One of these should be to grow the community outside of the current niche, IMHO. More on this below under "orphaned projects" * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. I don't think the proposal needs to include the "e.g. with sub-projects" part. Just noting that your goals in the incubator are to address the need to have different release cycles for core and extensions is sufficient. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. How much, if any, of this community do you expect to engage via the customary project lists once NiFi is established within the ASF? Will the project be able to leverage this established group? === Orphaned Products === Risk of orphaning is minimal. The project user and developer base is substantial, growing, and there is already extensive operational use of NiFi. Given that the established base is internal to the U.S. government, I'd encourage the podling to consider the risk of a bifurcated project should a substantial outside community fail to emerge or if those internal users should fail to engage with the outside community. You cover a related issue in your "Homogenous Developers" section. But I think building on the "Community" section of the current state to call this out as an independent issue is worthwhile. possible. This environment includes widely accessible source code repositories, published artifacts, ticket tracking, and extensive documentation. We also encourage contributions and frequent debate and hold regular, collaborative discussions through e-mail, chat rooms, and in-person meet-ups. Do you anticipate any difficulties moving these established communication mechanisms to ASF public lists? === Documentation === At this time there is no NiFi documentation on the web. However, we have extensive documentation included within the application that details usage of the many functions. We will be rapidly expanding the available documentation to cover things like installation, developer guide, frequently asked questions, best practices, and more. This documentation will be posted to the NiFi wiki at apache.org. I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say "this documentation will be made available via the NiFi project's use of incubator infra." (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)< chris.a.mattm...@jpl.nasa.gov> wrote: This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF! you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++ Chris Mattmann, Ph.D. Chief Arc
Re: [PROPOSAL] NiFi for Incubation
I didn't make an assertion *about* Accumulo, what I said exactly was: "that Accumulo may or may not suffer from "bifurcation" is debatable" I assert there are two sides to that argument (I've heard them both), merely to suggest that Sean's concerns should not be dismissed as they were. On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser wrote: > I would tend to disagree with your assertion about Accumulo, but this is > neither the time nor the place to discuss Accumulo's strengths and > weaknesses. > > I'll make the assumption that request to enter incubation is a strong > intent to grow outside of the project's current home. > > +1 (non-binding) for entry > > > Andrew Purtell wrote: > >> From what I understand, that Accumulo may or may not suffer from >> "bifurcation" is debatable despite this blanket claim otherwise. >> >> Regardless, Sean's point is valid, even if only a theoretical concern for >> NiFi at this point. If the podling is interested in growing outside >> community I'd be happy to sign on as a mentor to help that happen. >> >> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies >> wrote: >> >> Sean, >>> >>> The precedent of Accumulo is that the govt people and agencies involved >>> are >>> ready and able to have their staff collaborate openly in an Apache >>> community. There's no need to contemplate bifurcation; we have this >>> proposal because the management recognizes that this collaboration >>> produces >>> better stuff that solves more problems than the 'inside the tent' >>> alternative. >>> >>> --benson >>> >>> >>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey >>> wrote: >>> >>> I'm really excited to see NiFi come to the incubator; it'd be a great addition to the ASF. A few points in the proposal: == Initial Goals == >> > One of these should be to grow the community outside of the current >>> niche, >>> IMHO. More on this below under "orphaned projects" * Determine and establish a mechanism, possibly including a >> sub-project construct, that allows for extensions to the core >> application to occur at a pace that differs from the core application >> itself. >> > I don't think the proposal needs to include the "e.g. with sub-projects" part. Just noting that your goals in the incubator are to address the need to have >>> different >>> release cycles for core and extensions is sufficient. === Community === >> Over the past several years, NiFi has developed a strong community of >> both developers and operators within the U.S. government. We look >> forward to helping grow this to a broader base of industries. >> >> How much, if any, of this community do you expect to engage via the customary project lists once NiFi is established within the ASF? Will the project be able >>> to >>> leverage this established group? === Orphaned Products === >> Risk of orphaning is minimal. The project user and developer base is >> substantial, growing, and there is already extensive operational use >> of NiFi. >> > Given that the established base is internal to the U.S. government, I'd encourage the podling to consider the risk of a bifurcated project should a substantial outside community fail to emerge or if those internal users should fail to engage with the outside community. You cover a related issue in your "Homogenous Developers" section. But I think building on the "Community" section of the current state to call this out as an independent issue is worthwhile. possible. This environment includes widely accessible source code >> repositories, published artifacts, ticket tracking, and extensive >> documentation. We also encourage contributions and frequent debate and >> hold regular, collaborative discussions through e-mail, chat rooms, >> and in-person meet-ups. >> > Do you anticipate any difficulties moving these established communication mechanisms to ASF public lists? === Documentation === >> At this time there is no NiFi documentation on the web. However, we >> have extensive documentation included within the application that >> details usage of the many functions. We will be rapidly expanding the >> available documentation to cover things like installation, developer >> guide, frequently asked questions, best practices, and more. This >> documentation will be posted to the NiFi wiki at apache.org. >> > I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will >>> be >>> posted to the NiFi wiki, since that's an implementation detail. Just say "this documentation will be made available via the NiFi project's
Re: [PROPOSAL] NiFi for Incubation
Thanks for the clarification -- that wasn't how it came across to me. Andrew Purtell wrote: I didn't make an assertion *about* Accumulo, what I said exactly was: "that Accumulo may or may not suffer from "bifurcation" is debatable" I assert there are two sides to that argument (I've heard them both), merely to suggest that Sean's concerns should not be dismissed as they were. On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser wrote: I would tend to disagree with your assertion about Accumulo, but this is neither the time nor the place to discuss Accumulo's strengths and weaknesses. I'll make the assumption that request to enter incubation is a strong intent to grow outside of the project's current home. +1 (non-binding) for entry Andrew Purtell wrote: From what I understand, that Accumulo may or may not suffer from "bifurcation" is debatable despite this blanket claim otherwise. Regardless, Sean's point is valid, even if only a theoretical concern for NiFi at this point. If the podling is interested in growing outside community I'd be happy to sign on as a mentor to help that happen. On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies wrote: Sean, The precedent of Accumulo is that the govt people and agencies involved are ready and able to have their staff collaborate openly in an Apache community. There's no need to contemplate bifurcation; we have this proposal because the management recognizes that this collaboration produces better stuff that solves more problems than the 'inside the tent' alternative. --benson On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey wrote: I'm really excited to see NiFi come to the incubator; it'd be a great addition to the ASF. A few points in the proposal: == Initial Goals == One of these should be to grow the community outside of the current niche, IMHO. More on this below under "orphaned projects" * Determine and establish a mechanism, possibly including a sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. I don't think the proposal needs to include the "e.g. with sub-projects" part. Just noting that your goals in the incubator are to address the need to have different release cycles for core and extensions is sufficient. === Community === Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. How much, if any, of this community do you expect to engage via the customary project lists once NiFi is established within the ASF? Will the project be able to leverage this established group? === Orphaned Products === Risk of orphaning is minimal. The project user and developer base is substantial, growing, and there is already extensive operational use of NiFi. Given that the established base is internal to the U.S. government, I'd encourage the podling to consider the risk of a bifurcated project should a substantial outside community fail to emerge or if those internal users should fail to engage with the outside community. You cover a related issue in your "Homogenous Developers" section. But I think building on the "Community" section of the current state to call this out as an independent issue is worthwhile. possible. This environment includes widely accessible source code repositories, published artifacts, ticket tracking, and extensive documentation. We also encourage contributions and frequent debate and hold regular, collaborative discussions through e-mail, chat rooms, and in-person meet-ups. Do you anticipate any difficulties moving these established communication mechanisms to ASF public lists? === Documentation === At this time there is no NiFi documentation on the web. However, we have extensive documentation included within the application that details usage of the many functions. We will be rapidly expanding the available documentation to cover things like installation, developer guide, frequently asked questions, best practices, and more. This documentation will be posted to the NiFi wiki at apache.org. I love projects that start with documentation. :) I don't think the proposal needs to include that the documentation will be posted to the NiFi wiki, since that's an implementation detail. Just say "this documentation will be made available via the NiFi project's use of incubator infra." (I'll save detail for the eventual dev@ list, but you should strongly consider not using the wiki to host this documentation.) -Sean On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland wrote: Hi Joe, I know you've done a tremendous amount of work to make this happen so I am extremely happy this is *finally* making it's way to the incubator! I look forward to helping in anyway I can. Cheers! Brock On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris
Re: [VOTE] accept NiFi into the incubator
+1 (non-binding) rb On 11/21/2014 01:54 PM, Jake Farrell wrote: +1 (binding) -Jake On Fri, Nov 21, 2014 at 1:55 PM, Benson Margulies wrote: http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and positive conversation, so I offer this vote. Vote will be open for the usual 72 hours ... Here is my [+1] -- Ryan Blue Software Engineer Cloudera, Inc. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] NiFi for Incubation
Andrew - appreciate your comments and offer to assist as a mentor. Now that the vote for incubation has kicked off i'm not sure what i'm allowed to do or not. Will check will Benson. For what it is worth I agree Sean raised a great point. I think it will be an important part of our community building effort. And to echo Josh's point this proposal represents not just our intent to grow the community outside our current home but rather move homes - to the ASF. We're very excited to have this opportunity. Thanks Joe On Fri, Nov 21, 2014 at 5:19 PM, Josh Elser wrote: > Thanks for the clarification -- that wasn't how it came across to me. > > > Andrew Purtell wrote: > >> I didn't make an assertion *about* Accumulo, what I said exactly was: >> >> "that Accumulo may or may not suffer from "bifurcation" is >> debatable" >> >> I assert there are two sides to that argument (I've heard them both), >> merely to suggest that Sean's concerns should not be dismissed as they >> were. >> >> >> On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser wrote: >> >> I would tend to disagree with your assertion about Accumulo, but this is >>> neither the time nor the place to discuss Accumulo's strengths and >>> weaknesses. >>> >>> I'll make the assumption that request to enter incubation is a strong >>> intent to grow outside of the project's current home. >>> >>> +1 (non-binding) for entry >>> >>> >>> Andrew Purtell wrote: >>> >>> From what I understand, that Accumulo may or may not suffer from "bifurcation" is debatable despite this blanket claim otherwise. Regardless, Sean's point is valid, even if only a theoretical concern for NiFi at this point. If the podling is interested in growing outside community I'd be happy to sign on as a mentor to help that happen. On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies>>> > wrote: Sean, > The precedent of Accumulo is that the govt people and agencies involved > are > ready and able to have their staff collaborate openly in an Apache > community. There's no need to contemplate bifurcation; we have this > proposal because the management recognizes that this collaboration > produces > better stuff that solves more problems than the 'inside the tent' > alternative. > > --benson > > > On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey > wrote: > > I'm really excited to see NiFi come to the incubator; it'd be a great > >> addition to the ASF. >> >> A few points in the proposal: >> >> == Initial Goals == >> >>> One of these should be to grow the community outside of the current >>> >> niche, > > IMHO. >> >> More on this below under "orphaned projects" >> >> * Determine and establish a mechanism, possibly including a >> >>> sub-project construct, that allows for extensions to the core application to occur at a pace that differs from the core application itself. I don't think the proposal needs to include the "e.g. with >>> >> sub-projects" >> part. Just noting >> that your goals in the incubator are to address the need to have >> >> different > > release cycles >> for core and extensions is sufficient. >> >> >> === Community === >> >>> Over the past several years, NiFi has developed a strong community of both developers and operators within the U.S. government. We look forward to helping grow this to a broader base of industries. How much, if any, of this community do you expect to engage via the >>> customary project >> lists once NiFi is established within the ASF? Will the project be >> able >> >> to > > leverage this >> established group? >> >> >> === Orphaned Products === >> >>> Risk of orphaning is minimal. The project user and developer base is substantial, growing, and there is already extensive operational use of NiFi. Given that the established base is internal to the U.S. >>> government, I'd >>> >> encourage the >> podling to consider the risk of a bifurcated project should a >> substantial >> outside >> community fail to emerge or if those internal users should fail to >> engage >> with the >> outside community. >> >> You cover a related issue in your "Homogenous Developers" section. >> But I >> think >> building on the "Community" section of the current state to call this >> out >> as an >> independent issue is worthwhile. >> >> >> possible. This environment includes widely accessible source code >> >>> repositories, published artifacts, ticket tracking, and extensive documentation. We also encourage contribution
Re: [VOTE] accept NiFi into the incubator
+1 ( binding) On Fri, Nov 21, 2014 at 10:55 AM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [VOTE] accept NiFi into the incubator
On Fri, Nov 21, 2014 at 7:55 PM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. +1 -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] accept NiFi into the incubator
+1 (binding) Regards, Arvind Prabhakar On Fri, Nov 21, 2014 at 1:51 PM, Benson Margulies wrote: > = NiFi Proposal = > > == Abstract == > Ni``Fi is a dataflow system based on the concepts of flow-based > programming. > > == Proposal == > Ni``Fi supports powerful and scalable directed graphs of data routing, > transformation, and system mediation logic. Some of the high-level > capabilities and objectives of Ni``Fi include: > * Web-based user interface for seamless experience between design, > control, feedback, and monitoring of data flows > * Highly configurable along several dimensions of quality of service > such as loss tolerant versus guaranteed delivery, low latency versus > high throughput, and priority based queuing > * Fine-grained data provenance for all data received, forked, > joined, cloned, modified, sent, and ultimately dropped as data reaches > its configured end-state > * Component-based extension model along well defined interfaces > enabling rapid development and effective testing > > == Background == > Reliable and effective dataflow between systems can be difficult > whether you're running scripts on a laptop or have a massive > distributed computing system operated by numerous teams and > organizations. As the volume and rate of data grows and as the number > of systems, protocols, and formats increase and evolve so too does the > complexity and need for greater insight and agility. These are the > dataflow challenges that Ni``Fi was built to tackle. > > Ni``Fi is designed in a manner consistent with the core concepts > described in flow-based programming as originally documented by J. > Paul Morrison in the 1970s. http://www.jpaulmorrison.com/fbp/ This > model lends itself well to visual diagramming, concurrency, > componentization, testing, and reuse. In addition to staying close to > the fundamentals of flow-based programming, Ni``Fi provides > integration system specific features such as: guaranteed delivery; > back pressure; ability to gracefully handle backlogs and data surges; > and an operator interface that enables on-the-fly data flow > generation, modification, and observation. > > == Rationale == > Ni``Fi provides a reliable, scalable, manageable and accountable > platform for developers and technical staff to create and evolve > powerful data flows. Such a system is useful in many contexts > including large-scale enterprise integration, interaction with cloud > services and frameworks, business to business, intra-departmental, and > inter-departmental flows. Ni``Fi fits well within the Apache Software > Foundation (ASF) family as it depends on numerous ASF projects and > integrates with several others. We also anticipate developing > extensions for several other ASF projects such as Cassandra, Kafka, > and Storm in the near future. > > == Initial Goals == > * Ensure all dependencies are compliant with Apache License version > 2.0 and all that all code and documentation artifacts have the correct > Apache licensing markings and notice. > * Establish a formal release process and schedule, allowing for > dependable release cycles in a manner consistent with the Apache > development process. > * Establish a process which allows different release cycles for the > core framework and extensions. > * Grow the community to establish diversity of background and expertise. > > == Current Status == > === Meritocracy === > An integration platform is only as good as its ability to integrate > systems in a reliable, timely, and repeatable manner. The same can be > said of its ability to attract talent and a variety of perspectives as > integration systems by their nature are always evolving. We will > actively seek help and encourage promotion of influence in the project > through meritocracy. > > === Community === > Over the past several years, Ni``Fi has developed a strong community > of both developers and operators within the U.S. government. In open > sourcing Ni``Fi we plan to grow the community to a broader base of > industries and will work to align the interaction of our existing > community. > > === Core Developers === > The initial core developers are employed by the National Security > Agency and defense contractors. We will work to grow the community > among a more diverse set of developers and industries. > > === Alignment === > From its inception, Ni``Fi was developed with an open source > philosophy in mind and with the hopes of eventually being truly open > sourced. The Apache way is consistent with the approach we have taken > to date. The ASF clearly provides a mature and effective environment > for successful development as is evident across the spectrum of > well-known projects. Further, Ni``Fi depends on numerous ASF > libraries and projects including; ActiveMQ, Ant, Commons, Lucene, > Hadoop, Http``Client, Jakarta and Maven. We also anticipate > extensions and dependencies with several more ASF projects, including > Accumulo, Av
Re: [VOTE] accept NiFi into the incubator
+1 - binding On Nov 21, 2014, at 10:55 AM, Benson Margulies wrote: > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful and > positive conversation, so I offer this vote. > > Vote will be open for the usual 72 hours ... > > Here is my [+1] - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Accept Kylin into the Apache Incubator
+1 binding Regards, Alan On Nov 20, 2014, at 2:31 PM, Luke Han wrote: > Following the discussion earlier in the thread: > > http://mail-archives.apache.org/mod_mbox/incubator-general/201411.mbox/%3ccakmqrob22+n+r++date33f3pcpyujhfoeaqrms3t-udjwk6...@mail.gmail.com%3e > > I would like to call a VOTE for accepting Kylin as a new incubator project. > > The proposal is available at: > https://wiki.apache.org/incubator/KylinProposal > > and posted the text of the proposal below also. > > Vote is open until 24th November 2014, 23:59:00 UTC > > [ ] +1 accept Kylin in the Incubator > [ ] ±0 > [ ] -1 because... > - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] accept NiFi into the incubator
+1 (non-binding) Great proposal! On Fri, Nov 21, 2014 at 16:22 Alan D. Cabrera wrote: > +1 - binding > > On Nov 21, 2014, at 10:55 AM, Benson Margulies > wrote: > > > http://wiki.apache.org/incubator/NiFiProposal has elicited a cheerful > and > > positive conversation, so I offer this vote. > > > > Vote will be open for the usual 72 hours ... > > > > Here is my [+1] > > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [PROPOSAL] NiFi for Incubation
Andrew - I've updated the proposal to reflect you as a mentor. * Andrew Purtell (Salesforce) , IPMC Member Thanks again Joe On Fri, Nov 21, 2014 at 5:29 PM, Joe Witt wrote: > Andrew - appreciate your comments and offer to assist as a mentor. Now > that the vote for incubation has kicked off i'm not sure what i'm allowed > to do or not. Will check will Benson. > > For what it is worth I agree Sean raised a great point. I think it will > be an important part of our community building effort. And to echo Josh's > point this proposal represents not just our intent to grow the community > outside our current home but rather move homes - to the ASF. We're very > excited to have this opportunity. > > Thanks > Joe > > On Fri, Nov 21, 2014 at 5:19 PM, Josh Elser wrote: > >> Thanks for the clarification -- that wasn't how it came across to me. >> >> >> Andrew Purtell wrote: >> >>> I didn't make an assertion *about* Accumulo, what I said exactly was: >>> >>> "that Accumulo may or may not suffer from "bifurcation" is >>> debatable" >>> >>> I assert there are two sides to that argument (I've heard them both), >>> merely to suggest that Sean's concerns should not be dismissed as they >>> were. >>> >>> >>> On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser wrote: >>> >>> I would tend to disagree with your assertion about Accumulo, but this is neither the time nor the place to discuss Accumulo's strengths and weaknesses. I'll make the assumption that request to enter incubation is a strong intent to grow outside of the project's current home. +1 (non-binding) for entry Andrew Purtell wrote: From what I understand, that Accumulo may or may not suffer from > "bifurcation" is debatable despite this blanket claim otherwise. > > Regardless, Sean's point is valid, even if only a theoretical concern > for > NiFi at this point. If the podling is interested in growing outside > community I'd be happy to sign on as a mentor to help that happen. > > On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies com> > wrote: > > Sean, > >> The precedent of Accumulo is that the govt people and agencies >> involved >> are >> ready and able to have their staff collaborate openly in an Apache >> community. There's no need to contemplate bifurcation; we have this >> proposal because the management recognizes that this collaboration >> produces >> better stuff that solves more problems than the 'inside the tent' >> alternative. >> >> --benson >> >> >> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey >> wrote: >> >> I'm really excited to see NiFi come to the incubator; it'd be a >> great >> >>> addition to the ASF. >>> >>> A few points in the proposal: >>> >>> == Initial Goals == >>> One of these should be to grow the community outside of the current >>> niche, >> >> IMHO. >>> >>> More on this below under "orphaned projects" >>> >>> * Determine and establish a mechanism, possibly including a >>> sub-project construct, that allows for extensions to the core > application to occur at a pace that differs from the core > application > itself. > > I don't think the proposal needs to include the "e.g. with >>> sub-projects" >>> part. Just noting >>> that your goals in the incubator are to address the need to have >>> >>> different >> >> release cycles >>> for core and extensions is sufficient. >>> >>> >>> === Community === >>> Over the past several years, NiFi has developed a strong community > of > both developers and operators within the U.S. government. We look > forward to helping grow this to a broader base of industries. > > How much, if any, of this community do you expect to engage via > the > customary project >>> lists once NiFi is established within the ASF? Will the project be >>> able >>> >>> to >> >> leverage this >>> established group? >>> >>> >>> === Orphaned Products === >>> Risk of orphaning is minimal. The project user and developer base > is > substantial, growing, and there is already extensive operational > use > of NiFi. > > Given that the established base is internal to the U.S. government, I'd >>> encourage the >>> podling to consider the risk of a bifurcated project should a >>> substantial >>> outside >>> community fail to emerge or if those internal users should fail to >>> engage >>> with the >>> outside community. >>> >>> You cover a related issue in your "Homogenous Developers" section. >>>
