This is *fan freakin¹ tastic* Sounds like an awesome project and glad to hear a relationship to Tika! Awesome to see more government projects coming into the ASF!
you already have a great set of mentors and I don¹t really have more time on my plate, but really happy and will try and monitor and help on the lists. Cheers! Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Joe Witt <joe.w...@gmail.com> Reply-To: "general@incubator.apache.org" <general@incubator.apache.org> Date: Thursday, November 20, 2014 at 3:02 AM To: "general@incubator.apache.org" <general@incubator.apache.org> Subject: [PROPOSAL] NiFi for Incubation >Hello, > > >I would like to propose NiFi as an Apache Incubator Project. > >In addition to the copy provided below the Wiki version of the >proposal can be found here: >http://wiki.apache.org/incubator/NiFiProposal > >Thanks > >Joe > > >= NiFi Proposal = > >== Abstract == >NiFi is a dataflow system based on the concepts of flow-based programming. > >== Proposal == >NiFi supports powerful and scalable directed graphs of data routing, >transformation, and system mediation logic. Some of the high-level >capabilities and objectives of NiFi include: > * Web-based user interface for seamless experience between design, >control, feedback, and monitoring of data flows > * Highly configurable along several dimensions of quality of service >such as loss tolerant versus guaranteed delivery, low latency versus >high throughput, and priority based queuing > * Fine-grained data provenance for all data received, forked, >joined, cloned, modified, sent, and ultimately dropped as data reaches >its configured end-state > * Component-based extension model along well defined interfaces >enabling rapid development and effective testing > >== Background == >Reliable and effective dataflow between systems can be difficult >whether you're running scripts on a laptop or have a massive >distributed computing system operated by numerous teams and >organizations. As the volume and rate of data grows and as the number >of systems, protocols, and formats increase and evolve so too does the >complexity and need for greater insight and agility. These are the >dataflow challenges that NiFi was built to tackle. > >NiFi is designed in a manner consistent with the core concepts >described in flow-based programming as originally documented by J. >Paul Morrison in the 1970s. This model lends itself well to visual >diagramming, concurrency, componentization, testing, and reuse. In >addition to staying close to the fundamentals of flow-based >programming, NiFi provides integration system specific features such >as: guaranteed delivery; back pressure; ability to gracefully handle >backlogs and data surges; and an operator interface that enables >on-the-fly data flow generation, modification, and observation. > >== Rationale == >NiFi provides a reliable, scalable, manageable and accountable >platform for developers and technical staff to create and evolve >powerful data flows. Such a system is useful in many contexts >including large-scale enterprise integration, interaction with cloud >services and frameworks, business to business, intra-departmental, and >inter-departmental flows. NiFi fits well within the Apache Software >Foundation (ASF) family as it depends on numerous ASF projects and >integrates with several others. We also anticipate developing >extensions for several other ASF projects such as Cassandra, Kafka, >and Storm in the near future. > >== Initial Goals == > * Ensure all dependencies are compliant with Apache License version >2.0 and all that all code and documentation artifacts have the correct >Apache licensing markings and notice. > * Establish a formal release process and schedule, allowing for >dependable release cycles in a manner consistent with the Apache >development process. > * Determine and establish a mechanism, possibly including a >sub-project construct, that allows for extensions to the core >application to occur at a pace that differs from the core application >itself. > >== Current Status == >=== Meritocracy === >An integration platform is only as good as its ability to integrate >systems in a reliable, timely, and repeatable manner. The same can be >said of its ability to attract talent and a variety of perspectives as >integration systems by their nature are always evolving. We will >actively seek help and encourage promotion of influence in the project >through meritocracy. > >=== Community === >Over the past several years, NiFi has developed a strong community of >both developers and operators within the U.S. government. We look >forward to helping grow this to a broader base of industries. > >=== Core Developers === >The initial core developers are employed by the National Security >Agency and defense contractors. We will work to grow the community >among a more diverse set of developers and industries. > >=== Alignment === >From its inception, NiFi was developed with an open source philosophy >in mind and with the hopes of eventually being truly open sourced. >The Apache way is consistent with the approach we have taken to date. >The ASF clearly provides a mature and effective environment for >successful development as is evident across the spectrum of well-known >projects. Further, NiFi depends on numerous ASF libraries and >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, >HttpClient, Jakarta and Maven. We also anticipate extensions and >dependencies with several more ASF projects, including Accumulo, Avro, >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others. >This existing alignment with Apache and the desired community makes >the Apache Incubator a good fit for NiFi. > >== Known Risks == >=== Orphaned Products === >Risk of orphaning is minimal. The project user and developer base is >substantial, growing, and there is already extensive operational use >of NiFi. > >=== Inexperience with Open Source === >The initial committers to NiFi have limited experience with true open >source software development. However, despite the project origins >being from closed source development we have modelled our behavior and >community development on The Apache Way to the greatest extent >possible. This environment includes widely accessible source code >repositories, published artifacts, ticket tracking, and extensive >documentation. We also encourage contributions and frequent debate and >hold regular, collaborative discussions through e-mail, chat rooms, >and in-person meet-ups. We are committed to the ideals of open source >software and will eagerly seek out mentors and sponsors who can help >us quickly come up to speed. > >=== Homogenous Developers === >The initial committers of NiFi come from a limited set of entities >though we are committed to recruiting and developing additional >committers from a broad spectrum of industries and backgrounds. > >=== Reliance on Salaried Developers === >We expect NiFi development to continue on salaried time and through >volunteer time. The initial committers are paid by their employers to >contribute to this project. We are committed to developing and >recruiting participation from developers both salaried and >non-salaried. > >=== Relationship with other Apache Projects === >As described in the alignment section, NiFi is already heavily >dependent on other ASF projects and we anticipate further dependence >and integration with new and emerging projects in the Apache family. > >=== An Excessive Fascination with the Apache Brand === >We respect the laudable Apache brand and that is certainly a factor in >the decision to propose NiFi for the Apache Incubator. We intend to >provide a great deal of energy and capability to the ASF through this >project. We will be sensitive to and respectful of any overuse of the >Apache brand and ensure our focus remains on how we benefit the Apache >community. > >=== Documentation === >At this time there is no NiFi documentation on the web. However, we >have extensive documentation included within the application that >details usage of the many functions. We will be rapidly expanding the >available documentation to cover things like installation, developer >guide, frequently asked questions, best practices, and more. This >documentation will be posted to the NiFi wiki at apache.org. > >== Initial Source == >NiFi has been in active development since late 2006 with contributions >from dozens of developers and feedback from hundreds of users and >developers. The core codebase is written in Java and includes >detailed Javadocs and feature documentation. > >== Source and Intellectual Property Submission == >Previously referred to as Niagarafiles, the NiFi code and >documentation materials will be submitted by the National Security >Agency. NiFi has been developed by a mix of government employees and >private companies under government contract. Material developed by >the government employees is in the public domain and no U.S. copyright >exists in works of the federal government. For the contractor >developed material in the initial submission, the U.S. Government has >sufficient authority to open source per DFARS 252.227-7014. NSA has >submitted the Software Grant Agreement and Corporate Contributor >License Agreement to the Apache Software Foundation. > >== External Dependencies == >We have at least one dependency on an LGPL library which we will >promptly address. Otherwise, we believe all current dependencies are >compatible with the ASF guidelines. Our dependency licenses come from >the following license styles: Apache v 2.0, BSD, Public Domain, >Eclipse Public v1, MIT, CDDL v1. > >== Cryptography == >Consistent with http://www.apache.org/licenses/exports/ we believe >NiFi is classified as ECCN 5D002. NiFi doesn't implement any >cryptographic algorithms but is designed to use algorithms provided by >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc. >These cryptographic algorithm providers are used to support SSL, >SSH/SFTP, and the encryption and decryption of sensitive properties. >In the event that it becomes necessary we will engage with appropriate >Apache members to ensure we file any necessary paperwork or clarified >any cryptographic export license concerns. > >== Required Resources == >=== Mailing Lists === > * u...@nifi.incubator.apache.org > * d...@nifi.incubator.apache.org > * priv...@nifi.incubator.apache.org > * comm...@nifi.incubator.apache.org > >=== Source Control === >NiFi requests use of Git for source control >(git://git.apache.org/nifi.git). We request a writeable Git repo for >NiFi with mirroring to be setup to Github through INFRA. We request >sponsor Benson Margulies (bimargulies) to assist with creating the >INFRA ticket for this. > >=== Issue Tracking === >JIRA NiFi (NIFI) > >=== Initial Committers === > * Brandon Devries <brandon.devries at gmail dot com> > * Matt Gilman <matt.c.gilman at gmail dot com> > * Tony Kurc <trkurc at gmail dot com> > * Mark Payne <markap14 at hotmail dot com> > * Adam Taft <adam at adamtaft dot com> > * Joseph Witt <joewitt at gmail dot com> > >=== Affiliations === > * Brandon Devries (Requitest, Inc.) > * Matt Gilman (Raytheon) > * Tony Kurc (National Security Agency) > * Mark Payne (Sotera Defense Solutions, Inc.) > * Adam Taft (Requitest, Inc.) > * Joseph Witt (National Security Agency) > >== Sponsors == >=== Champion === > * Benson Margulies (Basis Technology) <bimargulies at apache dot org> > >=== Nominated Mentors === > * Drew Farris (Booz Allen Hamilton) <drew at apache dot org> > * Brock Noland (Cloudera) <brock at apache dot org> > * Billie Rinaldi (Hortonworks) <billie at apache dot org> > >=== Sponsoring Entity === >We request the Apache Incubator to sponsor this project. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
