Personally I would stick to a single proto with adapters to systems such as Kafka. IMHO it would be very hard to maintain support for several given that they all had different design considerations.
Sent from my iPhone > On Jan 10, 2016, at 08:10, Joe Witt <joe.w...@gmail.com> wrote: > > Joshua, > > Definitely agree with the bullet points you listed. As for the > differentiation it is not, in my view, so much a differentiation > concern but a 'there should be a better dataflow management and > provenance story' concern. > > My personal view is with MiNiFi, just as in NiFi, we should not > promote any single protocol but rather support those that clearly fit > its purposes and help it easily integrate with other systems. Those > you mentioned MQTT, AMQP, etc.. make good sense as strong candidates. > So too does delivery to Kafka and use of NiFi's site to site. > > Thanks > Joe > >> On Sun, Jan 10, 2016 at 9:02 AM, Joshua Davis <jda...@hortonworks.com> wrote: >> Some thoughts as well. >> >> - This will really differentiate NIFI from other open source projects that >> have a central model similar to Nifi but are too large to place on small >> devices. >> - This will enable anyone creating a project with devices whether they are >> refrigerators, security systems, power generation systems, or turbines to >> have real time data accurately and securely to push data to a system for >> processing. >> - Building the agents in other environments outside of the JVM will expand >> the reach of NIFI far beyond where it is today. >> >> Some questions: >> >> Will MiNifi be protocol based or API based? I am asking this question >> because if we make NIFI protocol based, then others can be free to create >> new agent implementations. >> >> Are we going to use an existing protocol such as MQTT, AMQP, or STOMP? >> >> Joshua Davis >> >> >> >> >> >> >>> On 1/9/16, 7:29 PM, "Joe Witt" <joew...@apache.org> wrote: >>> >>> NiFi Community, >>> >>> I'd like to initiate discussion around a proposal to create our first >>> sub-project of NiFi. A possible name for it is "MiNiFi" a sort of >>> play on Mini-NiFi. >>> >>> The idea is to provide a complementary data collection agent to NiFi's >>> current approach of dataflow management. As noted in our ASF TLP >>> resolution NiFi is to provide "an automated and durable data broker >>> between systems providing interactive command and control and detailed >>> chain of custody for data." MiNiFi would be consistent with that >>> scope with a specific focus on the first-mile challenge so common in >>> dataflow. >>> >>> Specific goals of MiNiFi would be to provide a small, lightweight, >>> centrally managed agent that natively generates data provenance and >>> seamlessly integrates with NiFi for follow-on dataflow management and >>> maintenance of the chain of custody provided by the powerful data >>> provenance features of NiFi. >>> >>> MiNiFi should be designed to operate directly on or adjacent to the >>> source sensor, system, server generating the events as a resource >>> sensitive tenant. There are numerous agent models in existence today >>> but they do not offer the command and control or provenance that is so >>> important to the philosophy and scope of NiFi. >>> >>> These agents would necessarily have a different interactive command >>> and control model than NiFi as you'd not expect consistent behavior, >>> capability, or accessibility of all instances of the agents at any >>> given time. >>> >>> Multiple implementations of MiNiFi are envisioned including those that >>> operate on the JVM and those that do not. >>> >>> As the discussion advances we can put together wiki pages, concept >>> diagrams, and requirements to help better articulate how this might >>> evolve. We should also discuss the mechanics of how this might work >>> in terms of infrastructure, code repository, and more. >>> >>> Thanks >>> Joe >
