Joe S, Just to elaborate a little bit on what Matt mentioned about the versioning of components/extensions...
That capability is described in a feature proposal called "Multiple Versions of the Same Extension" [1], and as Matt pointed out this work is underway and being tracked under NIFI-3380. When that work is completed then someone could deploy my-custom-processor-1.nar and my-custom-processor-2.nar which have MyCustomProcessor v1 and v2 respectively, and NiFi would be able to run these at the same. I didn't originally mention this just because on its own it is not dependent on a registry concept. Once that part is in place, it would then allow NiFi to make use of an extension registry where someone could see that my-custom-processor-3.nar was available and easily it bring it into a running NiFi and start using it along side v1 and v2 if desired. Oleg, I definitely agree that there are still some aspects of the extension registry that need to be designed and thought through, and by no means did I intend to dictate how it would be implemented. I definitely agree with your idea of possibly having a set of strategies as a way to achieve the best of worlds, and I was thinking of the same concept for the flow registry in terms of having a pluggable back-end for the persistence of flows so that anyone could build their own persistence mechanism. I could definitely see this approach working well for the community for both the flow and extension registries. Thanks, Bryan [1] https://cwiki.apache.org/confluence/display/NIFI/Multiple+Versions+of+the+Same+Extension On Thu, Feb 9, 2017 at 9:26 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> wrote: > Joe > > Versioning is probably the second main driver for the externalization as a > concept. I believe in it so strongly that I would probably cast -1 if > versioning was not part of it ;) > And some of the existing ready to use solutions already provide versioning of > artifacts (Binary, GitHub etc), which sets the expectations, so not having i > would be sad. . . so sad ;) > > Cheers > Oleg >> On Feb 9, 2017, at 8:45 AM, Joe Skora <jsk...@gmail.com> wrote: >> >> +1 as well. Glad to help if needed. >> >> Do you think this will include support for versioned Processors and >> Controller Services, such that I could have SuperWidgetProcessor 1.0 and >> SuperWidgetProcessor 1.5 on the same flow? >> >> On Thu, Feb 9, 2017 at 8:42 AM, Matt Gilman <matt.c.gil...@gmail.com> wrote: >> >>> +1. I really like this idea. Should ease deployments between instances and >>> facilitate a better UX throughout the lifecycle of a dataflow. >>> >>> Matt >>> >>> On Thu, Feb 9, 2017 at 7:52 AM, Koji Kawamura <ijokaruma...@gmail.com> >>> wrote: >>> >>>> Huge +1! I was excited to read the design documents :) >>>> Agree with flow versioning at ProcessGroup level. >>>> >>>> I don't know if this is helpful, but here is an experimental project >>>> of mine which tries to achieve the same goal, versioning ProcessGroup. >>>> https://github.com/ijokarumawak/nifi-deploy-process-group >>>> >>>> It contains logics that will probably need to be implemented such as >>>> checking remaining flow files in the queues around ProcessGroup, or >>>> checking number of input/output ports from/to the ProcessGroup ... etc >>>> >>>> Hope that helps in some way, and I'd like to help make this come true! >>>> >>>> Thanks, >>>> Koji >>>> >>>> On Thu, Feb 9, 2017 at 9:43 PM, Joe Gresock <jgres...@gmail.com> wrote: >>>>> +1, I've been waiting for this idea since NiFi went open source! >>>>> >>>>> On Thu, Feb 9, 2017 at 4:53 AM, Ricky Saltzer <ri...@cloudera.com> >>>> wrote: >>>>> >>>>>> I'm a big +1 to this proposal. It would solve a huge burden that is >>>> keeping >>>>>> NARs up to date in environments where there's alot of teams that share >>>> NARs >>>>>> but have separate NiFi deployments and repositories. >>>>>> >>>>>> On Feb 8, 2017 7:09 PM, "Peter Wicks (pwicks)" <pwi...@micron.com> >>>> wrote: >>>>>> >>>>>>> I think a lot of us are facing the same challenges, and this sounds >>>> like >>>>>> a >>>>>>> step in the right direction. >>>>>>> I had actually started to dig into a Flow Configuration plugin that >>>> would >>>>>>> use Git branches to copy/sync flows between instances/environments, >>>> and >>>>>>> keep them versioned; hadn't gotten very far. >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Jeremy Dyer [mailto:jdy...@gmail.com] >>>>>>> Sent: Wednesday, February 08, 2017 3:54 PM >>>>>>> To: dev@nifi.apache.org >>>>>>> Subject: Re: [DISCUSS] Proposal for an Apache NiFi sub-project - >>> NiFi >>>>>>> Registry >>>>>>> >>>>>>> Bryan - I think this is a fantastic idea. I would also think this >>>> would >>>>>> be >>>>>>> a good place to add a "device registry" as well. It makes much more >>>> sense >>>>>>> in my mind to have these efforts in sub projects outside of the >>>>>> nifi/minifi >>>>>>> core. >>>>>>> >>>>>>> On Wed, Feb 8, 2017 at 4:50 PM, Bryan Bende <bbe...@gmail.com> >>> wrote: >>>>>>> >>>>>>>> NiFi Community, >>>>>>>> >>>>>>>> I'd like to initiate a discussion around creating a sub-project of >>>>>>>> NiFi to encompass the registry capabilities outlined in several of >>>> the >>>>>>>> feature proposals on the Wiki [1]. A possible name for this >>>>>>>> sub-project is simply "NiFi Registry". >>>>>>>> >>>>>>>> Currently there are two feature proposals that call for NiFi to >>>>>>>> interact with an external registry: >>>>>>>> >>>>>>>> Configuration Management of Flows [2] - This feature proposal >>> calls >>>>>>>> for a flow registry where versioned flows can be published and >>>>>>>> consumed, allowing flows to be easily migrated between >>> environments >>>> . >>>>>>>> >>>>>>>> Extension Registry [3] - This feature proposal calls for a place >>> to >>>>>>>> publish NARs containing extensions, allowing NiFi to decouple >>> itself >>>>>>>> from including all of the NARs in the main distribution, and >>>> allowing >>>>>>>> better discovery of available extensions. >>>>>>>> >>>>>>>> The idea would be to create a NiFi Registry sub-project, with >>>>>>>> sub-modules for the various registries. These registries could >>> then >>>> be >>>>>>>> packaged and distributed as a single artifact and run as a >>>>>>>> complimentary application to NiFi and MiNiFi. NiFi would not >>> require >>>>>>>> the registry application, however, a given NiFi could be >>> configured >>>> to >>>>>>>> know about one or more flow registries, or one or more extension >>>>>>>> registries. >>>>>>>> >>>>>>>> Creating a sub-project would allow the registry code to evolve >>>>>>>> independently of NiFi and be released on it's own timeline. In >>>>>>>> addition, it would make tracking issues/work much clearer through >>> a >>>>>>>> separate JIRA. >>>>>>>> >>>>>>>> Please discuss and provide and thoughts or feedback. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Bryan >>>>>>>> >>>>>>>> [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+ >>>>>>>> Feature+Proposals >>>>>>>> [2] https://cwiki.apache.org/confluence/display/NIFI/ >>>>>>>> Configuration+Management+of+Flows >>>>>>>> [3] https://cwiki.apache.org/confluence/display/NIFI/ >>>>>>>> Extension+Repositories+%28aka+Extension+Registry%29+for+ >>>>>>>> Dynamically-loaded+Extensions >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> I know what it is to be in need, and I know what it is to have >>> plenty. I >>>>> have learned the secret of being content in any and every situation, >>>>> whether well fed or hungry, whether living in plenty or in want. I can >>>> do >>>>> all this through him who gives me strength. *-Philippians 4:12-13* >>>> >>> >
