Bryan, I remember the feature discussion, but some how missed NIFI-3380. That looks great, having those details in NARs will be useful for a lot of things. The growth of selection lists is the only potential concern, but that shouldn't be a problem unless we start keeping a lot of old versions in the release.
I agree that versioning doesn't need the registry, but the registries will be even more useful because of versioning. Thanks, Joe On Thu, Feb 9, 2017 at 10:18 AM, Bryan Bende <bbe...@gmail.com> wrote: > Joe S, > > Just to elaborate a little bit on what Matt mentioned about the > versioning of components/extensions... > > That capability is described in a feature proposal called "Multiple > Versions of the Same Extension" [1], and as Matt pointed out this work > is underway and being tracked under NIFI-3380. When that work is > completed then someone could deploy my-custom-processor-1.nar and > my-custom-processor-2.nar which have MyCustomProcessor v1 and v2 > respectively, and NiFi would be able to run these at the same. I > didn't originally mention this just because on its own it is not > dependent on a registry concept. Once that part is in place, it would > then allow NiFi to make use of an extension registry where someone > could see that my-custom-processor-3.nar was available and easily it > bring it into a running NiFi and start using it along side v1 and v2 > if desired. > > Oleg, > > I definitely agree that there are still some aspects of the extension > registry that need to be designed and thought through, and by no means > did I intend to dictate how it would be implemented. I definitely > agree with your idea of possibly having a set of strategies as a way > to achieve the best of worlds, and I was thinking of the same concept > for the flow registry in terms of having a pluggable back-end for the > persistence of flows so that anyone could build their own persistence > mechanism. I could definitely see this approach working well for the > community for both the flow and extension registries. > > Thanks, > > Bryan > > [1] https://cwiki.apache.org/confluence/display/NIFI/ > Multiple+Versions+of+the+Same+Extension > > On Thu, Feb 9, 2017 at 9:26 AM, Oleg Zhurakousky > <ozhurakou...@hortonworks.com> wrote: > > Joe > > > > Versioning is probably the second main driver for the externalization as > a concept. I believe in it so strongly that I would probably cast -1 if > versioning was not part of it ;) > > And some of the existing ready to use solutions already provide > versioning of artifacts (Binary, GitHub etc), which sets the expectations, > so not having i would be sad. . . so sad ;) > > > > Cheers > > Oleg > >> On Feb 9, 2017, at 8:45 AM, Joe Skora <jsk...@gmail.com> wrote: > >> > >> +1 as well. Glad to help if needed. > >> > >> Do you think this will include support for versioned Processors and > >> Controller Services, such that I could have SuperWidgetProcessor 1.0 and > >> SuperWidgetProcessor 1.5 on the same flow? > >> > >> On Thu, Feb 9, 2017 at 8:42 AM, Matt Gilman <matt.c.gil...@gmail.com> > wrote: > >> > >>> +1. I really like this idea. Should ease deployments between instances > and > >>> facilitate a better UX throughout the lifecycle of a dataflow. > >>> > >>> Matt > >>> > >>> On Thu, Feb 9, 2017 at 7:52 AM, Koji Kawamura <ijokaruma...@gmail.com> > >>> wrote: > >>> > >>>> Huge +1! I was excited to read the design documents :) > >>>> Agree with flow versioning at ProcessGroup level. > >>>> > >>>> I don't know if this is helpful, but here is an experimental project > >>>> of mine which tries to achieve the same goal, versioning ProcessGroup. > >>>> https://github.com/ijokarumawak/nifi-deploy-process-group > >>>> > >>>> It contains logics that will probably need to be implemented such as > >>>> checking remaining flow files in the queues around ProcessGroup, or > >>>> checking number of input/output ports from/to the ProcessGroup ... etc > >>>> > >>>> Hope that helps in some way, and I'd like to help make this come true! > >>>> > >>>> Thanks, > >>>> Koji > >>>> > >>>> On Thu, Feb 9, 2017 at 9:43 PM, Joe Gresock <jgres...@gmail.com> > wrote: > >>>>> +1, I've been waiting for this idea since NiFi went open source! > >>>>> > >>>>> On Thu, Feb 9, 2017 at 4:53 AM, Ricky Saltzer <ri...@cloudera.com> > >>>> wrote: > >>>>> > >>>>>> I'm a big +1 to this proposal. It would solve a huge burden that is > >>>> keeping > >>>>>> NARs up to date in environments where there's alot of teams that > share > >>>> NARs > >>>>>> but have separate NiFi deployments and repositories. > >>>>>> > >>>>>> On Feb 8, 2017 7:09 PM, "Peter Wicks (pwicks)" <pwi...@micron.com> > >>>> wrote: > >>>>>> > >>>>>>> I think a lot of us are facing the same challenges, and this sounds > >>>> like > >>>>>> a > >>>>>>> step in the right direction. > >>>>>>> I had actually started to dig into a Flow Configuration plugin that > >>>> would > >>>>>>> use Git branches to copy/sync flows between instances/environments, > >>>> and > >>>>>>> keep them versioned; hadn't gotten very far. > >>>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Jeremy Dyer [mailto:jdy...@gmail.com] > >>>>>>> Sent: Wednesday, February 08, 2017 3:54 PM > >>>>>>> To: dev@nifi.apache.org > >>>>>>> Subject: Re: [DISCUSS] Proposal for an Apache NiFi sub-project - > >>> NiFi > >>>>>>> Registry > >>>>>>> > >>>>>>> Bryan - I think this is a fantastic idea. I would also think this > >>>> would > >>>>>> be > >>>>>>> a good place to add a "device registry" as well. It makes much more > >>>> sense > >>>>>>> in my mind to have these efforts in sub projects outside of the > >>>>>> nifi/minifi > >>>>>>> core. > >>>>>>> > >>>>>>> On Wed, Feb 8, 2017 at 4:50 PM, Bryan Bende <bbe...@gmail.com> > >>> wrote: > >>>>>>> > >>>>>>>> NiFi Community, > >>>>>>>> > >>>>>>>> I'd like to initiate a discussion around creating a sub-project of > >>>>>>>> NiFi to encompass the registry capabilities outlined in several of > >>>> the > >>>>>>>> feature proposals on the Wiki [1]. A possible name for this > >>>>>>>> sub-project is simply "NiFi Registry". > >>>>>>>> > >>>>>>>> Currently there are two feature proposals that call for NiFi to > >>>>>>>> interact with an external registry: > >>>>>>>> > >>>>>>>> Configuration Management of Flows [2] - This feature proposal > >>> calls > >>>>>>>> for a flow registry where versioned flows can be published and > >>>>>>>> consumed, allowing flows to be easily migrated between > >>> environments > >>>> . > >>>>>>>> > >>>>>>>> Extension Registry [3] - This feature proposal calls for a place > >>> to > >>>>>>>> publish NARs containing extensions, allowing NiFi to decouple > >>> itself > >>>>>>>> from including all of the NARs in the main distribution, and > >>>> allowing > >>>>>>>> better discovery of available extensions. > >>>>>>>> > >>>>>>>> The idea would be to create a NiFi Registry sub-project, with > >>>>>>>> sub-modules for the various registries. These registries could > >>> then > >>>> be > >>>>>>>> packaged and distributed as a single artifact and run as a > >>>>>>>> complimentary application to NiFi and MiNiFi. NiFi would not > >>> require > >>>>>>>> the registry application, however, a given NiFi could be > >>> configured > >>>> to > >>>>>>>> know about one or more flow registries, or one or more extension > >>>>>>>> registries. > >>>>>>>> > >>>>>>>> Creating a sub-project would allow the registry code to evolve > >>>>>>>> independently of NiFi and be released on it's own timeline. In > >>>>>>>> addition, it would make tracking issues/work much clearer through > >>> a > >>>>>>>> separate JIRA. > >>>>>>>> > >>>>>>>> Please discuss and provide and thoughts or feedback. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> > >>>>>>>> Bryan > >>>>>>>> > >>>>>>>> [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+ > >>>>>>>> Feature+Proposals > >>>>>>>> [2] https://cwiki.apache.org/confluence/display/NIFI/ > >>>>>>>> Configuration+Management+of+Flows > >>>>>>>> [3] https://cwiki.apache.org/confluence/display/NIFI/ > >>>>>>>> Extension+Repositories+%28aka+Extension+Registry%29+for+ > >>>>>>>> Dynamically-loaded+Extensions > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> I know what it is to be in need, and I know what it is to have > >>> plenty. I > >>>>> have learned the secret of being content in any and every situation, > >>>>> whether well fed or hungry, whether living in plenty or in want. I > can > >>>> do > >>>>> all this through him who gives me strength. *-Philippians 4:12-13* > >>>> > >>> > > >
