I have thought about this in the past, too. Here's a scenario where I could never really lay down an approach I was happy with.
Consider that a NiFi user searches the NiFi registry and finds the PutMongo processor. Registry knows the PutMongo processor is in the nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a dependency on a controller service interface in nifi-mongodb-client-service-api-nar. Great, the user can then download and install those two nars. How would we then suggest that the user also needs a MongoDBClientService controller service implementation, such as that in the nifi-mongodb-services-nar? I'm not looking for an answer now, of course, but I just wanted to feed the discussion. Thanks, -- Mike On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bbe...@gmail.com> wrote: > Mark, > > I think there are a couple of ways that could be solved, but > ultimately it would be up to how the users choose to setup/manage the > registry, or registries. > > The NiFi Registry security model is based around permissions to > buckets (read/write), and all versioned items belong to a bucket. So > you could think of each bucket as a mini extension repository for > which you can control access to specific users and groups, so there > could be a bucket of extensions for each of the NiFi instances in your > example. There can also be multiple registry instances registered with > a given NiFi so extensions can be pulled from multiple registries. > > The extensions an instance needs is based on the flows it is running, > the flows already have specific bundle coordinates for every component > in the flow. You can think of it similar to Maven where you declared a > dependency on library foo and the build goes out and gets it for you, > in this case it is a flow that declares a dependency on a bundle. > > Mike, > > Bundles would need to be uploaded to NiFi Registry (to a specific > bucket) as part of some TBD release process. At a minimum I was > envisioning NiFi CLI commands that can be pointed to a file or a > directory and upload the given bundles to registry. There could be > other options as well, possibly through a Maven plugin to release > directly into registry, or possibly to have a type of extension in > NiFi Registry that actually points to an external location, i.e. all > the NARs that end up in Maven central could somehow be imported into > NiFi Registry, but with pointers back to the content which actually > comes from Maven central. Lot of things to figure out here. > > -Bryan > On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > Group selection based on tag names for bundles could probably do that. > > Meaning it could be a sorting/filtering mechanism in the NiFi/Registry > > interface perhaps. Will be good to consider that UX as that > > progresses. > > > > As far as the different environments NiFi instances would certainly be > > able to load only referenced Nars for versioned flows so you'll get > > the optimal set (at runtime) automatically. Very powerful. > > On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <mark.o.b...@gmail.com> wrote: > > > > > > Joe, > > > > > > I envision the Registry being able to provide a subset of NARs > required for > > > a specific NiFi instance. The user may have a relatively small set of > NARs > > > required for a NiFi used for basic routing/distribution, and a > different > > > more extensive set of NARs required for a more robust NiFi instance > which > > > performs various forms of processing/transformations. The grouping I am > > > describing would be a way to select multiple NARs required for a > specific > > > NiFi instance. > > > > > > Expanding the scenario a little farther, suppose an integration/test > > > environment defines the group. Then, the production environment can > use the > > > group definition to pull (or ensure it possesses) only the relevant > NARs > > > necessary. > > > > > > -Mark > > > > > > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > Mark > > > > > > > > Can you describe your use case from the user perspective both for the > > > > entity that would upload the items and demarcate them as a group as > > > > well as the user that would consume those bundles? > > > > > > > > I ask because the point here is that nars are themselves a 'group' in > > > > that they are a logical/contained grouping of extensions. These can > > > > have relationships to other nars as we know. And flows are designed > > > > against specific components that come from certain nars for which we > > > > know the precise coordinates. When importing flows that depend on > > > > these the system will be able to automatically acquire all that it > > > > needs. > > > > > > > > Thanks > > > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <mark.o.b...@gmail.com> > wrote: > > > > > > > > > > I would like to see a "group" capability in the Registry for NAR > bundles. > > > > > Then, users can create their own customized grouping of relevant > NARs. > > > > > Managing bundles one at a time will become tedious. > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.w...@gmail.com> > wrote: > > > > > > > > > > > Sivaprasanna - yes absolutely. That is the core point I think of > > > > > > Bryan's first bullet under the 'NiFi' section. > > > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna < > > > > sivaprasanna...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > One quick question. With the extension registry, my > understanding is > > > > that > > > > > > > we would try to reduce the overall NiFi size by offloading > certain > > > > > > existing > > > > > > > NAR bundles to the extension registry. So are we expecting the > > > > extension > > > > > > > registry to also come up with the ability/mechanism that lets > an > > > > user to > > > > > > > download these bundles ? > > > > > > > > > > > > > > - > > > > > > > Sivaprasanna > > > > > > > > > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.w...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > Bryan > > > > > > > > > > > > > > > > Very exciting to see this under way!!! We desperately have > to get > > > > our > > > > > > > > core nifi build size way down and make it far easier for > people to > > > > > > > > contribute new processors. We have a long line of > extensions that > > > > > > > > could be really useful/valuable and this will help unlock > that > > > > while > > > > > > > > improving the user experience tremendously. > > > > > > > > > > > > > > > > For the doc/concerns noted above we should also add/mention > that > > > > the > > > > > > > > relationship between nars (dependencies between nars for > > > > > > > > apis/controller services/parent nars/etc..) we want to have > a way > > > > to > > > > > > > > manage/show that or a user experience for it. Maybe not a > day-1 > > > > thing > > > > > > > > but important to call out. > > > > > > > > > > > > > > > > Thanks! > > > > > > > > Joe > > > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende < > bbe...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > All, > > > > > > > > > > > > > > > > > > We've needed the elusive extension registry for quite some > time > > > > now, > > > > > > > > > and with NiFi Registry I think we are in a good place to > make > > > > some > > > > > > > > > progress in this area. > > > > > > > > > > > > > > > > > > I've started looking into adding "extension bundles" to > NiFi > > > > Registry > > > > > > > > > as the next type of versioned item, along side the existing > > > > versioned > > > > > > > > > flows, and I wanted to take a minute to outline how that > approach > > > > > > > > > could work before getting too far into it. > > > > > > > > > > > > > > > > > > Also, I'd like to focus this discussion on the design and > > > > > > > > > functionality of the extension registry, and not on how the > > > > community > > > > > > > > > is going to use it. Topics like hosting a central registry, > > > > changing > > > > > > > > > the build process, restructuring the git repo, releasing > NARs, > > > > etc, > > > > > > > > > are all important topics, but first we need an extension > registry > > > > > > > > > before we can do any of that :) > > > > > > > > > > > > > > > > > > Here is a high-level description of what needs to be > done... > > > > > > > > > > > > > > > > > > NiFi Registry > > > > > > > > > > > > > > > > > > - Add a new type of item called an extension bundle, where > each > > > > > > bundle > > > > > > > > > can contain one ore extensions or APIs > > > > > > > > > > > > > > > > > > - Support bundles for traditional NiFi (aka NARs) and also > > > > bundles > > > > > > for > > > > > > > > > MiNiFi CPP > > > > > > > > > > > > > > > > > > - Ability to upload the binary artifact for a bundle and > extract > > > > the > > > > > > > > > metadata about the bundle, and metadata about the > extensions > > > > > > contained > > > > > > > > > in the bundle (more on this later) > > > > > > > > > > > > > > > > > > - Provide a pluggable storage provider for saving the > content of > > > > each > > > > > > > > > extension bundle so that we can have different > implementations > > > > like > > > > > > > > > local fileysystem, S3, and other object stores > > > > > > > > > > > > > > > > > > - Provide a REST API for listing and retrieving available > > > > bundles, > > > > > > > > > integrate this into the registry Java client and NiFi CLI > > > > > > > > > > > > > > > > > > NAR Maven Plugin > > > > > > > > > > > > > > > > > > - Generate a descriptor for each component in the NAR > which will > > > > > > > > > provide information like the description, tags, restricted > or > > > > not, > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > - These descriptors will be parsed by NiFi Registry when a > NAR is > > > > > > > > > being uploaded so that NiFi Registry will know about the > > > > extensions > > > > > > > > > contained with in the NAR > > > > > > > > > > > > > > > > > > NiFi > > > > > > > > > > > > > > > > > > - Provide some type of extension manager experience where > users > > > > can > > > > > > > > > search, browse, & install bundles that are available in > any of > > > > the > > > > > > > > > registry clients that have been defined > > > > > > > > > > > > > > > > > > - Introduce a new security policy to control which users > are > > > > allowed > > > > > > > > > to access the extension manager > > > > > > > > > > > > > > > > > > - Installing a bundle should load the NAR and make the > extensions > > > > > > > > > available leveraging the recent work done to auto-load new > NARs > > > > > > > > > > > > > > > > > > - Importing versioned flows from registry should provide > an easy > > > > way > > > > > > > > > to install bundles that are required by the flow, but > missing > > > > from > > > > > > the > > > > > > > > > local NiFi instance > > > > > > > > > > > > > > > > > > > > > > > > > > > If anyone has any thoughts or concerns about this approach, > > > > please > > > > > > let > > > > > > > > me know. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Bryan > > > > > > > > > > > > > > > > > > >
