I think we should be able to kick 0.4.0 very soon, just a couple of items to get in. Assuming those get reviewed and merged soon we could possibly kick out an RC early next week. I can plan to be RM unless anyone else is interested.
On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <[email protected]> wrote: > > I think Kevin summarized it nicely. Auto-pulling the extensions with a > versioned flow is definitely the ultimate vision, but we need to take > steps to get there. > > The process of automatically bringing in the extensions is something > that would be implemented on the NiFi side and will require a bit of > thought and design around the user experience. > > Before we can do any of that, we first need somewhere to store and > retrieve versioned extensions, which is the work that is underway on > the NiFi Registry side. > > On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <[email protected]> wrote: > > > > Hi Ryan, > > > > Bryan can speak more to this than I can, but, yes, providing the > > experience you describe is the eventual goal! Aside from the NiFi > > Extension Registry support though (the initial bits of which is what > > Bryan is referring to in 0.4.0), getting the full, streamlined > > experience in NiFi will require changes to NiFi as well, for example > > in the logic to import a flow as it now also has to reach back to > > Registry to pull extensions referenced by the flow during import. So > > the changes to NiFi Registry to support hosting NiFi Extensions is > > only half of the puzzle. This is why Bryan also mentioned it might be > > worth holding back 0.4.0 (or releasing it, but marking the new > > extension stuff as experimental), as it is likely the current > > implementation and interfaces will need to change once the NiFi work > > begins. > > > > Cheers, > > Kevin > > > > On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson > > <[email protected]> wrote: > > > > > > We'd like to connect a NiFi to a Registry, Pull down a Process Group, and > > > get any NARs required for that without having to manually install them on > > > the NiFi. > > > > > > That's what I'm understanding the Extension part of the 0.4.0 release to > > > be, am I on point, or way off? > > > > > > Thanks, > > > Ryan > > > > > > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <[email protected]> > > > wrote: > > > > > > > Hey Pierre, > > > > > > > > You can version the metadata db in the root dir of the git repo with > > > > event > > > > hooks. > > > > > > > > With an h2 jdbc url of > > > > > > > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE' > > > > > > > > You can connect to the db in a separate process, we're using that with a > > > > shell event hook to dump the db to sql script when mutations happen (now > > > > tenant events too :) ), storing that in the git repo using > > > > org.h2.tools.Script, and committing it. Then we can restore from that > > > > sql > > > > script on startup before calling the stock startup script. > > > > > > > > Side benefit is you can see metadata state changes in the git history > > > > and > > > > manually inspect the sql. > > > > > > > > > > > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard < > > > > [email protected]> > > > > wrote: > > > > > > > > > Bryan, > > > > > > > > > > Thanks for the details. On my side, the main feature I (and others I > > > > > suppose) am looking for is the ability to rebuild the metadata DB from > > > > the > > > > > Git repo. But to be honest, this is not a blocker at all: I can live > > > > with a > > > > > custom Docker image containing this feature. > > > > > > > > > > So, from my point of view, I don't have any issue with #2 but maybe #1 > > > > > would also give access to experimental features and have interesting > > > > > feedback from the community. But we would, indeed, need to make it > > > > crystal > > > > > clear that the APIs could change in ulterior release(s). > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <[email protected]> a écrit : > > > > > > > > > > > Ryan, > > > > > > > > > > > > The short answer to your question is no, there isn't actually > > > > > > anything > > > > > > new in the UI. The registry UI is setup in a generic way to list > > > > > > "versioned items", so versioned extension bundles will just > > > > > > automatically show up in the list with versioned flows. > > > > > > > > > > > > Let me put together a wiki page that outlines all of the moving > > > > > > parts > > > > > > for the extension registry work and where we are currently at. I'll > > > > > > send the link on this thread when I have something. > > > > > > > > > > > > > > > > > > For Pierre and others, > > > > > > > > > > > > I was thinking more about this discussion of releasing 0.4.0, and I > > > > > > think there are two of approaches we could take... > > > > > > > > > > > > 1) If there is a desire for the community to release 0.4.0 sooner > > > > > > rather than later, then we can mark all of the extension bundle > > > > > > related APIs as experimental, which would then give us the freedom > > > > > > to > > > > > > continue evolving them until we have the full extension registry > > > > > > experience, and likely mark them as stable in 0.5.0, or whichever > > > > > > future release we decide. The reason for this is that there will > > > > > > likely be API changes needed while building out the NiFi side of > > > > > > things, and I don't want us to get stuck in a spot where we can't > > > > > > change some API since it has been released, and so far the NiFi side > > > > > > of the work hasn't been started yet. > > > > > > > > > > > > 2) Wait until more of the extension registry work is finalized and > > > > > > use > > > > > > that as the driver of when to release 0.4.0, with the obvious > > > > > > downside > > > > > > that it may take a bit longer to get a release out which then holds > > > > > > up > > > > > > anything else that is not related to extension registry. > > > > > > > > > > > > So I guess we just have to decide the driving factor for releasing > > > > > > 0.4.0... If its the non-extension related features/fixes, then #1 is > > > > > > probably the best approach. If its the extension related stuff, then > > > > > > I'd vote for #2 since it isn't really ready yet. > > > > > > > > > > > > -Bryan > > > > > > > > > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Bryan, > > > > > > > Is there a new GUI part of the registry for the extension > > > > > > > registry > > > > > > work > > > > > > > in master? I compiled 0.4.0-SNAPSHOT and didn't see anything > > > > > > immediately. > > > > > > > You mentioned docs and sys admin stuff... Could you point me in > > > > > > > the > > > > > > > quick-start path if I wanted to try to kick-the-tires a little on > > > > this? > > > > > > > > > > > > > > Thanks, > > > > > > > Ryan > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > Mike, > > > > > > > > > > > > > > > > All record-oriented components are extensions and thus can make > > > > > > > > use > > > > > of > > > > > > > > versioned components (i.e. nothing related to records is baked > > > > > > > > into > > > > > > > > the core framework of NiFi). > > > > > > > > > > > > > > > > The record API is essentially the module named > > > > > > > > nifi-record-serialization-services-api which at runtime is > > > > > > > > included > > > > > in > > > > > > > > nifi-standard-services-api-nar. Any record oriented processors > > > > > > > > you > > > > > > > > build are dependent on a specific version of > > > > > > > > nifi-standard-services-api-nar since that is where the Java > > > > interface > > > > > > > > will be that your processor was compiled against. > > > > > > > > > > > > > > > > Right now, even without extension registry, you could deploy > > > > multiple > > > > > > > > versions of nifi-standard-services-api-nar to a NiFi instance, > > > > > > > > and > > > > > > > > then deploy multiple versions of your record processing NAR that > > > > > > > > correspond to the versions of your API NAR. > > > > > > > > > > > > > > > > Going forward with extension registry, we probably want to > > > > > > > > consider > > > > > > > > breaking up nifi-standard-services-api-nar into smaller API > > > > > > > > NARs, > > > > as > > > > > > > > well as nifi-standard-bundle into smaller processor bundles. For > > > > > > > > example, there could be a record API NAR and a record processors > > > > NAR, > > > > > > > > that would both be split out of from the standard NARs. > > > > > > > > > > > > > > > > -Bryan > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Great news about the extension registry! As we get closer to > > > > > > > > > that > > > > > > being > > > > > > > > > ready, I'd like to add in some discussion about versioning the > > > > > > record API > > > > > > > > > separately. A lot of the custom processors we build now are > > > > Record > > > > > > > > > API-based, and it would be great to be able to decouple them > > > > > > > > > from > > > > > one > > > > > > > > > specific release of NiFi. > > > > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Pierre, > > > > > > > > > > > > > > > > > > > > I think we are definitely close to an 0.4.0 release. A major > > > > > chunk > > > > > > of > > > > > > > > > > extension registry work has already landed in master, and I > > > > still > > > > > > have > > > > > > > > one > > > > > > > > > > other jira that I almost have ready and wanted to include, > > > > > > NIFIREG-233 > > > > > > > > for > > > > > > > > > > generating extensions docs. Plus we probably need to make a > > > > > > > > > > few > > > > > > > > > > additions/updates to the admin guide. > > > > > > > > > > > > > > > > > > > > -Bryan > > > > > > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard < > > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > All, > > > > > > > > > > > > > > > > > > > > > > Some really nice features have been included since the > > > > > > > > > > > NiFi > > > > > > Registry > > > > > > > > > > 0.3.0 > > > > > > > > > > > release and I'm wondering if it'd be a good time to > > > > > > > > > > > consider > > > > a > > > > > > 0.4.0 > > > > > > > > > > > release. > > > > > > > > > > > > > > > > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened > > > > > > > > > > > pull > > > > > > > > requests, > > > > > > > > > > plus > > > > > > > > > > > interesting discussions / JIRAS on the mailing lists. > > > > > > > > > > > > > > > > > > > > > > Are there any reasons to hold off on a 0.4.0 release? Are > > > > > there > > > > > > > > > > particular > > > > > > > > > > > JIRAs that the community considers necessary to have as > > > > > > > > > > > part > > > > of > > > > > > the > > > > > > > > > > > release? > > > > > > > > > > > > > > > > > > > > > > If not, happy to give it a try at performing RM duties! > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Sent from Gmail Mobile > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
