Excellent Bryan. Looking forward to this release. Thanks for taking the initiative here.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 16, 2019, at 12:34 PM, Bryan Bende <[email protected]> wrote: > > The last few items tagged for 0.4.0 have been merged so we should be > good to put out an RC for 0.4.0. > > I'll start working on getting everything together and try to get > something out soon. > > On Fri, May 10, 2019 at 9:00 AM Bryan Bende <[email protected]> wrote: >> >> I think we should be able to kick 0.4.0 very soon, just a couple of >> items to get in. Assuming those get reviewed and merged soon we could >> possibly kick out an RC early next week. I can plan to be RM unless >> anyone else is interested. >> >> On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <[email protected]> wrote: >>> >>> I think Kevin summarized it nicely. Auto-pulling the extensions with a >>> versioned flow is definitely the ultimate vision, but we need to take >>> steps to get there. >>> >>> The process of automatically bringing in the extensions is something >>> that would be implemented on the NiFi side and will require a bit of >>> thought and design around the user experience. >>> >>> Before we can do any of that, we first need somewhere to store and >>> retrieve versioned extensions, which is the work that is underway on >>> the NiFi Registry side. >>> >>> On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <[email protected]> wrote: >>>> >>>> Hi Ryan, >>>> >>>> Bryan can speak more to this than I can, but, yes, providing the >>>> experience you describe is the eventual goal! Aside from the NiFi >>>> Extension Registry support though (the initial bits of which is what >>>> Bryan is referring to in 0.4.0), getting the full, streamlined >>>> experience in NiFi will require changes to NiFi as well, for example >>>> in the logic to import a flow as it now also has to reach back to >>>> Registry to pull extensions referenced by the flow during import. So >>>> the changes to NiFi Registry to support hosting NiFi Extensions is >>>> only half of the puzzle. This is why Bryan also mentioned it might be >>>> worth holding back 0.4.0 (or releasing it, but marking the new >>>> extension stuff as experimental), as it is likely the current >>>> implementation and interfaces will need to change once the NiFi work >>>> begins. >>>> >>>> Cheers, >>>> Kevin >>>> >>>> On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson >>>> <[email protected]> wrote: >>>>> >>>>> We'd like to connect a NiFi to a Registry, Pull down a Process Group, and >>>>> get any NARs required for that without having to manually install them on >>>>> the NiFi. >>>>> >>>>> That's what I'm understanding the Extension part of the 0.4.0 release to >>>>> be, am I on point, or way off? >>>>> >>>>> Thanks, >>>>> Ryan >>>>> >>>>> On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey Pierre, >>>>>> >>>>>> You can version the metadata db in the root dir of the git repo with >>>>>> event >>>>>> hooks. >>>>>> >>>>>> With an h2 jdbc url of >>>>>> >>>>>> 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE' >>>>>> >>>>>> You can connect to the db in a separate process, we're using that with a >>>>>> shell event hook to dump the db to sql script when mutations happen (now >>>>>> tenant events too :) ), storing that in the git repo using >>>>>> org.h2.tools.Script, and committing it. Then we can restore from that >>>>>> sql >>>>>> script on startup before calling the stock startup script. >>>>>> >>>>>> Side benefit is you can see metadata state changes in the git history and >>>>>> manually inspect the sql. >>>>>> >>>>>> >>>>>> On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard < >>>>>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> Bryan, >>>>>>> >>>>>>> Thanks for the details. On my side, the main feature I (and others I >>>>>>> suppose) am looking for is the ability to rebuild the metadata DB from >>>>>> the >>>>>>> Git repo. But to be honest, this is not a blocker at all: I can live >>>>>> with a >>>>>>> custom Docker image containing this feature. >>>>>>> >>>>>>> So, from my point of view, I don't have any issue with #2 but maybe #1 >>>>>>> would also give access to experimental features and have interesting >>>>>>> feedback from the community. But we would, indeed, need to make it >>>>>> crystal >>>>>>> clear that the APIs could change in ulterior release(s). >>>>>>> >>>>>>> Pierre >>>>>>> >>>>>>> >>>>>>> >>>>>>> Le mer. 27 mars 2019 à 18:11, Bryan Bende <[email protected]> a écrit : >>>>>>> >>>>>>>> Ryan, >>>>>>>> >>>>>>>> The short answer to your question is no, there isn't actually anything >>>>>>>> new in the UI. The registry UI is setup in a generic way to list >>>>>>>> "versioned items", so versioned extension bundles will just >>>>>>>> automatically show up in the list with versioned flows. >>>>>>>> >>>>>>>> Let me put together a wiki page that outlines all of the moving parts >>>>>>>> for the extension registry work and where we are currently at. I'll >>>>>>>> send the link on this thread when I have something. >>>>>>>> >>>>>>>> >>>>>>>> For Pierre and others, >>>>>>>> >>>>>>>> I was thinking more about this discussion of releasing 0.4.0, and I >>>>>>>> think there are two of approaches we could take... >>>>>>>> >>>>>>>> 1) If there is a desire for the community to release 0.4.0 sooner >>>>>>>> rather than later, then we can mark all of the extension bundle >>>>>>>> related APIs as experimental, which would then give us the freedom to >>>>>>>> continue evolving them until we have the full extension registry >>>>>>>> experience, and likely mark them as stable in 0.5.0, or whichever >>>>>>>> future release we decide. The reason for this is that there will >>>>>>>> likely be API changes needed while building out the NiFi side of >>>>>>>> things, and I don't want us to get stuck in a spot where we can't >>>>>>>> change some API since it has been released, and so far the NiFi side >>>>>>>> of the work hasn't been started yet. >>>>>>>> >>>>>>>> 2) Wait until more of the extension registry work is finalized and use >>>>>>>> that as the driver of when to release 0.4.0, with the obvious downside >>>>>>>> that it may take a bit longer to get a release out which then holds up >>>>>>>> anything else that is not related to extension registry. >>>>>>>> >>>>>>>> So I guess we just have to decide the driving factor for releasing >>>>>>>> 0.4.0... If its the non-extension related features/fixes, then #1 is >>>>>>>> probably the best approach. If its the extension related stuff, then >>>>>>>> I'd vote for #2 since it isn't really ready yet. >>>>>>>> >>>>>>>> -Bryan >>>>>>>> >>>>>>>> On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson >>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>> Bryan, >>>>>>>>> Is there a new GUI part of the registry for the extension registry >>>>>>>> work >>>>>>>>> in master? I compiled 0.4.0-SNAPSHOT and didn't see anything >>>>>>>> immediately. >>>>>>>>> You mentioned docs and sys admin stuff... Could you point me in the >>>>>>>>> quick-start path if I wanted to try to kick-the-tires a little on >>>>>> this? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ryan >>>>>>>>> >>>>>>>>> On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <[email protected]> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> Mike, >>>>>>>>>> >>>>>>>>>> All record-oriented components are extensions and thus can make use >>>>>>> of >>>>>>>>>> versioned components (i.e. nothing related to records is baked into >>>>>>>>>> the core framework of NiFi). >>>>>>>>>> >>>>>>>>>> The record API is essentially the module named >>>>>>>>>> nifi-record-serialization-services-api which at runtime is included >>>>>>> in >>>>>>>>>> nifi-standard-services-api-nar. Any record oriented processors you >>>>>>>>>> build are dependent on a specific version of >>>>>>>>>> nifi-standard-services-api-nar since that is where the Java >>>>>> interface >>>>>>>>>> will be that your processor was compiled against. >>>>>>>>>> >>>>>>>>>> Right now, even without extension registry, you could deploy >>>>>> multiple >>>>>>>>>> versions of nifi-standard-services-api-nar to a NiFi instance, and >>>>>>>>>> then deploy multiple versions of your record processing NAR that >>>>>>>>>> correspond to the versions of your API NAR. >>>>>>>>>> >>>>>>>>>> Going forward with extension registry, we probably want to consider >>>>>>>>>> breaking up nifi-standard-services-api-nar into smaller API NARs, >>>>>> as >>>>>>>>>> well as nifi-standard-bundle into smaller processor bundles. For >>>>>>>>>> example, there could be a record API NAR and a record processors >>>>>> NAR, >>>>>>>>>> that would both be split out of from the standard NARs. >>>>>>>>>> >>>>>>>>>> -Bryan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen < >>>>>> [email protected] >>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Great news about the extension registry! As we get closer to that >>>>>>>> being >>>>>>>>>>> ready, I'd like to add in some discussion about versioning the >>>>>>>> record API >>>>>>>>>>> separately. A lot of the custom processors we build now are >>>>>> Record >>>>>>>>>>> API-based, and it would be great to be able to decouple them from >>>>>>> one >>>>>>>>>>> specific release of NiFi. >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <[email protected]> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Pierre, >>>>>>>>>>>> >>>>>>>>>>>> I think we are definitely close to an 0.4.0 release. A major >>>>>>> chunk >>>>>>>> of >>>>>>>>>>>> extension registry work has already landed in master, and I >>>>>> still >>>>>>>> have >>>>>>>>>> one >>>>>>>>>>>> other jira that I almost have ready and wanted to include, >>>>>>>> NIFIREG-233 >>>>>>>>>> for >>>>>>>>>>>> generating extensions docs. Plus we probably need to make a few >>>>>>>>>>>> additions/updates to the admin guide. >>>>>>>>>>>> >>>>>>>>>>>> -Bryan >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> All, >>>>>>>>>>>>> >>>>>>>>>>>>> Some really nice features have been included since the NiFi >>>>>>>> Registry >>>>>>>>>>>> 0.3.0 >>>>>>>>>>>>> release and I'm wondering if it'd be a good time to consider >>>>>> a >>>>>>>> 0.4.0 >>>>>>>>>>>>> release. >>>>>>>>>>>>> >>>>>>>>>>>>> There is a JIRA tagged for 0.4.0 and there are 2 opened pull >>>>>>>>>> requests, >>>>>>>>>>>> plus >>>>>>>>>>>>> interesting discussions / JIRAS on the mailing lists. >>>>>>>>>>>>> >>>>>>>>>>>>> Are there any reasons to hold off on a 0.4.0 release? Are >>>>>>> there >>>>>>>>>>>> particular >>>>>>>>>>>>> JIRAs that the community considers necessary to have as part >>>>>> of >>>>>>>> the >>>>>>>>>>>>> release? >>>>>>>>>>>>> >>>>>>>>>>>>> If not, happy to give it a try at performing RM duties! >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Pierre >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sent from Gmail Mobile >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>
