Team, I'm going through the outstanding JIRAs/PRs and flagging which look like they should be 'must have' for 1.20 and then will work the RC as soon as those land.
Hopefully have the RC up within a day or two but we'll see how these land as some have review comments pending action. Thanks On Thu, Jan 12, 2023 at 2:53 AM Isha Lamboo <isha.lam...@virtualsciences.nl> wrote: > Hi all, > > I would like to contribute to the migration tooling (mostly testing I > suppose) when that comes up. > > My team's largest client has a completely template-based pipeline with > external scripts replacing variable values before deploying to target > clusters, so we've already started looking at this when the goals for 2.0 > were discussed and approved. The migration to flowdefinitions and > parameters is quite complex and we've hit several blockers when we tried to > implement a direct translation. > > I expect that any time I spend helping to improve the tooling will pay off > handsomely for our clients. > > Regards, > > Isha > > -----Oorspronkelijk bericht----- > Van: Adam Taft <a...@adamtaft.com> > Verzonden: woensdag 11 januari 2023 23:42 > Aan: dev@nifi.apache.org > Onderwerp: Re: [discuss] NiFi 1.20 and NiFi 2.0 > > This is really insightful and spot on ... > > Kevin wrote: > > Good migration tooling will take a while to develop and test, and the > > core contributors to that effort may not have sufficient variety of > > flows to evaluate when the migration tools are "done" for the majority > > of the community to have success upgrading to 2.x. A milestone release > > would > allow > > us get more feedback on migration over a longer period than the vote > window > > of an RC candidate. > > It's exactly this case, that an early 2.0 release might not have had time > to fully work its way through existing production deployments, that's > concerning. The pace and voting of an "RC" is much too short to get any > quality feedback from users in the field. > > I think it's really smart to consider the "Milestone" release approach > here. We release 2.0.0-M1, 2.0.0-M2, ... waiting an adequate amount of time > for feedback. We can put these milestones on a calendar, as needed, so that > feedback is required some 'x' number of weeks/months after each milestone. > > And to this end, I'd personally rather see us keep the 'main' branch > current with the 1.x line _until_ we're ready and are satisfied with the > end goals of the 2.0 release objectives. When the milestone releases have > been completed and there's a comfort level with the 2.x line, it's at the > point we'd isolate the 1.x line into its own branch and switch main over to > the 2.x line. > > This is an attractive way of: > a) continuing business-as-usual with the 1.x line > b) making headway on the 2.x release milestones > c) giving adequate time for feedback against the 2.0 milestones coming > from the field > > I don't mind the known-unknowns. But it's really the unknown-unknowns that > are going to drive a delay in the 2.0 release. I think it's smart to be > able to get some of the unknowns ironed out before we finalize the 2.0 > release ceremony. The milestone approach really helps with that. > > /Adam > > On Wed, Jan 11, 2023 at 11:11 AM Kevin Doran <kdo...@apache.org> wrote: > > > Sorry, Joe, I was not clear, and to be honest the two thoughts are > > somewhat unrelated in my mind too :) > > > > I agree that good migration tooling is key. Otherwise, we risk users > > staying on 1.x or creating a schism of 1.x and 2.x users. > > > > Good migration tooling will take a while to develop and test, and the > > core contributors to that effort may not have sufficient variety of > > flows to evaluate when the migration tools are "done" for the majority > > of the community to have success upgrading to 2.x. A milestone release > > would allow us get more feedback on migration over a longer period > > than the vote window of an RC candidate. > > > > Perhaps we could continue to release from the 1.x line (including > > minor releases with some features) until we are ready to drop the > "milestone" > > qualifier from 2.0.0, and only then put 1.x into maintenance-only status. > > It would be the same proposal to move main to target 2.0.0-M1, but > > relax restrictions for what can land on the 1.x branch and be open to > > a 1.21, 1.22, etc. if 2.0.0 work takes longer than anticipated. For > > example, maybe we would be open to landing new/backported processors > > on the 1.x branch, but not core framework features or API changes. > > > > This might not be necessary, but I think it is fair that saying "no > > new features on 1.x" and also "no new features in 2.0.0" puts the > > project in a rough place if 2.0.0 takes longer than a few months, so > > if we go that route, we need to commit to a quick release of 2.0.0 > > that most users can move to easily. > > > > Thanks, > > Kevin > > > > On Jan 11, 2023 at 12:32:46, Joe Witt <joe.w...@gmail.com> wrote: > > > > > Kevin, > > > > > > Yeah we can do whatever we want as far as 'releases' of 2.0 that are > > prior > > > to us officially considering it 2.0/stable. > > > > > > That said - the migration tooling will be key to provide as we need > > > to > > make > > > the bridge as solid and stable as possible to help someone move from > > > 1.x > > to > > > 2.x. I dont know how related these two concepts (milestone releases > > > and 1.x to 2.x ease really are). > > > > > > Thanks > > > > > > On Wed, Jan 11, 2023 at 10:27 AM Kevin Doran <kdo...@apache.org> > wrote: > > > > > > [hit the wrong keyboard shortcut, here is the rest of my thoughts] > > > > > > > > > On this point from David: > > > > > > > > > We may need to have a longer release candidate period, or more > > incremental > > > > > > > fix releases > > > > > > > for the initial 2.0.0 release train, but I do not expect delaying > > > > a > > 2.0.0 > > > > > > > release for new features, as that is not part of the release goals. > > > > > > > > > > > > > > > > Would the community benefit from one or more milestone releases of > > > 2.0, > > to > > > > > > allow for a wider group to run / live on the proposed 2.0 prior to > > > > > > releasing it as "stable"? I know we've never done a milestone > > > release in > > > > > > the past, and I'm not sure what ASF guidance is on the topic, but if > > > it > > > > > > could be beneficial we could look into that. > > > > > > > > > Cheers, > > > > > > Kevin > > > > > > > > > On Jan 11, 2023 at 12:22:43, Kevin Doran <kdo...@apache.org> wrote: > > > > > > > > > > I think this is a good, practical discussion. > > > > > > > > > > > > > > On the one hand, we can't put off 2.x any longer as we really need > > > > to > > > > > > > updated the minimum required Java to 11. Updating main development > > > > to > > > > > > > target 2.x feels like a good way drive progress on that along with > > > > the > > > > > > > other 2.0 goals. > > > > > > > > > > > > > > On the other hand, the concerns are valid: moving all development > > > > to > > > > > > > target 2.x puts the project at risk if we cannot release 2.0.0 on > > > > a > > > > > > > reasonable timeline. The restricted scope of 2.0 helps, but this > > > > stated > > > > > > > release goal feels risky to me: > > > > > > > > > > > > > > Implement Migration Tools for Upgrading Flows > > > > > > > > > > > > > > > > > > > > > - Implement automated migration where possible to remap > > > > properties > > and > > > > > > > features > > > > > > > - Implement migration tools for manual conversion of XML > > Templates > > > > > > > to JSON Flow Definitions > > > > > > > - Create documentation for manual steps necessary where > > > > > > > programmatic migration cannot be implemented > > > > > > > - NiFi 2.0 should be capable of starting with ghosted > > > > components > > > > > > > for removed Processors or Controller Services > > > > > > > > > > > > > > > > > > > > > Removing deprecated components should be fairly straightforward > > > > and > > > > > > quick, > > > > > > > but automating and documenting migration is a large effort. > > > > > > > > > > > > > > On this po > > > > > > > > > > > > > > > > > > > > > On Jan 10, 2023 at 09:32:31, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > > > > > > > > >> The plan as I understand it is not to diverge and create separate > > > > > > feature > > > > > > >> development on the 1.x line, so I would expect all PRs to > > > >> continue to > > be > > > > > > >> submitted only to main. We would release 1.x as needed with major > > > >> bug > > > > > > >> fixes > > > > > > >> or critical security updates, and these would be cherry-picked > > > >> and/or > > > > > > >> backported as necessary, mostly without the need for PRs, the > > > >> same as > > we > > > > > > >> would do if we were bringing fixes from main (1.20.0-SNAPSHOT) > > > >> back > > to a > > > > > > >> maintenance line like (1.19.x). For precedent, we followed this > > > >> same > > > > > > >> approach going from the 0.x line to 1.0.0 and there wasn't any > > > >> major > > > > > > >> issue. > > > > > > >> > > > > > > >> On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler > > > >> <ottobackwa...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >> It was also mentioned in another thread that we need to have > > agreement > > > > > > on > > > > > > >> > > > > > > >> our explicit strategy and support for 1.x going forward, did that > > > > > > happen? > > > > > > >> > > > > > > >> > > > > > > >> From: Otto Fowler <ottobackwa...@gmail.com> > > > >> <ottobackwa...@gmail.com> > > > > > > >> > > > > > > >> Reply: Otto Fowler <ottobackwa...@gmail.com> > > > >> <ottobackwa...@gmail.com > > > > > > > > > >> > > > > > > >> Date: January 10, 2023 at 07:02:34 > > > > > > >> > > > > > > >> To: dev@nifi.apache.org <dev@nifi.apache.org> > > > >> <dev@nifi.apache.org> > > > > > > >> > > > > > > >> Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 > > > > > > >> > > > > > > >> > > > > > > >> There needs to be an update to the contributing guide as to how > > > >> to > > > > > > submit > > > > > > >> > > > > > > >> PRs to 1.x or 2.x etc. > > > > > > >> > > > > > > >> > > > > > > >> From: Joe Witt <joew...@apache.org> <joew...@apache.org> > > > > > > >> > > > > > > >> Reply: dev@nifi.apache.org <dev@nifi.apache.org> > > > >> <dev@nifi.apache.org > > > > > > > > > >> > > > > > > >> Date: January 9, 2023 at 15:53:16 > > > > > > >> > > > > > > >> To: dev@nifi.apache.org <dev@nifi.apache.org> > > > >> <dev@nifi.apache.org> > > > > > > >> > > > > > > >> Subject: [discuss] NiFi 1.20 and NiFi 2.0 > > > > > > >> > > > > > > >> > > > > > > >> Team, > > > > > > >> > > > > > > >> > > > > > > >> As David mentioned in [1] following a successful NiFi 2.0 release > > > >> goal > > > > > > >> > > > > > > >> planning - we are now going to start moving the 'main' line to be > > > >> the > > > > > > NiFi > > > > > > >> > > > > > > >> 2.0 line which will allow for the key work to take place. We will > > > >> also > > > > > > >> > > > > > > >> move niFi 1.x to its appropriate support line. > > > > > > >> > > > > > > >> > > > > > > >> It is also the case that we have nearly 100 JIRAs on NiFi 1.20 > > > >> and we > > > > > > have > > > > > > >> > > > > > > >> work in there including security items so it is time to make a > > release. > > > > > > >> > > > > > > >> The intent then is to initiate 1.20 and immediate after that > > > >> change > > > > > > 'main' > > > > > > >> > > > > > > >> to 2.0. > > > > > > >> > > > > > > >> > > > > > > >> Going forward then all work on the 1.x line should be focused on > > > > > > >> > > > > > > >> maintaining existing features, dependencies, and helping 1.x > > > >> users > > > > > > migrate > > > > > > >> > > > > > > >> to the 2.x line. Otherwise, new feature work will happen on > > > >> 'main' as > > it > > > > > > >> > > > > > > >> normally does and will come out in the NiFi 2.x release line. > > > > > > >> > > > > > > >> > > > > > > >> Please flag key outstanding items as we narrow down the release > > > > > > candidate > > > > > > >> > > > > > > >> for NiFi 1.20. > > > > > > >> > > > > > > >> > > > > > > >> Thanks > > > > > > >> > > > > > > >> Joe > > > > > > >> > > > > > > >> > > > > > > >> [1] > > > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2 > > > >> Flists.apache.org%2Fthread%2Fqo4vvdw46235y7vy2crcd6l4m11wl7jz&dat > > > >> a=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7Ccbea974a2c1f479d48 > > > >> 9d08daf42521f1%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C63809 > > > >> 0737572228694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo > > > >> > iV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zSqOK9zZPqXxLuxwo0QcqKGEAc7aXjfnnm4i%2BQt2B98%3D&reserved=0 > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > >
