Team Long story short we are now able to build and run NiFi main/2.x line on Java 21!
We've made a lot of progress this week on top of all the great work already done for getting to NiFi 2.0 but specifically to this thread of making NiFi 2.0 be based on Java 21. There are a couple PRs outstanding but once they land I'll put up a PR for this commit [1] and we will be building with Java 21-ea on the main line. The full clean build with all tests and all profiles I could find is now working locally and is also now running in my fork before I put up the PR [2], [3]. NiFi also runs on Java 21. We did have to make a bunch of updates to all things Groovy and we're reducing/eliminating a lot of pieces that are poorly maintained or need fundamentally different implementations in Java 21. The toolkit is likely to be removed it seems and we can later introduce back specific pieces if/as needed but designed better/more easily maintained. What would be ideal is we land a couple more key pieces like ensuring every deprecated component is actually removed and ensuring the flow.xml.gz is entirely gone. Then we kick out a NiFi 2.0 M1 release for people to work with. [1] https://github.com/joewitt/nifi/commit/c3d16d949f8153441c64a1f98fd641cf80178f43 [2] https://github.com/joewitt/nifi/actions/runs/6203527626 [3] https://github.com/joewitt/nifi/actions/runs/6203527625 Thanks Joe On Fri, Sep 15, 2023 at 1:06 PM Pierre Villard <[email protected]> wrote: > Templates are already out. Flow.xml being removed should be reviewed soon > (PR was rebased today). I'm working on the removal of variables. I hope to > get a PR for this in the next few days. > > Le ven. 15 sept. 2023, 19:22, Joe Witt <[email protected]> a écrit : > > > Timeline - we remain in full blitz mode to get things ready for 2.0. No > > clear ETA but we need to be getting it out soon. At least a milestone > > release of it for people to work with. There is a big change needed to > get > > rid of the flow.xml.gz in favor of the json form and that is in progress. > > I am not sure offhand whether templates got the boot yet. > > > > Latest fun is wrestling our rather messy situation with Groovy in the > build > > as that seems not ready for Java 21 generally. > > > > > > > > On Fri, Sep 15, 2023 at 10:19 AM Ryan Hendrickson < > > [email protected]> wrote: > > > > > I think NiFi 2.x going to Java 21 for all the reasons outlined makes a > > lot > > > of sense. > > > > > > Is there a timeline for 2.x? > > > > > > On Mon, Sep 11, 2023 at 5:00 AM Pierre Villard < > > > [email protected]> > > > wrote: > > > > > > > Thanks Joe, it makes total sense and I agree that the ones that would > > > > likely be slow at adopting Java 21 would not go to NiFi 2.0 super > > quickly > > > > anyway. Being able to bring the latest and greatest in NiFi is great > > and > > > > given all of the features announced in Java 21, I imagine a lot of > > > projects > > > > we depend on will be doing the same. > > > > > > > > Le jeu. 7 sept. 2023 à 19:36, Joe Witt <[email protected]> a écrit > : > > > > > > > > > Pierre > > > > > > > > > > A few concerns you raised so want to address my view on each: > > > > > > > > > > Will users be able to adopt Java 21 fast enough? > > > > > I share Brandon's view on that in terms of their adoption timeline. > > It > > > > > will likely align well with NiFi 2.0 itself in my estimation. > > > > > > > > > > Will this delay NiFi 2.0? > > > > > If it would then I'd not be supportive. I don't think we need to > > > bother > > > > > with adopting any of the features now. What I would like us to > have > > is > > > > the > > > > > option to adopt them as we progress. We should get 2.0 done asap > and > > > if > > > > > this added delay then I'd be way less interested in this idea. > > > > > > > > > > Feature benefits of 21 and what will that bring? > > > > > Mark spoke well to the key one that stood out to me which was the > new > > > > > threading model available. It would be awfully nice to leverage > that > > > for > > > > > the efficiency it represents and especially if it can reduce some > of > > > our > > > > > heap usage which is valuable in cloud/shared compute contexts. > > > > > > > > > > Performance benefits of Java 21? > > > > > It appears from some analysis found with googling that Java 21 > brings > > > out > > > > > of the box 4-5% performance increases generally. Not amazing but > > > useful. > > > > > > > > > > User experience otherwise with Java 21? > > > > > I believe it would be consistent with Java 17 for their point of > view > > > in > > > > > terms of install/config/etc.. > > > > > > > > > > My motivation for this is fairly pure honestly. Since we're > setting > > > our > > > > > new minimum bar that lives for as long as the 2.x release line > lives > > > I'd > > > > > like to set it at the current LTS available when we ship that line > as > > > > well. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > On Thu, Sep 7, 2023 at 4:22 AM Brandon DeVries < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > +1 to requiring java 21. Starting off as "up to date" as possibly > > > > makes a > > > > > > lot of sense, and some of the new features seem especially > relevant > > > to > > > > > NiFi. > > > > > > > > > > > > I definitely understand the concerns about organizations being > > > willing > > > > / > > > > > > able to approve Java 21... But those same organizations might > also > > be > > > > > > hesitant to move to NiFi 2.0. We will continue to support java > 17 & > > > > NiFi > > > > > > 1.x for some time, so hopefully those groups will have the time > > they > > > > need > > > > > > to get approvals, do evaluations, and upgrade. > > > > > > > > > > > > Brandon > > > > > > ________________________________ > > > > > > From: Pierre Villard <[email protected]> > > > > > > Sent: Thursday, September 7, 2023 6:15:58 AM > > > > > > To: [email protected] <[email protected]> > > > > > > Subject: Re: [discuss] nifi 2.0 and Java 21… > > > > > > > > > > > > Hi all, > > > > > > > > > > > > I share the concerns raised by Chris regarding how quickly users > of > > > > NiFi > > > > > > will be able to adopt Java 21. > > > > > > > > > > > > While I'm definitely in favor of using the latest and greatest, > > > > > especially > > > > > > when it brings to the table such significant features, we need to > > be > > > > > > careful as it may significantly delay the adoption of NiFi 2.0 in > > big > > > > > > companies where deploying Java 21 will take time. I acknowledge > > that > > > > > going > > > > > > from Java 8 to Java 17 is certainly the same effort as going from > > > Java > > > > 8 > > > > > to > > > > > > Java 21 but how quickly security-sensitive environments will > adopt > > a > > > > new > > > > > > release of Java that is completely new? > > > > > > > > > > > > In addition to that, it sounds like we would add a significant > > rework > > > > of > > > > > > the framework in NiFi 2.0 assuming we adopt Java 21 as the > minimum > > > > > version. > > > > > > Do we think this is going to significantly delay the first > release > > of > > > > > NiFi > > > > > > 2.0? We still have work to do but adding this on top could delay > > the > > > > > > release, no? > > > > > > > > > > > > Finally, the features that Java 21 are bringing sound super > > > interesting > > > > > in > > > > > > the context of NiFi but do we already have an idea of what it > will > > > > > improve? > > > > > > is it the user experience, and if so, how will it change? is it > the > > > > > > performance, and if so, do we have an idea of how things will > > > improve? > > > > > > > > > > > > Thanks, > > > > > > Pierre > > > > > > > > > > > > Le mer. 6 sept. 2023 à 23:07, Chris Sampson > > > > > > <[email protected]> a écrit : > > > > > > > > > > > > > Yeah, I understand the need to move to 21 as a minimum to take > > > > > advantage > > > > > > of > > > > > > > its features. Hopefully the wider java ecosystem won't be an > > issue > > > in > > > > > the > > > > > > > short term. > > > > > > > > > > > > > > I just wanted the discussion to be clear about this being a > > change > > > to > > > > > the > > > > > > > Java baseline/minimum for NiFi 2.0. > > > > > > > > > > > > > > It's a +1 from me. > > > > > > > > > > > > > > On Wed, 6 Sept 2023, 19:01 Joe Witt, <[email protected]> > wrote: > > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > My suggestion is rooted in making Java 21 the minimum of the > > NiFi > > > > 2.0 > > > > > > > > line. It would not work on Java 17. The reason for this is > so > > > > that > > > > > we > > > > > > > can > > > > > > > > leverage the longest duration of a given LTS line while also > > > > > benefiting > > > > > > > > from the language improvements that affords. Maintaining > > > > > compatibility > > > > > > > > with future versions we generally have to do. But whatever > the > > > > > minimum > > > > > > > > version we accept dictates which language features we can > > > leverage. > > > > > So > > > > > > > if > > > > > > > > it is 17 then we can't leverage anything from the 21 line for > > > > > example. > > > > > > > > > > > > > > > > GIven the nature and timelines of LTS I don't really think > > there > > > is > > > > > the > > > > > > > > same burn in logic that we'd have all known in the past > before > > > the > > > > > > > > LTS/STS/etc.. release constructs existed. > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Wed, Sep 6, 2023 at 10:53 AM Chris Sampson > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > To be clear, is the discussion one of making Java 21 the > > > minimum > > > > > > > > > requirement for NiFi 2.0.0, or rather NiFi 2.x be > compatible > > > with > > > > > > Java > > > > > > > > 21, > > > > > > > > > while retaining Java 17 as a minimum? > > > > > > > > > > > > > > > > > > If we moved straight to a Java 21 requirement, will we run > > into > > > > > > > > > compatibility issues that delay initial NiFi 2 release? > Will > > > the > > > > > move > > > > > > > to > > > > > > > > > Java 21 mean some organisations delay their migration to > > NiFi 2 > > > > > > through > > > > > > > > not > > > > > > > > > wanting to move to the latest Java LTS version until it's > > had a > > > > > time > > > > > > > for > > > > > > > > > "settling" through security/bug patches, etc.? And are > either > > > of > > > > > > these > > > > > > > > > sufficient concern to pause Java 21 becoming the > requirement, > > > as > > > > we > > > > > > may > > > > > > > > > then need to extend NiFi 1.x maintenance for longer into > the > > > > > future? > > > > > > > > > > > > > > > > > > Generally, I'm in favour of moving to "latest and > greatest", > > > > > > > particularly > > > > > > > > > for LTS versions of technologies, but the rate of Java > > version > > > > > > adoption > > > > > > > > > across the community gives me pause. > > > > > > > > > > > > > > > > > > I certainly see the advantage of new Java features for NiFi > > in > > > > Java > > > > > > 21, > > > > > > > > > such as the already mentioned virtual threads. > > > > > > > > > > > > > > > > > > On Wed, 6 Sept 2023, 18:34 Mike Thomsen, < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 100% > > > > > > > > > > > > > > > > > > > > On Wed, Sep 6, 2023 at 11:48 AM Adam Taft < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Yes, please. +1 Exactly what Mark said. Virtual threads > > > have > > > > > > > > potential > > > > > > > > > to > > > > > > > > > > > be extremely impactful to applications like NiFi. > > > > > > > > > > > > > > > > > > > > > > /Adam > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 6, 2023 at 7:26 AM Mark Payne < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for bringing his up, Joe. > > > > > > > > > > > > > > > > > > > > > > > > I would definitely be a +1. I think the new virtual > > > thread > > > > > > > concept > > > > > > > > > > would > > > > > > > > > > > > have great impact on us. > > > > > > > > > > > > It would allow us to significantly simplify our > > > scheduling > > > > > > logic, > > > > > > > > > which > > > > > > > > > > > > would help with code maintainability > > > > > > > > > > > > but would also make configuration simpler. This is > one > > of > > > > the > > > > > > > most > > > > > > > > > > > > difficult things for users to configure, > > > > > > > > > > > > and I would very much welcome the ability to simplify > > > this. > > > > > It > > > > > > > > would > > > > > > > > > > > > likely also yield better off-heap memory > > > > > > > > > > > > utilization by reducing the number of native threads > > > > > necessary. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > -Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 6, 2023, at 10:20 AM, Joe Witt < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Team > > > > > > > > > > > > > > > > > > > > > > > > > > Thought it might be worth relighting this thread > with > > > > Java > > > > > 21 > > > > > > > GA > > > > > > > > > > > > imminent. > > > > > > > > > > > > > Given the timing we should give consideration to > > having > > > > > Java > > > > > > 21 > > > > > > > > as > > > > > > > > > > the > > > > > > > > > > > > > basis for nifi 2.x to buy maximum time with LTS > > > > alignment. > > > > > > > There > > > > > > > > > are > > > > > > > > > > > > also > > > > > > > > > > > > > a couple interesting language features we can > likely > > > take > > > > > > > > advantage > > > > > > > > > > of. > > > > > > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 21, 2023 at 6:21 AM David Handermann < > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Dirk, > > > > > > > > > > > > >> > > > > > > > > > > > > >> Thanks for summarizing your findings in the > > referenced > > > > > Jira > > > > > > > > > issues. > > > > > > > > > > It > > > > > > > > > > > > >> sounds like subsequent discussion of Nashorn > support > > > may > > > > > be > > > > > > > > better > > > > > > > > > > on > > > > > > > > > > > a > > > > > > > > > > > > new > > > > > > > > > > > > >> thread. > > > > > > > > > > > > >> > > > > > > > > > > > > >> The Spring 6 and Jetty 11 upgrades are going to > > > require > > > > > > > > > significant > > > > > > > > > > > > work. > > > > > > > > > > > > >> One incremental step in that direction was making > > Java > > > > 17 > > > > > > the > > > > > > > > > > minimum > > > > > > > > > > > > >> version, and upgrading to Jetty 10 should also > help > > > move > > > > > > > things > > > > > > > > > > > forward. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Although compiling NiFi modules with a reference > to > > > the > > > > > > > > standalone > > > > > > > > > > > > Nashorn > > > > > > > > > > > > >> library may introduce issues, there should be > other > > > > > options > > > > > > > for > > > > > > > > > > > > referencing > > > > > > > > > > > > >> the library at runtime. That requires custom class > > > > > loading, > > > > > > > > which > > > > > > > > > > some > > > > > > > > > > > > >> Processors support, so that seems like the general > > > > > direction > > > > > > > to > > > > > > > > > go. > > > > > > > > > > > > >> > > > > > > > > > > > > >> If you have additional findings, feel free to > start > > a > > > > new > > > > > > > > > developer > > > > > > > > > > > list > > > > > > > > > > > > >> thread and that may gather additional feedback. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Regards, > > > > > > > > > > > > >> David Handermann > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Wed, Jun 21, 2023 at 12:17 AM Dirk Arends < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >>> Since initially raising concerns about the move > to > > > Java > > > > > 17 > > > > > > > > losing > > > > > > > > > > > > >> Nashorn, > > > > > > > > > > > > >>> I have been investigating the suggestion to use > > > Nashorn > > > > > as > > > > > > a > > > > > > > > > > > standalone > > > > > > > > > > > > >>> package as potential easier alternative to > GraalVM. > > > [1] > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> While making some progress, a number of issues > have > > > > been > > > > > > > > > > encountered > > > > > > > > > > > > >> which > > > > > > > > > > > > >>> I haven't been able to resolve as yet. More > details > > > are > > > > > > > > included > > > > > > > > > in > > > > > > > > > > > > >>> relevant JIRA tickets, but summarising: > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> - Building NiFi with a recent Nashorn dependency > > > leads > > > > to > > > > > > > > errors > > > > > > > > > > > > >>> "Unsupported class file major version 61" [2] > > > > > > > > > > > > >>> - Building NiFi using Java 17 highlights issues > > with > > > > the > > > > > > > > current > > > > > > > > > > > Jetty > > > > > > > > > > > > >>> version, which I believe would require an upgrade > > > from > > > > > > 9.4.51 > > > > > > > > to > > > > > > > > > > > > 11.0.15 > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > >>> - Jetty 11 then requires an upgrade of the Spring > > > > > Framework > > > > > > > > > > version 5 > > > > > > > > > > > > to > > > > > > > > > > > > >> 6. > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> The current steps to remove references to > > > "Javascript" > > > > > as a > > > > > > > > > > > > preinstalled > > > > > > > > > > > > >>> scripting language [5] are understandable, but it > > > does > > > > > seem > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > > >> still > > > > > > > > > > > > >>> quite a bit to do before Nashorn or another > > external > > > > > > > scripting > > > > > > > > > > engine > > > > > > > > > > > > >> could > > > > > > > > > > > > >>> be used. > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> [1] > > https://issues.apache.org/jira/browse/NIFI-11700 > > > : > > > > > Java > > > > > > > 17 > > > > > > > > > > > Nashorn > > > > > > > > > > > > >>> standalone support > > > > > > > > > > > > >>> [2] > > https://issues.apache.org/jira/browse/NIFI-11701 > > > : > > > > > > > Support > > > > > > > > > > > building > > > > > > > > > > > > >>> with > > > > > > > > > > > > >>> version 61 class files > > > > > > > > > > > > >>> [3] > > https://issues.apache.org/jira/browse/NIFI-11702 > > > : > > > > > > > Upgrade > > > > > > > > > > Jetty > > > > > > > > > > > to > > > > > > > > > > > > >>> version 11 > > > > > > > > > > > > >>> [4] > > https://issues.apache.org/jira/browse/NIFI-11703 > > > : > > > > > > > Upgrade > > > > > > > > > > Spring > > > > > > > > > > > > >>> Framework to version 6 > > > > > > > > > > > > >>> [5] > > https://issues.apache.org/jira/browse/NIFI-11713 > > > : > > > > > > Remove > > > > > > > > > > > > Deprecated > > > > > > > > > > > > >>> ECMAScript Support > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> Regards, > > > > > > > > > > > > >>> Dirk Arends > > > > > > > > > > > > >>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
