I'd say we'll have to address the branching strategy but not yet. Let's get a sense of the scope/specific bits we want to tackle in 2.0 then worry about best way to go about it.
On Fri, Jul 23, 2021 at 9:39 AM David Handermann <exceptionfact...@gmail.com> wrote: > > Thanks to everyone who has provided feedback thus far, it is good to see > general interest in moving forward with a release focusing on technical > debt removal. > > It seems like it would be helpful to start gathering removal candidates on > a Confluence wiki page, and then turning those into Jira issues. I am open > to suggestions here, but it seems like creating several epics related to > high level goals would be a good way to organize the work. I can start with > creating a Confluence wiki page to capture some of these initial details. > > A release version 2.0.0 already exists in Jira, so that could be used for > tagging planned issues. > > From a branching and maintenance perspective, perhaps PMC members could > describe how this should work? Should the main branch become the branch > planned for 2.0.0 work, and does that require a vote for approval to > proceed? Or should a new 2.0 branch be created and new commits applied > selectively? > > Regards, > David Handermann > > On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess <mattyb...@apache.org> wrote: > > > Along with the itemized list for ancient components we should look at > > updating versions of drivers, SDKs, etc. for external systems such as > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > > is probably the right time to get things up to date to make them more > > useful to more people. > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough <thena...@gmail.com> wrote: > > > > > > I'm a +1 for removing pretty much all of this stuff. There are security > > > implications to keeping old dependencies around, so the more old code we > > > can remove the better. I agree that eventually we need to move to > > > supporting only Java 11+, and as our next release will probably be about > > 4 > > > - 6 months from now that doesn't seem too soon. We could potentially > > break > > > this in two and remove the deprecated processors and leave 1.x on Java 8, > > > and finally start on 2.x which would support only Java 11. I'm unsure of > > > what implications changing the date and time handling would have - for > > > running systems that use long term historical logs, unexpected impacts to > > > time logging could be a problem. > > > > > > As Joe says I think feature work will have to be dedicated to 2.x and we > > > could support 1.x for security fixes for some period of time. 2.x seems > > > like a gargantuan task but it's probably time to get started. Not sure > > how > > > we handle all open PRs and the transition between 1.x and 2.x. > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > Jon > > > > > > > > You're right we have to be careful and you're right there are still > > > > significant Java 8 users out there. But we also have to be careful > > > > about security and sustainability of the codebase. If we had talked > > > > about this last year when that article came out I'd have agreed it is > > > > too early. Interestingly that link seems to get updated and I tried > > > > [1] and found more recent data (not sure how recent). Anyway it > > > > suggests Java 8 is still the top dog but we see good growth on 11. In > > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem > > > > to care about Java 11 until later half last year and now suddenly it > > > > is all over the place. > > > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in > > > > work on the 1.x line just being blunt. We did this many years ago > > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > > > > so) but it was purely bug fix/security related bits. We would need to > > > > do something similar. But feature work would almost certainly go to > > > > the 2.x line. Maybe there are other workable models but my instinct > > > > suggests this is likely to follow a similar path. > > > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We > > > > need to make the call in both the interests of the user base and the > > > > contributor base of the community. > > > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > Thanks > > > > Joe > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > > > > Russ > > > > > > > > > > Yeah the flow registry is a key part of it. But also now you can > > > > > download the flow definition in JSON (upload i think is there now > > > > > too). Templates offered a series of challenges such as we store them > > > > > in the flow definition which has made flows massive in an unintended > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > We have a couple cases where we headed down a particular concept and > > > > > came up with better approaches later. We need to reconcile these > > with > > > > > the benefit of hindsight, and while being careful to be not overly > > > > > disruptive to existing users, to reduce the codebase/maintenance > > > > > burden and allow continued evolution of the project. > > > > > > > > > > Thanks > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > r...@windofkeltia.com> > > > > wrote: > > > > > > > > > > > > Joe, > > > > > > > > > > > > I apologize for the off-topic intrusion, but what replaces > > templates? > > > > > > The Registry? Templates rocked and we have used them since 0.5.x. > > > > > > > > > > > > Russ > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > > David, > > > > > > > > > > > > > > I think this is a highly reasonable approach and such a focus > > will > > > > > > > greatly help make a 2.0 release far more approachable to knock > > out. > > > > > > > Not only that but tech debt reduction would help make work > > towards > > > > > > > major features we'd think about in a 'major release' sense more > > > > > > > approachable. > > > > > > > > > > > > > > We should remove all deprecated things (as well as verify we > > have the > > > > > > > right list). We should remove/consider removal of deprecated > > > > concepts > > > > > > > like templates. We should consider whether we can resolve the > > > > various > > > > > > > ways we've handled what are now parameters down to one clean > > > > approach. > > > > > > > We should remove options in the nifi.properties which turn out to > > > > > > > never be used quite right (if there are). There is quite a bit > > we > > > > can > > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > > > Lots to consider here but I think this is the right discussion. > > > > > > > > > > > > > > Than ks > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende <bbe...@gmail.com> > > > > wrote: > > > > > > >> I'm a +1 for this... Not sure if this falls under "Removing > > > > Deprecated > > > > > > >> Components", but I think we should also look at anything that > > has > > > > been > > > > > > >> marked as deprecated throughout the code base as a candidate for > > > > > > >> removal. There are quite a few classes, methods, properties, etc > > > > that > > > > > > >> have been waiting for a chance to be removed. > > > > > > >> > > > > > > >> On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > >> <exceptionfact...@apache.org> wrote: > > > > > > >>> Team, > > > > > > >>> > > > > > > >>> With all of the excellent work that many have contributed to > > NiFi > > > > over the > > > > > > >>> years, the code base has also accumulated some amount of > > technical > > > > debt. A > > > > > > >>> handful of components have been marked as deprecated, and some > > > > components > > > > > > >>> remain in the code base to support integration with old > > versions > > > > of various > > > > > > >>> products. Following the principles of semantic versioning, > > > > introducing a > > > > > > >>> major release would provide the opportunity to remove these > > > > deprecated and > > > > > > >>> unsupported components. > > > > > > >>> > > > > > > >>> Rather than focusing the next major release on new features, > > what > > > > do you > > > > > > >>> think about focusing on technical debt removal? This approach > > > > would not > > > > > > >>> make for the most interesting release, but it provides the > > > > opportunity to > > > > > > >>> clean up elements that involve breaking changes. > > > > > > >>> > > > > > > >>> Focusing on technical debt, at least three primary goals come > > to > > > > mind for > > > > > > >>> the next major release: > > > > > > >>> > > > > > > >>> 1. Removal of deprecated and unmaintained components > > > > > > >>> 2. Require Java 11 as the minimum supported version > > > > > > >>> 3. Transition internal date and time handling to JSR 310 > > java.time > > > > > > >>> components > > > > > > >>> > > > > > > >>> *Removing Deprecated Components* > > > > > > >>> > > > > > > >>> Removing support for older and deprecated components provides a > > > > great > > > > > > >>> opportunity to improve the overall security posture when it > > comes > > > > to > > > > > > >>> maintaining dependencies. The OWASP dependency plugin report > > > > currently > > > > > > >>> generates 50 MB of HTML for questionable dependencies, many of > > > > which are > > > > > > >>> related to old versions of various libraries. > > > > > > >>> > > > > > > >>> As a starting point, here are a handful of components and > > > > extension modules > > > > > > >>> that could be targeted for removal in a major version: > > > > > > >>> > > > > > > >>> - PostHTTP and GetHTTP > > > > > > >>> - ListenLumberjack and the entire nifi-lumberjack-bundle > > > > > > >>> - ListenBeats and the entire nifi-beats-bundle > > > > > > >>> - Elasticsearch 5 components > > > > > > >>> - Hive 1 and 2 components > > > > > > >>> > > > > > > >>> *Requiring Java 11* > > > > > > >>> > > > > > > >>> Java 8 is now over seven years old, and NiFi has supported > > general > > > > > > >>> compatibility with Java 11 for several years. NiFi 1.14.0 > > > > incorporated > > > > > > >>> internal improvements specifically related to TLS 1.3, which > > > > allowed > > > > > > >>> closing out the long-running Java 11 compatibility epic > > NIFI-5174. > > > > Making > > > > > > >>> Java 11 the minimum required version provides the opportunity > > to > > > > address > > > > > > >>> any lingering edge cases and put NiFi in a better position to > > > > support > > > > > > >>> current Java versions. > > > > > > >>> > > > > > > >>> *JSR 310 for Date and Time Handling* > > > > > > >>> > > > > > > >>> Without making the scope too broad, transitioning internal date > > > > and time > > > > > > >>> handling to use DateTimeFormatter instead of SimpleDateFormat > > > > would provide > > > > > > >>> a number of advantages. The Java Time components provide much > > > > better > > > > > > >>> clarity when it comes to handling localized date and time > > > > representations, > > > > > > >>> and also avoid the inherent confusion of java.sql.Date > > extending > > > > > > >>> java.util.Date. Many internal components, specifically > > > > Record-oriented > > > > > > >>> processors and services, rely on date parsing, leading to > > > > confusion and > > > > > > >>> various workarounds. The pattern formats of SimpleDateFormat > > and > > > > > > >>> DateTimeFormatter are very similar, but there are a few subtle > > > > differences. > > > > > > >>> Making this transition would provide a much better foundation > > > > going forward. > > > > > > >>> > > > > > > >>> *Conclusion* > > > > > > >>> > > > > > > >>> Thanks for giving this proposal some consideration. Many of you > > > > have been > > > > > > >>> developing NiFi for years and I look forward to your feedback. > > I > > > > would be > > > > > > >>> glad to put together a more formalized recommendation on > > > > Confluence and > > > > > > >>> write up Jira epics if this general approach sounds agreeable > > to > > > > the > > > > > > >>> community. > > > > > > >>> > > > > > > >>> Regards, > > > > > > >>> David Handermann > > > > > > > > > > > >
