Thanks for the push Otto. They are created: [NIFI-9026 <https://issues.apache.org/jira/browse/NIFI-9026>] - Integration with auto-scaling container environments [NIFI-9027 <https://issues.apache.org/jira/browse/NIFI-9027>] - Integration of S3 buckets with Remote Process Groups for outage events and data overruns [NIFI-9028 <https://issues.apache.org/jira/browse/NIFI-9028>] - Large Canvas Management
Best, Ryan On Fri, Aug 6, 2021 at 1:34 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > You don’t have to assign it to yourself. User submitted jiras are very > important, because they include use cases and details that may not occur to > a developer who just picks it up. It is a great way to contribute! > > From: Ryan Hendrickson <ryan.andrew.hendrick...@gmail.com> > <ryan.andrew.hendrick...@gmail.com> > Reply: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > Date: August 6, 2021 at 03:23:43 > To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > If I put Jira's in, then I'd have to admit I'm a developer vs a user.... > :-) > > I'm just a power user at heart :-) ... > > Ryan > > On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > Ryan, these are awesome, are there jiras for them? It would be best to > > capture the requirements / use cases > > > > From: Ryan Hendrickson <ryan.andrew.hendrick...@gmail.com> > > <ryan.andrew.hendrick...@gmail.com> > > Reply: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > Date: August 4, 2021 at 01:14:23 > > To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > I'll preface with what I'm about to say is not rooted in any real > > understanding of the lift required or other customer experiences... We > > have used NiFi for a long time and are excited for what new features may > > lay ahead in the 2.x line. > > > > ---After having written this, I think our succinct pain points focus > > around large data flow management, auto-scaling container environments, > and > > integration of external analytic/enrichment creation.--- > > > > Thanks, > > Ryan > > > > > > 1 - Integration with auto-scaling container environments > > We often struggle with scaling custom processors in large flows. We've > > implemented a recent design pattern where we write custom python code > > packaged into a docker image, fronted with a web-service, and deployed > into > > a container environment. In NiFi we use InvokeHTTP to call out to the > > web-service. If the load becomes too great, we can easily spin-up more > > than one external web-service without sacrificing the NiFi's overall > > thruput and thread contention. In this scenario, we're using NiFi to > > manage the overall dataflow, routes, ETL services, etc, but not so much > > smarts around auto-scaling processing, etc. > > > > This makes it really easy to receive a dataflow analytic/enrichment from > an > > external team, wrap it in a docker image, and deploy it into an > > auto-scaling container environment. This also lets us avoid conversations > > about how performant the code is and if it's meeting a minimum throughput > > requirement - typically XXGB/5minutes. > > > > The request for 2.x would be to make interacting with container deployed > > services a more seamless, streamlined process, possibly through a > > normalized API or as a "Remote Container Environment" that can bind to a > > service with a health endpoint and verify it's alive and well, etc. > > > > 2 - Integration of S3 buckets with Remote Process Groups for outage > events > > and data overruns > > Another frequent struggle is the occasional death of a server in a NiFi > > flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis > > and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together > > with Remote Process Groups to perform the overall dataflow goals. If any > > one of the 15 experience an outage, data begins to backup on the previous > > NiFi, hitting backpressure limits, etc. When the down server comes back > > online, it receives a flood of data, often overwhelming it, necessitating > > careful flowfile limits and backpressure points. > > > > The design pattern we're exploring, but would like to see baked into > NiFi, > > is that when a Remote Process Group becomes unavailable due to > > server-to-server comms failures AND the backpressure limit is reached on > a > > relationship, the Remote Process Group would begin sending data to an S3 > > bucket as an automatic failover procedure. When the downed server comes > > back-online, the Remote Process Group port would know to pull data from > S3 > > AND receive data via the port from the chained server. > > > > 3 - Large Canvas Management > > Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2 > > NiFi clusters, each canvas is managed individually. To see the entire > > canvas, systems engineering diagrams need to be kept up-to-date > identifying > > all the input/output ports. To move processors from server to server > often > > requires the verification of the existence of scripts, configuration > files, > > NiFi custom nars, and using the templating or Registry to migrate > portions > > of the dataflow. > > > > Our current design pattern to accomplish some of these tasks include > using > > Ansible, Jenkins, and Nexus to verify script & configuration files are up > > to date. We use Confluence's Gliffy to document the NiFi communication > > paths between the servers. > > > > Ideally, we'd be able to see an overall management view of the Canvases > > together in a single place - it would indicate which server the portions > of > > the canvas are deployed on and tie together where the Remote Process > Group > > ports are connected. Additionally, NiFi would allow the user to move > > Processors and Process Groups seamlessly from one server to another > server > > to help manually distribute and balance data flow processing cycles. > > > > > > > > On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > > Kevin, > > > > > > This may well be a very interesting approach which opens up a lot of > > > options for us in how we tackle a 2.x... > > > > > > Thanks > > > > > > On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kdoran.apa...@gmail.com> > > > wrote: > > > > > > > > Outside of core NiFi, for MiNiFi, I think it would be beneficial to > > > align MiNiFi Java and MiNiFI C++ to use the same method(s) for command > > and > > > control / flow deployment. This was discussed when MiNiFi Java was > merged > > > into the NiFi codebase [1]. > > > > > > > > My proposal would be to create a Java implementation of the C2 > Protocol > > > used by MiNiFi C++, both server-side and client-side, to allow > deploying > > to > > > MiNiFi Java instances from a centralize flow definition. Thanks to the > > > unification of MiNiFi Java and NiFi, this could likely also be added as > a > > > capability of NiFi at that same time. > > > > > > > > If we are able to do this on the 1.x line, then I would also support > > > deprecating other methods of flow deployment to MiNiFi outlined in [1] > > and > > > removing them in a 2.x release. > > > > > > > > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 > < > > > https://github.com/apache/nifi/pull/4933#issuecomment-808411977> > > > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design < > > > https://cwiki.apache.org/confluence/display/MINIFI/C2+Design> > > > > > > > > Thanks, > > > > Kevin > > > > > > > > > On Aug 3, 2021, at 10:07 AM, David Handermann < > > > exceptionfact...@gmail.com> wrote: > > > > > > > > > > Thanks for the continued discussion around what can or should be > > > removed. > > > > > Having a clear migration path from version 1 to version 2 will be > > very > > > > > important for supporting current deployments. > > > > > > > > > > Following the example of some other projects, one way to consider > the > > > > > upgrade is from the point of view of deprecation warnings. If > > > implemented > > > > > correctly, an installation of NiFi running the latest minor release > > of > > > > > version 1, and showing no deprecation warnings, should be > upgradeable > > > to > > > > > version 2 without any changes. Making this work involves accurately > > > tagging > > > > > components and methods with deprecation indicators, and providing a > > > clear > > > > > way to observe deprecation warnings. With the current version > > > containing > > > > > both deprecated components and deprecated methods in various > classes, > > > this > > > > > would involve some work to get right. A simple approach might be a > > > named > > > > > logger for deprecation warnings, which would write a separate log > > > file. A > > > > > more advanced approach might involve a tool to analyze the system > and > > > flow > > > > > configurations. Either way, it seems that some additional work in a > > > minor > > > > > release version is necessary before considering a major release > > > version. > > > > > > > > > > One other point on compatibility: as long as the core component API > > > > > definition remains backwards compatible, it should be possible to > run > > > older > > > > > components. This provides a transition option for customers using > > > > > components such as PostHTTP, or any others that are not actively > > > > > maintained. At some point it may become necessary to break > > > compatibility at > > > > > that level, but at least for version 2, maintaining component API > > > > > compatibility is key. > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <mark.o.b...@gmail.com> > > > wrote: > > > > > > > > > >> I created a JIRA ticket to investigate and improve the performance > > of > > > > >> InvokeHTTP. It includes a flow definition for benchmarking the > > > performance > > > > >> of both PostHTTP and InvokeHTTP. > > > > >> > > > > >> https://issues.apache.org/jira/browse/NIFI-8968 > > > > >> > > > > >> Thanks, > > > > >> Mark > > > > >> > > > > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <a...@adamtaft.com> > > wrote: > > > > >> > > > > >>> I'm not seeing the side thread that was going to discuss > > deprecation > > > of > > > > >>> PostHTTP. Has that thread started and I just don't see it? > > > > >>> > > > > >>> One (significant?) concern with PostHTTP is the smooth > integration > > of > > > > >>> NiFi-to-NiFi communication that is very transparently enabled > with > > > the > > > > >>> ListenHTTP and PostHTTP processors. There's some special logic in > > > there > > > > >> for > > > > >>> handling flowfiles that InvokeHTTP doesn't really (nor should > > really) > > > > >> have. > > > > >>> > > > > >>> I know of several (large) NiFi installations that rely on the > > > PostHTTP / > > > > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication > > for > > > > >> folks > > > > >>> reluctant or unable to enable site-to-site type configuration. > > > > >>> > > > > >>> Honestly, I don't know why we'd want to "deprecate" any > processor, > > as > > > > >>> opposed to just moving it to a new location. If these processors > > can > > > be > > > > >>> ported and maintained to whatever the 2.0 API looks like, there's > > > > >> possibly > > > > >>> little harm keeping them around. > > > > >>> > > > > >>> And by the way, what happened to the "marketplace" concept? Is > this > > > being > > > > >>> considered for 2.0 as well? Because relocating the deprecated > > > processors > > > > >>> to an external nar might be the best solution. Losing PostHTTP > > > entirely I > > > > >>> think would be a mistake, but I'd conceptually support relocating > > it. > > > > >>> > > > > >>> Thanks, > > > > >>> > > > > >>> /Adam > > > > >>> > > > > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <joe.w...@gmail.com> > > wrote: > > > > >>> > > > > >>>> Looks like we just need to knock out 5 JIRAs :) [1] > > > > >>>> > > > > >>>> I felt like we had a label folks were using at one point but > > quickly > > > > >>>> looking revealed nothing exciting. After this confluence page > > > > >>>> stabilizes a bit we can probably knock out some JIRAs and such. > > > > >>>> > > > > >>>> [1] > > https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > >>>> > > > > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler < > > > ottobackwa...@gmail.com> > > > > >>>> wrote: > > > > >>>>> > > > > >>>>> I find myself wishing I had a list of all the jiras / issues > that > > > > >> have > > > > >>>>> been put off for a 2.0 release because they required some > change > > or > > > > >>>> another > > > > >>>>> :( > > > > >>>>> > > > > >>>>> From: Joe Witt <joe.w...@gmail.com> <joe.w...@gmail.com> > > > > >>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < > > > > >> dev@nifi.apache.org> > > > > >>>>> Date: July 27, 2021 at 12:30:35 > > > > >>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> < > > dev@nifi.apache.org > > > > > > > > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > >>>>> > > > > >>>>> A few thoughts: > > > > >>>>> > > > > >>>>> 1. I would love to see deprecation notices show up in the UI in > > > > >>>>> various ways to help motivate users to move off things to more > > > > >>>>> supportable things. That is not a prerequisite for anything > > > happening > > > > >>>>> however. Just a good feature/nice thing to do for users when > > > someone > > > > >>>>> is able to tackle it. > > > > >>>>> > > > > >>>>> 2. The decision to deprecate something and to further remove it > > > need > > > > >>>>> not mean there is a superior solution available. If that thing > > > itself > > > > >>>>> isn't getting the love/attention it needs to be > > > > >>>>> maintained/supported/healthy going forward that alone is enough > > to > > > > >>>>> remove it. That might well be the case with PostHTTP [1] and > for > > > > >>>>> comparison you can see how much effort has gone into InvokeHTTP > > > [2]. > > > > >>>>> > > > > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must > do' > > > the > > > > >>>>> further away from reality such a release will become. We'll > have > > to > > > > >>>>> get very specific about 'musts' vs 'wants'. > > > > >>>>> > > > > >>>>> [1] > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > > > >>>>> [2] > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > >>>>> > > > > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > > > >>>>> <exceptionfact...@apache.org> wrote: > > > > >>>>>> > > > > >>>>>> Thanks Mark, providing a template or comparison statistics > with > > > > >> Java > > > > >>>>>> versions and component configuration details would be very > > > helpful. > > > > >>> If > > > > >>>> it > > > > >>>>>> is possible to run tests using a public API or deployable > > service, > > > > >>> that > > > > >>>>>> would also help confirm potential differences. > > > > >>>>>> > > > > >>>>>> Showing a deprecation notice in the UI could be helpful, > perhaps > > > > >> as a > > > > >>>>>> configurable option. NIFI-8650 describes a general Flow > Analysis > > > > >>>>>> capability, and it seems like that might be one possible way > to > > > > >>> surface > > > > >>>>>> deprecation warnings. For something more specific to component > > > > >>>>> deprecation, > > > > >>>>>> it seems important to find a balance between making it obvious > > and > > > > >>>> making > > > > >>>>>> it something that ends up getting ignored. > > > > >>>>>> > > > > >>>>>> Regards, > > > > >>>>>> David Handermann > > > > >>>>>> > > > > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean < > > mark.o.b...@gmail.com > > > > > > > > >>>> wrote: > > > > >>>>>> > > > > >>>>>>> I'll start a new thread for PostHTTP when I get a template > > and/or > > > > >>>>> detailed > > > > >>>>>>> stats. > > > > >>>>>>> > > > > >>>>>>> I know the deprecation is noted in the documentation. That's > a > > > > >>>>> necessary > > > > >>>>>>> and minimum level of notification. I was suggesting it be > more > > > > >>>> obvious > > > > >>>>> in > > > > >>>>>>> the UI. I think it would be beneficial to somehow be aware of > > the > > > > >>>>>>> deprecation status simply by looking at the flow (perhaps > even > > on > > > > >>> the > > > > >>>>>>> summary pages too), and without having to open the > > documentation > > > > >>> for > > > > >>>>> every > > > > >>>>>>> processor to confirm whether or not a component is marked as > > > > >>>>> deprecated. > > > > >>>>>>> > > > > >>>>>>> Thanks, > > > > >>>>>>> Mark > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > > > > >>>>>>> exceptionfact...@apache.org> wrote: > > > > >>>>>>> > > > > >>>>>>>> Mark, > > > > >>>>>>>> > > > > >>>>>>>> Thanks for the feedback. It may be better to start a > separate > > > > >>>> thread > > > > >>>>> on > > > > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating > > the > > > > >>>>>>> performance > > > > >>>>>>>> differences between PostHTTP and InvokeHTTP? > > > > >>>>>>>> > > > > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas > > > > >>> InvokeHTTP > > > > >>>>> uses > > > > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version > updates > > > > >> for > > > > >>>>> OkHttp, > > > > >>>>>>>> so it would be important to test with the most recent > version. > > > > >> It > > > > >>>> is > > > > >>>>> also > > > > >>>>>>>> worth noting that test classes for PostHTTP are now > > integration > > > > >>>> tests > > > > >>>>> as > > > > >>>>>>>> opposed to unit tests, which means they are not executed as > > > > >> part > > > > >>> of > > > > >>>>> the > > > > >>>>>>>> automated builds. > > > > >>>>>>>> > > > > >>>>>>>> The NiFi documentation includes the contents of the > > > > >>>> DeprecationNotice > > > > >>>>> for > > > > >>>>>>>> PostHTTP: > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > >>>>>>>> > > > > >>>>>>>> Regards, > > > > >>>>>>>> David Handermann > > > > >>>>>>>> > > > > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean < > > > > >> mark.o.b...@gmail.com > > > > >>>> > > > > >>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving > > > > >>>>> deliberately > > > > >>>>>>> to a > > > > >>>>>>>>> 2.0 release. I have a concern with just one processor that > is > > > > >>>>> currently > > > > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated > > > > >>>> specifically > > > > >>>>> any > > > > >>>>>>>>> other deprecated components; I cannot say if there are or > are > > > > >>> not > > > > >>>>>>> similar > > > > >>>>>>>>> issues elsewhere.) I understand the rationale for > deprecating > > > > >>>> this > > > > >>>>>>>>> processor in that it eliminates a processor whose > > > > >> functionality > > > > >>>> is > > > > >>>>>>>>> available elsewhere, specifically in the more flexible > > > > >>>> InvokeHTTP. > > > > >>>>>>>> However, > > > > >>>>>>>>> in my experience and testing, PostHTTP performs transfers > > > > >> with > > > > >>>> far > > > > >>>>>>>> greater > > > > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of > > > > >> removing > > > > >>>>>>> PostHTTP > > > > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its > > > > >>> throughput > > > > >>>>>>>>> capability. > > > > >>>>>>>>> > > > > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP > for > > > > >>>>> similar > > > > >>>>>>>>> reasons? Or, is there a performance-related configuration > for > > > > >>>>>>> InvokeHTTP > > > > >>>>>>>> I > > > > >>>>>>>>> may have missed? > > > > >>>>>>>>> > > > > >>>>>>>>> Also, in order to help facilitate a smooth transition to > 2.0 > > > > >>>> from a > > > > >>>>>>> user > > > > >>>>>>>>> perspective, would it be advisable to add some sort of > visual > > > > >>>>> indicator > > > > >>>>>>>> in > > > > >>>>>>>>> the UI for components that are currently annotated as > > > > >>>> @Deprecated? > > > > >>>>> This > > > > >>>>>>>>> might help users proactively modify their flow prior to a > > > > >>> release > > > > >>>>> that > > > > >>>>>>>>> would otherwise break it. > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende < > > > > >> bbe...@gmail.com> > > > > >>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi > repo, > > > > >>>> we've > > > > >>>>>>>>>> actually gone more towards a mono-repo where everything is > > > > >>>>> released > > > > >>>>>>>>>> together. Eventually it would still be nice to have a > > > > >> smaller > > > > >>>>> base > > > > >>>>>>>>>> distribution containing just the framework and standard > > > > >> NARs, > > > > >>>> but > > > > >>>>> it > > > > >>>>>>>>>> is hard to tackle that until we provide a way for users to > > > > >>>> easily > > > > >>>>>>>>>> obtain the NARs in some other way. > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < > > > > >>>>> edward.ar...@gmail.com > > > > >>>>>>>> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Given the major version number shift and the spliting up > > > > >> of > > > > >>>>>>>> processors > > > > >>>>>>>>>> into > > > > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start > > > > >>>> individually > > > > >>>>>>>>> versioning > > > > >>>>>>>>>>> individual NARs/ bundles. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> I can see this bringing a large number of benifits > > > > >>> including > > > > >>>>> making > > > > >>>>>>>>> Nifi > > > > >>>>>>>>>>> more deployable with things RPM, but also potentially > > > > >>>> allowing > > > > >>>>>>> those > > > > >>>>>>>>> that > > > > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Edward > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, < > > > > >>>> ottobackwa...@gmail.com> > > > > >>>>> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any > > > > >>> one > > > > >>>>> of > > > > >>>>>>> the > > > > >>>>>>>>>>>> processors. > > > > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not > > > > >>>> using > > > > >>>>> a > > > > >>>>>>>> high > > > > >>>>>>>>>> level > > > > >>>>>>>>>>>> purpose build client and may break. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> If we change the aws version, we need to coordinate in > > > > >>>> such a > > > > >>>>> way > > > > >>>>>>>>> that > > > > >>>>>>>>>> they > > > > >>>>>>>>>>>> all > > > > >>>>>>>>>>>> can come along reasonably. > > > > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK? > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> From: David Handermann <exceptionfact...@apache.org> > > > > >>>>>>>>>>>> <exceptionfact...@apache.org> > > > > >>>>>>>>>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < > > > > >>>>>>>>> dev@nifi.apache.org> > > > > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42 > > > > >>>>>>>>>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> < > > > > >>>>>>> dev@nifi.apache.org > > > > >>>>>>>>> > > > > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Chris, > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like > > > > >>>> some > > > > >>>>> of > > > > >>>>>>> the > > > > >>>>>>>>>> work to > > > > >>>>>>>>>>>> reorganize the module structure could be done outside > > > > >> of > > > > >>> a > > > > >>>>> major > > > > >>>>>>>>>> release, > > > > >>>>>>>>>>>> but it would be great to target any breaking changes > > > > >> for > > > > >>>> 2.0. > > > > >>>>>>>>> Perhaps a > > > > >>>>>>>>>>>> separate feature proposal on module restructuring, with > > > > >>> the > > > > >>>>> goal > > > > >>>>>>> of > > > > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to > > > > >>> move > > > > >>>>> that > > > > >>>>>>>> part > > > > >>>>>>>>>> of > > > > >>>>>>>>>>>> the discussion forward. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like > > > > >>> that > > > > >>>>> might > > > > >>>>>>>> be > > > > >>>>>>>>>>>> possible now. I haven't taken a close look at the > > > > >>>> referencing > > > > >>>>>>>>>> components, > > > > >>>>>>>>>>>> so I'm not sure about the level of effort involved. > > > > >> Minor > > > > >>>>> NiFi > > > > >>>>>>>>> version > > > > >>>>>>>>>>>> updates have incorporated major new versions of > > > > >>>> dependencies. > > > > >>>>> For > > > > >>>>>>>>>> example, > > > > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4 > > > > >> to > > > > >>> 5. > > > > >>>>> On > > > > >>>>>>> the > > > > >>>>>>>>> one > > > > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major > > > > >>>> release > > > > >>>>>>> seems > > > > >>>>>>>>>>>> helpful, but unless there are changes that break > > > > >> existing > > > > >>>>>>> component > > > > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked > > > > >>>>> independently. > > > > >>>>>>>>>> Others may > > > > >>>>>>>>>>>> have more insight into particular usage of that > > > > >> library. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Regards, > > > > >>>>>>>>>>>> David Handermann > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > > >>>>>>>>>>>> <chris.samp...@naimuri.com.invalid> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> Might be worth considering refactoring the build as > > > > >>> part > > > > >>>> of > > > > >>>>>>> this > > > > >>>>>>>>> work > > > > >>>>>>>>>>>> too, > > > > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a > > > > >>>>> commit, > > > > >>>>>>>> etc. > > > > >>>>>>>>> - > > > > >>>>>>>>>>>>> discussed briefly in previous threads but don't think > > > > >>> any > > > > >>>>>>> changes > > > > >>>>>>>>>> made > > > > >>>>>>>>>>>> yet. > > > > >>>>>>>>>>>>> If NARs/components are likely to be split up and > > > > >>>> refactored > > > > >>>>>>> then > > > > >>>>>>>>> such > > > > >>>>>>>>>>>> work > > > > >>>>>>>>>>>>> around the build probably makes sense to consider. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> I've a couple of PRs open that include updates to > > > > >>>>> Elasticsearch > > > > >>>>>>>>>> versions > > > > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which > > > > >>>> Elastic > > > > >>>>>>>> changed > > > > >>>>>>>>>>>> licence) > > > > >>>>>>>>>>>>> in case there were licence concerns. But more work > > > > >> can > > > > >>> be > > > > >>>>> done > > > > >>>>>>> to > > > > >>>>>>>>>> tidy up > > > > >>>>>>>>>>>>> the processors, absolutely. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a > > > > >>>>> refactor > > > > >>>>>>> of > > > > >>>>>>>>>> those > > > > >>>>>>>>>>>>> processors as well. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Cheers, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Chris Sampson > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > > >>>>>>>>>>>> exceptionfact...@apache.org> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles, > > > > >>> Mark. > > > > >>>>> There > > > > >>>>>>>>> are a > > > > >>>>>>>>>>>>> number > > > > >>>>>>>>>>>>>> of components in the standard NAR bundles with > > > > >>>> particular > > > > >>>>>>>>>> dependencies > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>> would make more sense in separate NARs. > > > > >> Reorganizing > > > > >>>> the > > > > >>>>>>>> standard > > > > >>>>>>>>>> NAR > > > > >>>>>>>>>>>> to > > > > >>>>>>>>>>>>>> components with limited dependencies and wide > > > > >>>>> applicability > > > > >>>>>>>> would > > > > >>>>>>>>>>>>>> definitely help with future maintenance. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Regards, > > > > >>>>>>>>>>>>>> David Handermann > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > > > > >>>>>>>>> marka...@hotmail.com> > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> There’s also some code that exists in order to > > > > >>>> maintain > > > > >>>>>>>>> backward > > > > >>>>>>>>>>>>>>> compatibility in the repositories. I would very > > > > >>> much > > > > >>>>> like > > > > >>>>>>> the > > > > >>>>>>>>>>>>>> repositories > > > > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file > > > > >>> format > > > > >>>>>>> supports > > > > >>>>>>>>>> really > > > > >>>>>>>>>>>>> old > > > > >>>>>>>>>>>>>>> formats. And the old impls of the repositories > > > > >>>>> themselves, > > > > >>>>>>>> like > > > > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv > > > > >> Repo, > > > > >>>> etc. > > > > >>>>>>> Lots > > > > >>>>>>>> of > > > > >>>>>>>>>> stuff > > > > >>>>>>>>>>>>>> there > > > > >>>>>>>>>>>>>>> that can be removed. And some methods in > > > > >>>> ProcessSession > > > > >>>>>>> that > > > > >>>>>>>>> are > > > > >>>>>>>>>>>> never > > > > >>>>>>>>>>>>>> used > > > > >>>>>>>>>>>>>>> by any processor in the codebase but exists in > > > > >> the > > > > >>>>> public > > > > >>>>>>> API > > > > >>>>>>>>> so > > > > >>>>>>>>>>>> can’t > > > > >>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>> removed till 2.0. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I think his is also a great time to clean up the > > > > >>>>> “standard > > > > >>>>>>>>> nar.” > > > > >>>>>>>>>> At > > > > >>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the > > > > >>>>>>> components > > > > >>>>>>>>>> there > > > > >>>>>>>>>>>> are > > > > >>>>>>>>>>>>>> not > > > > >>>>>>>>>>>>>>> really “standard” - things like connecting to > > > > >> FTP & > > > > >>>>> SFTP > > > > >>>>>>>>>> servers, XML > > > > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could > > > > >> potentially > > > > >>> be > > > > >>>>> moved > > > > >>>>>>>>> into > > > > >>>>>>>>>>>> other > > > > >>>>>>>>>>>>>>> nars. The > > > > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war > > > > >>>>>>> is > > > > >>>>>>>>>> 6.9 MB > > > > >>>>>>>>>>>> is > > > > >>>>>>>>>>>>>> not > > > > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of > > > > >>>> things > > > > >>>>>>>> probably > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>>>>> reconsider within the standard nar. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach, > > > > >>> to > > > > >>>>> allow > > > > >>>>>>>> for > > > > >>>>>>>>> a > > > > >>>>>>>>>> 2.0 > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>> is not a huge feature release but allows the > > > > >>> project > > > > >>>> to > > > > >>>>> be > > > > >>>>>>>>>> simpler > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>>>> more > > > > >>>>>>>>>>>>>>> nimble. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Thanks > > > > >>>>>>>>>>>>>>> -Mark > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > > > >>>>>>>>>> mikerthom...@gmail.com > > > > >>>>>>>>>>>>>> <mailto: > > > > >>>>>>>>>>>>>>> mikerthom...@gmail.com>> wrote: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Russell, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low > > > > >>> level > > > > >>>>> REST > > > > >>>>>>>>>> client is > > > > >>>>>>>>>>>>>>> still fine. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs > > > > >>> at > > > > >>>>>>> present. > > > > >>>>>>>>> One > > > > >>>>>>>>>>>> uses > > > > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic > > > > >>> REST > > > > >>>>>>> client. > > > > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for > > > > >> the > > > > >>>>> moment. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Mike > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > > >>>>>>>>>>>> r...@windofkeltia.com > > > > >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> wrote: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the > > > > >>> Elastic > > > > >>>>>>>> framework > > > > >>>>>>>>>> has > > > > >>>>>>>>>>>> just > > > > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to > > > > >>>>> acknowledge > > > > >>>>>>>>> that, > > > > >>>>>>>>>>>> maybe > > > > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not > > > > >>>>> understanding > > > > >>>>>>>>>> exactly > > > > >>>>>>>>>>>> how > > > > >>>>>>>>>>>>>>> this sort of thing is considered in a > > > > >> large-scale, > > > > >>>>>>>> world-class > > > > >>>>>>>>>>>> software > > > > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor, > > > > >>>> just > > > > >>>>> a > > > > >>>>>>>>> grateful > > > > >>>>>>>>>>>>>>> consumer.) > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Russ > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > >>>>>>>>>>>>>>> Along with the itemized list for ancient > > > > >> components > > > > >>>> we > > > > >>>>>>> should > > > > >>>>>>>>>> look at > > > > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for > > > > >>> external > > > > >>>>>>> systems > > > > >>>>>>>>>> such as > > > > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be > > > > >>> breaking > > > > >>>>>>> changes > > > > >>>>>>>>> but > > > > >>>>>>>>>> 2.0 > > > > >>>>>>>>>>>>>>> is probably the right time to get things up to > > > > >> date > > > > >>>> to > > > > >>>>> make > > > > >>>>>>>>> them > > > > >>>>>>>>>> more > > > > >>>>>>>>>>>>>>> useful to more people. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > > > >>>>>>>>>> thena...@gmail.com > > > > >>>>>>>>>>>>>> <mailto: > > > > >>>>>>>>>>>>>>> thena...@gmail.com>> wrote: > > > > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this > > > > >>> stuff. > > > > >>>>> There > > > > >>>>>>>> are > > > > >>>>>>>>>>>> security > > > > >>>>>>>>>>>>>>> implications to keeping old dependencies around, > > > > >> so > > > > >>>> the > > > > >>>>>>> more > > > > >>>>>>>>> old > > > > >>>>>>>>>> code > > > > >>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we > > > > >>>> need > > > > >>>>> to > > > > >>>>>>>> move > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release > > > > >>>> will > > > > >>>>>>>> probably > > > > >>>>>>>>>> be > > > > >>>>>>>>>>>>> about > > > > >>>>>>>>>>>>>> 4 > > > > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon. > > > > >> We > > > > >>>>> could > > > > >>>>>>>>>> potentially > > > > >>>>>>>>>>>>>> break > > > > >>>>>>>>>>>>>>> this in two and remove the deprecated processors > > > > >>> and > > > > >>>>> leave > > > > >>>>>>>> 1.x > > > > >>>>>>>>> on > > > > >>>>>>>>>>>> Java > > > > >>>>>>>>>>>>> 8, > > > > >>>>>>>>>>>>>>> and finally start on 2.x which would support only > > > > >>>> Java > > > > >>>>> 11. > > > > >>>>>>>> I'm > > > > >>>>>>>>>> unsure > > > > >>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>> what implications changing the date and time > > > > >>> handling > > > > >>>>> would > > > > >>>>>>>>> have > > > > >>>>>>>>>> - > > > > >>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>> running systems that use long term historical > > > > >> logs, > > > > >>>>>>>> unexpected > > > > >>>>>>>>>>>> impacts > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>> time logging could be a problem. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be > > > > >>>>> dedicated > > > > >>>>>>> to > > > > >>>>>>>>>> 2.x and > > > > >>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>> could support 1.x for security fixes for some > > > > >>> period > > > > >>>> of > > > > >>>>>>> time. > > > > >>>>>>>>> 2.x > > > > >>>>>>>>>>>> seems > > > > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to > > > > >>> get > > > > >>>>>>> started. > > > > >>>>>>>>> Not > > > > >>>>>>>>>>>> sure > > > > >>>>>>>>>>>>>> how > > > > >>>>>>>>>>>>>>> we handle all open PRs and the transition between > > > > >>> 1.x > > > > >>>>> and > > > > >>>>>>>> 2.x. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < > > > > >>>>>>>> joe.w...@gmail.com > > > > >>>>>>>>>>>> <mailto: > > > > >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Jon > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> You're right we have to be careful and you're > > > > >> right > > > > >>>>> there > > > > >>>>>>> are > > > > >>>>>>>>>> still > > > > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also > > > > >>> have > > > > >>>> to > > > > >>>>> be > > > > >>>>>>>>>> careful > > > > >>>>>>>>>>>>>>> about security and sustainability of the > > > > >> codebase. > > > > >>> If > > > > >>>>> we > > > > >>>>>>> had > > > > >>>>>>>>>> talked > > > > >>>>>>>>>>>>>>> about this last year when that article came out > > > > >> I'd > > > > >>>>> have > > > > >>>>>>>> agreed > > > > >>>>>>>>>> it is > > > > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get > > > > >>>> updated > > > > >>>>>>> and I > > > > >>>>>>>>>> tried > > > > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how > > > > >>> recent). > > > > >>>>>>> Anyway > > > > >>>>>>>> it > > > > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see > > > > >>> good > > > > >>>>> growth > > > > >>>>>>>> on > > > > >>>>>>>>>> 11. In > > > > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too. > > > > >>>>> Customers > > > > >>>>>>>> didn't > > > > >>>>>>>>>> seem > > > > >>>>>>>>>>>>>>> to care about Java 11 until later half last year > > > > >>> and > > > > >>>>> now > > > > >>>>>>>>>> suddenly it > > > > >>>>>>>>>>>>>>> is all over the place. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd > > > > >> see > > > > >>>>> rapid > > > > >>>>>>>>>> decrease in > > > > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did > > > > >> this > > > > >>>> many > > > > >>>>>>> years > > > > >>>>>>>>> ago > > > > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a > > > > >> while > > > > >>>>> (maybe > > > > >>>>>>> a > > > > >>>>>>>>>> year or > > > > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related > > > > >>> bits. > > > > >>>> We > > > > >>>>>>> would > > > > >>>>>>>>>> need to > > > > >>>>>>>>>>>>>>> do something similar. But feature work would > > > > >> almost > > > > >>>>>>> certainly > > > > >>>>>>>>> go > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable > > > > >> models > > > > >>>> but > > > > >>>>> my > > > > >>>>>>>>>> instinct > > > > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to > > > > >>>> dump > > > > >>>>> Java > > > > >>>>>>>> 8. > > > > >>>>>>>>> We > > > > >>>>>>>>>>>>>>> need to make the call in both the interests of > > > > >> the > > > > >>>> user > > > > >>>>>>> base > > > > >>>>>>>>> and > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>>> contributor base of the community. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> [1] > > > > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Thanks > > > > >>>>>>>>>>>>>>> Joe > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < > > > > >>>>>>> joe.w...@gmail.com > > > > >>>>>>>>>> <mailto: > > > > >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: > > > > >>>>>>>>>>>>>>> Russ > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But > > > > >>> also > > > > >>>>> now > > > > >>>>>>> you > > > > >>>>>>>>> can > > > > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i > > > > >>> think > > > > >>>> is > > > > >>>>>>> there > > > > >>>>>>>>> now > > > > >>>>>>>>>>>>>>> too). Templates offered a series of challenges > > > > >> such > > > > >>>> as > > > > >>>>> we > > > > >>>>>>>> store > > > > >>>>>>>>>> them > > > > >>>>>>>>>>>>>>> in the flow definition which has made flows > > > > >> massive > > > > >>>> in > > > > >>>>> an > > > > >>>>>>>>>> unintended > > > > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> We have a couple cases where we headed down a > > > > >>>>> particular > > > > >>>>>>>>> concept > > > > >>>>>>>>>> and > > > > >>>>>>>>>>>>>>> came up with better approaches later. We need to > > > > >>>>> reconcile > > > > >>>>>>>>> these > > > > >>>>>>>>>> with > > > > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful > > > > >>> to > > > > >>>> be > > > > >>>>> not > > > > >>>>>>>>>> overly > > > > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the > > > > >>>>>>>>> codebase/maintenance > > > > >>>>>>>>>>>>>>> burden and allow continued evolution of the > > > > >>> project. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Thanks > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > > >>>>>>>>>>>> r...@windofkeltia.com > > > > >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>> Joe, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what > > > > >>>>> replaces > > > > >>>>>>>>>> templates? > > > > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used > > > > >>> them > > > > >>>>> since > > > > >>>>>>>>> 0.5.x. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Russ > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote: > > > > >>>>>>>>>>>>>>> David, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and > > > > >>>> such a > > > > >>>>>>> focus > > > > >>>>>>>>>> will > > > > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more > > > > >>> approachable > > > > >>>>> to > > > > >>>>>>>> knock > > > > >>>>>>>>>> out. > > > > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help > > > > >>> make > > > > >>>>> work > > > > >>>>>>>>>> towards > > > > >>>>>>>>>>>>>>> major features we'd think about in a 'major > > > > >>> release' > > > > >>>>> sense > > > > >>>>>>>> more > > > > >>>>>>>>>>>>>>> approachable. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> We should remove all deprecated things (as well > > > > >> as > > > > >>>>> verify > > > > >>>>>>> we > > > > >>>>>>>>>> have the > > > > >>>>>>>>>>>>>>> right list). We should remove/consider removal of > > > > >>>>>>> deprecated > > > > >>>>>>>>>>>>>>> concepts > > > > >>>>>>>>>>>>>>> like templates. We should consider whether we can > > > > >>>>> resolve > > > > >>>>>>> the > > > > >>>>>>>>>>>>>>> various > > > > >>>>>>>>>>>>>>> ways we've handled what are now parameters down > > > > >> to > > > > >>>> one > > > > >>>>>>> clean > > > > >>>>>>>>>>>>>>> approach. > > > > >>>>>>>>>>>>>>> We should remove options in the nifi.properties > > > > >>> which > > > > >>>>> turn > > > > >>>>>>>> out > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> never be used quite right (if there are). There > > > > >> is > > > > >>>>> quite a > > > > >>>>>>>> bit > > > > >>>>>>>>> we > > > > >>>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Lots to consider here but I think this is the > > > > >> right > > > > >>>>>>>> discussion. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Than ks > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < > > > > >>>>>>>> bbe...@gmail.com > > > > >>>>>>>>>>>> <mailto: > > > > >>>>>>>>>>>>>>> bbe...@gmail.com>> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under > > > > >>>>> "Removing > > > > >>>>>>>>>>>>>>> Deprecated > > > > >>>>>>>>>>>>>>> Components", but I think we should also look at > > > > >>>>> anything > > > > >>>>>>> that > > > > >>>>>>>>> has > > > > >>>>>>>>>>>>>>> been > > > > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as > > > > >> a > > > > >>>>>>> candidate > > > > >>>>>>>>> for > > > > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods, > > > > >>>>>>> properties, > > > > >>>>>>>>> etc > > > > >>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>> have been waiting for a chance to be removed. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > >>>>>>>>>>>>>>> <exceptionfact...@apache.org<mailto: > > > > >>>>>>>>> exceptionfact...@apache.org > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>> Team, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> With all of the excellent work that many have > > > > >>>>> contributed > > > > >>>>>>> to > > > > >>>>>>>>> NiFi > > > > >>>>>>>>>>>>>>> over the > > > > >>>>>>>>>>>>>>> years, the code base has also accumulated some > > > > >>> amount > > > > >>>>> of > > > > >>>>>>>>>> technical > > > > >>>>>>>>>>>>>>> debt. A > > > > >>>>>>>>>>>>>>> handful of components have been marked as > > > > >>> deprecated, > > > > >>>>> and > > > > >>>>>>>> some > > > > >>>>>>>>>>>>>>> components > > > > >>>>>>>>>>>>>>> remain in the code base to support integration > > > > >> with > > > > >>>> old > > > > >>>>>>>>> versions > > > > >>>>>>>>>>>>>>> of various > > > > >>>>>>>>>>>>>>> products. Following the principles of semantic > > > > >>>>> versioning, > > > > >>>>>>>>>>>>>>> introducing a > > > > >>>>>>>>>>>>>>> major release would provide the opportunity to > > > > >>> remove > > > > >>>>> these > > > > >>>>>>>>>>>>>>> deprecated and > > > > >>>>>>>>>>>>>>> unsupported components. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Rather than focusing the next major release on > > > > >> new > > > > >>>>>>> features, > > > > >>>>>>>>> what > > > > >>>>>>>>>>>>>>> do you > > > > >>>>>>>>>>>>>>> think about focusing on technical debt removal? > > > > >>> This > > > > >>>>>>> approach > > > > >>>>>>>>>>>>>>> would not > > > > >>>>>>>>>>>>>>> make for the most interesting release, but it > > > > >>>> provides > > > > >>>>> the > > > > >>>>>>>>>>>>>>> opportunity to > > > > >>>>>>>>>>>>>>> clean up elements that involve breaking changes. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Focusing on technical debt, at least three > > > > >> primary > > > > >>>>> goals > > > > >>>>>>> come > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> mind for > > > > >>>>>>>>>>>>>>> the next major release: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained > > > > >>> components > > > > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported > > > > >> version > > > > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to > > > > >>> JSR > > > > >>>>> 310 > > > > >>>>>>>>>> java.time > > > > >>>>>>>>>>>>>>> components > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> *Removing Deprecated Components* > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Removing support for older and deprecated > > > > >>> components > > > > >>>>>>>> provides a > > > > >>>>>>>>>>>>>>> great > > > > >>>>>>>>>>>>>>> opportunity to improve the overall security > > > > >> posture > > > > >>>>> when it > > > > >>>>>>>>> comes > > > > >>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency > > > > >>> plugin > > > > >>>>>>> report > > > > >>>>>>>>>>>>>>> currently > > > > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable > > > > >>>> dependencies, > > > > >>>>> many > > > > >>>>>>>> of > > > > >>>>>>>>>>>>>>> which are > > > > >>>>>>>>>>>>>>> related to old versions of various libraries. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> As a starting point, here are a handful of > > > > >>> components > > > > >>>>> and > > > > >>>>>>>>>>>>>>> extension modules > > > > >>>>>>>>>>>>>>> that could be targeted for removal in a major > > > > >>>> version: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP > > > > >>>>>>>>>>>>>>> - ListenLumberjack and the entire > > > > >>>>> nifi-lumberjack-bundle > > > > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle > > > > >>>>>>>>>>>>>>> - Elasticsearch 5 components > > > > >>>>>>>>>>>>>>> - Hive 1 and 2 components > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> *Requiring Java 11* > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has > > > > >>>>> supported > > > > >>>>>>>>>> general > > > > >>>>>>>>>>>>>>> compatibility with Java 11 for several years. > > > > >> NiFi > > > > >>>>> 1.14.0 > > > > >>>>>>>>>>>>>>> incorporated > > > > >>>>>>>>>>>>>>> internal improvements specifically related to TLS > > > > >>>> 1.3, > > > > >>>>>>> which > > > > >>>>>>>>>>>>>>> allowed > > > > >>>>>>>>>>>>>>> closing out the long-running Java 11 > > > > >> compatibility > > > > >>>> epic > > > > >>>>>>>>>> NIFI-5174. > > > > >>>>>>>>>>>>>>> Making > > > > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the > > > > >>>>>>> opportunity > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> address > > > > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better > > > > >>>>> position > > > > >>>>>>> to > > > > >>>>>>>>>>>>>>> support > > > > >>>>>>>>>>>>>>> current Java versions. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling* > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning > > > > >>>>> internal > > > > >>>>>>>> date > > > > >>>>>>>>>>>>>>> and time > > > > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of > > > > >>>>>>> SimpleDateFormat > > > > >>>>>>>>>>>>>>> would provide > > > > >>>>>>>>>>>>>>> a number of advantages. The Java Time components > > > > >>>>> provide > > > > >>>>>>> much > > > > >>>>>>>>>>>>>>> better > > > > >>>>>>>>>>>>>>> clarity when it comes to handling localized date > > > > >>> and > > > > >>>>> time > > > > >>>>>>>>>>>>>>> representations, > > > > >>>>>>>>>>>>>>> and also avoid the inherent confusion of > > > > >>>> java.sql.Date > > > > >>>>>>>>> extending > > > > >>>>>>>>>>>>>>> java.util.Date. Many internal components, > > > > >>>> specifically > > > > >>>>>>>>>>>>>>> Record-oriented > > > > >>>>>>>>>>>>>>> processors and services, rely on date parsing, > > > > >>>> leading > > > > >>>>> to > > > > >>>>>>>>>>>>>>> confusion and > > > > >>>>>>>>>>>>>>> various workarounds. The pattern formats of > > > > >>>>>>> SimpleDateFormat > > > > >>>>>>>>> and > > > > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there > > > > >> are a > > > > >>>> few > > > > >>>>>>>> subtle > > > > >>>>>>>>>>>>>>> differences. > > > > >>>>>>>>>>>>>>> Making this transition would provide a much > > > > >> better > > > > >>>>>>> foundation > > > > >>>>>>>>>>>>>>> going forward. > > > > >>>>>>>>>>>>>>> *Conclusion* > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Thanks for giving this proposal some > > > > >> consideration. > > > > >>>>> Many of > > > > >>>>>>>> you > > > > >>>>>>>>>>>>>>> have been > > > > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to > > > > >>> your > > > > >>>>>>>> feedback. > > > > >>>>>>>>> I > > > > >>>>>>>>>>>>>>> would be > > > > >>>>>>>>>>>>>>> glad to put together a more formalized > > > > >>> recommendation > > > > >>>>> on > > > > >>>>>>>>>>>>>>> Confluence and > > > > >>>>>>>>>>>>>>> write up Jira epics if this general approach > > > > >> sounds > > > > >>>>>>> agreeable > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>> community. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Regards, > > > > >>>>>>>>>>>>>>> David Handermann > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > >
