I'll preface with what I'm about to say is not rooted in any real understanding of the lift required or other customer experiences... We have used NiFi for a long time and are excited for what new features may lay ahead in the 2.x line.
---After having written this, I think our succinct pain points focus around large data flow management, auto-scaling container environments, and integration of external analytic/enrichment creation.--- Thanks, Ryan 1 - Integration with auto-scaling container environments We often struggle with scaling custom processors in large flows. We've implemented a recent design pattern where we write custom python code packaged into a docker image, fronted with a web-service, and deployed into a container environment. In NiFi we use InvokeHTTP to call out to the web-service. If the load becomes too great, we can easily spin-up more than one external web-service without sacrificing the NiFi's overall thruput and thread contention. In this scenario, we're using NiFi to manage the overall dataflow, routes, ETL services, etc, but not so much smarts around auto-scaling processing, etc. This makes it really easy to receive a dataflow analytic/enrichment from an external team, wrap it in a docker image, and deploy it into an auto-scaling container environment. This also lets us avoid conversations about how performant the code is and if it's meeting a minimum throughput requirement - typically XXGB/5minutes. The request for 2.x would be to make interacting with container deployed services a more seamless, streamlined process, possibly through a normalized API or as a "Remote Container Environment" that can bind to a service with a health endpoint and verify it's alive and well, etc. 2 - Integration of S3 buckets with Remote Process Groups for outage events and data overruns Another frequent struggle is the occasional death of a server in a NiFi flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together with Remote Process Groups to perform the overall dataflow goals. If any one of the 15 experience an outage, data begins to backup on the previous NiFi, hitting backpressure limits, etc. When the down server comes back online, it receives a flood of data, often overwhelming it, necessitating careful flowfile limits and backpressure points. The design pattern we're exploring, but would like to see baked into NiFi, is that when a Remote Process Group becomes unavailable due to server-to-server comms failures AND the backpressure limit is reached on a relationship, the Remote Process Group would begin sending data to an S3 bucket as an automatic failover procedure. When the downed server comes back-online, the Remote Process Group port would know to pull data from S3 AND receive data via the port from the chained server. 3 - Large Canvas Management Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2 NiFi clusters, each canvas is managed individually. To see the entire canvas, systems engineering diagrams need to be kept up-to-date identifying all the input/output ports. To move processors from server to server often requires the verification of the existence of scripts, configuration files, NiFi custom nars, and using the templating or Registry to migrate portions of the dataflow. Our current design pattern to accomplish some of these tasks include using Ansible, Jenkins, and Nexus to verify script & configuration files are up to date. We use Confluence's Gliffy to document the NiFi communication paths between the servers. Ideally, we'd be able to see an overall management view of the Canvases together in a single place - it would indicate which server the portions of the canvas are deployed on and tie together where the Remote Process Group ports are connected. Additionally, NiFi would allow the user to move Processors and Process Groups seamlessly from one server to another server to help manually distribute and balance data flow processing cycles. On Tue, Aug 3, 2021 at 12:45 PM Joe Witt <joe.w...@gmail.com> wrote: > Kevin, > > This may well be a very interesting approach which opens up a lot of > options for us in how we tackle a 2.x... > > Thanks > > On Tue, Aug 3, 2021 at 8:45 AM Kevin Doran <kdoran.apa...@gmail.com> > wrote: > > > > Outside of core NiFi, for MiNiFi, I think it would be beneficial to > align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and > control / flow deployment. This was discussed when MiNiFi Java was merged > into the NiFi codebase [1]. > > > > My proposal would be to create a Java implementation of the C2 Protocol > used by MiNiFi C++, both server-side and client-side, to allow deploying to > MiNiFi Java instances from a centralize flow definition. Thanks to the > unification of MiNiFi Java and NiFi, this could likely also be added as a > capability of NiFi at that same time. > > > > If we are able to do this on the 1.x line, then I would also support > deprecating other methods of flow deployment to MiNiFi outlined in [1] and > removing them in a 2.x release. > > > > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 < > https://github.com/apache/nifi/pull/4933#issuecomment-808411977> > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design < > https://cwiki.apache.org/confluence/display/MINIFI/C2+Design> > > > > Thanks, > > Kevin > > > > > On Aug 3, 2021, at 10:07 AM, David Handermann < > exceptionfact...@gmail.com> wrote: > > > > > > Thanks for the continued discussion around what can or should be > removed. > > > Having a clear migration path from version 1 to version 2 will be very > > > important for supporting current deployments. > > > > > > Following the example of some other projects, one way to consider the > > > upgrade is from the point of view of deprecation warnings. If > implemented > > > correctly, an installation of NiFi running the latest minor release of > > > version 1, and showing no deprecation warnings, should be upgradeable > to > > > version 2 without any changes. Making this work involves accurately > tagging > > > components and methods with deprecation indicators, and providing a > clear > > > way to observe deprecation warnings. With the current version > containing > > > both deprecated components and deprecated methods in various classes, > this > > > would involve some work to get right. A simple approach might be a > named > > > logger for deprecation warnings, which would write a separate log > file. A > > > more advanced approach might involve a tool to analyze the system and > flow > > > configurations. Either way, it seems that some additional work in a > minor > > > release version is necessary before considering a major release > version. > > > > > > One other point on compatibility: as long as the core component API > > > definition remains backwards compatible, it should be possible to run > older > > > components. This provides a transition option for customers using > > > components such as PostHTTP, or any others that are not actively > > > maintained. At some point it may become necessary to break > compatibility at > > > that level, but at least for version 2, maintaining component API > > > compatibility is key. > > > > > > Regards, > > > David Handermann > > > > > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <mark.o.b...@gmail.com> > wrote: > > > > > >> I created a JIRA ticket to investigate and improve the performance of > > >> InvokeHTTP. It includes a flow definition for benchmarking the > performance > > >> of both PostHTTP and InvokeHTTP. > > >> > > >> https://issues.apache.org/jira/browse/NIFI-8968 > > >> > > >> Thanks, > > >> Mark > > >> > > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <a...@adamtaft.com> wrote: > > >> > > >>> I'm not seeing the side thread that was going to discuss deprecation > of > > >>> PostHTTP. Has that thread started and I just don't see it? > > >>> > > >>> One (significant?) concern with PostHTTP is the smooth integration of > > >>> NiFi-to-NiFi communication that is very transparently enabled with > the > > >>> ListenHTTP and PostHTTP processors. There's some special logic in > there > > >> for > > >>> handling flowfiles that InvokeHTTP doesn't really (nor should really) > > >> have. > > >>> > > >>> I know of several (large) NiFi installations that rely on the > PostHTTP / > > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication for > > >> folks > > >>> reluctant or unable to enable site-to-site type configuration. > > >>> > > >>> Honestly, I don't know why we'd want to "deprecate" any processor, as > > >>> opposed to just moving it to a new location. If these processors can > be > > >>> ported and maintained to whatever the 2.0 API looks like, there's > > >> possibly > > >>> little harm keeping them around. > > >>> > > >>> And by the way, what happened to the "marketplace" concept? Is this > being > > >>> considered for 2.0 as well? Because relocating the deprecated > processors > > >>> to an external nar might be the best solution. Losing PostHTTP > entirely I > > >>> think would be a mistake, but I'd conceptually support relocating it. > > >>> > > >>> Thanks, > > >>> > > >>> /Adam > > >>> > > >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <joe.w...@gmail.com> wrote: > > >>> > > >>>> Looks like we just need to knock out 5 JIRAs :) [1] > > >>>> > > >>>> I felt like we had a label folks were using at one point but quickly > > >>>> looking revealed nothing exciting. After this confluence page > > >>>> stabilizes a bit we can probably knock out some JIRAs and such. > > >>>> > > >>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > >>>> > > >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler < > ottobackwa...@gmail.com> > > >>>> wrote: > > >>>>> > > >>>>> I find myself wishing I had a list of all the jiras / issues that > > >> have > > >>>>> been put off for a 2.0 release because they required some change or > > >>>> another > > >>>>> :( > > >>>>> > > >>>>> From: Joe Witt <joe.w...@gmail.com> <joe.w...@gmail.com> > > >>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < > > >> dev@nifi.apache.org> > > >>>>> Date: July 27, 2021 at 12:30:35 > > >>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org > > > > >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > >>>>> > > >>>>> A few thoughts: > > >>>>> > > >>>>> 1. I would love to see deprecation notices show up in the UI in > > >>>>> various ways to help motivate users to move off things to more > > >>>>> supportable things. That is not a prerequisite for anything > happening > > >>>>> however. Just a good feature/nice thing to do for users when > someone > > >>>>> is able to tackle it. > > >>>>> > > >>>>> 2. The decision to deprecate something and to further remove it > need > > >>>>> not mean there is a superior solution available. If that thing > itself > > >>>>> isn't getting the love/attention it needs to be > > >>>>> maintained/supported/healthy going forward that alone is enough to > > >>>>> remove it. That might well be the case with PostHTTP [1] and for > > >>>>> comparison you can see how much effort has gone into InvokeHTTP > [2]. > > >>>>> > > >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do' > the > > >>>>> further away from reality such a release will become. We'll have to > > >>>>> get very specific about 'musts' vs 'wants'. > > >>>>> > > >>>>> [1] > > >>>>> > > >>>> > > >>> > > >> > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > >>>>> [2] > > >>>>> > > >>>> > > >>> > > >> > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > >>>>> > > >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > >>>>> <exceptionfact...@apache.org> wrote: > > >>>>>> > > >>>>>> Thanks Mark, providing a template or comparison statistics with > > >> Java > > >>>>>> versions and component configuration details would be very > helpful. > > >>> If > > >>>> it > > >>>>>> is possible to run tests using a public API or deployable service, > > >>> that > > >>>>>> would also help confirm potential differences. > > >>>>>> > > >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps > > >> as a > > >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis > > >>>>>> capability, and it seems like that might be one possible way to > > >>> surface > > >>>>>> deprecation warnings. For something more specific to component > > >>>>> deprecation, > > >>>>>> it seems important to find a balance between making it obvious and > > >>>> making > > >>>>>> it something that ends up getting ignored. > > >>>>>> > > >>>>>> Regards, > > >>>>>> David Handermann > > >>>>>> > > >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <mark.o.b...@gmail.com > > > > >>>> wrote: > > >>>>>> > > >>>>>>> I'll start a new thread for PostHTTP when I get a template and/or > > >>>>> detailed > > >>>>>>> stats. > > >>>>>>> > > >>>>>>> I know the deprecation is noted in the documentation. That's a > > >>>>> necessary > > >>>>>>> and minimum level of notification. I was suggesting it be more > > >>>> obvious > > >>>>> in > > >>>>>>> the UI. I think it would be beneficial to somehow be aware of the > > >>>>>>> deprecation status simply by looking at the flow (perhaps even on > > >>> the > > >>>>>>> summary pages too), and without having to open the documentation > > >>> for > > >>>>> every > > >>>>>>> processor to confirm whether or not a component is marked as > > >>>>> deprecated. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Mark > > >>>>>>> > > >>>>>>> > > >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > > >>>>>>> exceptionfact...@apache.org> wrote: > > >>>>>>> > > >>>>>>>> Mark, > > >>>>>>>> > > >>>>>>>> Thanks for the feedback. It may be better to start a separate > > >>>> thread > > >>>>> on > > >>>>>>>> PostHTTP, but can you provide an example flow demonstrating the > > >>>>>>> performance > > >>>>>>>> differences between PostHTTP and InvokeHTTP? > > >>>>>>>> > > >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas > > >>> InvokeHTTP > > >>>>> uses > > >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates > > >> for > > >>>>> OkHttp, > > >>>>>>>> so it would be important to test with the most recent version. > > >> It > > >>>> is > > >>>>> also > > >>>>>>>> worth noting that test classes for PostHTTP are now integration > > >>>> tests > > >>>>> as > > >>>>>>>> opposed to unit tests, which means they are not executed as > > >> part > > >>> of > > >>>>> the > > >>>>>>>> automated builds. > > >>>>>>>> > > >>>>>>>> The NiFi documentation includes the contents of the > > >>>> DeprecationNotice > > >>>>> for > > >>>>>>>> PostHTTP: > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >>> > > >> > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > >>>>>>>> > > >>>>>>>> Regards, > > >>>>>>>> David Handermann > > >>>>>>>> > > >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean < > > >> mark.o.b...@gmail.com > > >>>> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving > > >>>>> deliberately > > >>>>>>> to a > > >>>>>>>>> 2.0 release. I have a concern with just one processor that is > > >>>>> currently > > >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated > > >>>> specifically > > >>>>> any > > >>>>>>>>> other deprecated components; I cannot say if there are or are > > >>> not > > >>>>>>> similar > > >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating > > >>>> this > > >>>>>>>>> processor in that it eliminates a processor whose > > >> functionality > > >>>> is > > >>>>>>>>> available elsewhere, specifically in the more flexible > > >>>> InvokeHTTP. > > >>>>>>>> However, > > >>>>>>>>> in my experience and testing, PostHTTP performs transfers > > >> with > > >>>> far > > >>>>>>>> greater > > >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of > > >> removing > > >>>>>>> PostHTTP > > >>>>>>>>> unless/until InvokeHTTP is refactored to increase its > > >>> throughput > > >>>>>>>>> capability. > > >>>>>>>>> > > >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for > > >>>>> similar > > >>>>>>>>> reasons? Or, is there a performance-related configuration for > > >>>>>>> InvokeHTTP > > >>>>>>>> I > > >>>>>>>>> may have missed? > > >>>>>>>>> > > >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0 > > >>>> from a > > >>>>>>> user > > >>>>>>>>> perspective, would it be advisable to add some sort of visual > > >>>>> indicator > > >>>>>>>> in > > >>>>>>>>> the UI for components that are currently annotated as > > >>>> @Deprecated? > > >>>>> This > > >>>>>>>>> might help users proactively modify their flow prior to a > > >>> release > > >>>>> that > > >>>>>>>>> would otherwise break it. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende < > > >> bbe...@gmail.com> > > >>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo, > > >>>> we've > > >>>>>>>>>> actually gone more towards a mono-repo where everything is > > >>>>> released > > >>>>>>>>>> together. Eventually it would still be nice to have a > > >> smaller > > >>>>> base > > >>>>>>>>>> distribution containing just the framework and standard > > >> NARs, > > >>>> but > > >>>>> it > > >>>>>>>>>> is hard to tackle that until we provide a way for users to > > >>>> easily > > >>>>>>>>>> obtain the NARs in some other way. > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < > > >>>>> edward.ar...@gmail.com > > >>>>>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> Given the major version number shift and the spliting up > > >> of > > >>>>>>>> processors > > >>>>>>>>>> into > > >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start > > >>>> individually > > >>>>>>>>> versioning > > >>>>>>>>>>> individual NARs/ bundles. > > >>>>>>>>>>> > > >>>>>>>>>>> I can see this bringing a large number of benifits > > >>> including > > >>>>> making > > >>>>>>>>> Nifi > > >>>>>>>>>>> more deployable with things RPM, but also potentially > > >>>> allowing > > >>>>>>> those > > >>>>>>>>> that > > >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade. > > >>>>>>>>>>> > > >>>>>>>>>>> Edward > > >>>>>>>>>>> > > >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, < > > >>>> ottobackwa...@gmail.com> > > >>>>> > > >>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any > > >>> one > > >>>>> of > > >>>>>>> the > > >>>>>>>>>>>> processors. > > >>>>>>>>>>>> the Web Gateway API invoke processor for example is not > > >>>> using > > >>>>> a > > >>>>>>>> high > > >>>>>>>>>> level > > >>>>>>>>>>>> purpose build client and may break. > > >>>>>>>>>>>> > > >>>>>>>>>>>> If we change the aws version, we need to coordinate in > > >>>> such a > > >>>>> way > > >>>>>>>>> that > > >>>>>>>>>> they > > >>>>>>>>>>>> all > > >>>>>>>>>>>> can come along reasonably. > > >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK? > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> From: David Handermann <exceptionfact...@apache.org> > > >>>>>>>>>>>> <exceptionfact...@apache.org> > > >>>>>>>>>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < > > >>>>>>>>> dev@nifi.apache.org> > > >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42 > > >>>>>>>>>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> < > > >>>>>>> dev@nifi.apache.org > > >>>>>>>>> > > >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > >>>>>>>>>>>> > > >>>>>>>>>>>> Chris, > > >>>>>>>>>>>> > > >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like > > >>>> some > > >>>>> of > > >>>>>>> the > > >>>>>>>>>> work to > > >>>>>>>>>>>> reorganize the module structure could be done outside > > >> of > > >>> a > > >>>>> major > > >>>>>>>>>> release, > > >>>>>>>>>>>> but it would be great to target any breaking changes > > >> for > > >>>> 2.0. > > >>>>>>>>> Perhaps a > > >>>>>>>>>>>> separate feature proposal on module restructuring, with > > >>> the > > >>>>> goal > > >>>>>>> of > > >>>>>>>>>>>> supporting optimized builds, would be a helpful way to > > >>> move > > >>>>> that > > >>>>>>>> part > > >>>>>>>>>> of > > >>>>>>>>>>>> the discussion forward. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like > > >>> that > > >>>>> might > > >>>>>>>> be > > >>>>>>>>>>>> possible now. I haven't taken a close look at the > > >>>> referencing > > >>>>>>>>>> components, > > >>>>>>>>>>>> so I'm not sure about the level of effort involved. > > >> Minor > > >>>>> NiFi > > >>>>>>>>> version > > >>>>>>>>>>>> updates have incorporated major new versions of > > >>>> dependencies. > > >>>>> For > > >>>>>>>>>> example, > > >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4 > > >> to > > >>> 5. > > >>>>> On > > >>>>>>> the > > >>>>>>>>> one > > >>>>>>>>>>>> hand, including the AWS SDK update as part of a major > > >>>> release > > >>>>>>> seems > > >>>>>>>>>>>> helpful, but unless there are changes that break > > >> existing > > >>>>>>> component > > >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked > > >>>>> independently. > > >>>>>>>>>> Others may > > >>>>>>>>>>>> have more insight into particular usage of that > > >> library. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Regards, > > >>>>>>>>>>>> David Handermann > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > >>>>>>>>>>>> <chris.samp...@naimuri.com.invalid> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Might be worth considering refactoring the build as > > >>> part > > >>>> of > > >>>>>>> this > > >>>>>>>>> work > > >>>>>>>>>>>> too, > > >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a > > >>>>> commit, > > >>>>>>>> etc. > > >>>>>>>>> - > > >>>>>>>>>>>>> discussed briefly in previous threads but don't think > > >>> any > > >>>>>>> changes > > >>>>>>>>>> made > > >>>>>>>>>>>> yet. > > >>>>>>>>>>>>> If NARs/components are likely to be split up and > > >>>> refactored > > >>>>>>> then > > >>>>>>>>> such > > >>>>>>>>>>>> work > > >>>>>>>>>>>>> around the build probably makes sense to consider. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I've a couple of PRs open that include updates to > > >>>>> Elasticsearch > > >>>>>>>>>> versions > > >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which > > >>>> Elastic > > >>>>>>>> changed > > >>>>>>>>>>>> licence) > > >>>>>>>>>>>>> in case there were licence concerns. But more work > > >> can > > >>> be > > >>>>> done > > >>>>>>> to > > >>>>>>>>>> tidy up > > >>>>>>>>>>>>> the processors, absolutely. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a > > >>>>> refactor > > >>>>>>> of > > >>>>>>>>>> those > > >>>>>>>>>>>>> processors as well. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Chris Sampson > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, < > > >>>>>>>>>>>> exceptionfact...@apache.org> > > >>>>>>>>>>>> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles, > > >>> Mark. > > >>>>> There > > >>>>>>>>> are a > > >>>>>>>>>>>>> number > > >>>>>>>>>>>>>> of components in the standard NAR bundles with > > >>>> particular > > >>>>>>>>>> dependencies > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>> would make more sense in separate NARs. > > >> Reorganizing > > >>>> the > > >>>>>>>> standard > > >>>>>>>>>> NAR > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> components with limited dependencies and wide > > >>>>> applicability > > >>>>>>>> would > > >>>>>>>>>>>>>> definitely help with future maintenance. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>> David Handermann > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > > >>>>>>>>> marka...@hotmail.com> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> There’s also some code that exists in order to > > >>>> maintain > > >>>>>>>>> backward > > >>>>>>>>>>>>>>> compatibility in the repositories. I would very > > >>> much > > >>>>> like > > >>>>>>> the > > >>>>>>>>>>>>>> repositories > > >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file > > >>> format > > >>>>>>> supports > > >>>>>>>>>> really > > >>>>>>>>>>>>> old > > >>>>>>>>>>>>>>> formats. And the old impls of the repositories > > >>>>> themselves, > > >>>>>>>> like > > >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv > > >> Repo, > > >>>> etc. > > >>>>>>> Lots > > >>>>>>>> of > > >>>>>>>>>> stuff > > >>>>>>>>>>>>>> there > > >>>>>>>>>>>>>>> that can be removed. And some methods in > > >>>> ProcessSession > > >>>>>>> that > > >>>>>>>>> are > > >>>>>>>>>>>> never > > >>>>>>>>>>>>>> used > > >>>>>>>>>>>>>>> by any processor in the codebase but exists in > > >> the > > >>>>> public > > >>>>>>> API > > >>>>>>>>> so > > >>>>>>>>>>>> can’t > > >>>>>>>>>>>>> be > > >>>>>>>>>>>>>>> removed till 2.0. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I think his is also a great time to clean up the > > >>>>> “standard > > >>>>>>>>> nar.” > > >>>>>>>>>> At > > >>>>>>>>>>>>> this > > >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the > > >>>>>>> components > > >>>>>>>>>> there > > >>>>>>>>>>>> are > > >>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>> really “standard” - things like connecting to > > >> FTP & > > >>>>> SFTP > > >>>>>>>>>> servers, XML > > >>>>>>>>>>>>>>> processing, Jolt transform, etc. could > > >> potentially > > >>> be > > >>>>> moved > > >>>>>>>>> into > > >>>>>>>>>>>> other > > >>>>>>>>>>>>>>> nars. The > > >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war > > >>>>>>> is > > >>>>>>>>>> 6.9 MB > > >>>>>>>>>>>> is > > >>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of > > >>>> things > > >>>>>>>> probably > > >>>>>>>>>> to > > >>>>>>>>>>>>>>> reconsider within the standard nar. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I definitely think this is a reasonable approach, > > >>> to > > >>>>> allow > > >>>>>>>> for > > >>>>>>>>> a > > >>>>>>>>>> 2.0 > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>>> is not a huge feature release but allows the > > >>> project > > >>>> to > > >>>>> be > > >>>>>>>>>> simpler > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>> more > > >>>>>>>>>>>>>>> nimble. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>>>> -Mark > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > >>>>>>>>>> mikerthom...@gmail.com > > >>>>>>>>>>>>>> <mailto: > > >>>>>>>>>>>>>>> mikerthom...@gmail.com>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Russell, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low > > >>> level > > >>>>> REST > > >>>>>>>>>> client is > > >>>>>>>>>>>>>>> still fine. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>> > > >>> > > >> > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs > > >>> at > > >>>>>>> present. > > >>>>>>>>> One > > >>>>>>>>>>>> uses > > >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic > > >>> REST > > >>>>>>> client. > > >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for > > >> the > > >>>>> moment. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Mike > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > >>>>>>>>>>>> r...@windofkeltia.com > > >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the > > >>> Elastic > > >>>>>>>> framework > > >>>>>>>>>> has > > >>>>>>>>>>>> just > > >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to > > >>>>> acknowledge > > >>>>>>>>> that, > > >>>>>>>>>>>> maybe > > >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not > > >>>>> understanding > > >>>>>>>>>> exactly > > >>>>>>>>>>>> how > > >>>>>>>>>>>>>>> this sort of thing is considered in a > > >> large-scale, > > >>>>>>>> world-class > > >>>>>>>>>>>> software > > >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor, > > >>>> just > > >>>>> a > > >>>>>>>>> grateful > > >>>>>>>>>>>>>>> consumer.) > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Russ > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote: > > >>>>>>>>>>>>>>> Along with the itemized list for ancient > > >> components > > >>>> we > > >>>>>>> should > > >>>>>>>>>> look at > > >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for > > >>> external > > >>>>>>> systems > > >>>>>>>>>> such as > > >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be > > >>> breaking > > >>>>>>> changes > > >>>>>>>>> but > > >>>>>>>>>> 2.0 > > >>>>>>>>>>>>>>> is probably the right time to get things up to > > >> date > > >>>> to > > >>>>> make > > >>>>>>>>> them > > >>>>>>>>>> more > > >>>>>>>>>>>>>>> useful to more people. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > >>>>>>>>>> thena...@gmail.com > > >>>>>>>>>>>>>> <mailto: > > >>>>>>>>>>>>>>> thena...@gmail.com>> wrote: > > >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this > > >>> stuff. > > >>>>> There > > >>>>>>>> are > > >>>>>>>>>>>> security > > >>>>>>>>>>>>>>> implications to keeping old dependencies around, > > >> so > > >>>> the > > >>>>>>> more > > >>>>>>>>> old > > >>>>>>>>>> code > > >>>>>>>>>>>>> we > > >>>>>>>>>>>>>>> can remove the better. I agree that eventually we > > >>>> need > > >>>>> to > > >>>>>>>> move > > >>>>>>>>> to > > >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release > > >>>> will > > >>>>>>>> probably > > >>>>>>>>>> be > > >>>>>>>>>>>>> about > > >>>>>>>>>>>>>> 4 > > >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon. > > >> We > > >>>>> could > > >>>>>>>>>> potentially > > >>>>>>>>>>>>>> break > > >>>>>>>>>>>>>>> this in two and remove the deprecated processors > > >>> and > > >>>>> leave > > >>>>>>>> 1.x > > >>>>>>>>> on > > >>>>>>>>>>>> Java > > >>>>>>>>>>>>> 8, > > >>>>>>>>>>>>>>> and finally start on 2.x which would support only > > >>>> Java > > >>>>> 11. > > >>>>>>>> I'm > > >>>>>>>>>> unsure > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>>> what implications changing the date and time > > >>> handling > > >>>>> would > > >>>>>>>>> have > > >>>>>>>>>> - > > >>>>>>>>>>>> for > > >>>>>>>>>>>>>>> running systems that use long term historical > > >> logs, > > >>>>>>>> unexpected > > >>>>>>>>>>>> impacts > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>>> time logging could be a problem. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> As Joe says I think feature work will have to be > > >>>>> dedicated > > >>>>>>> to > > >>>>>>>>>> 2.x and > > >>>>>>>>>>>>> we > > >>>>>>>>>>>>>>> could support 1.x for security fixes for some > > >>> period > > >>>> of > > >>>>>>> time. > > >>>>>>>>> 2.x > > >>>>>>>>>>>> seems > > >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to > > >>> get > > >>>>>>> started. > > >>>>>>>>> Not > > >>>>>>>>>>>> sure > > >>>>>>>>>>>>>> how > > >>>>>>>>>>>>>>> we handle all open PRs and the transition between > > >>> 1.x > > >>>>> and > > >>>>>>>> 2.x. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < > > >>>>>>>> joe.w...@gmail.com > > >>>>>>>>>>>> <mailto: > > >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Jon > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> You're right we have to be careful and you're > > >> right > > >>>>> there > > >>>>>>> are > > >>>>>>>>>> still > > >>>>>>>>>>>>>>> significant Java 8 users out there. But we also > > >>> have > > >>>> to > > >>>>> be > > >>>>>>>>>> careful > > >>>>>>>>>>>>>>> about security and sustainability of the > > >> codebase. > > >>> If > > >>>>> we > > >>>>>>> had > > >>>>>>>>>> talked > > >>>>>>>>>>>>>>> about this last year when that article came out > > >> I'd > > >>>>> have > > >>>>>>>> agreed > > >>>>>>>>>> it is > > >>>>>>>>>>>>>>> too early. Interestingly that link seems to get > > >>>> updated > > >>>>>>> and I > > >>>>>>>>>> tried > > >>>>>>>>>>>>>>> [1] and found more recent data (not sure how > > >>> recent). > > >>>>>>> Anyway > > >>>>>>>> it > > >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see > > >>> good > > >>>>> growth > > >>>>>>>> on > > >>>>>>>>>> 11. In > > >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too. > > >>>>> Customers > > >>>>>>>> didn't > > >>>>>>>>>> seem > > >>>>>>>>>>>>>>> to care about Java 11 until later half last year > > >>> and > > >>>>> now > > >>>>>>>>>> suddenly it > > >>>>>>>>>>>>>>> is all over the place. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd > > >> see > > >>>>> rapid > > >>>>>>>>>> decrease in > > >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did > > >> this > > >>>> many > > >>>>>>> years > > >>>>>>>>> ago > > >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a > > >> while > > >>>>> (maybe > > >>>>>>> a > > >>>>>>>>>> year or > > >>>>>>>>>>>>>>> so) but it was purely bug fix/security related > > >>> bits. > > >>>> We > > >>>>>>> would > > >>>>>>>>>> need to > > >>>>>>>>>>>>>>> do something similar. But feature work would > > >> almost > > >>>>>>> certainly > > >>>>>>>>> go > > >>>>>>>>>> to > > >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable > > >> models > > >>>> but > > >>>>> my > > >>>>>>>>>> instinct > > >>>>>>>>>>>>>>> suggests this is likely to follow a similar path. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to > > >>>> dump > > >>>>> Java > > >>>>>>>> 8. > > >>>>>>>>> We > > >>>>>>>>>>>>>>> need to make the call in both the interests of > > >> the > > >>>> user > > >>>>>>> base > > >>>>>>>>> and > > >>>>>>>>>> the > > >>>>>>>>>>>>>>> contributor base of the community. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> [1] > > >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/ > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>>>> Joe > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < > > >>>>>>> joe.w...@gmail.com > > >>>>>>>>>> <mailto: > > >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: > > >>>>>>>>>>>>>>> Russ > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But > > >>> also > > >>>>> now > > >>>>>>> you > > >>>>>>>>> can > > >>>>>>>>>>>>>>> download the flow definition in JSON (upload i > > >>> think > > >>>> is > > >>>>>>> there > > >>>>>>>>> now > > >>>>>>>>>>>>>>> too). Templates offered a series of challenges > > >> such > > >>>> as > > >>>>> we > > >>>>>>>> store > > >>>>>>>>>> them > > >>>>>>>>>>>>>>> in the flow definition which has made flows > > >> massive > > >>>> in > > >>>>> an > > >>>>>>>>>> unintended > > >>>>>>>>>>>>>>> way which isn't fun for cluster behavior. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> We have a couple cases where we headed down a > > >>>>> particular > > >>>>>>>>> concept > > >>>>>>>>>> and > > >>>>>>>>>>>>>>> came up with better approaches later. We need to > > >>>>> reconcile > > >>>>>>>>> these > > >>>>>>>>>> with > > >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful > > >>> to > > >>>> be > > >>>>> not > > >>>>>>>>>> overly > > >>>>>>>>>>>>>>> disruptive to existing users, to reduce the > > >>>>>>>>> codebase/maintenance > > >>>>>>>>>>>>>>> burden and allow continued evolution of the > > >>> project. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > >>>>>>>>>>>> r...@windofkeltia.com > > >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> Joe, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what > > >>>>> replaces > > >>>>>>>>>> templates? > > >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used > > >>> them > > >>>>> since > > >>>>>>>>> 0.5.x. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Russ > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote: > > >>>>>>>>>>>>>>> David, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I think this is a highly reasonable approach and > > >>>> such a > > >>>>>>> focus > > >>>>>>>>>> will > > >>>>>>>>>>>>>>> greatly help make a 2.0 release far more > > >>> approachable > > >>>>> to > > >>>>>>>> knock > > >>>>>>>>>> out. > > >>>>>>>>>>>>>>> Not only that but tech debt reduction would help > > >>> make > > >>>>> work > > >>>>>>>>>> towards > > >>>>>>>>>>>>>>> major features we'd think about in a 'major > > >>> release' > > >>>>> sense > > >>>>>>>> more > > >>>>>>>>>>>>>>> approachable. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> We should remove all deprecated things (as well > > >> as > > >>>>> verify > > >>>>>>> we > > >>>>>>>>>> have the > > >>>>>>>>>>>>>>> right list). We should remove/consider removal of > > >>>>>>> deprecated > > >>>>>>>>>>>>>>> concepts > > >>>>>>>>>>>>>>> like templates. We should consider whether we can > > >>>>> resolve > > >>>>>>> the > > >>>>>>>>>>>>>>> various > > >>>>>>>>>>>>>>> ways we've handled what are now parameters down > > >> to > > >>>> one > > >>>>>>> clean > > >>>>>>>>>>>>>>> approach. > > >>>>>>>>>>>>>>> We should remove options in the nifi.properties > > >>> which > > >>>>> turn > > >>>>>>>> out > > >>>>>>>>> to > > >>>>>>>>>>>>>>> never be used quite right (if there are). There > > >> is > > >>>>> quite a > > >>>>>>>> bit > > >>>>>>>>> we > > >>>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>> do purely in the name of tech debt reduction. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Lots to consider here but I think this is the > > >> right > > >>>>>>>> discussion. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Than ks > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < > > >>>>>>>> bbe...@gmail.com > > >>>>>>>>>>>> <mailto: > > >>>>>>>>>>>>>>> bbe...@gmail.com>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under > > >>>>> "Removing > > >>>>>>>>>>>>>>> Deprecated > > >>>>>>>>>>>>>>> Components", but I think we should also look at > > >>>>> anything > > >>>>>>> that > > >>>>>>>>> has > > >>>>>>>>>>>>>>> been > > >>>>>>>>>>>>>>> marked as deprecated throughout the code base as > > >> a > > >>>>>>> candidate > > >>>>>>>>> for > > >>>>>>>>>>>>>>> removal. There are quite a few classes, methods, > > >>>>>>> properties, > > >>>>>>>>> etc > > >>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>> have been waiting for a chance to be removed. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > >>>>>>>>>>>>>>> <exceptionfact...@apache.org<mailto: > > >>>>>>>>> exceptionfact...@apache.org > > >>>>>>>>>>>> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> Team, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> With all of the excellent work that many have > > >>>>> contributed > > >>>>>>> to > > >>>>>>>>> NiFi > > >>>>>>>>>>>>>>> over the > > >>>>>>>>>>>>>>> years, the code base has also accumulated some > > >>> amount > > >>>>> of > > >>>>>>>>>> technical > > >>>>>>>>>>>>>>> debt. A > > >>>>>>>>>>>>>>> handful of components have been marked as > > >>> deprecated, > > >>>>> and > > >>>>>>>> some > > >>>>>>>>>>>>>>> components > > >>>>>>>>>>>>>>> remain in the code base to support integration > > >> with > > >>>> old > > >>>>>>>>> versions > > >>>>>>>>>>>>>>> of various > > >>>>>>>>>>>>>>> products. Following the principles of semantic > > >>>>> versioning, > > >>>>>>>>>>>>>>> introducing a > > >>>>>>>>>>>>>>> major release would provide the opportunity to > > >>> remove > > >>>>> these > > >>>>>>>>>>>>>>> deprecated and > > >>>>>>>>>>>>>>> unsupported components. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Rather than focusing the next major release on > > >> new > > >>>>>>> features, > > >>>>>>>>> what > > >>>>>>>>>>>>>>> do you > > >>>>>>>>>>>>>>> think about focusing on technical debt removal? > > >>> This > > >>>>>>> approach > > >>>>>>>>>>>>>>> would not > > >>>>>>>>>>>>>>> make for the most interesting release, but it > > >>>> provides > > >>>>> the > > >>>>>>>>>>>>>>> opportunity to > > >>>>>>>>>>>>>>> clean up elements that involve breaking changes. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Focusing on technical debt, at least three > > >> primary > > >>>>> goals > > >>>>>>> come > > >>>>>>>>> to > > >>>>>>>>>>>>>>> mind for > > >>>>>>>>>>>>>>> the next major release: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained > > >>> components > > >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported > > >> version > > >>>>>>>>>>>>>>> 3. Transition internal date and time handling to > > >>> JSR > > >>>>> 310 > > >>>>>>>>>> java.time > > >>>>>>>>>>>>>>> components > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> *Removing Deprecated Components* > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Removing support for older and deprecated > > >>> components > > >>>>>>>> provides a > > >>>>>>>>>>>>>>> great > > >>>>>>>>>>>>>>> opportunity to improve the overall security > > >> posture > > >>>>> when it > > >>>>>>>>> comes > > >>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency > > >>> plugin > > >>>>>>> report > > >>>>>>>>>>>>>>> currently > > >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable > > >>>> dependencies, > > >>>>> many > > >>>>>>>> of > > >>>>>>>>>>>>>>> which are > > >>>>>>>>>>>>>>> related to old versions of various libraries. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> As a starting point, here are a handful of > > >>> components > > >>>>> and > > >>>>>>>>>>>>>>> extension modules > > >>>>>>>>>>>>>>> that could be targeted for removal in a major > > >>>> version: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> - PostHTTP and GetHTTP > > >>>>>>>>>>>>>>> - ListenLumberjack and the entire > > >>>>> nifi-lumberjack-bundle > > >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle > > >>>>>>>>>>>>>>> - Elasticsearch 5 components > > >>>>>>>>>>>>>>> - Hive 1 and 2 components > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> *Requiring Java 11* > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has > > >>>>> supported > > >>>>>>>>>> general > > >>>>>>>>>>>>>>> compatibility with Java 11 for several years. > > >> NiFi > > >>>>> 1.14.0 > > >>>>>>>>>>>>>>> incorporated > > >>>>>>>>>>>>>>> internal improvements specifically related to TLS > > >>>> 1.3, > > >>>>>>> which > > >>>>>>>>>>>>>>> allowed > > >>>>>>>>>>>>>>> closing out the long-running Java 11 > > >> compatibility > > >>>> epic > > >>>>>>>>>> NIFI-5174. > > >>>>>>>>>>>>>>> Making > > >>>>>>>>>>>>>>> Java 11 the minimum required version provides the > > >>>>>>> opportunity > > >>>>>>>>> to > > >>>>>>>>>>>>>>> address > > >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better > > >>>>> position > > >>>>>>> to > > >>>>>>>>>>>>>>> support > > >>>>>>>>>>>>>>> current Java versions. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling* > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Without making the scope too broad, transitioning > > >>>>> internal > > >>>>>>>> date > > >>>>>>>>>>>>>>> and time > > >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of > > >>>>>>> SimpleDateFormat > > >>>>>>>>>>>>>>> would provide > > >>>>>>>>>>>>>>> a number of advantages. The Java Time components > > >>>>> provide > > >>>>>>> much > > >>>>>>>>>>>>>>> better > > >>>>>>>>>>>>>>> clarity when it comes to handling localized date > > >>> and > > >>>>> time > > >>>>>>>>>>>>>>> representations, > > >>>>>>>>>>>>>>> and also avoid the inherent confusion of > > >>>> java.sql.Date > > >>>>>>>>> extending > > >>>>>>>>>>>>>>> java.util.Date. Many internal components, > > >>>> specifically > > >>>>>>>>>>>>>>> Record-oriented > > >>>>>>>>>>>>>>> processors and services, rely on date parsing, > > >>>> leading > > >>>>> to > > >>>>>>>>>>>>>>> confusion and > > >>>>>>>>>>>>>>> various workarounds. The pattern formats of > > >>>>>>> SimpleDateFormat > > >>>>>>>>> and > > >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there > > >> are a > > >>>> few > > >>>>>>>> subtle > > >>>>>>>>>>>>>>> differences. > > >>>>>>>>>>>>>>> Making this transition would provide a much > > >> better > > >>>>>>> foundation > > >>>>>>>>>>>>>>> going forward. > > >>>>>>>>>>>>>>> *Conclusion* > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks for giving this proposal some > > >> consideration. > > >>>>> Many of > > >>>>>>>> you > > >>>>>>>>>>>>>>> have been > > >>>>>>>>>>>>>>> developing NiFi for years and I look forward to > > >>> your > > >>>>>>>> feedback. > > >>>>>>>>> I > > >>>>>>>>>>>>>>> would be > > >>>>>>>>>>>>>>> glad to put together a more formalized > > >>> recommendation > > >>>>> on > > >>>>>>>>>>>>>>> Confluence and > > >>>>>>>>>>>>>>> write up Jira epics if this general approach > > >> sounds > > >>>>>>> agreeable > > >>>>>>>>> to > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>> community. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>> David Handermann > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>> > > >>> > > >> > > > >
