Outside of core NiFi, for MiNiFi, I think it would be beneficial to align MiNiFi Java and MiNiFI C++ to use the same method(s) for command and control / flow deployment. This was discussed when MiNiFi Java was merged into the NiFi codebase [1].
My proposal would be to create a Java implementation of the C2 Protocol used by MiNiFi C++, both server-side and client-side, to allow deploying to MiNiFi Java instances from a centralize flow definition. Thanks to the unification of MiNiFi Java and NiFi, this could likely also be added as a capability of NiFi at that same time. If we are able to do this on the 1.x line, then I would also support deprecating other methods of flow deployment to MiNiFi outlined in [1] and removing them in a 2.x release. [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <https://github.com/apache/nifi/pull/4933#issuecomment-808411977> [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <https://cwiki.apache.org/confluence/display/MINIFI/C2+Design> Thanks, Kevin > On Aug 3, 2021, at 10:07 AM, David Handermann <exceptionfact...@gmail.com> > wrote: > > Thanks for the continued discussion around what can or should be removed. > Having a clear migration path from version 1 to version 2 will be very > important for supporting current deployments. > > Following the example of some other projects, one way to consider the > upgrade is from the point of view of deprecation warnings. If implemented > correctly, an installation of NiFi running the latest minor release of > version 1, and showing no deprecation warnings, should be upgradeable to > version 2 without any changes. Making this work involves accurately tagging > components and methods with deprecation indicators, and providing a clear > way to observe deprecation warnings. With the current version containing > both deprecated components and deprecated methods in various classes, this > would involve some work to get right. A simple approach might be a named > logger for deprecation warnings, which would write a separate log file. A > more advanced approach might involve a tool to analyze the system and flow > configurations. Either way, it seems that some additional work in a minor > release version is necessary before considering a major release version. > > One other point on compatibility: as long as the core component API > definition remains backwards compatible, it should be possible to run older > components. This provides a transition option for customers using > components such as PostHTTP, or any others that are not actively > maintained. At some point it may become necessary to break compatibility at > that level, but at least for version 2, maintaining component API > compatibility is key. > > Regards, > David Handermann > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean <mark.o.b...@gmail.com> wrote: > >> I created a JIRA ticket to investigate and improve the performance of >> InvokeHTTP. It includes a flow definition for benchmarking the performance >> of both PostHTTP and InvokeHTTP. >> >> https://issues.apache.org/jira/browse/NIFI-8968 >> >> Thanks, >> Mark >> >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <a...@adamtaft.com> wrote: >> >>> I'm not seeing the side thread that was going to discuss deprecation of >>> PostHTTP. Has that thread started and I just don't see it? >>> >>> One (significant?) concern with PostHTTP is the smooth integration of >>> NiFi-to-NiFi communication that is very transparently enabled with the >>> ListenHTTP and PostHTTP processors. There's some special logic in there >> for >>> handling flowfiles that InvokeHTTP doesn't really (nor should really) >> have. >>> >>> I know of several (large) NiFi installations that rely on the PostHTTP / >>> ListenHTTP combination. It has enabled NiFi to NiFi communication for >> folks >>> reluctant or unable to enable site-to-site type configuration. >>> >>> Honestly, I don't know why we'd want to "deprecate" any processor, as >>> opposed to just moving it to a new location. If these processors can be >>> ported and maintained to whatever the 2.0 API looks like, there's >> possibly >>> little harm keeping them around. >>> >>> And by the way, what happened to the "marketplace" concept? Is this being >>> considered for 2.0 as well? Because relocating the deprecated processors >>> to an external nar might be the best solution. Losing PostHTTP entirely I >>> think would be a mistake, but I'd conceptually support relocating it. >>> >>> Thanks, >>> >>> /Adam >>> >>> On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <joe.w...@gmail.com> wrote: >>> >>>> Looks like we just need to knock out 5 JIRAs :) [1] >>>> >>>> I felt like we had a label folks were using at one point but quickly >>>> looking revealed nothing exciting. After this confluence page >>>> stabilizes a bit we can probably knock out some JIRAs and such. >>>> >>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 >>>> >>>> On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <ottobackwa...@gmail.com> >>>> wrote: >>>>> >>>>> I find myself wishing I had a list of all the jiras / issues that >> have >>>>> been put off for a 2.0 release because they required some change or >>>> another >>>>> :( >>>>> >>>>> From: Joe Witt <joe.w...@gmail.com> <joe.w...@gmail.com> >>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < >> dev@nifi.apache.org> >>>>> Date: July 27, 2021 at 12:30:35 >>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> <dev@nifi.apache.org> >>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals >>>>> >>>>> A few thoughts: >>>>> >>>>> 1. I would love to see deprecation notices show up in the UI in >>>>> various ways to help motivate users to move off things to more >>>>> supportable things. That is not a prerequisite for anything happening >>>>> however. Just a good feature/nice thing to do for users when someone >>>>> is able to tackle it. >>>>> >>>>> 2. The decision to deprecate something and to further remove it need >>>>> not mean there is a superior solution available. If that thing itself >>>>> isn't getting the love/attention it needs to be >>>>> maintained/supported/healthy going forward that alone is enough to >>>>> remove it. That might well be the case with PostHTTP [1] and for >>>>> comparison you can see how much effort has gone into InvokeHTTP [2]. >>>>> >>>>> 3. When discussing a 2.0 release each thing we add as a 'must do' the >>>>> further away from reality such a release will become. We'll have to >>>>> get very specific about 'musts' vs 'wants'. >>>>> >>>>> [1] >>>>> >>>> >>> >> https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java >>>>> [2] >>>>> >>>> >>> >> https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java >>>>> >>>>> On Tue, Jul 27, 2021 at 8:47 AM David Handermann >>>>> <exceptionfact...@apache.org> wrote: >>>>>> >>>>>> Thanks Mark, providing a template or comparison statistics with >> Java >>>>>> versions and component configuration details would be very helpful. >>> If >>>> it >>>>>> is possible to run tests using a public API or deployable service, >>> that >>>>>> would also help confirm potential differences. >>>>>> >>>>>> Showing a deprecation notice in the UI could be helpful, perhaps >> as a >>>>>> configurable option. NIFI-8650 describes a general Flow Analysis >>>>>> capability, and it seems like that might be one possible way to >>> surface >>>>>> deprecation warnings. For something more specific to component >>>>> deprecation, >>>>>> it seems important to find a balance between making it obvious and >>>> making >>>>>> it something that ends up getting ignored. >>>>>> >>>>>> Regards, >>>>>> David Handermann >>>>>> >>>>>> On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <mark.o.b...@gmail.com> >>>> wrote: >>>>>> >>>>>>> I'll start a new thread for PostHTTP when I get a template and/or >>>>> detailed >>>>>>> stats. >>>>>>> >>>>>>> I know the deprecation is noted in the documentation. That's a >>>>> necessary >>>>>>> and minimum level of notification. I was suggesting it be more >>>> obvious >>>>> in >>>>>>> the UI. I think it would be beneficial to somehow be aware of the >>>>>>> deprecation status simply by looking at the flow (perhaps even on >>> the >>>>>>> summary pages too), and without having to open the documentation >>> for >>>>> every >>>>>>> processor to confirm whether or not a component is marked as >>>>> deprecated. >>>>>>> >>>>>>> Thanks, >>>>>>> Mark >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 27, 2021 at 11:16 AM David Handermann < >>>>>>> exceptionfact...@apache.org> wrote: >>>>>>> >>>>>>>> Mark, >>>>>>>> >>>>>>>> Thanks for the feedback. It may be better to start a separate >>>> thread >>>>> on >>>>>>>> PostHTTP, but can you provide an example flow demonstrating the >>>>>>> performance >>>>>>>> differences between PostHTTP and InvokeHTTP? >>>>>>>> >>>>>>>> PostHTTP uses the Apache HttpComponents library, whereas >>> InvokeHTTP >>>>> uses >>>>>>>> OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates >> for >>>>> OkHttp, >>>>>>>> so it would be important to test with the most recent version. >> It >>>> is >>>>> also >>>>>>>> worth noting that test classes for PostHTTP are now integration >>>> tests >>>>> as >>>>>>>> opposed to unit tests, which means they are not executed as >> part >>> of >>>>> the >>>>>>>> automated builds. >>>>>>>> >>>>>>>> The NiFi documentation includes the contents of the >>>> DeprecationNotice >>>>> for >>>>>>>> PostHTTP: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html >>>>>>>> >>>>>>>> Regards, >>>>>>>> David Handermann >>>>>>>> >>>>>>>> On Tue, Jul 27, 2021 at 9:56 AM Mark Bean < >> mark.o.b...@gmail.com >>>> >>>>> wrote: >>>>>>>> >>>>>>>>> I'm strongly in favor of reducing tech debt, and moving >>>>> deliberately >>>>>>> to a >>>>>>>>> 2.0 release. I have a concern with just one processor that is >>>>> currently >>>>>>>>> marked as deprecated: PostHTTP. (I have not evaluated >>>> specifically >>>>> any >>>>>>>>> other deprecated components; I cannot say if there are or are >>> not >>>>>>> similar >>>>>>>>> issues elsewhere.) I understand the rationale for deprecating >>>> this >>>>>>>>> processor in that it eliminates a processor whose >> functionality >>>> is >>>>>>>>> available elsewhere, specifically in the more flexible >>>> InvokeHTTP. >>>>>>>> However, >>>>>>>>> in my experience and testing, PostHTTP performs transfers >> with >>>> far >>>>>>>> greater >>>>>>>>> throughput than InvokeHTTP. I would not be in favor of >> removing >>>>>>> PostHTTP >>>>>>>>> unless/until InvokeHTTP is refactored to increase its >>> throughput >>>>>>>>> capability. >>>>>>>>> >>>>>>>>> Has anyone else continued to use PostHTTP over InvokeHTTP for >>>>> similar >>>>>>>>> reasons? Or, is there a performance-related configuration for >>>>>>> InvokeHTTP >>>>>>>> I >>>>>>>>> may have missed? >>>>>>>>> >>>>>>>>> Also, in order to help facilitate a smooth transition to 2.0 >>>> from a >>>>>>> user >>>>>>>>> perspective, would it be advisable to add some sort of visual >>>>> indicator >>>>>>>> in >>>>>>>>> the UI for components that are currently annotated as >>>> @Deprecated? >>>>> This >>>>>>>>> might help users proactively modify their flow prior to a >>> release >>>>> that >>>>>>>>> would otherwise break it. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende < >> bbe...@gmail.com> >>>>> wrote: >>>>>>>>> >>>>>>>>>> With the merging of MiNiFi and registry into the NiFi repo, >>>> we've >>>>>>>>>> actually gone more towards a mono-repo where everything is >>>>> released >>>>>>>>>> together. Eventually it would still be nice to have a >> smaller >>>>> base >>>>>>>>>> distribution containing just the framework and standard >> NARs, >>>> but >>>>> it >>>>>>>>>> is hard to tackle that until we provide a way for users to >>>> easily >>>>>>>>>> obtain the NARs in some other way. >>>>>>>>>> >>>>>>>>>> On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < >>>>> edward.ar...@gmail.com >>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Given the major version number shift and the spliting up >> of >>>>>>>> processors >>>>>>>>>> into >>>>>>>>>>> multiple NAR's. I'd like to suggest that we start >>>> individually >>>>>>>>> versioning >>>>>>>>>>> individual NARs/ bundles. >>>>>>>>>>> >>>>>>>>>>> I can see this bringing a large number of benifits >>> including >>>>> making >>>>>>>>> Nifi >>>>>>>>>>> more deployable with things RPM, but also potentially >>>> allowing >>>>>>> those >>>>>>>>> that >>>>>>>>>>> have to deploy managed Nifi instances easier to upgrade. >>>>>>>>>>> >>>>>>>>>>> Edward >>>>>>>>>>> >>>>>>>>>>> On Mon, 26 Jul 2021, 20:42 Otto Fowler, < >>>> ottobackwa...@gmail.com> >>>>> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> The issue with updating the aws sdk is if it breaks any >>> one >>>>> of >>>>>>> the >>>>>>>>>>>> processors. >>>>>>>>>>>> the Web Gateway API invoke processor for example is not >>>> using >>>>> a >>>>>>>> high >>>>>>>>>> level >>>>>>>>>>>> purpose build client and may break. >>>>>>>>>>>> >>>>>>>>>>>> If we change the aws version, we need to coordinate in >>>> such a >>>>> way >>>>>>>>> that >>>>>>>>>> they >>>>>>>>>>>> all >>>>>>>>>>>> can come along reasonably. >>>>>>>>>>>> IE: what happens if 1 or 2 break but the rest or OK? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From: David Handermann <exceptionfact...@apache.org> >>>>>>>>>>>> <exceptionfact...@apache.org> >>>>>>>>>>>> Reply: dev@nifi.apache.org <dev@nifi.apache.org> < >>>>>>>>> dev@nifi.apache.org> >>>>>>>>>>>> Date: July 26, 2021 at 09:33:42 >>>>>>>>>>>> To: dev@nifi.apache.org <dev@nifi.apache.org> < >>>>>>> dev@nifi.apache.org >>>>>>>>> >>>>>>>>>>>> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals >>>>>>>>>>>> >>>>>>>>>>>> Chris, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the reply and recommendations. It seems like >>>> some >>>>> of >>>>>>> the >>>>>>>>>> work to >>>>>>>>>>>> reorganize the module structure could be done outside >> of >>> a >>>>> major >>>>>>>>>> release, >>>>>>>>>>>> but it would be great to target any breaking changes >> for >>>> 2.0. >>>>>>>>> Perhaps a >>>>>>>>>>>> separate feature proposal on module restructuring, with >>> the >>>>> goal >>>>>>> of >>>>>>>>>>>> supporting optimized builds, would be a helpful way to >>> move >>>>> that >>>>>>>> part >>>>>>>>>> of >>>>>>>>>>>> the discussion forward. >>>>>>>>>>>> >>>>>>>>>>>> Regarding updating AWS SDK to version 2, it seems like >>> that >>>>> might >>>>>>>> be >>>>>>>>>>>> possible now. I haven't taken a close look at the >>>> referencing >>>>>>>>>> components, >>>>>>>>>>>> so I'm not sure about the level of effort involved. >> Minor >>>>> NiFi >>>>>>>>> version >>>>>>>>>>>> updates have incorporated major new versions of >>>> dependencies. >>>>> For >>>>>>>>>> example, >>>>>>>>>>>> NiFi 1.14 included an upgrade from Spring Framework 4 >> to >>> 5. >>>>> On >>>>>>> the >>>>>>>>> one >>>>>>>>>>>> hand, including the AWS SDK update as part of a major >>>> release >>>>>>> seems >>>>>>>>>>>> helpful, but unless there are changes that break >> existing >>>>>>> component >>>>>>>>>>>> properties, upgrading the AWS SDK could be worked >>>>> independently. >>>>>>>>>> Others may >>>>>>>>>>>> have more insight into particular usage of that >> library. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> David Handermann >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson >>>>>>>>>>>> <chris.samp...@naimuri.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Might be worth considering refactoring the build as >>> part >>>> of >>>>>>> this >>>>>>>>> work >>>>>>>>>>>> too, >>>>>>>>>>>>> e.g. only building the bits of the repo affected by a >>>>> commit, >>>>>>>> etc. >>>>>>>>> - >>>>>>>>>>>>> discussed briefly in previous threads but don't think >>> any >>>>>>> changes >>>>>>>>>> made >>>>>>>>>>>> yet. >>>>>>>>>>>>> If NARs/components are likely to be split up and >>>> refactored >>>>>>> then >>>>>>>>> such >>>>>>>>>>>> work >>>>>>>>>>>>> around the build probably makes sense to consider. >>>>>>>>>>>>> >>>>>>>>>>>>> I've a couple of PRs open that include updates to >>>>> Elasticsearch >>>>>>>>>> versions >>>>>>>>>>>>> already, although I stopped at 7.10.2 (after which >>>> Elastic >>>>>>>> changed >>>>>>>>>>>> licence) >>>>>>>>>>>>> in case there were licence concerns. But more work >> can >>> be >>>>> done >>>>>>> to >>>>>>>>>> tidy up >>>>>>>>>>>>> the processors, absolutely. >>>>>>>>>>>>> >>>>>>>>>>>>> AWS libraries to v2 would seem a sensible move and a >>>>> refactor >>>>>>> of >>>>>>>>>> those >>>>>>>>>>>>> processors as well. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> >>>>>>>>>>>>> Chris Sampson >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, 24 Jul 2021, 17:47 David Handermann, < >>>>>>>>>>>> exceptionfact...@apache.org> >>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for pointing out the standard NAR bundles, >>> Mark. >>>>> There >>>>>>>>> are a >>>>>>>>>>>>> number >>>>>>>>>>>>>> of components in the standard NAR bundles with >>>> particular >>>>>>>>>> dependencies >>>>>>>>>>>>> that >>>>>>>>>>>>>> would make more sense in separate NARs. >> Reorganizing >>>> the >>>>>>>> standard >>>>>>>>>> NAR >>>>>>>>>>>> to >>>>>>>>>>>>>> components with limited dependencies and wide >>>>> applicability >>>>>>>> would >>>>>>>>>>>>>> definitely help with future maintenance. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> David Handermann >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < >>>>>>>>> marka...@hotmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> There’s also some code that exists in order to >>>> maintain >>>>>>>>> backward >>>>>>>>>>>>>>> compatibility in the repositories. I would very >>> much >>>>> like >>>>>>> the >>>>>>>>>>>>>> repositories >>>>>>>>>>>>>>> to contain no unnecessary code. And swap file >>> format >>>>>>> supports >>>>>>>>>> really >>>>>>>>>>>>> old >>>>>>>>>>>>>>> formats. And the old impls of the repositories >>>>> themselves, >>>>>>>> like >>>>>>>>>>>>>>> PersistentProvRepo instead of WriteAheadProv >> Repo, >>>> etc. >>>>>>> Lots >>>>>>>> of >>>>>>>>>> stuff >>>>>>>>>>>>>> there >>>>>>>>>>>>>>> that can be removed. And some methods in >>>> ProcessSession >>>>>>> that >>>>>>>>> are >>>>>>>>>>>> never >>>>>>>>>>>>>> used >>>>>>>>>>>>>>> by any processor in the codebase but exists in >> the >>>>> public >>>>>>> API >>>>>>>>> so >>>>>>>>>>>> can’t >>>>>>>>>>>>> be >>>>>>>>>>>>>>> removed till 2.0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think his is also a great time to clean up the >>>>> “standard >>>>>>>>> nar.” >>>>>>>>>> At >>>>>>>>>>>>> this >>>>>>>>>>>>>>> point, it’s something like 70 MB. And many of the >>>>>>> components >>>>>>>>>> there >>>>>>>>>>>> are >>>>>>>>>>>>>> not >>>>>>>>>>>>>>> really “standard” - things like connecting to >> FTP & >>>>> SFTP >>>>>>>>>> servers, XML >>>>>>>>>>>>>>> processing, Jolt transform, etc. could >> potentially >>> be >>>>> moved >>>>>>>>> into >>>>>>>>>>>> other >>>>>>>>>>>>>>> nars. The >>>>> nifi-standard-content-viewer-1.15.0-SNAPSHOT.war >>>>>>> is >>>>>>>>>> 6.9 MB >>>>>>>>>>>> is >>>>>>>>>>>>>> not >>>>>>>>>>>>>>> necessary for stateless or minifi java. Lots of >>>> things >>>>>>>> probably >>>>>>>>>> to >>>>>>>>>>>>>>> reconsider within the standard nar. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I definitely think this is a reasonable approach, >>> to >>>>> allow >>>>>>>> for >>>>>>>>> a >>>>>>>>>> 2.0 >>>>>>>>>>>>> that >>>>>>>>>>>>>>> is not a huge feature release but allows the >>> project >>>> to >>>>> be >>>>>>>>>> simpler >>>>>>>>>>>> and >>>>>>>>>>>>>> more >>>>>>>>>>>>>>> nimble. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> -Mark >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Jul 24, 2021, at 10:59 AM, Mike Thomsen < >>>>>>>>>> mikerthom...@gmail.com >>>>>>>>>>>>>> <mailto: >>>>>>>>>>>>>>> mikerthom...@gmail.com>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Russell, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AFAICT from looking at Elastic's repos, the low >>> level >>>>> REST >>>>>>>>>> client is >>>>>>>>>>>>>>> still fine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Our Elasticsearch support is spread over two NARs >>> at >>>>>>> present. >>>>>>>>> One >>>>>>>>>>>> uses >>>>>>>>>>>>>>> OkHttp and the other uses that low level Elastic >>> REST >>>>>>> client. >>>>>>>>>>>>>>> Therefore, I think we're fine on licensing for >> the >>>>> moment. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mike >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < >>>>>>>>>>>> r...@windofkeltia.com >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Bringing up Elastic also reminds me that the >>> Elastic >>>>>>>> framework >>>>>>>>>> has >>>>>>>>>>>> just >>>>>>>>>>>>>>> recently transitioned out of Open Source, so to >>>>> acknowledge >>>>>>>>> that, >>>>>>>>>>>> maybe >>>>>>>>>>>>>>> some effort toward OpenSearch--I say this not >>>>> understanding >>>>>>>>>> exactly >>>>>>>>>>>> how >>>>>>>>>>>>>>> this sort of thing is considered in a >> large-scale, >>>>>>>> world-class >>>>>>>>>>>> software >>>>>>>>>>>>>>> project like Apache NiFi. (I'm not a contributor, >>>> just >>>>> a >>>>>>>>> grateful >>>>>>>>>>>>>>> consumer.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Russ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 7/23/21 10:28 AM, Matt Burgess wrote: >>>>>>>>>>>>>>> Along with the itemized list for ancient >> components >>>> we >>>>>>> should >>>>>>>>>> look at >>>>>>>>>>>>>>> updating versions of drivers, SDKs, etc. for >>> external >>>>>>> systems >>>>>>>>>> such as >>>>>>>>>>>>>>> Elasticsearch, Cassandra, etc. There may be >>> breaking >>>>>>> changes >>>>>>>>> but >>>>>>>>>> 2.0 >>>>>>>>>>>>>>> is probably the right time to get things up to >> date >>>> to >>>>> make >>>>>>>>> them >>>>>>>>>> more >>>>>>>>>>>>>>> useful to more people. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < >>>>>>>>>> thena...@gmail.com >>>>>>>>>>>>>> <mailto: >>>>>>>>>>>>>>> thena...@gmail.com>> wrote: >>>>>>>>>>>>>>> I'm a +1 for removing pretty much all of this >>> stuff. >>>>> There >>>>>>>> are >>>>>>>>>>>> security >>>>>>>>>>>>>>> implications to keeping old dependencies around, >> so >>>> the >>>>>>> more >>>>>>>>> old >>>>>>>>>> code >>>>>>>>>>>>> we >>>>>>>>>>>>>>> can remove the better. I agree that eventually we >>>> need >>>>> to >>>>>>>> move >>>>>>>>> to >>>>>>>>>>>>>>> supporting only Java 11+, and as our next release >>>> will >>>>>>>> probably >>>>>>>>>> be >>>>>>>>>>>>> about >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>>> - 6 months from now that doesn't seem too soon. >> We >>>>> could >>>>>>>>>> potentially >>>>>>>>>>>>>> break >>>>>>>>>>>>>>> this in two and remove the deprecated processors >>> and >>>>> leave >>>>>>>> 1.x >>>>>>>>> on >>>>>>>>>>>> Java >>>>>>>>>>>>> 8, >>>>>>>>>>>>>>> and finally start on 2.x which would support only >>>> Java >>>>> 11. >>>>>>>> I'm >>>>>>>>>> unsure >>>>>>>>>>>>> of >>>>>>>>>>>>>>> what implications changing the date and time >>> handling >>>>> would >>>>>>>>> have >>>>>>>>>> - >>>>>>>>>>>> for >>>>>>>>>>>>>>> running systems that use long term historical >> logs, >>>>>>>> unexpected >>>>>>>>>>>> impacts >>>>>>>>>>>>> to >>>>>>>>>>>>>>> time logging could be a problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As Joe says I think feature work will have to be >>>>> dedicated >>>>>>> to >>>>>>>>>> 2.x and >>>>>>>>>>>>> we >>>>>>>>>>>>>>> could support 1.x for security fixes for some >>> period >>>> of >>>>>>> time. >>>>>>>>> 2.x >>>>>>>>>>>> seems >>>>>>>>>>>>>>> like a gargantuan task but it's probably time to >>> get >>>>>>> started. >>>>>>>>> Not >>>>>>>>>>>> sure >>>>>>>>>>>>>> how >>>>>>>>>>>>>>> we handle all open PRs and the transition between >>> 1.x >>>>> and >>>>>>>> 2.x. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < >>>>>>>> joe.w...@gmail.com >>>>>>>>>>>> <mailto: >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You're right we have to be careful and you're >> right >>>>> there >>>>>>> are >>>>>>>>>> still >>>>>>>>>>>>>>> significant Java 8 users out there. But we also >>> have >>>> to >>>>> be >>>>>>>>>> careful >>>>>>>>>>>>>>> about security and sustainability of the >> codebase. >>> If >>>>> we >>>>>>> had >>>>>>>>>> talked >>>>>>>>>>>>>>> about this last year when that article came out >> I'd >>>>> have >>>>>>>> agreed >>>>>>>>>> it is >>>>>>>>>>>>>>> too early. Interestingly that link seems to get >>>> updated >>>>>>> and I >>>>>>>>>> tried >>>>>>>>>>>>>>> [1] and found more recent data (not sure how >>> recent). >>>>>>> Anyway >>>>>>>> it >>>>>>>>>>>>>>> suggests Java 8 is still the top dog but we see >>> good >>>>> growth >>>>>>>> on >>>>>>>>>> 11. In >>>>>>>>>>>>>>> my $dayjob this aligns to what I'm seeing too. >>>>> Customers >>>>>>>> didn't >>>>>>>>>> seem >>>>>>>>>>>>>>> to care about Java 11 until later half last year >>> and >>>>> now >>>>>>>>>> suddenly it >>>>>>>>>>>>>>> is all over the place. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think once we put out a NiFi 2.0 release we'd >> see >>>>> rapid >>>>>>>>>> decrease in >>>>>>>>>>>>>>> work on the 1.x line just being blunt. We did >> this >>>> many >>>>>>> years >>>>>>>>> ago >>>>>>>>>>>>>>> with 0.x to 1.x and we stood behind 0.x for a >> while >>>>> (maybe >>>>>>> a >>>>>>>>>> year or >>>>>>>>>>>>>>> so) but it was purely bug fix/security related >>> bits. >>>> We >>>>>>> would >>>>>>>>>> need to >>>>>>>>>>>>>>> do something similar. But feature work would >> almost >>>>>>> certainly >>>>>>>>> go >>>>>>>>>> to >>>>>>>>>>>>>>> the 2.x line. Maybe there are other workable >> models >>>> but >>>>> my >>>>>>>>>> instinct >>>>>>>>>>>>>>> suggests this is likely to follow a similar path. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ...anyway I agree it isn't that easy of a call to >>>> dump >>>>> Java >>>>>>>> 8. >>>>>>>>> We >>>>>>>>>>>>>>> need to make the call in both the interests of >> the >>>> user >>>>>>> base >>>>>>>>> and >>>>>>>>>> the >>>>>>>>>>>>>>> contributor base of the community. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>> https://www.jetbrains.com/lp/devecosystem-2021/java/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Joe >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < >>>>>>> joe.w...@gmail.com >>>>>>>>>> <mailto: >>>>>>>>>>>>>>> joe.w...@gmail.com>> wrote: >>>>>>>>>>>>>>> Russ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yeah the flow registry is a key part of it. But >>> also >>>>> now >>>>>>> you >>>>>>>>> can >>>>>>>>>>>>>>> download the flow definition in JSON (upload i >>> think >>>> is >>>>>>> there >>>>>>>>> now >>>>>>>>>>>>>>> too). Templates offered a series of challenges >> such >>>> as >>>>> we >>>>>>>> store >>>>>>>>>> them >>>>>>>>>>>>>>> in the flow definition which has made flows >> massive >>>> in >>>>> an >>>>>>>>>> unintended >>>>>>>>>>>>>>> way which isn't fun for cluster behavior. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have a couple cases where we headed down a >>>>> particular >>>>>>>>> concept >>>>>>>>>> and >>>>>>>>>>>>>>> came up with better approaches later. We need to >>>>> reconcile >>>>>>>>> these >>>>>>>>>> with >>>>>>>>>>>>>>> the benefit of hindsight, and while being careful >>> to >>>> be >>>>> not >>>>>>>>>> overly >>>>>>>>>>>>>>> disruptive to existing users, to reduce the >>>>>>>>> codebase/maintenance >>>>>>>>>>>>>>> burden and allow continued evolution of the >>> project. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < >>>>>>>>>>>> r...@windofkeltia.com >>>>>>>>>>>>>>> <mailto:r...@windofkeltia.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> Joe, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I apologize for the off-topic intrusion, but what >>>>> replaces >>>>>>>>>> templates? >>>>>>>>>>>>>>> The Registry? Templates rocked and we have used >>> them >>>>> since >>>>>>>>> 0.5.x. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Russ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 7/23/21 8:31 AM, Joe Witt wrote: >>>>>>>>>>>>>>> David, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think this is a highly reasonable approach and >>>> such a >>>>>>> focus >>>>>>>>>> will >>>>>>>>>>>>>>> greatly help make a 2.0 release far more >>> approachable >>>>> to >>>>>>>> knock >>>>>>>>>> out. >>>>>>>>>>>>>>> Not only that but tech debt reduction would help >>> make >>>>> work >>>>>>>>>> towards >>>>>>>>>>>>>>> major features we'd think about in a 'major >>> release' >>>>> sense >>>>>>>> more >>>>>>>>>>>>>>> approachable. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We should remove all deprecated things (as well >> as >>>>> verify >>>>>>> we >>>>>>>>>> have the >>>>>>>>>>>>>>> right list). We should remove/consider removal of >>>>>>> deprecated >>>>>>>>>>>>>>> concepts >>>>>>>>>>>>>>> like templates. We should consider whether we can >>>>> resolve >>>>>>> the >>>>>>>>>>>>>>> various >>>>>>>>>>>>>>> ways we've handled what are now parameters down >> to >>>> one >>>>>>> clean >>>>>>>>>>>>>>> approach. >>>>>>>>>>>>>>> We should remove options in the nifi.properties >>> which >>>>> turn >>>>>>>> out >>>>>>>>> to >>>>>>>>>>>>>>> never be used quite right (if there are). There >> is >>>>> quite a >>>>>>>> bit >>>>>>>>> we >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>> do purely in the name of tech debt reduction. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Lots to consider here but I think this is the >> right >>>>>>>> discussion. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Than ks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < >>>>>>>> bbe...@gmail.com >>>>>>>>>>>> <mailto: >>>>>>>>>>>>>>> bbe...@gmail.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> I'm a +1 for this... Not sure if this falls under >>>>> "Removing >>>>>>>>>>>>>>> Deprecated >>>>>>>>>>>>>>> Components", but I think we should also look at >>>>> anything >>>>>>> that >>>>>>>>> has >>>>>>>>>>>>>>> been >>>>>>>>>>>>>>> marked as deprecated throughout the code base as >> a >>>>>>> candidate >>>>>>>>> for >>>>>>>>>>>>>>> removal. There are quite a few classes, methods, >>>>>>> properties, >>>>>>>>> etc >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> have been waiting for a chance to be removed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 23, 2021 at 10:13 AM David Handermann >>>>>>>>>>>>>>> <exceptionfact...@apache.org<mailto: >>>>>>>>> exceptionfact...@apache.org >>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> Team, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> With all of the excellent work that many have >>>>> contributed >>>>>>> to >>>>>>>>> NiFi >>>>>>>>>>>>>>> over the >>>>>>>>>>>>>>> years, the code base has also accumulated some >>> amount >>>>> of >>>>>>>>>> technical >>>>>>>>>>>>>>> debt. A >>>>>>>>>>>>>>> handful of components have been marked as >>> deprecated, >>>>> and >>>>>>>> some >>>>>>>>>>>>>>> components >>>>>>>>>>>>>>> remain in the code base to support integration >> with >>>> old >>>>>>>>> versions >>>>>>>>>>>>>>> of various >>>>>>>>>>>>>>> products. Following the principles of semantic >>>>> versioning, >>>>>>>>>>>>>>> introducing a >>>>>>>>>>>>>>> major release would provide the opportunity to >>> remove >>>>> these >>>>>>>>>>>>>>> deprecated and >>>>>>>>>>>>>>> unsupported components. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rather than focusing the next major release on >> new >>>>>>> features, >>>>>>>>> what >>>>>>>>>>>>>>> do you >>>>>>>>>>>>>>> think about focusing on technical debt removal? >>> This >>>>>>> approach >>>>>>>>>>>>>>> would not >>>>>>>>>>>>>>> make for the most interesting release, but it >>>> provides >>>>> the >>>>>>>>>>>>>>> opportunity to >>>>>>>>>>>>>>> clean up elements that involve breaking changes. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Focusing on technical debt, at least three >> primary >>>>> goals >>>>>>> come >>>>>>>>> to >>>>>>>>>>>>>>> mind for >>>>>>>>>>>>>>> the next major release: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Removal of deprecated and unmaintained >>> components >>>>>>>>>>>>>>> 2. Require Java 11 as the minimum supported >> version >>>>>>>>>>>>>>> 3. Transition internal date and time handling to >>> JSR >>>>> 310 >>>>>>>>>> java.time >>>>>>>>>>>>>>> components >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Removing Deprecated Components* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Removing support for older and deprecated >>> components >>>>>>>> provides a >>>>>>>>>>>>>>> great >>>>>>>>>>>>>>> opportunity to improve the overall security >> posture >>>>> when it >>>>>>>>> comes >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> maintaining dependencies. The OWASP dependency >>> plugin >>>>>>> report >>>>>>>>>>>>>>> currently >>>>>>>>>>>>>>> generates 50 MB of HTML for questionable >>>> dependencies, >>>>> many >>>>>>>> of >>>>>>>>>>>>>>> which are >>>>>>>>>>>>>>> related to old versions of various libraries. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As a starting point, here are a handful of >>> components >>>>> and >>>>>>>>>>>>>>> extension modules >>>>>>>>>>>>>>> that could be targeted for removal in a major >>>> version: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - PostHTTP and GetHTTP >>>>>>>>>>>>>>> - ListenLumberjack and the entire >>>>> nifi-lumberjack-bundle >>>>>>>>>>>>>>> - ListenBeats and the entire nifi-beats-bundle >>>>>>>>>>>>>>> - Elasticsearch 5 components >>>>>>>>>>>>>>> - Hive 1 and 2 components >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Requiring Java 11* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Java 8 is now over seven years old, and NiFi has >>>>> supported >>>>>>>>>> general >>>>>>>>>>>>>>> compatibility with Java 11 for several years. >> NiFi >>>>> 1.14.0 >>>>>>>>>>>>>>> incorporated >>>>>>>>>>>>>>> internal improvements specifically related to TLS >>>> 1.3, >>>>>>> which >>>>>>>>>>>>>>> allowed >>>>>>>>>>>>>>> closing out the long-running Java 11 >> compatibility >>>> epic >>>>>>>>>> NIFI-5174. >>>>>>>>>>>>>>> Making >>>>>>>>>>>>>>> Java 11 the minimum required version provides the >>>>>>> opportunity >>>>>>>>> to >>>>>>>>>>>>>>> address >>>>>>>>>>>>>>> any lingering edge cases and put NiFi in a better >>>>> position >>>>>>> to >>>>>>>>>>>>>>> support >>>>>>>>>>>>>>> current Java versions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *JSR 310 for Date and Time Handling* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Without making the scope too broad, transitioning >>>>> internal >>>>>>>> date >>>>>>>>>>>>>>> and time >>>>>>>>>>>>>>> handling to use DateTimeFormatter instead of >>>>>>> SimpleDateFormat >>>>>>>>>>>>>>> would provide >>>>>>>>>>>>>>> a number of advantages. The Java Time components >>>>> provide >>>>>>> much >>>>>>>>>>>>>>> better >>>>>>>>>>>>>>> clarity when it comes to handling localized date >>> and >>>>> time >>>>>>>>>>>>>>> representations, >>>>>>>>>>>>>>> and also avoid the inherent confusion of >>>> java.sql.Date >>>>>>>>> extending >>>>>>>>>>>>>>> java.util.Date. Many internal components, >>>> specifically >>>>>>>>>>>>>>> Record-oriented >>>>>>>>>>>>>>> processors and services, rely on date parsing, >>>> leading >>>>> to >>>>>>>>>>>>>>> confusion and >>>>>>>>>>>>>>> various workarounds. The pattern formats of >>>>>>> SimpleDateFormat >>>>>>>>> and >>>>>>>>>>>>>>> DateTimeFormatter are very similar, but there >> are a >>>> few >>>>>>>> subtle >>>>>>>>>>>>>>> differences. >>>>>>>>>>>>>>> Making this transition would provide a much >> better >>>>>>> foundation >>>>>>>>>>>>>>> going forward. >>>>>>>>>>>>>>> *Conclusion* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for giving this proposal some >> consideration. >>>>> Many of >>>>>>>> you >>>>>>>>>>>>>>> have been >>>>>>>>>>>>>>> developing NiFi for years and I look forward to >>> your >>>>>>>> feedback. >>>>>>>>> I >>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>> glad to put together a more formalized >>> recommendation >>>>> on >>>>>>>>>>>>>>> Confluence and >>>>>>>>>>>>>>> write up Jira epics if this general approach >> sounds >>>>>>> agreeable >>>>>>>>> to >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> community. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> David Handermann >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>> >>> >>
