Re: [DISCUSS] NiFi 2.0 Release Goals

[Bryan Bende]

With the merging of MiNiFi and registry into the NiFi repo, we've
actually gone more towards a mono-repo where everything is released
together. Eventually it would still be nice to have a smaller base
distribution containing just the framework and standard NARs, but it
is hard to tackle that until we provide a way for users to easily
obtain the NARs in some other way.

On Mon, Jul 26, 2021 at 4:20 PM Edward Armes  wrote:
>
> Given the major version number shift and the spliting up of processors into
> multiple NAR's. I'd like to suggest that we start individually versioning
> individual NARs/ bundles.
>
> I can see this bringing a large number of benifits including making Nifi
> more deployable with things RPM, but also potentially allowing those that
> have to deploy managed Nifi instances easier to upgrade.
>
> Edward
>
> On Mon, 26 Jul 2021, 20:42 Otto Fowler,  wrote:
>
> >  The issue with updating the aws sdk is if it breaks any one of the
> > processors.
> > the Web Gateway API invoke processor for example is not using a high level
> > purpose build client and may break.
> >
> > If we change the aws version, we need to coordinate in such a way that they
> > all
> > can come along reasonably.
> > IE:  what happens if 1 or 2 break but the rest or OK?
> >
> >
> >
> > From: David Handermann 
> > 
> > Reply: dev@nifi.apache.org  
> > Date: July 26, 2021 at 09:33:42
> > To: dev@nifi.apache.org  
> > Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals
> >
> > Chris,
> >
> > Thanks for the reply and recommendations. It seems like some of the work to
> > reorganize the module structure could be done outside of a major release,
> > but it would be great to target any breaking changes for 2.0. Perhaps a
> > separate feature proposal on module restructuring, with the goal of
> > supporting optimized builds, would be a helpful way to move that part of
> > the discussion forward.
> >
> > Regarding updating AWS SDK to version 2, it seems like that might be
> > possible now. I haven't taken a close look at the referencing components,
> > so I'm not sure about the level of effort involved. Minor NiFi version
> > updates have incorporated major new versions of dependencies. For example,
> > NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
> > hand, including the AWS SDK update as part of a major release seems
> > helpful, but unless there are changes that break existing component
> > properties, upgrading the AWS SDK could be worked independently. Others may
> > have more insight into particular usage of that library.
> >
> > Regards,
> > David Handermann
> >
> > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
> >  wrote:
> >
> > > Might be worth considering refactoring the build as part of this work
> > too,
> > > e.g. only building the bits of the repo affected by a commit, etc. -
> > > discussed briefly in previous threads but don't think any changes made
> > yet.
> > > If NARs/components are likely to be split up and refactored then such
> > work
> > > around the build probably makes sense to consider.
> > >
> > > I've a couple of PRs open that include updates to Elasticsearch versions
> > > already, although I stopped at 7.10.2 (after which Elastic changed
> > licence)
> > > in case there were licence concerns. But more work can be done to tidy up
> > > the processors, absolutely.
> > >
> > > AWS libraries to v2 would seem a sensible move and a refactor of those
> > > processors as well.
> > >
> > >
> > > Cheers,
> > >
> > > Chris Sampson
> > >
> > > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> > exceptionfact...@apache.org>
> >
> > > wrote:
> > >
> > > > Thanks for pointing out the standard NAR bundles, Mark. There are a
> > > number
> > > > of components in the standard NAR bundles with particular dependencies
> > > that
> > > > would make more sense in separate NARs. Reorganizing the standard NAR
> > to
> > > > components with limited dependencies and wide applicability would
> > > > definitely help with future maintenance.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne 
> > > wrote:
> > > >
> > > > > There’s also some code that exists in order to maintain backward
> > > > > compatibility in the repositories. I would very much like the
> > > > repositories
> > > > > to contain no unnecessary code. And swap file format supports really
> > > old
> > > > > formats. And the old impls of the repositories themselves, like
> > > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > > > there
> > > > > that can be removed. And some methods in ProcessSession that are
> > never
> > > > used
> > > > > by any processor in the codebase but exists in the public API so
> > can’t
> > > be
> > > > > removed till 2.0.
> > > > >
> > > > > I think his is also a great time to clean up the “standard nar.” At
> > > this
> > > > > point, it’s something like 70 MB. And many of the components there
> > are
> > > > not
> > > > > 

Re: [DISCUSS] NiFi 2.0 Release Goals

[Edward Armes]

Given the major version number shift and the spliting up of processors into
multiple NAR's. I'd like to suggest that we start individually versioning
individual NARs/ bundles.

I can see this bringing a large number of benifits including making Nifi
more deployable with things RPM, but also potentially allowing those that
have to deploy managed Nifi instances easier to upgrade.

Edward

On Mon, 26 Jul 2021, 20:42 Otto Fowler,  wrote:

>  The issue with updating the aws sdk is if it breaks any one of the
> processors.
> the Web Gateway API invoke processor for example is not using a high level
> purpose build client and may break.
>
> If we change the aws version, we need to coordinate in such a way that they
> all
> can come along reasonably.
> IE:  what happens if 1 or 2 break but the rest or OK?
>
>
>
> From: David Handermann 
> 
> Reply: dev@nifi.apache.org  
> Date: July 26, 2021 at 09:33:42
> To: dev@nifi.apache.org  
> Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals
>
> Chris,
>
> Thanks for the reply and recommendations. It seems like some of the work to
> reorganize the module structure could be done outside of a major release,
> but it would be great to target any breaking changes for 2.0. Perhaps a
> separate feature proposal on module restructuring, with the goal of
> supporting optimized builds, would be a helpful way to move that part of
> the discussion forward.
>
> Regarding updating AWS SDK to version 2, it seems like that might be
> possible now. I haven't taken a close look at the referencing components,
> so I'm not sure about the level of effort involved. Minor NiFi version
> updates have incorporated major new versions of dependencies. For example,
> NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
> hand, including the AWS SDK update as part of a major release seems
> helpful, but unless there are changes that break existing component
> properties, upgrading the AWS SDK could be worked independently. Others may
> have more insight into particular usage of that library.
>
> Regards,
> David Handermann
>
> On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
>  wrote:
>
> > Might be worth considering refactoring the build as part of this work
> too,
> > e.g. only building the bits of the repo affected by a commit, etc. -
> > discussed briefly in previous threads but don't think any changes made
> yet.
> > If NARs/components are likely to be split up and refactored then such
> work
> > around the build probably makes sense to consider.
> >
> > I've a couple of PRs open that include updates to Elasticsearch versions
> > already, although I stopped at 7.10.2 (after which Elastic changed
> licence)
> > in case there were licence concerns. But more work can be done to tidy up
> > the processors, absolutely.
> >
> > AWS libraries to v2 would seem a sensible move and a refactor of those
> > processors as well.
> >
> >
> > Cheers,
> >
> > Chris Sampson
> >
> > On Sat, 24 Jul 2021, 17:47 David Handermann, <
> exceptionfact...@apache.org>
>
> > wrote:
> >
> > > Thanks for pointing out the standard NAR bundles, Mark. There are a
> > number
> > > of components in the standard NAR bundles with particular dependencies
> > that
> > > would make more sense in separate NARs. Reorganizing the standard NAR
> to
> > > components with limited dependencies and wide applicability would
> > > definitely help with future maintenance.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne 
> > wrote:
> > >
> > > > There’s also some code that exists in order to maintain backward
> > > > compatibility in the repositories. I would very much like the
> > > repositories
> > > > to contain no unnecessary code. And swap file format supports really
> > old
> > > > formats. And the old impls of the repositories themselves, like
> > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > > there
> > > > that can be removed. And some methods in ProcessSession that are
> never
> > > used
> > > > by any processor in the codebase but exists in the public API so
> can’t
> > be
> > > > removed till 2.0.
> > > >
> > > > I think his is also a great time to clean up the “standard nar.” At
> > this
> > > > point, it’s something like 70 MB. And many of the components there
> are
> > > not
> > > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > > processing, Jolt transform, etc. could potentially be moved into
> other
> > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
> is
> > > not
> > > > necessary for stateless or minifi java. Lots of things probably to
> > > > reconsider within the standard nar.
> > > >
> > > > I definitely think this is a reasonable approach, to allow for a 2.0
> > that
> > > > is not a huge feature release but allows the project to be simpler
> and
> > > more
> > > > nimble.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen  > >  > 

Re: [DISCUSS] NiFi 2.0 Release Goals

[Otto Fowler]

 The issue with updating the aws sdk is if it breaks any one of the
processors.
the Web Gateway API invoke processor for example is not using a high level
purpose build client and may break.

If we change the aws version, we need to coordinate in such a way that they
all
can come along reasonably.
IE:  what happens if 1 or 2 break but the rest or OK?



From: David Handermann 

Reply: dev@nifi.apache.org  
Date: July 26, 2021 at 09:33:42
To: dev@nifi.apache.org  
Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals

Chris,

Thanks for the reply and recommendations. It seems like some of the work to
reorganize the module structure could be done outside of a major release,
but it would be great to target any breaking changes for 2.0. Perhaps a
separate feature proposal on module restructuring, with the goal of
supporting optimized builds, would be a helpful way to move that part of
the discussion forward.

Regarding updating AWS SDK to version 2, it seems like that might be
possible now. I haven't taken a close look at the referencing components,
so I'm not sure about the level of effort involved. Minor NiFi version
updates have incorporated major new versions of dependencies. For example,
NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
hand, including the AWS SDK update as part of a major release seems
helpful, but unless there are changes that break existing component
properties, upgrading the AWS SDK could be worked independently. Others may
have more insight into particular usage of that library.

Regards,
David Handermann

On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
 wrote:

> Might be worth considering refactoring the build as part of this work
too,
> e.g. only building the bits of the repo affected by a commit, etc. -
> discussed briefly in previous threads but don't think any changes made
yet.
> If NARs/components are likely to be split up and refactored then such
work
> around the build probably makes sense to consider.
>
> I've a couple of PRs open that include updates to Elasticsearch versions
> already, although I stopped at 7.10.2 (after which Elastic changed
licence)
> in case there were licence concerns. But more work can be done to tidy up
> the processors, absolutely.
>
> AWS libraries to v2 would seem a sensible move and a refactor of those
> processors as well.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sat, 24 Jul 2021, 17:47 David Handermann, 

> wrote:
>
> > Thanks for pointing out the standard NAR bundles, Mark. There are a
> number
> > of components in the standard NAR bundles with particular dependencies
> that
> > would make more sense in separate NARs. Reorganizing the standard NAR
to
> > components with limited dependencies and wide applicability would
> > definitely help with future maintenance.
> >
> > Regards,
> > David Handermann
> >
> > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne 
> wrote:
> >
> > > There’s also some code that exists in order to maintain backward
> > > compatibility in the repositories. I would very much like the
> > repositories
> > > to contain no unnecessary code. And swap file format supports really
> old
> > > formats. And the old impls of the repositories themselves, like
> > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > there
> > > that can be removed. And some methods in ProcessSession that are
never
> > used
> > > by any processor in the codebase but exists in the public API so
can’t
> be
> > > removed till 2.0.
> > >
> > > I think his is also a great time to clean up the “standard nar.” At
> this
> > > point, it’s something like 70 MB. And many of the components there
are
> > not
> > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > processing, Jolt transform, etc. could potentially be moved into
other
> > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
is
> > not
> > > necessary for stateless or minifi java. Lots of things probably to
> > > reconsider within the standard nar.
> > >
> > > I definitely think this is a reasonable approach, to allow for a 2.0
> that
> > > is not a huge feature release but allows the project to be simpler
and
> > more
> > > nimble.
> > >
> > > Thanks
> > > -Mark
> > >
> > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen  >  > > mikerthom...@gmail.com>> wrote:
> > >
> > > Russell,
> > >
> > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > still fine.
> > >
> >
>
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >
> > > Our Elasticsearch support is spread over two NARs at present. One
uses
> > > OkHttp and the other uses that low level Elastic REST client.
> > > Therefore, I think we're fine on licensing for the moment.
> > >
> > > Mike
> > >
> > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman  > > > wrote:
> > >
> > > Bringing up Elastic also reminds me that the 

Re: [DISCUSS] NiFi 2.0 Release Goals

[David Handermann]

Chris,

Thanks for the reply and recommendations. It seems like some of the work to
reorganize the module structure could be done outside of a major release,
but it would be great to target any breaking changes for 2.0. Perhaps a
separate feature proposal on module restructuring, with the goal of
supporting optimized builds, would be a helpful way to move that part of
the discussion forward.

Regarding updating AWS SDK to version 2, it seems like that might be
possible now. I haven't taken a close look at the referencing components,
so I'm not sure about the level of effort involved. Minor NiFi version
updates have incorporated major new versions of dependencies. For example,
NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
hand, including the AWS SDK update as part of a major release seems
helpful, but unless there are changes that break existing component
properties, upgrading the AWS SDK could be worked independently. Others may
have more insight into particular usage of that library.

Regards,
David Handermann

On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
 wrote:

> Might be worth considering refactoring the build as part of this work too,
> e.g. only building the bits of the repo affected by a commit, etc. -
> discussed briefly in previous threads but don't think any changes made yet.
> If NARs/components are likely to be split up and refactored then such work
> around the build probably makes sense to consider.
>
> I've a couple of PRs open that include updates to Elasticsearch versions
> already, although I stopped at 7.10.2 (after which Elastic changed licence)
> in case there were licence concerns. But more work can be done to tidy up
> the processors, absolutely.
>
> AWS libraries to v2 would seem a sensible move and a refactor of those
> processors as well.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sat, 24 Jul 2021, 17:47 David Handermann, 
> wrote:
>
> > Thanks for pointing out the standard NAR bundles, Mark.  There are a
> number
> > of components in the standard NAR bundles with particular dependencies
> that
> > would make more sense in separate NARs. Reorganizing the standard NAR to
> > components with limited dependencies and wide applicability would
> > definitely help with future maintenance.
> >
> > Regards,
> > David Handermann
> >
> > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne 
> wrote:
> >
> > > There’s also some code that exists in order to maintain backward
> > > compatibility in the repositories. I would very much like the
> > repositories
> > > to contain no unnecessary code. And swap file format supports really
> old
> > > formats. And the old impls of the repositories themselves, like
> > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > there
> > > that can be removed. And some methods in ProcessSession that are never
> > used
> > > by any processor in the codebase but exists in the public API so can’t
> be
> > > removed till 2.0.
> > >
> > > I think his is also a great time to clean up the “standard nar.” At
> this
> > > point, it’s something like 70 MB. And many of the components there are
> > not
> > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > processing, Jolt transform, etc. could potentially be moved into other
> > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is
> > not
> > > necessary for stateless or minifi java. Lots of things probably to
> > > reconsider within the standard nar.
> > >
> > > I definitely think this is a reasonable approach, to allow for a 2.0
> that
> > > is not a huge feature release but allows the project to be simpler and
> > more
> > > nimble.
> > >
> > > Thanks
> > > -Mark
> > >
> > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen  >  > > mikerthom...@gmail.com>> wrote:
> > >
> > > Russell,
> > >
> > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > still fine.
> > >
> >
> https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >
> > > Our Elasticsearch support is spread over two NARs at present. One uses
> > > OkHttp and the other uses that low level Elastic REST client.
> > > Therefore, I think we're fine on licensing for the moment.
> > >
> > > Mike
> > >
> > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman  > > > wrote:
> > >
> > > Bringing up Elastic also reminds me that the Elastic framework has just
> > > recently transitioned out of Open Source, so to acknowledge that, maybe
> > > some effort toward OpenSearch--I say this not understanding exactly how
> > > this sort of thing is considered in a large-scale, world-class software
> > > project like Apache NiFi. (I'm not a contributor, just a grateful
> > > consumer.)
> > >
> > > Russ
> > >
> > > On 7/23/21 10:28 AM, Matt Burgess wrote:
> > > Along with the itemized list for ancient components we should look at
> > > updating