Re: Apache Nifi gets an "Out of Memory Error" by the Java Socket

[Joe Witt]

Hello

It is really difficult to review that output as it came through.  Can you
please create a JIRA and/or find a way to format/share that better.

While there is an OutofMemory error there also appears to be some
fundamental Jython challenges to sort out.

Thanks

On Wed, Nov 6, 2024 at 7:56 AM DangKim, Michael
 wrote:

> Hello all,
>
> I got a problem with our nifi instance and could not find a solution for
> this problem.
> The Nifi is running as a docker container. I tried to restart the
> container, but without solution.
>
> I get this failure info by mail:
>
> >>
> [{"Timestamp":"2024-11-06
> 13:18:24,002","loglevel":"ERROR","logmessage":"[Timer-Driven Process
> Thread-16] o.a.n.p.standard.EvaluateJsonPath
> EvaluateJsonPath[id=bcbf35e2-a94c-1ac0-8815-6c0edfcf36ab] Processing
> halted: yielding [1 sec]","stackTrace":"java.lang.OutOfMemoryError: Java
> heap space","_raw":"2024-11-06 13:18:24,002 ERROR [Timer-Driven Process
> Thread-16] o.a.n.p.standard.EvaluateJsonPath
> EvaluateJsonPath[id=bcbf35e2-a94c-1ac0-8815-6c0edfcf36ab] Processing
> halted: yielding [1 sec]\njava.lang.OutOfMemoryError: Java heap
> space"},{"Timestamp":"2024-11-06
> 13:18:29,252","loglevel":"ERROR","logmessage":"[Packet Reader for
> 10.34.129.22] com.hierynomus.smbj.session.Session Caught exception while
> closing TreeConnect with id:
> 1","stackTrace":"com.hierynomus.protocol.transport.TransportException:
> java.net.SocketException: Broken pipe (Write failed)\n\tat
> com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:86)\n\tat
> com.hierynomus.smbj.connection.Connection.send(Connection.java:238)\n\tat
> com.hierynomus.smbj.session.Session.send(Session.java:303)\n\tat
> com.hierynomus.smbj.share.TreeConnect.close(TreeConnect.java:69)\n\tat
> com.hierynomus.smbj.share.Share.close(Share.java:116)\n\tat
> com.hierynomus.smbj.session.Session.logoff(Session.java:236)\n\tat
> com.hierynomus.smbj.session.Session.close(Session.java:279)\n\tat
> com.hierynomus.smbj.connection.Connection.close(Connection.java:178)\n\tat
> com.hierynomus.smbj.connection.Connection.close(Connection.java:155)\n\tat
> com.hierynomus.smbj.connection.Connection.handleError(Connection.java:295)\n\tat
> com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)\n\tat
> java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
> java.net.SocketException: Broken pipe (Write failed)\n\tat java.base/
> java.net.SocketOutputStream.socketWrite0(Native Method)\n\tat java.base/
> java.net.SocketOutputStream.socketWrite(Unknown Source)\n\tat java.base/
> java.net.SocketOutputStream.write(Unknown Source)\n\tat 
> java.base/java.io.BufferedOutputStream.flushBuffer(Unknown
> Source)\n\tat java.base/java.io.BufferedOutputStream.flush(Unknown
> Source)\n\tat
> com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:83)\n\t...
> 11 common frames omitted","_raw":"2024-11-06 13:18:29,252 ERROR [Packet
> Reader for 10.34.129.22] com.hierynomus.smbj.session.Session Caught
> exception while closing TreeConnect with id:
> 1\ncom.hierynomus.protocol.transport.TransportException:
> java.net.SocketException: Broken pipe (Write failed)\n\tat
> com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:86)\n\tat
> com.hierynomus.smbj.connection.Connection.send(Connection.java:238)\n\tat
> com.hierynomus.smbj.session.Session.send(Session.java:303)\n\tat
> com.hierynomus.smbj.share.TreeConnect.close(TreeConnect.java:69)\n\tat
> com.hierynomus.smbj.share.Share.close(Share.java:116)\n\tat
> com.hierynomus.smbj.session.Session.logoff(Session.java:236)\n\tat
> com.hierynomus.smbj.session.Session.close(Session.java:279)\n\tat
> com.hierynomus.smbj.connection.Connection.close(Connection.java:178)\n\tat
> com.hierynomus.smbj.connection.Connection.close(Connection.java:155)\n\tat
> com.hierynomus.smbj.connection.Connection.handleError(Connection.java:295)\n\tat
> com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)\n\tat
> java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
> java.net.SocketException: Broken pipe (Write failed)\n\tat java.base/
> java.net.SocketOutputStream.socketWrite0(Native Method)\n\tat java.base/
> java.net.SocketOutputStream.socketWrite(Unknown Source)\n\tat java.base/
> java.net.SocketOutputStream.write(Unknown Source)\n\tat 
> java.base/java.io.BufferedOutputStream.flushBuffer(Unknown
> Source)\n\tat java.base/java.io.BufferedOutputStream.flush(Unknown
> Source)\n\tat
> com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:83)\n\t...
> 11 common frames omitted"},{"Timestamp":"2024-11-06
> 13:26:58,229","loglevel":"ERROR","logmessage":"[Timer-Driven Process
> Thread-5] o.a.n.p.standard.GenerateFlowFile
> GenerateFlowFile[id=bcbf36a5-a94c-1ac0-6366-b97b138e8ad9] Failed to
> properly initialize Processor. If still scheduled to run, NiFi will attempt
> to initialize and run the Processor again after the 

Re: [DISCUSS] End-of-life timing for NiFi 1

[Joe Witt]

To be actionable and concrete here is a proposal:

1. Declare the NiFi 1.28.x line as the 'End of Life/End of Support' line.
This means we may still do periodic bug fixes or when possible/reasonable
bump vulnerable libs.  But we will not be doing further analysis/triage of
security reports nor adding features.

2. Add a DISCLAIMER to the source and key binaries of the 1.x/1.28 line [1].

3. Update the downloads page making the links for nifi source and the main
nifi assembly of whatever is the latest NiFi 1.28.x release there but
clearly articulated as the end of support line for which bug fixes and some
narrow dependency updates may occur.  Advise users of the 1.x line of the
importance of planning to migrate to the 2.x line.

4. Conduct a VOTE to codify this.

5. Conduct an Apache NiFi 1.28.1 release to pickle up (2) and the bug fixes
already available.

6. As we gather more user input on things which are helpful to them we
factor these into migration guidance/tooling as appropriate on the 2.x line.

Thanks

[1] https://github.com/apache/nifi/pull/9491

On Tue, Nov 5, 2024 at 9:04 AM Joe Witt  wrote:

> Team,
>
> I will start a discussion thread on the users list so we can hear more
> inputs from them and from that perspective.
>
> This thread needs to focus on what the contributors/committers/PMC in the
> community can/will/should do and the PMC in particular as we're obligated
> to ensure we're putting out software for which we can stand behind its
> security posture.
>
> We do not need to get worried about customers.  They have vendors that
> support them.  What we need to worry about and continue to do an excellent
> job caring for is the apache nifi user base and we need to ensure they
> don't have the belief that the NiFi 1.x line will be fixed in the presence
> of vulnerability reports.  I'll ask on the users list how folks would like
> us to communicate about the state of things.
>
> What I think we need to ask here is more in the spirit of what this thread
> was started about.  When do we as a contributor/committer/PMC base want to
> make it official in our own sense that we will not be producing releases
> for the 1.x line?  How we best communicate/help the user base then follows
> from that.  Stated another way those who feel they will be in a good
> position to do security reviews, vulnerability scans and remediation,
> conduct releases for some period of time please share what you think you'll
> be able to do and roughly for how long.
>
> Thanks
>
> On Tue, Nov 5, 2024 at 3:36 AM Pierre Villard 
> wrote:
>
>> Hi all,
>>
>> While I think we could set an EOL date a bit further in the future, it is
>> important to keep in mind what EOL means. It only means we won't be
>> providing security fixes / bug fixes through new releases. It does not
>> mean that NiFi 1.x is gone. If that is a big concern for some users when
>> running EOL software then we should remind those users that they've been
>> doing it for 2+ years already when using NiFi 1.x (taking Jetty as an
>> example here). And Joe is definitely right when saying that we have a
>> smaller and smaller group of people willing to spend an extensive amount
>> of
>> time taking care of PR/reviews, of release candidates, testing/validating
>> RCs, etc, for the 1.x line.
>>
>> I also agree that, even if many users are already using NiFi 2 in
>> production, many places have strict policies to not adopt a new major
>> release. I don't want to start a debate whether this is making sense or
>> not
>> but we know those rules exist in many places :) And the fact that we had
>> milestone releases for one year is not going to be enough of an argument.
>>
>> Given what we've seen in the past, we usually make a new release every 3
>> months or so. It's probably fair to assume a 2.1.0 release will happen
>> early next year. With that in mind, I tend to agree with Michael
>> suggesting
>> an EOL date at the end of January (3 months from now). We could also say
>> that 1.28.1 will happen at this time and will be the last one in the
>> community.
>>
>> Vendors have already announced support for NiFi 1.x for multiple
>> additional
>> years so this approach follows what we see in other projects where
>> extended
>> support is only provided through paid options with specific companies.
>>
>> It is awesome to finally see 2.0 out and this decision will help drive
>> users to that new release, which is much better in so many ways...
>>
>> Thanks,
>> Pierre
>>
>>
>> Le mar. 5 nov. 2024 à 10:36, Isha Lamboo 
>> a
>> écrit :
>>
>>

Re: [DISCUSS] Release Timing for NiFi 1.28.0

[Joe Witt]

Ferenc,

Thanks for the RM work on the 1.28 release.

Are you up for doing a 1.28.1 release in the coming days?

There are a few bugs fixed worth grabbing then a security relevant update
both in the form of a DISCLAIMER file that will help communicate to users
about the go forward nature of the 1.x line.

Thanks

On Fri, Oct 18, 2024 at 4:06 AM Ferenc Kis  wrote:

> Great, thanks Matt
>
> On Fri, Oct 18, 2024 at 12:00 AM Matt Burgess 
> wrote:
>
> > +1 I was informed that fix was backported already under a separate Jira
> and
> > PR, so I'm good with going forward.
> >
> > On Thu, Oct 17, 2024 at 9:47 AM Matt Burgess 
> wrote:
> >
> > > As with 2.0.0, if we can backport [1] into 1.28 I think we should.
> > >
> > > Thanks Ferenc!
> > >
> > > [1] https://github.com/apache/nifi/pull/9273
> > >
> > > On Thu, Oct 17, 2024 at 8:11 AM Ferenc Kis 
> > > wrote:
> > >
> > >> Hi Team,
> > >>
> > >> all of the remaining tickets are now merged, so we can proceed with
> the
> > >> release.
> > >> https://issues.apache.org/jira/projects/NIFI/versions/12354883
> > >>
> > >> Please let me know if there are still any tickets in progress which
> > could
> > >> be blockers for this release.
> > >>
> > >> I plan to start the release candidate process before the end of this
> > >> week, if there are no objections.
> > >>
> > >> Regards
> > >> Ferenc
> > >>
> > >> On Thu, Oct 10, 2024 at 10:47 PM David Handermann <
> > >> exceptionfact...@apache.org> wrote:
> > >>
> > >> > Ferenc,
> > >> >
> > >> > Thanks for the reply! I will continue reviewing that soon to get
> > >> > things ready for the release.
> > >> >
> > >> > Regards,
> > >> > David Handermann
> > >> >
> > >> > On Thu, Oct 10, 2024 at 1:51 PM Ferenc Kis  >
> > >> > wrote:
> > >> > >
> > >> > > Thanks David!
> > >> > >
> > >> > > We are still waiting for
> > >> > https://issues.apache.org/jira/browse/NIFI-13744,
> > >> > > it seems to be important to get into 1.28
> > >> > > Once it is merged and backported to 1.28, I plan to start to
> release
> > >> > process
> > >> > >
> > >> > > Regards
> > >> > > Ferenc
> > >> > >
> > >> > > On Thu, Oct 10, 2024 at 3:16 PM David Handermann <
> > >> > > exceptionfact...@apache.org> wrote:
> > >> > >
> > >> > > > Ferenc,
> > >> > > >
> > >> > > > Thanks for summarizing the status of key issues.
> > >> > > >
> > >> > > > With the dependency upgrades addressed and the other noted pull
> > >> > > > requests merged, the support branch seems to be in a good
> position
> > >> for
> > >> > > > a release candidate build.
> > >> > > >
> > >> > > > Regards,
> > >> > > > David Handermann
> > >> > > >
> > >> > > > On Fri, Oct 4, 2024 at 6:52 AM Ferenc Kis <
> > briansolo1...@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi Team,
> > >> > > > >
> > >> > > > > let's recap on the current status.
> > >> > > > >
> > >> > > > > * These tickets are still in progress, but seem to get close
> to
> > >> merge
> > >> > > > > https://issues.apache.org/jira/browse/NIFI-13744 -
> > >> > > > > https://github.com/apache/nifi/pull/9285
> > >> > > > > https://issues.apache.org/jira/browse/NIFI-13776 -
> > >> > > > > https://github.com/apache/nifi/pull/9286
> > >> > > > >
> > >> > > > > * As discussed I went through the dependencies, and updated as
> > >> much
> > >> > as I
> > >> > > > > could.
> > >> > > > > Details are on the Jira, but in a nutshell: went through the
> > >> commits
> > >> > from
> > >> > > > > the 2024 year which contain dependency upgrades. These commits
> > are
> > >> > from
> > >> > > > the
> > >> > > > > 1.26 and the 1.27 release period, so the list should reflect
> > well
> > >> > which
> > >> > > > > dependencies are usually upgraded from release to release.
> > >> > > > > Unfortunately not all dependency upgrades were feasible due to
> > >> > various
> > >> > > > > reasons. If I remember correctly this was the reason why those
> > >> > components
> > >> > > > > were deprecated and removed from the NiFi 2.x line.
> > >> > > > > *This PR needs to be reviewed.*
> > >> > > > > https://issues.apache.org/jira/browse/NIFI-13836 -
> > >> > > > > https://github.com/apache/nifi/pull/9340
> > >> > > > >
> > >> > > > > * As requested I created a Jira for backporting NIFI-13529.
> The
> > >> > commit
> > >> > > > > couldn't be simply cherry-picked to support/1.x due to major
> > >> > refactorings
> > >> > > > > on the 2.x line.
> > >> > > > > *This PR needs to be reviewed.*
> > >> > > > > https://issues.apache.org/jira/browse/NIFI-13837 -
> > >> > > > > https://github.com/apache/nifi/pull/9341
> > >> > > > >
> > >> > > > > Regards
> > >> > > > > Ferenc
> > >> > > > >
> > >> > > > > On Thu, Oct 3, 2024 at 10:31 AM Mathew Kiprop <
> > >> > mathewkipro...@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi team,
> > >> > > > > >
> > >> > > > > > Is this issue[1] backported to the 1.x line and will it be
> > >> > available in
> > >> > > > > > 1.28.0? I haven't seen it tracked here [2]
> > >> > > > > >
> > >> > > > > > [1] https:

Re: [DISCUSS] End-of-life timing for NiFi 1

[Joe Witt]

; > year-long migration at least.
> >
> > That said, are there any tools or scripts to make the migration smoother?
> > Configuring multiple levels of parameter contexts with inheritance is a
> > labor-intensive process if we are to mirror the current setup with
> > variables being inherited from main canvas, team PG, subject PG and flow
> > PG, etc. Anything that could go through the process groups and configure
> > this automatically would be greatly appreciated. I will look into that
> > myself too, but anything helps really.
> >
> > Regards,
> >
> > Isha
> >
> > -Oorspronkelijk bericht-
> > Van: Joe Witt 
> > Verzonden: maandag 4 november 2024 23:44
> > Aan: dev@nifi.apache.org
> > Onderwerp: Re: [DISCUSS] End-of-life timing for NiFi 1
> >
> > The EOL discussion is not here because we have a new problem.  It is here
> > because we finally have an answer.
> >
> > The inability to address reported vulnerabilities or fundamental end of
> > life status for key underlying components in the 1.x line is a problem
> that
> > was fully recognized three years ago.
> >
> > In that time we created a plan for what NiFi 2.0 would be and how we'd
> > manage both maintaining the 1.x line while building to the 2.x GA.  In
> the
> > past year we've conducted four milestone releases of NiFi 2.x and we've
> > continued putting out feature, bug fix, and security improvement releases
> > of 1.x.
> >
> > Feature bearing releases of 1.x are no longer appropriate as 2.x is here
> > and GA and that was the plan all along.
> >
> > Bug fixes are still reasonable in spirit but you need people to submit
> the
> > JIRAs, fix the JIRAs, peer review the changes, and to conduct releases
> and
> > make votes.  That is in increasingly short supply as it has been quite
> the
> > task splitting attention across two major lines and naturally developers
> > and users will gravitate toward the go forward path.
> >
> > Vulnerability/Security related considerations are where things are
> > fundamentally problematic.  We had a security report today about the
> super
> > old/outdated front-end libraries we use in 1.x.  That won't change.  We
> had
> > a report last week about Spring libraries needing updated except you
> can't
> > unless you have Pivotal support so not an option.  Those won't change. We
> > have had questions around Jetty changes but that is tied to Java 8.
> We've
> > had questions about Java 8 being end of life and even Java 11 and even
> now
> > Java 17 in terms of its codebase permissive licensing changing.  The
> things
> > we can reasonably address in the 1.x line are getting smaller and smaller
> > and the time required to address any new thing is higher and higher.
> >
> > We as a community, regardless of good intentions, cannot fix the illities
> > of the 1.x line and thus the 2.x line is here.  The 1.x line will
> > absolutely continue to atrophy and it will accelerate.  If we do not
> signal
> > EOL on 1.x that means we're saying we can keep fixing problems.  While
> that
> > is true for bugs, that is not true for vulnerabilities broadly and for
> our
> > most critical components.
> >
> > If you still fix bugs people assume this means you still reasonably fix
> > vulnerabilities/etc..  And unless we declare EOL on the 1.x line we will
> > continue to get non-serviceable reports and mislead the user base.
> >
> > The answer is to clearly signal that users should transition to the 2.x
> > line and focus our help on answering questions people might have on how
> to
> > do that.
> >
> > I am supportive of EOL for the 1.x line.  I also like the poetic nature
> of
> > the decade timing.
> >
> > On Mon, Nov 4, 2024 at 2:47 PM David Handermann <
> > exceptionfact...@apache.org>
> > wrote:
> >
>

Re: [DISCUSS] End-of-life timing for NiFi 1

[Joe Witt]

The EOL discussion is not here because we have a new problem.  It is here
because we finally have an answer.

The inability to address reported vulnerabilities or fundamental end of
life status for key underlying components in the 1.x line is a problem that
was fully recognized three years ago.

In that time we created a plan for what NiFi 2.0 would be and how we'd
manage both maintaining the 1.x line while building to the 2.x GA.  In
the past year we've conducted four milestone releases of NiFi 2.x and we've
continued putting out feature, bug fix, and security improvement releases
of 1.x.

Feature bearing releases of 1.x are no longer appropriate as 2.x is here
and GA and that was the plan all along.

Bug fixes are still reasonable in spirit but you need people to submit the
JIRAs, fix the JIRAs, peer review the changes, and to conduct releases and
make votes.  That is in increasingly short supply as it has been quite the
task splitting attention across two major lines and naturally developers
and users will gravitate toward the go forward path.

Vulnerability/Security related considerations are where things are
fundamentally problematic.  We had a security report today about the super
old/outdated front-end libraries we use in 1.x.  That won't change.  We had
a report last week about Spring libraries needing updated except you can't
unless you have Pivotal support so not an option.  Those won't change. We
have had questions around Jetty changes but that is tied to Java 8.  We've
had questions about Java 8 being end of life and even Java 11 and even now
Java 17 in terms of its codebase permissive licensing changing.  The things
we can reasonably address in the 1.x line are getting smaller and smaller
and the time required to address any new thing is higher and higher.

We as a community, regardless of good intentions, cannot fix the illities
of the 1.x line and thus the 2.x line is here.  The 1.x line will
absolutely continue to atrophy and it will accelerate.  If we do not signal
EOL on 1.x that means we're saying we can keep fixing problems.  While that
is true for bugs, that is not true for vulnerabilities broadly and for our
most critical components.

If you still fix bugs people assume this means you still reasonably fix
vulnerabilities/etc..  And unless we declare EOL on the 1.x line we will
continue to get non-serviceable reports and mislead the user base.

The answer is to clearly signal that users should transition to the 2.x
line and focus our help on answering questions people might have on how to
do that.

I am supportive of EOL for the 1.x line.  I also like the poetic nature of
the decade timing.

On Mon, Nov 4, 2024 at 2:47 PM David Handermann 
wrote:

> Mike,
>
> Thanks for the reply and thoughts on the timing, there is something to
> be said for a December 8, 2024 date given the 10 year anniversary.
>
> Regards,
> David Handermann
>
> On Mon, Nov 4, 2024 at 3:32 PM Michael Moser  wrote:
> >
> >   I agree with reducing support for the 1.x branch down to no longer
> allow
> > new features.  I don't think we should discontinue accepting bug fixes,
> > though.   We don't know if bugs will be found early in the life cycle of
> > the new features in 1.28.0.
> >
> > I would like to see one last release, called 1.28.1 and I think we should
> > give it more than 5 weeks of time.  Perhaps January 31, 2025 would be a
> > better target?  Though I just had a thought that December 8, 2024 would
> be
> > exactly 10 years after the very first commit for the initial code
> > contribution, and that would be satisfyingly poetic.
> >
> > -- Mike
> >
> >
> > On Mon, Nov 4, 2024 at 4:27 PM Russell Bateman 
> > wrote:
> >
> > > Folks,
> > >
> > > I understand the assertions based on Spring, Jetty and Angular. Those
> > > are what they are. And no new features are fine by me. Deprecation of
> > > existing or cautionary features are fine by me.
> > >
> > > However, I'm going to have trouble convincing my management to continue
> > > considering NiFi in our product when I reveal to them that one scant
> > > week has elapsed since the last version (1.28.0), but that now the
> whole
> > > product's reached end-of-life. It seems a strong indictment of Apache
> > > NiFi  to drop 2.0.0 only to say on the same day that there's no grace
> > > period for users to adopt a /major version dot 0/. While I may not
> > > myself have been paying serious enough attention to NiFi 2 (except to
> > > install and run it for some time now to make certain of no surprises),
> I
> > > feel like rev'ing 1.25.0's third number a couple of dozen (so, for
> > > nearly a full year) might have delivered more the message, "get quickly
> > > off 1.x 'cause it's dead by year's end." Even with a year's notice,
> we'd
> > > have a struggle to get our customers off of 1.x.
> > >
> > > Maybe it's just that the message needs more palatable crafting?
> > >
> > > "As we cannot provide security support for NiFi 1.x, [we] should
> > > discontinue a

Re: Request for maintainers on extensions with libraries that are dated/EOL

[Joe Witt]

Thanks Matt and Peter!

On Mon, Nov 4, 2024 at 12:15 PM Peter Turcsanyi 
wrote:

> Continuing the work on the Salesforce upgrade, PR [1] created for the 3.x
> => 4.x change.
>
> Regards,
> Peter
>
> [1] https://github.com/apache/nifi/pull/9484
>
> On Mon, Oct 28, 2024 at 9:07 PM Matt Burgess  wrote:
>
> > For Redis I wrote up [1] and put up a PR [2] in case we do agree to
> upgrade
> > to the latest version. I tested all Redis-backed components with no
> errors
> > or changes in behavior
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-13944
> > [2] https://github.com/apache/nifi/pull/9465
> >
> >
> > On Mon, Oct 28, 2024 at 3:13 PM Matt Burgess 
> wrote:
> >
> > > I added [1] and a PR [2] to upgrade the easy ones. I commented on the
> > > Salesforce version and we should decide if we want to update the Mongo
> > > components to 8, and redis to 6.4+
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-13943
> > > [2] https://github.com/apache/nifi/pull/9464
> > >
> > > On Mon, Oct 28, 2024 at 1:36 PM Joe Witt  wrote:
> > >
> > >> Team,
> > >>
> > >> These need help from maintainers to keep them updated or they risk
> > >> requiring removal.
> > >>
> > >> Problematically Dated libraries we have shown as:
> > >> package-name currVersion latestVersionAvail
> > >>
> > >> com.dropbox.core 5.4.6 7.0.0
> > >>  One year old
> > >>
> > >> org.apache.camel/salesforce 3.14.10 4.8.1
> > >>  Already EOL for two years.
> > >>
> > >> org.mongodb 4.11.4 5.2.0
> > >>Compatible with Mongo 6 and 7 which are still supported. (all older
> > are
> > >> EOL)
> > >>Not compatible with Mongo 8 which is latest.
> > >>
> > >> org.questdb 7.4.2 8.1.4
> > >>
> > >> org.snmp4j 2.8.18 3.8.2
> > >>
> > >> redis.clients 3.10.0 5.2.0
> > >>  Compatible with up to 6.2 which is EOL on Feb 2025
> > >>  Not compatible with 6.4 or 7.x which are latest
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0 (RC2)

[Joe Witt]

+1 binding

Checked all the things.
Ran a real flow.
Ran vulnerability analysis and the results are amazing!  Clean run which
has to be a first.

NiFi 2.0 is a multi-year journey and amazing work by the community.  It
sets the NiFi user base up for continued great success and growth of use
cases.

thanks everyone.  thanks david!

On Tue, Oct 29, 2024 at 8:25 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1294/
>
> Git Tag: nifi-2.0.0-RC2
> Git Commit ID: 2f13b609bdb77cb985aa35e8b66b4f04274d7c59
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/2f13b609bdb77cb985aa35e8b66b4f04274d7c59
>
> Checksums of nifi-2.0.0-source-release.zip
>
> SHA512:
> deff4c81842759ef14651874a1b3204f75d0287a578174505b409ab2869704ed77527a5e262dd030bb7a5d1531a8d55105b015444c5802f55caf80439fe11220
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 358
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354889
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Request for maintainers on extensions with libraries that are dated/EOL

[Joe Witt]

Team,

These need help from maintainers to keep them updated or they risk
requiring removal.

Problematically Dated libraries we have shown as:
package-name currVersion latestVersionAvail

com.dropbox.core 5.4.6 7.0.0
 One year old

org.apache.camel/salesforce 3.14.10 4.8.1
 Already EOL for two years.

org.mongodb 4.11.4 5.2.0
   Compatible with Mongo 6 and 7 which are still supported. (all older are
EOL)
   Not compatible with Mongo 8 which is latest.

org.questdb 7.4.2 8.1.4

org.snmp4j 2.8.18 3.8.2

redis.clients 3.10.0 5.2.0
 Compatible with up to 6.2 which is EOL on Feb 2025
 Not compatible with 6.4 or 7.x which are latest

Thanks
Joe

Re: [VOTE] Release Apache NiFi 1.28.0 (RC1)

[Joe Witt]

Team,

+1 binding (with strong reservations for any future 1.x releases that don't
turn the tide of reported vulnerable libs).

Did the normal check stuff and build and it seems good.

Where I have delayed in responding is in digging into the nature of the
reported vulnerabilities for this release using various scanning methods.

We continue to drift on the 1.x line further away from what our user base
does and should come to expect from us in terms of managing libraries with
reported vulnerabilities.  We know some of this but it also serves as a
clear signal we need to now actively move away from the 1.x line and focus
on making 2.x reflect the best of our efforts as well as help users move
there.

But for the findings...

Using the dependency-check plugin for comparison here.  This is not perfect
but it is a good first approximation.

The 1.x release has 169 dependencies (out of 2500 unique ones) with
reported vulnerabilities.  Of course that still does not mean they are
actually vulnerable as used in nifi nor does it mean there are not false
positives.  However, by comparison the 2.x release has 19.

On the 1.x line the bulk of these are related to various dependencies of
Hadoop, Iceberg, Parquet but also things like JRuby, Neo4j, Ignite, H2,
Derby, Cassandra, Jetty, and more.  It is pretty daunting and crazy to look
at.  Some of this is long standing and known but some may be and do appear
more recent.  Would we know?  Would we catch them or would they get lost in
the noise.

While there was a recent pass of updated dependency versions some of these
take real effort and digging into for them to go away.

Also this should serve as a general advisory to us that we should perhaps
no longer be so stringent on keeping certain components around simply
because we think there are a lot of users for them.  If we can't get rid of
vulnerabilities or manage them well because of components that seemingly
forever have reported vulnerabilities they should be strong candidates to
get dropped from our releases and codebase.

I will not be supportive of further 1.x releases unless they're of the
incremental variety and reduce these vulnerabilities.

Thanks

On Tue, Oct 22, 2024 at 3:13 PM Lehel Boér  wrote:

> +1 (non-binding)
>
> Ran through release helper, verified successful build.
>
> Conf:
> Apache Maven 3.9.9
> openjdk version "1.8.0_382"
> OpenJDK Runtime Environment Corretto-8.382.05.1 (build 1.8.0_382-b05)
> OpenJDK 64-Bit Server VM Corretto-8.382.05.1 (build 25.382-b05, mixed mode)
> Default locale: en_US, platform encoding: UTF-8
> ProductName: macOS
> ProductVersion: 14.5
> BuildVersion: 23F79
>
> Checked a few commits.
>
>
> Thanks for RM'ing Ferenc!
>
> 
> From: Dan S 
> Sent: Tuesday, October 22, 2024 15:58
> To: dev@nifi.apache.org 
> Subject: Re: [VOTE] Release Apache NiFi 1.28.0 (RC1)
>
> +1 non-binding
>
> Verified checksum, signatures and hashes
> Built the source code on
> Java version: 1.8.0_412, vendor: Red Hat, Inc., runtime:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.el8.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.18.0-513.24.1.el8_9.x86_64", arch: "amd64",
> family: "unix"
>
> Ran a simple flow to confirm the fix for NIFI-13418
> . As a result uncovered
> a
> bug reported in NIFI-13922
> .
>
> Thanks for Rming Ferenc!
>
> On Tue, Oct 22, 2024 at 4:41 PM Kedvessy Péter 
> wrote:
>
> > +1 non-binding
> >
> > Verified the followings:
> >   - checksum and signature
> >   - git commit hash and tag
> >   - NiFi build with contrib-check on the following environment:
> > - mac os x 14.3.1
> > - openjdk 17.0.11 2024-04-16
> >   OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
> >   OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed
> > mode)
> > - maven 3.9.9
> >   - simple flow execution
> >   - new minifi java agent features
> >
> > Thanks for managing this release, Ferenc.
> >
> > Regards,
> > Peter
> >
> > Arpad Boda  ezt írta (időpont: 2024. okt. 22., K,
> > 17:37):
> >
> > > +1 (binding)
> > >
> > > Verified checksum, signature.
> > > Built and executed tests on Mac, started a simple flow, everything
> worked
> > > as expected.
> > > Thanks for working on this release, Ferenc!
> > >
> > > Thanks,
> > > Arpad
> > >
> > >
> > > On Tue, Oct 22, 2024 at 12:46 PM Ferenc Kis 
> > > wrote:
> > >
> > > > Team,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi
> > > > 1.28.0.
> > > >
> > > > Please review the following guide for how to verify a release
> candidate
> > > > build:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > >
> > > > The source being voted on the and the convenience binaries are
> > available
> > > on
> > > > the Apache Distribution R

Re: [VOTE] Release Apache NiFi 2.0.0 (RC1)

[Joe Witt]

+1 (binding)

We kicked off the discussion thread for doing a 2.0 release in July 2021 :)

Thanks Everyone!

Joe

On Tue, Oct 22, 2024 at 12:37 PM Robert Fellows 
wrote:

> +1 (non-binding)
>
> I focused mainly on the new user interface. looks really good but noted a
> few issues that should get addressed. Nothing that I would suggest sinking
> the RC for though.
>
> Otherwise, It is looking good! So Exciting!
>
>- Light mode - Drag icons of components are nearly invisible
>   - https://issues.apache.org/jira/browse/NIFI-13920
>- Cancelling of edit dialog of current process group does not remove the
>"/edit" from the route
>   - https://issues.apache.org/jira/browse/NIFI-13921
>
>
> Java Details:
> openjdk version "21.0.1" 2023-10-17 LTS
> OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
> OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed mode,
> sharing)
>
> Built on Mac OS 14.7 (Sonoma)
>
> On Tue, Oct 22, 2024 at 1:03 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 2.0.0.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are
> > available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1293/
> >
> > Git Tag: nifi-2.0.0-RC1
> > Git Commit ID: 34160740980c2d9d5cd0a6c399e09c2bd06d5acb
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/34160740980c2d9d5cd0a6c399e09c2bd06d5acb
> >
> > Checksums of nifi-2.0.0-source-release.zip
> >
> > SHA512:
> >
> 09f37a4f8fd798a8b067d7970ea571ebc2f729c98cffd427e00d41a0b9312bbcd6b074a1f9109304939b89b5074ccdc48feeff712ecce3f3b1e1a358da7e4a57
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 323
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354889
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [] +1 Release this package as nifi-2.0.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>
>
> --
> ---
> Rob Fellows
>

Re: [DISCUSS] Preparing Release of Apache NiFi 2.0.0

[Joe Witt]

This is email so I guess emoji's as we'd have in the Slack world aren't a
thing.

Cue the flashing huge +++1 emoji.

Let's do it!

Thanks

On Wed, Oct 16, 2024 at 9:00 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> It has been almost one year since we released Milestone 1 of NiFi
> 2.0.0. Over the four milestone releases, we have resolved almost 2000
> Jira issues, added numerous capabilities, and removed large amounts of
> technical debt. Although there is always more that could be done, we
> have reached the point where we should release a stable 2.0.0 version.
>
> The current release version [1] has almost 300 Jira issues resolved.
> With the release of NiFi API 2.0.0 and the NiFi NAR Maven Plugin
> 2.1.0, we are in a solid position to go forward with a 2.0.0 version.
> Although there are bound to be additional areas we want to improve, we
> should consider those after a general availability release.
>
> With that background, I would be glad to handle Release Manager
> responsibilities for this version. Unless there are any critical
> blockers, I plan to start the release candidate process before the end
> of this week. That should allow time for a few more items to be
> completed.
>
> Regards,
> David Handermann
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354889
>

Re: NIFI processor getting hanged

[Joe Witt]

ArunV

Pod memory is one thing.  Think of that sort of like RAM to a vm.  There is
then the heap size for the JVM which is far more directly relevant.  In
general it requires fairly good understanding of RAM/Heap/various OS memory
usage aspects to dial these in correctly.

When you increase the heap and experience better behavior for longer but
then the same behavior - this is a strong signal to a memory leak and I'd
suggest to start by focusing in the code mentioned and the class member
variables.

Thanks

On Thu, Oct 10, 2024 at 7:51 AM Varadarajan, Arun <
arun.varadara...@paramount.com> wrote:

> Hi , thanks for your response. From the metrics, we see the POD memory is
> always less than what we have set as limit. And increasing the memory
> doesn’t help, though we see less GC over head errors, after increasing the
> memory.
>
>
>
>
>
> *From:* Joe Witt 
> *Sent:* Thursday, October 10, 2024 8:17 PM
> *To:* dev@nifi.apache.org
> *Subject:* Re: NIFI processor getting hanged
>
>
>
> 
>
>
>
> Arun V
>
>
>
> This is a strong indication to review any custom components or processors
> in the flow which can consume large quantities of memory.
>
>
>
> OutOfMemory errors in a general sense indicate either a single large thing
> caused the heap to be exhausted or you see errors like this about overhead
> limit exceeded which indicates another type which is there is simply too
> much retained information in the heap.
>
>
>
> Look at your code for the CableLabsDataMappingProcessor for instance.
> Does it have any class member variables that hold stuff like any kind of
> maps/caches/etc.. If so I'd start there.  This sounds, with limited
> information, like a memory leak.  When you increased the size of the memory
> did the flow stay online longer than before by some time interval?  Point
> is increasing memory buys time but does not solve heap exhaustion in the
> event of a memory leak.
>
>
>
> Thanks
>
>
>
> On Thu, Oct 10, 2024 at 5:51 AM Varadarajan, Arun <
> arun.varadara...@paramount.com> wrote:
>
>
> Hi Team,
>
> We see our NIFI workflow getting hanged intermittently once / twice per
> week. After which , even if we restart the processor [say ConsumeAMQP] ,
> this is not consuming the messages from the RMQ [though the queue had
> messages].
>
> After restarting the POD [in K8] , the workflow functions as expected.
> Please note , we had already increased the POD memory and the issue is
> still occurring.
>
> Also we are observing the below error for some time , and want to check if
> this is the cause and if so , what could be the fix.
>
> CableLabsDataMappingProcessor[id=8a6f5d24-1fa5-38c6-1748-d1e9fe54ee93]
> failed to process session due to java.lang.OutOfMemoryError: GC overhead
> limit exceeded; Processor Administratively Yielded for 1 sec:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2024-10-09 00:51:45,229 WARN [Timer-Driven Process Thread-1]
> o.a.n.controller.tasks.ConnectableTask  Administratively Yielding
> CableLabsDataMappingProcessor[id=8a6f5d24-1fa5-38c6-1748-d1e9fe54ee93] due
> to uncaught Exception: java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Any help would be greatly appreciated.
>
> Regards,
> Arun V
>
>

Re: NIFI processor getting hanged

[Joe Witt]

Arun V

This is a strong indication to review any custom components or processors
in the flow which can consume large quantities of memory.

OutOfMemory errors in a general sense indicate either a single large thing
caused the heap to be exhausted or you see errors like this about overhead
limit exceeded which indicates another type which is there is simply too
much retained information in the heap.

Look at your code for the CableLabsDataMappingProcessor for instance.  Does
it have any class member variables that hold stuff like any kind of
maps/caches/etc.. If so I'd start there.  This sounds, with limited
information, like a memory leak.  When you increased the size of the memory
did the flow stay online longer than before by some time interval?  Point
is increasing memory buys time but does not solve heap exhaustion in the
event of a memory leak.

Thanks

On Thu, Oct 10, 2024 at 5:51 AM Varadarajan, Arun <
arun.varadara...@paramount.com> wrote:

>
> Hi Team,
>
> We see our NIFI workflow getting hanged intermittently once / twice per
> week. After which , even if we restart the processor [say ConsumeAMQP] ,
> this is not consuming the messages from the RMQ [though the queue had
> messages].
>
> After restarting the POD [in K8] , the workflow functions as expected.
> Please note , we had already increased the POD memory and the issue is
> still occurring.
>
> Also we are observing the below error for some time , and want to check if
> this is the cause and if so , what could be the fix.
>
> CableLabsDataMappingProcessor[id=8a6f5d24-1fa5-38c6-1748-d1e9fe54ee93]
> failed to process session due to java.lang.OutOfMemoryError: GC overhead
> limit exceeded; Processor Administratively Yielded for 1 sec:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2024-10-09 00:51:45,229 WARN [Timer-Driven Process Thread-1]
> o.a.n.controller.tasks.ConnectableTask  Administratively Yielding
> CableLabsDataMappingProcessor[id=8a6f5d24-1fa5-38c6-1748-d1e9fe54ee93] due
> to uncaught Exception: java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Any help would be greatly appreciated.
>
> Regards,
> Arun V
>
>

Oct 24 Board report for Apache NiFi

[Joe Witt]

Team

Below is what I've submitted to the board for this quarter.  Thanks again!

Joe


---

## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High. Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (9 years ago) There are currently 67
committers and 37 PMC members in this project. The Committer-to-PMC ratio is
roughly 9:5.

Community changes, past quarter:
- No new PMC members. Last addition was Gabor Gyimesi on 2024-06-26.
- No new committers. Last additional was Lehel Boer on 2024-06-16.

## Project Activity:
The big push remains focused on making Apache NiFi 2.0.0 happen.  We've
completed four huge milestone releases to that and it looks likely our next
release will be officially NiFi 2.0.0 and for GA usage.

Toward making NiFi 2.0.0 official we've broken out into its own release the
NiFi API which helps codify our core contract of backwards compatibility and
points of extension for the project. We released the Apache NiFi API 2.0.0
on
September 23, 2024.

To leverage packaging NARs that understand the split of the NiFi API vs the
core framework version now we also have the latest NiFi Nar Maven Plugin
2.1.0
as of Sep 26th.

MiNiFi CPP continues to see JIRAs and PR activity as well.

## Community Health:
JIRA and Mailing list activity remains consistent.

The slack community grew by another 160 participants in the general channel
alone increasing from 3,284 in our previous report and consistent with
quarter
over quarter growth for some time.  Slack is generally quite busy with
threads
from people needing general user help or questions, development ideas, and
review requests.

As noted above we've split out the NiFi API.  Part of this was so we can
introduce a NiFI Improvement Process for changes to our core API.  This
process was inspired by the tremendous growth the communities of Kafka and
Airflow have produced and these communities have healthy discussions on how
to
introduce changes to critical sections.

We continue to produce frequent releases in the NiFi community and enjoy
rather active vote participation.  Releases we generate are substantial
along
all key dimensions of feature, improvements, bug fixes, and emphasis on
security.  The NiFi
2.0 release line is among the most vulnerability free we've ever had when it
comes to dependency management.

Mailing lists remain quite active despite the continued increase of activity
in Slack.

Re: [DISCUSS] Release Timing for NiFi 1.28.0

[Joe Witt]

Hello

Dependency maintenance was referenced but seems largely focused on cloud
service updates.  There are many other dependencies which need maintaining
and there are many such dependencies which are listed as having HIGH or
CRITICAL vulnerabilities in their version.

There needs to be considered effort to maintain dependencies (broadly) and
ensure vulnerability reports are reviewed.  This work is much more
laborious on the 1.x line than the 2.x line to be clear.  And some
dependencies cannot be addressed as they abandoned older JVM lines long
ago.  But the things which can be updated do need to be.

At a quick glance things we have there which need to be addressed solely
based on reported vulnerabilities (this is not even close to complete and
the outdated dependency list is dramatically larger):
JRuby (ftp/yaml,etc.. libs with high/critical)
Neo4j
Jetty (though we can't solve it entirely)
Various Hadoop/parquet/etc.. things which need to be updated to something
recent
h2
derby
cypher-gremlin
netty
legacy kafka clients
activemq
spring-web/spring-webmvc
protobuf-java many versions
commons-config
jjwt
sshd
guava
commons-httpclient
commons-compress
spring-*
questdb

It isn't reasonable for us to produce feature bearing releases without in
each case ensuring we're updating to the best of our options (Java 8/11/17
compatible libraries).

Thanks
Joe

On Fri, Sep 27, 2024 at 10:18 AM Michael Moser  wrote:

> There is a PR for https://issues.apache.org/jira/browse/NIFI-13549 which
> only affects the 1.x codebase.  It seems fairly straight forward, so should
> be considered for 1.28
>
> Thanks,
> -- Mike
>
>
> On Fri, Sep 27, 2024 at 10:13 AM Ferenc Kis 
> wrote:
>
> > Hi Team,
> >
> > Let's recap on the status.
> >
> > I'm aware of four tickets we are waiting for to get into the NiFi 1.28
> > release.
> >
> >- https://issues.apache.org/jira/browse/NIFI-13776: this is currently
> >being reviewed on the 2.x line. My guess is once it's been merged into
> >main, we can merge the PR for the 1.x line as well
> >- https://issues.apache.org/jira/browse/NIFI-13776: this is a 1.x
> only
> >bugfix, and being reviewed
> >- https://issues.apache.org/jira/browse/NIFI-13765: Jira is closed,
> it
> >turned out the issue does not impact the 1.x line
> >- https://issues.apache.org/jira/browse/NIFI-13782: it seems it's
> more
> >complex than expected and needs additional efforts to complete
> >
> > In summary I think we should wait for the first two tickets as those are
> > near to completion, then kick off the release.
> >
> > What do you think? Is there anything else we should add to the release?
> >
> > Regards
> > Ferenc Kis
> >
> > On Thu, Sep 19, 2024 at 4:51 PM Shane Ardell 
> > wrote:
> >
> > > I'm also working on https://issues.apache.org/jira/browse/NIFI-13765.
> > > Although not critical, it would be nice to have this in the next
> release.
> > > The fix is simple and I should have a PR open by tomorrow.
> > >
> > > Best,
> > > Shane
> > >
> > > On Thu, Sep 19, 2024 at 7:23 AM Ferenc Kis 
> > > wrote:
> > >
> > > > Thank you for your feedback
> > > >
> > > > I think it is worth to wait for both fixes as those seem to fix
> > critical
> > > > issues
> > > >
> > > > Let's reiterate on this next week
> > > >
> > > > Regards
> > > > Ferenc Kis
> > > >
> > > > On Thu, Sep 19, 2024 at 12:40 AM Dan S  wrote:
> > > >
> > > > > I am also working on NIFI-13744
> > > > >  which is a bug
> in
> > > how
> > > > > ExcelReader handles dates, times and timestamps. I am also
> targeting
> > > for
> > > > a
> > > > > fix by the end of the week.
> > > > >
> > > > > On Wed, Sep 18, 2024 at 4:31 PM Mike Thomsen <
> mikerthom...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I found a serious bug in how copys3object handles large files and
> > am
> > > > > > working on a fix. I should have it ready by the end of the week,
> > but
> > > > > > without it the processor is broken on large file copy actions.
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Sep 18, 2024, at 10:41 AM, Lucas Ottersbach <
> > > > > > lucas.ottersb...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hey Ferenc,
> > > > > > >
> > > > > > > thank you for starting the discussion. It sounds like a
> > reasonable
> > > > > amount
> > > > > > > of time has passed and changes were introduced since the last
> > > > release.
> > > > > > > Waiting for the one pending PR sound reasonable.
> > > > > > > Definitely a +1 from my side.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Lucas
> > > > > > >
> > > > > > > Ferenc Kis  schrieb am Mi., 18. Sept.
> > > 2024,
> > > > > > 12:33:
> > > > > > >
> > > > > > >> Hi Team,
> > > > > > >>
> > > > > > >> I'd like to initiate a discussion about the next release of
> NiFi
> > > > 1.x.
> > > > > > >> Since the release of Apache NiFi 1.27, several important bugs
> > [1]
> > > > > > including
> > > > > > 

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 2.1.0 (RC1)

[Joe Witt]

i appear to have missed the boat by approximately 30 seconds...

but +1 including testing a full nifi build with nifi api 2.0 and this.

thanks!

On Wed, Sep 25, 2024 at 2:33 PM David Handermann <
exceptionfact...@apache.org> wrote:

> +1 binding
>
> On Mon, Sep 23, 2024 at 2:03 PM David Handermann
>  wrote:
> >
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi NAR Maven Plugin 2.1.0.
> >
> > Please review the following guide for how to verify a release candidate
> build:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification+for+NiFi+NAR+Maven+Plugin
> >
> > The source being voted on the and the convenience binaries are
> > available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-2.1.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1283/
> >
> > Git Tag: nifi-maven-2.1.0-RC1
> > Git Commit ID: 33591a2b3cc2134218f6a56ed3fe4a5eb2e3fdad
> > GitHub Commit Link:
> >
> https://github.com/apache/nifi-maven/commit/33591a2b3cc2134218f6a56ed3fe4a5eb2e3fdad
> >
> > Checksums of nifi-nar-maven-plugin-2.1.0-source-release.zip
> >
> > SHA512:
> 098eeecc7e9217d5676320688c405eae26db502eb86098b2aafb7a02f5c9651da31558affec9804efbb387835d625d899a588b6f38ca059c1a82769b05e0e785
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 1
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12355125
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion2.1.0
> >
> > The vote will be open for 48 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [] +1 Release this package as nifi-nar-maven-plugin-2.1.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
>

Re: Inquiry regarding feasibility of enabling NewRelic APM for Nifi workflows

[Joe Witt]

Hello

Please subscribe to the mailing list so you see replies.  You received a
reply in this thread but it goes to the list and not you directly. I just
manually added you in cc here.

Thanks

On Wed, Sep 25, 2024 at 10:03 AM T, Murugan  wrote:

> Hi Team,
>
> Just want to follow up, Has anyone had a chance to check this ask ?
>
> Thanks,
> Murugan
>
> From: T, Murugan 
> Date: Monday, 23 September 2024 at 7:44 PM
> To: dev@nifi.apache.org 
> Cc: Varadarajan, Arun , Arumugam,
> SenthilKumar , Nagar, Pankaj <
> pankaj.na...@paramount.com>
> Subject: Inquiry regarding feasibility of enabling NewRelic APM for Nifi
> workflows
> Hi Team,
>
> I hope this message finds you well !
> I am writing to inquire about the feasibility of integrating Apache Nifi
> with New Relic for monitoring and performance management.
> We are exploring option to enable New Relic APM agent for one of our Nifi
> workflow, but we are facing issue that Nifi metrics was not captured and
> would like to understand if there are any existing solutions or
> recommendations for setting up this integration.
>
> Nifi version that we are currently using is, v1.27.0
> And we are deploying our Nifi in Kubernetes environment using Docker. We
> use below commands in our Docker file to download New Relic jar
>
> # Download the new relic jar
> ADD
> https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-agent.jar
> /opt/newrelic/newrelic.jar
> COPY config/newrelic.yml /opt/newrelic/
> RUN chmod -R 755 /opt/newrelic/
>
> Java agent command that we used in our start up script,
>
> exec java -javaagent:/opt/newrelic/newrelic.jar \
>   -jar nifi-deploy-config-viacom-1.3.1-jar-with-dependencies.jar \
>   -nifi http://$CONTAINER_IP:8080/nifi-api \
>   -conf musicvideo_ingestion_low.yml musicvideo_publish.yml \
>   -mode deploy 2>&1 >&1
>
> Specifically, we would like to monitor key performance metrics, system
> health and workflows in Nifi using New Relic’s platform. If this
> integration is possible, could you please provide guidance or documentation
> on how to achieve it ?
>
> Thank you for your assistance, look forward to your response.
>
> Regards,
> Murugan
>

Re: [DISCUSS] Python processor dependency management

[Joe Witt]

Gabor

Thanks.  While I understand the logical grouping *these all do doc parsing
things* why is it important for them to be in the same package? Why not
have separate document parsing packages each which can be built/deployed
separately?

Thanks

On Tue, Sep 24, 2024 at 9:29 AM Gábor Gyimesi  wrote:

> David, Joe,
>
> You are right, it's easier to understand such a use case with an
> example. We currently have a ParseDocument processor in our python
> extensions with PLAIN_TEXT, HTML, MARKDOWN, PDF, WORD, EXCEL,
> POWERPOINT input format support, using the unstructured library on its
> own or through langchain. The unstructured library has support for
> several input formats and depending on that format it provides
> additional support in extensions like unstructured[pdf],
> unstructured[csv] and so on.
>
> This ParseDocument could be quite cumbersome on its own so let's say
> we would like to create separate processors for each format. We would
> create a "parse_document" package and have separate processors for
> each document format like ParseText, ParsePdf, ParseCSV, then we would
> like to install only the format specific unstructured package
> extension for a processor. In this case a ParseText would only require
> the base unstructured pip package, but for ParsePdf that would require
> unstructured[pdf] which also has some very large transitive
> dependencies like nvidia cuda packages and pytorch. The
> unstructured[pdf] package installs almost 6GB of dependencies in the
> latest release, which is unnecessary if we only want to use a much
> more lightweight processor from the same package like a ParseText
> processor for example.
>
> But this is just one example, I think there could be other use cases
> as well, when the same package that contains logically inseparable
> processors could have different dependencies and should not be
> installed in a processor specific virtual environment if they are not
> used.
>
> Regards,
> Gabor Gyimesi
>
> On Tue, 24 Sept 2024 at 16:52, David Handermann
>  wrote:
> >
> > Gabor,
> >
> > On a similar note, it would be helpful to provide a concrete example.
> >
> > Unlike Java NARs, Python Processors do not have the same concept of
> > multiple layers of parent class loaders right now. Virtual
> > environments provide dependency sharing, but there isn't the same
> > concept of sharing dependencies. Attempting to implement something
> > similar to NAR hierarchy for Python Processors presents some important
> > questions that would have to be addressed.
> >
> > Having a concrete example where to consider these complexities would
> > be a helpful way to evaluate whether it makes sense to introduce
> > additional dependency loading concepts for Python.
> >
> > Regards,
> > David Handermann
> >
> > On Tue, Sep 24, 2024 at 9:41 AM Joe Witt  wrote:
> > >
> > > Gabor
> > >
> > > Can you please describe a specific case or cases where ProcessorA and
> > > ProcessorB should be in the same package/module and yet have such
> vastly
> > > different (100s of MB or even GB) of dependency requirements?
> > >
> > > Thanks
> > > Joe
> > >
> > > On Tue, Sep 24, 2024 at 7:32 AM Ferenc Gerlits 
> wrote:
> > >
> > > > Hi Gabor,
> > > >
> > > > I like this approach, and I think the restriction you propose (that
> > > > all utility files in the package use the same dependencies, and extra
> > > > dependencies for processor A are only used in ProcessorA.py) is
> > > > reasonable.  I would be happy to implement this if there are no
> > > > objections.
> > > >
> > > > Thanks,
> > > > Ferenc
> > > >
>

Commit messages in the mono-repo of nifi.git

[Joe Witt]

Team,

NIFI-13762 is a good example but we need to try and remember to have good
commit messages.  The repo as a whole is for nifi itself, minifi (java),
c2, the registry, etc..

While the context can always be assumed to be nifi if committing for
registry or minifi or the c2 protocol the commit message should make that
clear.

The commit that just went into the commit log says

"NIFI-13762 Expose processor metrics as a part of FlowInfo"

My immediate thought was there is no way we should be doing such a thing.
Then I traced the path to find this was added support for a minifi concept
in the c2 protocol.  Please do clarify in the commit messages then what the
change was.

A more helpful commit message then would be

NIFI-13762 Add support for exposing processor metrics in FlowInfo to the C2
protocol and use in MiNiF

Thanks
Joe

Re: [DISCUSS] Python processor dependency management

[Joe Witt]

Gabor

Can you please describe a specific case or cases where ProcessorA and
ProcessorB should be in the same package/module and yet have such vastly
different (100s of MB or even GB) of dependency requirements?

Thanks
Joe

On Tue, Sep 24, 2024 at 7:32 AM Ferenc Gerlits  wrote:

> Hi Gabor,
>
> I like this approach, and I think the restriction you propose (that
> all utility files in the package use the same dependencies, and extra
> dependencies for processor A are only used in ProcessorA.py) is
> reasonable.  I would be happy to implement this if there are no
> objections.
>
> Thanks,
> Ferenc
>

Re: [DISCUSS] Planning Release of nifi-nar-maven-plugin

[Joe Witt]

David

+1 lets do it.

Peter: good catch and thanks for putting up a PR for it!

Joe

On Mon, Sep 23, 2024 at 10:37 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> As we continue making progress towards NiFi 2.0.0, we need to release
> an updated version of the NiFi NAR Maven Plugin.
>
> Peter Turcsanyi addressed an issue that currently requires matching
> versions of nifi-api and nifi-framework-api in the NAR plugin,
> described in NIFI-13773 [1].
>
> With the changes reviewed and merged, releasing a new version will
> allow incorporation of the NiFi API 2.0.0 release back into the main
> NiFi repository.
>
> I will be glad to handle Release Manager duties for this version.
> Given the narrow scope of changes, I think a 48 hour window for voting
> seems reasonable. I plan to prepare the release and call for a vote
> soon.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-13773
>

Re: [VOTE] Release Apache NiFi API 2.0.0 (RC1)

[Joe Witt]

+1 binding!

Thanks

On Thu, Sep 19, 2024 at 11:43 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi API 2.0.0.
>
> This is the first release of nifi-api as a separate module, following
> the recent vote in favor:
>
> https://lists.apache.org/thread/p6d7qsyto8x8xqzr6scp3fp33lrbb82c
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification+for+NiFi+API
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-api-2.0.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1282/
>
> Git Tag: nifi-api-2.0.0-RC1
> Git Commit ID: 0a171b92813959cc4d84727b7b5c74c763fd4292
> GitHub Commit Link:
>
> https://github.com/apache/nifi-api/commit/0a171b92813959cc4d84727b7b5c74c763fd4292
>
> Checksums of nifi-api-2.0.0-source-release.zip
>
> SHA512:
> fe9790322797f80283c22b803ff8e8dbe17385e2e81349299a6b8d05be820a746d06dd75ea1ae26c9144f5e5f7c5ff349cc9681e5f4ab283ea91001411d74cbf
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 1
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12355111
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiAPIVersion2.0.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-api-2.0.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: Upgrading NiFi to latest compatible versions using JDK 17

[Joe Witt]

Adding to Mike's comments

The 1.x line should work on Java 8, 11, and 17.  21 will not work.

The 2.x line can absolutely be used for production applications and
certainly already is.  However, please note we may still break things prior
to officially removing the 'mX' nomenclature.  We periodically get close to
cutting the official 2.0 release then find other tech debt to destroy but
we're close!

Thanks



On Wed, Sep 4, 2024 at 7:17 AM Mike Thomsen  wrote:

> 1) Newer 1.X should still be 100% compatible with Java 11 and 17
> simultaneously.
>
> 2) I would recommend against that because while they're very stable,
> they're technically preview releases/release candidates so the optics with
> management could be really bad for you and your team.
>
> On Wed, Sep 4, 2024 at 9:05 AM Nagar, Pankaj 
> wrote:
>
> > Hi Team,
> >
> > We have been using NiFi 1.4.1 version from many years, but we are forced
> > to move to java 17 . But we found out that latest stable image of NiFi is
> > using Java 11. Can you please suggest on following :
> >
> > 1. Which NiFi version can be used along with JDK 17 ?
> > 2. Can milestone(2.0.0-Mx) versions be used for Production applications ?
> >
> > Please share any documents links related to this.
> >
> > Thanks,
> > Pankaj
> >
>

Re: [VOTE] Adopt NiFi Improvement Proposal Process and Move nifi-api

[Joe Witt]

+1 binding

On Thu, Sep 5, 2024 at 6:46 PM David Handermann 
wrote:

> Team,
>
> Following a positive discussion [1] I am pleased to be calling for a
> vote on the following:
>
> 1. Adopt the NiFi Improvement Proposal [2] process for future efforts
> 2. Move the nifi-api module [3] to a separate source repository
>
> The vote will be open until Monday, September 9.
>
> [] +1 Adopt the NiFi Improvement Proposal process and move the nifi-api
> module
> [] +0 No opinion
> [] -1 Do not adopt the NiFi Improvement Proposal process or move the
> nifi-api module
>
> Regards,
> David Handermann
>
> [1] https://lists.apache.org/thread/4j5d460oqrz5kjo2zyd2xoc3dp7lswt2
> [2]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process
> [3] https://github.com/apache/nifi/tree/main/nifi-api
>

Re: [DISCUSS] Introducing NiFi Improvement Proposals

[Joe Witt]

Thanks David.

Looks like a solid starting point.


On Fri, Aug 30, 2024 at 8:15 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I have drafted an initial version of the NiFi Improvement Proposal Process:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process
>
> The document outlines the scope of project components requiring a
> proposal, as well as the requirements for a proposal, and the review
> process.
>
> The general approach for review and approval is designed to be similar
> to the release process.
>
> The requirements for a proposal are similar to other project
> improvement proposals.
>
> I recommend creating a new Jira project for handling Improvement
> Proposals, as opposed to using Confluence pages.
>
> Please review the draft and I will look to call for a vote within the next
> week.
>
> As mentioned earlier, with the nifi-api module being the core contract
> for internal and external extensions, I would like to include moving
> the nifi-api to a separate Git repository as part of the vote for
> instituting the Improvement Proposal Process. If this proves to be too
> much of a concern, we can divide the question and vote separately on
> the Improvement Proposal Process and moving the nifi-api.
>
> Thanks for the feedback!
>
> Regards,
> David Handermann
>
> On Wed, Aug 21, 2024 at 11:09 AM David Handermann
>  wrote:
> >
> > Thanks for the replies!
> >
> > I will move forward with drafting a NiFi Improvements Proposal page to
> > keep the discussion going.
> >
> > Regards,
> > David Handermann
> >
> > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne  wrote:
> > >
> > > David,
> > >
> > > +1 to all of this. Especially for guaranteeing compatibility and
> maintainability, I think instrumenting a more
> > > formal approach for updating the API is a step in the right direction.
> The huge amount of purging, cleanup,
> > > and refactoring that has gone into 2.0 helps to highlight the
> importance of ensuring maintainability.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > > > On Aug 20, 2024, at 2:00 AM, Joe Witt  wrote:
> > > >
> > > > David
> > > >
> > > > Yeah well written and good points.
> > > >
> > > > I dont think this says we would relax our existing commitment we
> have shown
> > > > for things such as the http based api but rather is calling out
> specific
> > > > things which we will make changes to even more formalized.
> > > >
> > > > To that end +1 on the concept of proposals and +1 to breaking out the
> > > > nifi-api (fundamental designed extension points for flow
> development) in
> > > > particular.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft  wrote:
> > > >
> > > >> These are all really appreciated concepts, David. Thank you for
> putting the
> > > >> thoughts and time in on this!
> > > >>
> > > >> Regarding this:
> > > >>> The NiFi REST API does not have quite the same level of concern,
> but may
> > > >> warrant inclusion.
> > > >>
> > > >> I hear what you're saying. However, the REST API (from my
> > > >> observation/experience) has gathered quite a number of useful tools
> and
> > > >> "hacks" for NiFi. Quite often, many different monitoring and
> alerting tools
> > > >> have been developed against the REST API by third parties and/or
> > > >> integrators of NiFi against their internal workflows. Having stable
> API
> > > >> versioning in the REST API possibly makes just as much sense as
> having the
> > > >> same for the nifi-api itself. This is a prime entry point for
> extensions
> > > >> and other features developed alongside NiFi, maybe even the weird
> stuff
> > > >> that you can't do with the nifi-api directly.
> > > >>
> > > >> Food for thought of course, but I would hope that we can treat the
> REST API
> > > >> as a proper first-class citizen in terms of documented versioning.
> It turns
> > > >> out it's quite a useful means for interacting with a running NiFi
> instance.
> > > >>
> > > >> /Adam
> > > >>
> > > >> On Fri, Aug 16, 2024 at 9:

Re: [DISCUSS] Introducing NiFi Improvement Proposals

[Joe Witt]

David

Yeah well written and good points.

I dont think this says we would relax our existing commitment we have shown
for things such as the http based api but rather is calling out specific
things which we will make changes to even more formalized.

To that end +1 on the concept of proposals and +1 to breaking out the
nifi-api (fundamental designed extension points for flow development) in
particular.

Thanks
Joe

On Fri, Aug 16, 2024 at 8:20 PM Adam Taft  wrote:

> These are all really appreciated concepts, David. Thank you for putting the
> thoughts and time in on this!
>
> Regarding this:
> > The NiFi REST API does not have quite the same level of concern, but may
> warrant inclusion.
>
> I hear what you're saying. However, the REST API (from my
> observation/experience) has gathered quite a number of useful tools and
> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
> have been developed against the REST API by third parties and/or
> integrators of NiFi against their internal workflows. Having stable API
> versioning in the REST API possibly makes just as much sense as having the
> same for the nifi-api itself. This is a prime entry point for extensions
> and other features developed alongside NiFi, maybe even the weird stuff
> that you can't do with the nifi-api directly.
>
> Food for thought of course, but I would hope that we can treat the REST API
> as a proper first-class citizen in terms of documented versioning. It turns
> out it's quite a useful means for interacting with a running NiFi instance.
>
> /Adam
>
> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Team,
> >
> > As we wrap up remaining items for NiFi 2, we should consider how to
> > continue improving the quality and maintainability of the NiFi
> > ecosystem.
> >
> > One primary focus of NiFi 2 has been the reduction of technical debt,
> > which involved the removal of numerous modules and thousands of lines
> > of code. In that process, it is worth highlighting that the core NiFi
> > API, and the NiFi Framework API, and the NiFi REST API have had
> > comparatively few breaking changes. This a testament to the quality of
> > the API design itself. The NiFi Version Schema and API Compatibility
> > [1] has provided a strong direction thus far.
> >
> > With that background, we should consider adopting a more formal
> > process around changes that impact the fundamental API contracts that
> > NiFi provides. NiFi Feature Proposals [2] have provided elements of
> > this in the past, but did not include approval requirements. Kafka [3]
> > and Airflow [4] have more structured improvement proposal processes,
> > and that is what we should adopt going forward.
> >
> > Part of moving in this direction requires identifying the areas that
> > would require going through the Improvement Proposal process itself.
> > At minimum, this should include the nifi-api [5] module. The
> > nifi-framework-api [6] is also worth consideration for inclusion in
> > this category. The NiFi REST API does not have quite the same level of
> > concern, but may warrant inclusion.
> >
> > As part of this discussion, we should consider separating the nifi-api
> > module into its own repository, with its own versioning scheme. This
> > will provide a helpful distinction in terms of the scope of changes,
> > and allow the API to be released independently of the application,
> > providing strong version compatibility guarantees.
> >
> > Based on feedback for the general idea, I would be glad to draft a
> > NiFi Improvement Proposal page, outlining the recommended steps in
> > more detail so we can come to consensus on how this should work.
> >
> > Regards,
> > David Handermann
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > [2]
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > [3]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > [4]
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > [5] https://github.com/apache/nifi/tree/main/nifi-api
> > [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> >
>

Re: [Discuss] NiFi 2.0 milestones, where we are?

[Joe Witt]

Jens,

My comment regarding S2S was focused on the client in particular which
seems to cause some regrettable dependency chains we need to resolve.
There is not a plan to really do the groundwork necessary to promote that
protocol externally such that this type of client library should be
maintained.  I am asserting we should no longer support/maintain/carry code
for a s2s client to be used outside of nifi itself.

Now I do think there is debate to be had on the validity of S2S as a
protocol/capability in nifi itself but as you note for those that rely on
it this can be useful since it means less firewalls and centralized
policy.  It is however a lot to maintain so keeping it needs to be
supportable.  Not in scope for my comments above but an interesting area to
address eventually.

Thanks
Joe

On Mon, Aug 12, 2024 at 12:14 PM Joe Witt  wrote:

> Yep they'll be good to get to but they're not blockers.
>
> On Mon, Aug 12, 2024 at 12:10 PM Mark Bean  wrote:
>
>> There are only a couple issues remaining from NIFIDEVS-12400
>> <https://issues.apache.org/jira/browse/NIFI-12400>. Although, this issue
>> is
>> marked as resolved. I don't want the following unresolved issues to fall
>> through the cracks prior to the final 2.0.0 release:
>>
>> https://issues.apache.org/jira/browse/NIFI-13318
>> https://issues.apache.org/jira/browse/NIFI-13319
>>
>> These add the 'Stop & Configure' and 'Disable & Configure' options to the
>> UI of processors and controller services. This is an extremely useful
>> feature used frequently.
>>
>> Thanks,
>> Mark
>>
>> On Sun, Aug 11, 2024 at 7:04 AM Jens M. Kofoed 
>> wrote:
>>
>> > With access policies, I mean the possibility to only allow certain
>> "users"
>> > to see and access specifics input/output ports. So while system A can
>> share
>> > many S2S ports, system B can only see/access ports for system B and
>> system
>> > C can only see/access ports for system C. I know I can do something
>> similar
>> > with InvokeHTTP, inputs ports. But each processor will have to bind to
>> its
>> > own port, right?
>> > Kind Regards
>> > Jens M. Kofoed
>> >
>> > Den søn. 11. aug. 2024 kl. 12.54 skrev Jens M. Kofoed <
>> > jmkofoed@gmail.com>:
>> >
>> > > Dear Joe
>> > >
>> > > I'm not quite sure how to understand your comment about S2S. Will
>> > > Site-To-Site communication as it is today going to be removed from
>> NiFi
>> > in
>> > > the future?
>> > > I'm aware of many places where S2S is in use. The benefit of only
>> having
>> > 2
>> > > ports open in your firewall (https+S2S), and combining it with access
>> > > policies which is able to connect to specifics input/output ports is
>> very
>> > > useful. If S2S is going to be removed, what will be the replacement?
>> > >
>> > > Kind regards
>> > > Jens M. Kofoed
>> > >
>> > > Den fre. 9. aug. 2024 kl. 16.58 skrev Joe Witt :
>> > >
>> > >> It is incredibly tempting to keep removing crusty bits like the XML
>> and
>> > >> S2S
>> > >> as we are genuinely so much better off now having made so much tech
>> debt
>> > >> progress.  Huge thanks to you in particular, David!  But I do agree
>> with
>> > >> your view.
>> > >>
>> > >> I think right on the heels of the last vestige of old UI/content
>> viewing
>> > >> we
>> > >> push for the official 2.0 release.
>> > >>
>> > >> I'd actually think we consider a 3.0 to be one year later which dumps
>> > >> complex config models and S2S (at least the client impl) and
>> continues
>> > >> cleanup but otherwise incurs no user/migration impact.  But that is a
>> > >> fairly off the cuff mention.
>> > >>
>> > >> Thanks
>> > >>
>> > >> On Fri, Aug 9, 2024 at 7:21 AM David Handermann <
>> > >> exceptionfact...@apache.org>
>> > >> wrote:
>> > >>
>> > >> > Arpad,
>> > >> >
>> > >> > Thanks for initiating the discussion! I think we are getting very
>> > >> > close to ready for a GA release of NiFi 2.0.
>> > >> >
>> > >> > The last major element I am aware of right now is some reworking of
>> > >

Re: [Discuss] NiFi 2.0 milestones, where we are?

[Joe Witt]

Yep they'll be good to get to but they're not blockers.

On Mon, Aug 12, 2024 at 12:10 PM Mark Bean  wrote:

> There are only a couple issues remaining from NIFIDEVS-12400
> <https://issues.apache.org/jira/browse/NIFI-12400>. Although, this issue
> is
> marked as resolved. I don't want the following unresolved issues to fall
> through the cracks prior to the final 2.0.0 release:
>
> https://issues.apache.org/jira/browse/NIFI-13318
> https://issues.apache.org/jira/browse/NIFI-13319
>
> These add the 'Stop & Configure' and 'Disable & Configure' options to the
> UI of processors and controller services. This is an extremely useful
> feature used frequently.
>
> Thanks,
> Mark
>
> On Sun, Aug 11, 2024 at 7:04 AM Jens M. Kofoed 
> wrote:
>
> > With access policies, I mean the possibility to only allow certain
> "users"
> > to see and access specifics input/output ports. So while system A can
> share
> > many S2S ports, system B can only see/access ports for system B and
> system
> > C can only see/access ports for system C. I know I can do something
> similar
> > with InvokeHTTP, inputs ports. But each processor will have to bind to
> its
> > own port, right?
> > Kind Regards
> > Jens M. Kofoed
> >
> > Den søn. 11. aug. 2024 kl. 12.54 skrev Jens M. Kofoed <
> > jmkofoed@gmail.com>:
> >
> > > Dear Joe
> > >
> > > I'm not quite sure how to understand your comment about S2S. Will
> > > Site-To-Site communication as it is today going to be removed from NiFi
> > in
> > > the future?
> > > I'm aware of many places where S2S is in use. The benefit of only
> having
> > 2
> > > ports open in your firewall (https+S2S), and combining it with access
> > > policies which is able to connect to specifics input/output ports is
> very
> > > useful. If S2S is going to be removed, what will be the replacement?
> > >
> > > Kind regards
> > > Jens M. Kofoed
> > >
> > > Den fre. 9. aug. 2024 kl. 16.58 skrev Joe Witt :
> > >
> > >> It is incredibly tempting to keep removing crusty bits like the XML
> and
> > >> S2S
> > >> as we are genuinely so much better off now having made so much tech
> debt
> > >> progress.  Huge thanks to you in particular, David!  But I do agree
> with
> > >> your view.
> > >>
> > >> I think right on the heels of the last vestige of old UI/content
> viewing
> > >> we
> > >> push for the official 2.0 release.
> > >>
> > >> I'd actually think we consider a 3.0 to be one year later which dumps
> > >> complex config models and S2S (at least the client impl) and continues
> > >> cleanup but otherwise incurs no user/migration impact.  But that is a
> > >> fairly off the cuff mention.
> > >>
> > >> Thanks
> > >>
> > >> On Fri, Aug 9, 2024 at 7:21 AM David Handermann <
> > >> exceptionfact...@apache.org>
> > >> wrote:
> > >>
> > >> > Arpad,
> > >> >
> > >> > Thanks for initiating the discussion! I think we are getting very
> > >> > close to ready for a GA release of NiFi 2.0.
> > >> >
> > >> > The last major element I am aware of right now is some reworking of
> > >> > content viewer integration. Jira issue NIFI-13632 [1] highlights
> some
> > >> > general issues to be addressed, bringing the content viewer
> > >> > integration in line with the rest of the redesigned web user
> > >> > interface. I will defer to Matt Gilman and others for details.  I'm
> > >> > not aware of anything else in particular, but this particular area
> is
> > >> > important as it relates to the contract between the main application
> > >> > and content viewers.
> > >> >
> > >> > Beyond that, there are other framework issues I would like to
> address
> > >> > in the future, such as the configuration structure that requires
> > >> > multiple XML files, and the Site-to-Site client library. However, in
> > >> > the interest of keeping changes scoped, I believe those can be
> > >> > addressed down the road.
> > >> >
> > >> > With that background, as soon as we are in a good position with the
> > >> > content viewer integration, we should proceed to preparing for a
> 

Re: [Discuss] NiFi 2.0 milestones, where we are?

[Joe Witt]

One caveat to my earlier comments - went through JIRAs and found this one
hanging out.   Ran into this while doing the NIFI-12988 refactor work.

https://issues.apache.org/jira/browse/NIFI-13080

We should not let the 'nifi-properties' module remain in the root lib nor
the strange relationship to nifi-property-utils and we definitely do not
want NiFiProperties to be referenced by anything outside the core
framework.  It isn't nearly as bad as it had been in the past as now we see
only things which are framework extensions using it.  But we do need to get
that better handled and ensure nifi properties land as a jar within the
framework nar and not in the extensions class path.

That was the only thing I've found so far that really needs to be cleaned
up prior to calling this done for 2x

On Fri, Aug 9, 2024 at 9:39 AM Steven Matison 
wrote:

> This is awesome to hear.  I recently moved all my personal stuff into 2.0
> space and everything working smoothly as expected.  Now I can play with new
> and forget old.
>
>
> Amazing work from everyone who has helped the 2.x project get so far so
> fast!!
>
>
>
>
> On Fri, Aug 9, 2024 at 11:21 AM Douglas Reid 
> wrote:
>
> > Thanks Joe.What’s a rough estimate to wrap up the final bits of work?
> >
> >
> > > On Aug 9, 2024, at 10:58 AM, Joe Witt  wrote:
> > >
> > > It is incredibly tempting to keep removing crusty bits like the XML and
> > S2S
> > > as we are genuinely so much better off now having made so much tech
> debt
> > > progress.  Huge thanks to you in particular, David!  But I do agree
> with
> > > your view.
> > >
> > > I think right on the heels of the last vestige of old UI/content
> viewing
> > we
> > > push for the official 2.0 release.
> > >
> > > I'd actually think we consider a 3.0 to be one year later which dumps
> > > complex config models and S2S (at least the client impl) and continues
> > > cleanup but otherwise incurs no user/migration impact.  But that is a
> > > fairly off the cuff mention.
> > >
> > > Thanks
> > >
> > > On Fri, Aug 9, 2024 at 7:21 AM David Handermann <
> > exceptionfact...@apache.org>
> > > wrote:
> > >
> > >> Arpad,
> > >>
> > >> Thanks for initiating the discussion! I think we are getting very
> > >> close to ready for a GA release of NiFi 2.0.
> > >>
> > >> The last major element I am aware of right now is some reworking of
> > >> content viewer integration. Jira issue NIFI-13632 [1] highlights some
> > >> general issues to be addressed, bringing the content viewer
> > >> integration in line with the rest of the redesigned web user
> > >> interface. I will defer to Matt Gilman and others for details.  I'm
> > >> not aware of anything else in particular, but this particular area is
> > >> important as it relates to the contract between the main application
> > >> and content viewers.
> > >>
> > >> Beyond that, there are other framework issues I would like to address
> > >> in the future, such as the configuration structure that requires
> > >> multiple XML files, and the Site-to-Site client library. However, in
> > >> the interest of keeping changes scoped, I believe those can be
> > >> addressed down the road.
> > >>
> > >> With that background, as soon as we are in a good position with the
> > >> content viewer integration, we should proceed to preparing for a
> > >> release.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> [1] https://issues.apache.org/jira/browse/NIFI-13632
> > >>
> > >> On Thu, Aug 8, 2024 at 5:57 PM Arpad Boda  wrote:
> > >>>
> > >>> Team,
> > >>>
> > >>> Tremendous amount of effort has been put into NiFi 2.0 so far and I
> > >> haven't
> > >>> seen breaking changes introduced during the last few weeks, so I
> wonder
> > >>> where you think we are in the process of making it final(GA)?
> > >>> Do you see features missing, things that need to be either updated or
> > >>> deprecated, breaking api changes introduced?
> > >>>
> > >>> Don't get me wrong, I have no intent to push the community to release
> > 2.0
> > >>> as soon as we can, the goal of this thread is to have a common
> > >>> understanding of where we are in the process and what we need to
> > achieve
> > >> to
> > >>> get there.
> > >>>
> > >>> Thanks in advance for sharing your thoughts, opinions!
> > >>>
> > >>> Cheers,
> > >>> Arpad
> > >>
> >
> >
>

Re: [Discuss] NiFi 2.0 milestones, where we are?

[Joe Witt]

It is incredibly tempting to keep removing crusty bits like the XML and S2S
as we are genuinely so much better off now having made so much tech debt
progress.  Huge thanks to you in particular, David!  But I do agree with
your view.

I think right on the heels of the last vestige of old UI/content viewing we
push for the official 2.0 release.

I'd actually think we consider a 3.0 to be one year later which dumps
complex config models and S2S (at least the client impl) and continues
cleanup but otherwise incurs no user/migration impact.  But that is a
fairly off the cuff mention.

Thanks

On Fri, Aug 9, 2024 at 7:21 AM David Handermann 
wrote:

> Arpad,
>
> Thanks for initiating the discussion! I think we are getting very
> close to ready for a GA release of NiFi 2.0.
>
> The last major element I am aware of right now is some reworking of
> content viewer integration. Jira issue NIFI-13632 [1] highlights some
> general issues to be addressed, bringing the content viewer
> integration in line with the rest of the redesigned web user
> interface. I will defer to Matt Gilman and others for details.  I'm
> not aware of anything else in particular, but this particular area is
> important as it relates to the contract between the main application
> and content viewers.
>
> Beyond that, there are other framework issues I would like to address
> in the future, such as the configuration structure that requires
> multiple XML files, and the Site-to-Site client library. However, in
> the interest of keeping changes scoped, I believe those can be
> addressed down the road.
>
> With that background, as soon as we are in a good position with the
> content viewer integration, we should proceed to preparing for a
> release.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-13632
>
> On Thu, Aug 8, 2024 at 5:57 PM Arpad Boda  wrote:
> >
> > Team,
> >
> > Tremendous amount of effort has been put into NiFi 2.0 so far and I
> haven't
> > seen breaking changes introduced during the last few weeks, so I wonder
> > where you think we are in the process of making it final(GA)?
> > Do you see features missing, things that need to be either updated or
> > deprecated, breaking api changes introduced?
> >
> > Don't get me wrong, I have no intent to push the community to release 2.0
> > as soon as we can, the goal of this thread is to have a common
> > understanding of where we are in the process and what we need to achieve
> to
> > get there.
> >
> > Thanks in advance for sharing your thoughts, opinions!
> >
> > Cheers,
> > Arpad
>

Re: Cannot get build working

[Joe Witt]

Dan

You can get a couple thread dumps during the build being in a hung state
and that may be quite informative.

Thanks

On Sun, Jul 28, 2024 at 2:27 PM Dan S  wrote:

> Maven home: /opt/apache-maven-3.9.6 Java version: 21.0.3, vendor: Red Hat,
> Inc., runtime: /usr/lib/jvm/java-21-openjdk-21.0.3.0.9-1.el8.x86_64 Default
> locale: en_US, platform encoding: UTF-8 OS name: "linux", version:
> "4.18.0-513.24.1.el8_9.x86_64", arch: "amd64", family: "unix"
>
> The command I am running is from top level directory
> ./mvnw -Pcontrib-check
>
>
> On Sat, Jul 27, 2024 at 12:37 PM Chris Sampson
>  wrote:
>
> > Can you provide more information, such as:
> > what command(s) are you running to build?
> > what versions of Java & Maven in what OS?
> > what error(s) are you seeing resulting in the failure?
> >
> >
> > Cheers,
> >
> > ---
> > Chris Sampson
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > > On 26 Jul 2024, at 21:49, Dan S  wrote:
> > >
> > > I am currently working on a feature branch for NIFI-12709
> > > .
> > > I currently rebased from main where the latest commit was
> > > 2f73ced5a6dad469fb1942cfe9a6ac61031436a4
> > >
> > > I consistently cannot build anymore as the build keeps on hanging when
> > > running
> > org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest
> > > I have wiped out my local maven repo but I still am having the problem.
> > > Is anyone else having this issue? Any suggestions on what I can do?
> >
> >
>

Re: [DISCUSS] Python pip in Docker Images

[Joe Witt]

David

I had a feeling this was going to be a thing once that person posted about
it in nifi slack yesterday...

To the extent we want the docker images to be a developer
convenience mechanism then pip staying there is important.

To the extent we want the docker images to be production ready then pip
being there is not great for the reasons mentioned.  Though the images need
a lot of improvement.

Matt - Maven and Pip do indeed both get used to source dependencies.  Maven
is used at build time whereas Pip is used at runtime.  The necessary
security posture is different.  I don't think the point here is that the
CVE is specifically applicable but rather sourcing dependencies at runtime
is not an ideal production practice without some additional controls in
place.

Thanks
Joe

On Wed, Jul 24, 2024 at 4:51 PM Matthew Hawkins  wrote:

> Hi David,
>
> See the RHEL bug [1] for the shellacking this now rescinded CVE received.
>
> Removing pip from the python side should also be accompanied by removing
> maven from the Java side, if you are serious about addressing the actual
> security concern raised in this CVE.
> (That malicious content may exist somewhere on the internet, and people can
> download it via http). Also, remove every node from NiFi that allows the
> ingestion of data from any source, by the same token. That'll also help
> reduce developer maintenance workload on these nasty CVE filled nodes ;) ;)
> ;)
>
>
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1835736
>
>
> On Thu, 25 July 2024, 06:16 David Handermann,  >
> wrote:
>
> > Team,
> >
> > Apache NiFi 2.0.0-M1 introduced support for native Python Processors,
> > bringing new capabilities and some new questions. Part of that initial
> > release included the installation of Python pip for dynamic dependency
> > installation in the unofficial Docker images. Although this feature is
> > useful in some development scenarios, pip has several known
> > vulnerabilities, including CVE-2018-20225, related to the potential
> > for passing an option to reference additional repositories. Although
> > NiFi does not use this option, the presence of pip is a notable
> > security concern at the level of runtime package retrieval.
> >
> > With that background, it seems that we should remove pip from the
> > unofficial Docker image builds. This would bring closer alignment with
> > the convenience binary downloads, which do not have Python support
> > enabled in the default configuration. Although this would require
> > users to build their own images to add pip, it follows the general
> > principle of providing secure defaults. There is a developer usability
> > tradeoff, but we have had similar considerations in the past, and have
> > leaned in the direction of security.
> >
> > Regards,
> > David Handermann
> >
>

Re: [DISCUSS] 1.x Maintenance

[Joe Witt]

Adam,

We are all sympathetic to Pierre's view. As one of the people who
participates in contributing code, code reviews, discussions, conducting
releases, and more, his concerns and actions carry considerable weight for
those of us that do so as well and the broader community.

For repository encryption we should have raised a discussion thread.
Fortunately every code change is reversible.  Nothing is lost. If there is
anything needing undone let's discuss them.  There are no ships sailing.
Just contributors contributing.

If you keep the rest of the sentence in the quote you snipped it reads
"let's surface them and address them" which does the opposite of close off
conversation.

Thanks
Joe


On Mon, Jul 8, 2024 at 12:07 AM Adam Taft  wrote:

> Feeling sympathetic to Pierre's point of view here.
>
> Picking specifically on the repository encryption issue itself, we have
> deprecation of a fundamental and well-known capability of NiFi. Repository
> encryption has been a key highlight and feature for those using NiFi in
> highly secure enclaves that maintain the strictest of auditing oversight.
> This change will no doubt cause reassessment efforts and scrutiny over what
> will be perceived as a loss of information security.
>
> The justification for the change is probably warranted. Maybe repository
> encryption should never have been a problem NiFi attempted to solve. But
> regardless, the change was made without reasonable discussion, at minimum,
> here on the mailing lists where an Apache project is supposed to have such
> discussions.
>
> - The PR to remove this capability was created 5 days ago. [1]
> - The github PR was accepted and merged 3 days ago, on July 4th, a US
> national holiday no less. [2]
> - The deprecated components page was updated to reflect this change
> *retroactively* 1 day ago. [3]
>
> Granted, I might not have seen additional mailing list discussions on this
> topic. I grepped my inbox, but maybe I missed it. But still, for a security
> feature to be removed in the timeline outlined above, even if it was
> brought to the mailing list, would have been aggressive.
>
> So yes, there's a smell here. Clearly there needs to be a balance of doing
> small/normal changes without a lot (or any) needed discussion on the
> mailing lists, using Jira as the main project driver. But likewise, there
> has to be a sense of when to promote a discussion (or even awareness) to
> the mailing list level. A removed feature that has any sort of touchpoint
> to information security should most definitely be brought forward here.
>
> I'm supportive of the removal, to be clear. But I'm not supportive of how
> it was handled. And saying we should need "*specific examples where
> something looks like it was handled poorly*" is a good way to close off
> conversation. That's a lot of work looking back at a ship that has already
> sailed. It's better to trust that an impacting feature will be raised to
> the larger audience than to hunt out and find surprises after the fact.
> Community trust is an essential aspect of an Apache OSS project.
>
> [1] https://issues.apache.org/jira/browse/NIFI-13494
> [2] https://github.com/apache/nifi/pull/9039/commits
> [3]
>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=235834208
>
> On Sun, Jul 7, 2024 at 1:34 PM Joe Witt  wrote:
>
> > Pierre,
> >
> > We are seeing active maintenance on the 1.x line and it is by the same
> > people active on the 2.x line.  Is there a specific concern to highlight
> > and discuss?
> >
> > From the previous email the reference to things being removed quickly and
> > without discussion is something we need to ensure is being handled well.
> > If there are specific examples where something looks like it was handled
> > poorly then let's surface them and address them.
> >
> > The items you noted in your latest email specifically having concerns
> with:
> >
> > Kafka 2.6 parity concern - Can you share more on what is missing?
> > Obviously supporting Kafka users within NiFi is quite important but I
> think
> > we're making the right steps here.  Whatever gap we have now let's
> identify
> > and see who can knock it out.
> >
> > Kudu:  Releases look very infrequent  for Kudu.  We're on the latest. We
> > have vulnerable library hits and they've been there for a very long time.
> > This is an ideal candidate for a repository for components not likely to
> > see active changes.
> >
> > Repository Encryption: Such a component is both highly security sensitive
> > but also a tremendous code/maintenance issue. Considering how ke

Re: [DISCUSS] 1.x Maintenance

[Joe Witt]

Pierre,

We are seeing active maintenance on the 1.x line and it is by the same
people active on the 2.x line.  Is there a specific concern to highlight
and discuss?

>From the previous email the reference to things being removed quickly and
without discussion is something we need to ensure is being handled well.
If there are specific examples where something looks like it was handled
poorly then let's surface them and address them.

The items you noted in your latest email specifically having concerns with:

Kafka 2.6 parity concern - Can you share more on what is missing?
Obviously supporting Kafka users within NiFi is quite important but I think
we're making the right steps here.  Whatever gap we have now let's identify
and see who can knock it out.

Kudu:  Releases look very infrequent  for Kudu.  We're on the latest. We
have vulnerable library hits and they've been there for a very long time.
This is an ideal candidate for a repository for components not likely to
see active changes.

Repository Encryption: Such a component is both highly security sensitive
but also a tremendous code/maintenance issue. Considering how key
management for it worked to this point we should collaborate with users to
understand how to better achieve the desired outcome.

Legacy Kafka and Kudu are good candidates for an extensions repo for things
which will likely not see active maintenance.  Establishing such a
repository and the build process and so on for it is a considerable effort.
Just needs a volunteer to build that infra/process out.

Thanks



On Sun, Jul 7, 2024 at 11:04 AM Pierre Villard 
wrote:

> I'm not discussing the motivations behind the removal of what has been
> removed this week. It makes sense. But we can't expect to have users
> quickly move to 2.x when the amount of things users will have to take care
> of when moving to 2.x keeps growing and is very significant.
>
> For the extensions, I'm a bit surprised by the removal of some extensions
> such as Kudu, that we know is used a lot by NiFi users, and has been
> actively maintained over the last few years. The one thing that concerns me
> the most is the removal of the Kafka 2.6 components knowing that we don't
> have feature parity, yet, with the new approach. I understand that we have
> to do the cut at some point and there is no easy solution but again it
> means that we should likely actively maintain 1.x for a longer period of
> time.
>
> On the framework side, I'm a bit concerned by the removal of repository
> encryption which is also something many users are using. Not saying this is
> perfect and should not be removed but we can't just say "switch to OS level
> encryption / cloud provider encryption". Again, this will take a lot of
> time for users using all of those things to adjust.
>
> We are putting more and more expectations on users to move to 2.x when they
> have been using NiFi for many many years and have systems running smoothly.
> I'm not saying we should not clean things up, but by taking this approach,
> I think we owe our community an active maintenance of the 1.x branch as
> many users will need time to upgrade.
>
> Le dim. 7 juil. 2024 à 15:35, Joe Witt  a écrit :
>
> > Pierre
> >
> > Which things that have been removed are problematic and which bits got
> > removed too fast?  There is no intent to make migration harder but there
> is
> > to make it more sustainable.
> >
> > The focus is getting things with less known usage but highest tech debt
> > removed. These are things with problematic socket code, limited security
> > features, capabilities we made new versions for and ultimately which have
> > minimal maintenance.
> >
> > If there is something controversial getting removed lets discuss it.  I
> > dont think keeping any of this in the main codebase is workable but it
> can
> > prompt having an extensions repository for lesser maintained components.
> > This would help communicate these components have limited maintenance but
> > may still be useful.
> >
> > If the bar to clean things up is higher than the bar to add new we have a
> > known quality problem.  We either have a similar bar or we need other
> > solutions.
> >
> > If there is a desire to see some (all?) decomissioned components kept and
> > placed in an extensions repo to ease migration or just end user needs is
> > there anybody available and interested to do that work?
> >
> > Thanks
> >
> > On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard <
> pierre.villard...@gmail.com
> > >
> > wrote:
> >
> > > Given all the things that are being removed in 2.x without even having
> a
> > > single dis

Re: [DISCUSS] 1.x Maintenance

[Joe Witt]

Pierre

Which things that have been removed are problematic and which bits got
removed too fast?  There is no intent to make migration harder but there is
to make it more sustainable.

The focus is getting things with less known usage but highest tech debt
removed. These are things with problematic socket code, limited security
features, capabilities we made new versions for and ultimately which have
minimal maintenance.

If there is something controversial getting removed lets discuss it.  I
dont think keeping any of this in the main codebase is workable but it can
prompt having an extensions repository for lesser maintained components.
This would help communicate these components have limited maintenance but
may still be useful.

If the bar to clean things up is higher than the bar to add new we have a
known quality problem.  We either have a similar bar or we need other
solutions.

If there is a desire to see some (all?) decomissioned components kept and
placed in an extensions repo to ease migration or just end user needs is
there anybody available and interested to do that work?

Thanks

On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard 
wrote:

> Given all the things that are being removed in 2.x without even having a
> single discussion about it (PR opened and merged within 48 hours), we
> should not expect users (especially very large ones) to move from 1.x to
> 2.x any time soon. Things that have been removed over the last few days are
> going to be extremely impactful and will require users to do a significant
> amount of work / re-architecturing to move to 2.x. It's a bit sad to see so
> many things being done without any discussion in the community.
>
> Le ven. 5 juil. 2024 à 20:11, Michael Moser  a écrit :
>
> > My opinion is that the NiFi 1.x line is already in maintenance and
> doesn't
> > need new features or APIs, but bug fixes are still important to get in,
> and
> > security fixes are still critical.
> >
> > That said, this is an open source community-driven project, so if you can
> > find contributors and committers that want to put in extra work to
> support
> > 1.x, then why not?  I'm sure many of us have seen something new in NiFi
> and
> > thought "I wouldn't have done that".  But others in the community thought
> > it was a good idea, so why not?
> >
> > I don't really agree with the argument that putting too much work into
> 1.x
> > will delay people's transition to 2.0.  People will transition to 2.0 on
> > their own schedule, with myriad reasons for that schedule.  As a
> community,
> > we can simply make recommendations and provide solutions that help with a
> > preferred direction.
> >
> > Kind regards,
> > -- Mike
> >
> >
> > On Wed, Jul 3, 2024 at 1:27 PM Joe Witt  wrote:
> >
> > > Judith,
> > >
> > > The Apache NiFi 2.x line is built on and requires Java 21 and will not
> > > support any versions prior.
> > >
> > > NiFi 1.x line is built on Java 8 and requires either Java 8, 11, or 17.
> > >
> > > The dates you mention which extend to 2030/etc.. for a particular
> > > distributor of JVMs is a function of what sort of support that
> commercial
> > > vendor is making available to customers and it is different from what
> > that
> > > means for the public generally.
> > >
> > > Notably though it is also a very different concern from critical
> > > dependencies we have and how they evolve. For example we rely on things
> > > like Spring and Jetty and so on.  And these all choose to adopt
> language
> > > features or discontinue key lines which tie to certain Java versions,
> > > etc..  We can't control whether they choose to move to new major lines
> > when
> > > they fix vulnerabilities or whether they backport, etc..  We held the
> > line
> > > on Java 8 support far longer than I imagined we could but the current
> > > confluence of events/scenarios means we made the jump all the way to
> Java
> > > 21 (an LTS) release with the hopes that we can set a good clock for
> > people
> > > to work with for some time.
> > >
> > > The Java 1.x line will continue to be available for users to download
> and
> > > will still see general maintenance to the extent there are contributors
> > > available to do that.  You also have access to vendor supported paths
> to
> > > consider there which is similar to the stated 'Extended Support
> > > Availability' of Java.  But they too will have to navigate the
> complexity
> > > of the reported vulnerabilities

Re: [VOTE] Release Apache NiFi 1.27.0 (RC2)

[Joe Witt]

+1 binding

On Thu, Jul 4, 2024 at 1:28 PM David Handermann 
wrote:

> +1 binding
>
> - Verified signatures and hashes
> - Ran build using Maven Wrapper 3.9.6
> - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-412 x86_64
> - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64
> - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64
> - Ran NiFi on macOS 14.5 with Azul Zulu JDK 21.0.1 Aarch64
> - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64
>
> - Verified NIFI-13429 with EncryptContentPGP and DecryptContentPGP
> using sample JPEG
> - Verified NIFI-13464 NiFi Registry starts without issues following
> deprecation-log dependency adjustments
>
> Thanks Pierre!
>
> Regards,
> David Handermann
>
> On Wed, Jul 3, 2024 at 7:42 AM Pierre Villard
>  wrote:
> >
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.27.0.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are available
> on
> > the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1276
> >
> > Git Tag: nifi-1.27.0-RC2
> > Git Commit ID: e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18
> > GitHub Commit Link:
> >
> https://github.com/apache/nifi/commit/e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18
> >
> > Checksums of nifi-1.27.0-source-release.zip
> >
> > SHA256: 0ac01d54eeceb4a4f4863880bf67dfa71e6a585036c5caf0546c592bf55ced48
> > SHA512:
> >
> 675c7750752bf3092061c9eaac39a975955e9bf881e6bee3124c5842738d8c8626b6a551f33ef7a678018bd83e0323c1f0f0d79101255494d8ca91be7fc750f5
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/pvillard.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 37
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [] +1 Release this package as nifi-1.27.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
>

Re: Nifi 2.0.0 Major release timeline

[Joe Witt]

Ajay

We do not have a specific date in mind.  I believe largely we've completed
the major efforts we had intended for 2.0 and that even went further to
include a completely re-written UI which is a huge help.  We do continue to
find unmaintained or security relevant code with limited maintenance and
we're aggressively cleaning all that up.  We need to ensure we've cleaned
up all deprecated bits and are putting ourselves in the best position to
maintain, secure, and add capability as we go.

Not speaking for the community but my personal take is we're likely 1-2
months away now from minting a release as GA of 2.0 as what I see that
remains is pretty purely just code removal/clean-up.  There are always one
or two surprises that emerge but the 2.0 line is looking really good and is
already considerably in use for production purposes so stability is there.

Thanks
Joe

On Thu, Jul 4, 2024 at 6:16 AM Premarajan, Ajay
 wrote:

> Hi Team,
>
> We are currently using NiFi version 1.26 in our production environment. Is
> there a scheduled timeline for the major release of NiFi 2.0.0, or an
> estimate of how many milestone releases are planned before the major
> version is released?
>
>
> Mit freundlichen Grüßen / Kind regards
>
> Ajay Premarajan
>
>

Re: [DISCUSS] 1.x Maintenance

[Joe Witt]

Judith,

The Apache NiFi 2.x line is built on and requires Java 21 and will not
support any versions prior.

NiFi 1.x line is built on Java 8 and requires either Java 8, 11, or 17.

The dates you mention which extend to 2030/etc.. for a particular
distributor of JVMs is a function of what sort of support that commercial
vendor is making available to customers and it is different from what that
means for the public generally.

Notably though it is also a very different concern from critical
dependencies we have and how they evolve. For example we rely on things
like Spring and Jetty and so on.  And these all choose to adopt language
features or discontinue key lines which tie to certain Java versions,
etc..  We can't control whether they choose to move to new major lines when
they fix vulnerabilities or whether they backport, etc..  We held the line
on Java 8 support far longer than I imagined we could but the current
confluence of events/scenarios means we made the jump all the way to Java
21 (an LTS) release with the hopes that we can set a good clock for people
to work with for some time.

The Java 1.x line will continue to be available for users to download and
will still see general maintenance to the extent there are contributors
available to do that.  You also have access to vendor supported paths to
consider there which is similar to the stated 'Extended Support
Availability' of Java.  But they too will have to navigate the complexity
of the reported vulnerabilities.

My experience in the enterprise customer space is that tolerance for
components which contain reported vulnerabilities is far far lower than
ever before.  This in turn means the focus shifts to giving people strong
upgrade paths to consider.

Thanks
Joe

On Wed, Jul 3, 2024 at 10:10 AM Maucieri, Judith  wrote:

> If I may:  One large obstacle to "our" shift to v2.x is the absence of
> Java 8 support (unless I overlooked updates to the plan stated in November
> 2022 during release of version 1.19.0, Java 8 only remains in NiFi 1.x
> releases, which you hinted at remaining accurate).
>
> I make this statement in support of multiple scenarios where we employ
> Apache NiFi.  CVEs that are not addressed in v1.x would be a major
> concern.  In our case, per requirements levied upon us, we must maintain
> compatibility with all active Long Term Support (LTS) versions of Java.  I
> feel this is reinforced by LTS of RHEL 7, which just began and extends into
> 2028 and employs (by default) Java 8.
>
> Note the LTS dates currently indicated (reference
> https://en.wikipedia.org/wiki/Java_version_history):
> - Java 8 December 2030
> - Java 11 January 2032
> - Java 17 September 2029
> - Java 21 September 2031
>
> Thank you,
> JM
>
> -Original Message-
> From: Matt Burgess 
> Sent: Tuesday, July 2, 2024 4:05 PM
> To: dev@nifi.apache.org
> Subject: [DISCUSS] 1.x Maintenance
>
> !---|
>   This Message Is From an External Sender
>   Please use caution when clicking links or opening
>   attachments.
> |---!
>
> There have been some ongoing discussions [1,2] about what to bring back
> for PRs to 1.x vs trying to push forward with 2.x. There are of course
> great points from everyone. On the 2.x front, namely that 2.x has many
> improvements not just to components but the framework and UI as well, and
> that we've seen less contributions to the maintenance on the 1.x line. On
> the 1.x front that community members need to support 1.x for their users
> for the time being.
>
> I'm opening this discussion thread so we can collect everyone's thoughts
> so the PMC can make a sensible recommendation/decision moving forward. For
> myself, I think all bug PRs should be backported where prudent/possible,
> and even new features (although that can be a difficult backport due to
> moving the extensions in [3], but I recommend a separate PR against 1.x,
> hopefully a simple copy-paste if there are no Java 21-specific code issues,
> but please run contrib-check against Java 8 first).
>
> I disagree with the "atrophy" sentiment from [2], a mature release line
> normally experiences less contributions because it is mostly stable in its
> own right. Guiding users to 2.x IMO is the right call but many of us have
> to deal with the fact that it doesn't usually happen overnight, especially
> without a GA release.
>
> I opened this discussion with my opinions but if I may humbly speak for
> the PMC, we need your voice, and we look forward to all of your thoughts,
> opinions, and questions.
>
> Thank you,
> Matty B
>
> [1]
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/NIFI-13196__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7B8Ei8u6ou8_hDUuBxGEIQDtUKArfA$
> [2]
> https://urldefense.com/v3/__https://github.com/apache/nifi/pull/9018__;!!HkW_WPl1peM!oK4J-OlEsDT4Hl0lVpbgtfPr1qvI-MyLLzjIjPmf0O8cfZFlSIaP7

Re: [DISCUSS] 1.x Maintenance

[Joe Witt]

Russell - great perspective and thanks for sharing.

Matt,

If we reduce contribution and support for our existing 1.x line and its
users to just code commits the data suggests things are going well.

git shortlog --numbered --summary  --since "Jan 1 2023" --before "Jun 30
2024" upstream/main
git shortlog --numbered --summary  --since "Jan 1 2023" --before "Jun 30
2024" upstream/support/nifi-1.x

Quick analysis shows the same 7 people are present in the top 10 of both.
This is a strong indication there are already great efforts being taken
consistently by folks to backport where feasible and reasonable.

Three of the top 10 for main who are not in the top 10 for support did all
the work for the brand new UI.  Fundamentally the data suggests things are
going as they should and in general backporting happens as it should.

If you expand beyond just code contributions to look at code reviews,
slack, mailing lists, etc.. I think you'll also find a similar cast of
community characters helping people on the 1.x and 2.x line.  No sign of
abandoning.

We're still landing JIRAs, PRs, and releases on 1.x except now we have
different folks engaged in generating the release candidates - which is
awesome and so very much appreciated.

As for concerns about my statement of atrophy you referenced I would
appreciate your input on what is disagreeable. The sunsetting of Java 8,
Spring 5, Jetty 9, and so on are very important and are unique concerns of
the 1.x line. Some of these critical dependencies are vulnerable and we
cannot solve them on the 1.x line due to choices of those libraries and the
tie for specific versions to Java 8.  The atrophy I reference is baked in
due to these facts.

Now perhaps a concern that is fair to surface is regarding the level of
dependency maintenance happening on the main line vs that on the 1.x line.
I bring this up because I personally have been focusing on these on the 2.x
line and not on the 1.x line. Ironically the NIFi-12998 JIRA you reference
as making backports problematic is an important change which helps make
that line more dependency maintainable (among several other important
improvements on main).  NIFI-12998 would not have been feasible, and it was
brutal to be clear, had others not already done awesome work on the
2.x/main line thanks to the updated dependencies and benefits of bill of
materials patterns and so on.  I have found dependency maintenance work is
far more time consuming and error prone on the 1.x line than on the 2.x
line but your mileage may vary.

I read your note as raising concerns that the 1.x line isn't getting the
support and effort it deserves to take care of our community as a whole.
The data does not support that perspective.  Not giving credence to the
great work happening in the 2.x/main line to help the users have a strong
path forward also seems to miss an important perspective.  One of the most
effective ways to take care of our 1.x users is to diligently provide a
path to a strong and stable 2.x line.

I'm extremely impressed by the work happening in the community at large and
appreciative of everyone's efforts.

Thanks

On Tue, Jul 2, 2024 at 2:18 PM Russell Bateman 
wrote:

> I'm not too much of an end user though I get by. I have probably written
> 60 or 70 proprietary custom processors (to handle the ingestion of
> medical documents in my company's search engine). I want to call out
> superb test runners and mocks that make processor development possible,
> even a pleasure. How not to be grateful that what runs in JUnit while
> developing never fails to run later in production?
>
> Mostly though, I wanted to thank all that have crafted NiFi v1 over the
> years. Fixes have been prompt. Support via the mail groups and Slack has
> been first-rate and courteous.
>
> Apache NiFi is dynamite software. It's rare to find a complex, magical
> and so useful piece of software that you can get up and running in half
> an hour with something real to show for it. And I mean show for it
> because the UI makes it easy to demonstrate to management and other
> non-technical folk the wizardry while it's going down.
>
> Thanks to all.
>
> Russell Bateman
>
>
>
> On 7/2/24 14:05, Matt Burgess wrote:
> > There have been some ongoing discussions [1,2] about what to bring back
> for
> > PRs to 1.x vs trying to push forward with 2.x. There are of course great
> > points from everyone. On the 2.x front, namely that 2.x has many
> > improvements not just to components but the framework and UI as well, and
> > that we've seen less contributions to the maintenance on the 1.x line. On
> > the 1.x front that community members need to support 1.x for their users
> > for the time being.
> >
> > I'm opening this discussion thread so we can collect everyone's thoughts
> so
> > the PMC can make a sensible recommendation/decision moving forward. For
> > myself, I think all bug PRs should be backported where prudent/possible,
> > and even new features (although that can be a diff

board report July 2024

[Joe Witt]

Team,

Thanks again for another strong quarter for NiFi.


## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High. Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago) There are currently 67
committers and 37 PMC members in this project. The Committer-to-PMC ratio is
roughly 2:1.

Community changes, past quarter:
- Gabor Gyimesi was added as PMC member on 2024-06-26.
- Lehel Boer was added as committer on 2024-06-16.

## Project Activity:
We released Apache NiFi 2.0.0 milestone 4 (M4) on July 1st 2024. M4 contains
more than 150 JIRAs including moving completely to the new UI and providing
more integration options for Kafka, splitting packet capture data, and a new
Python source component API.

We released the NiFi NAR Maven Plugin 2.0.0 on May 28 2024 supporting the
new
rules extension API in build manifests.

We released the MiNiFi CPP 0.99.0 on May 18 2024 including many improvements
and MacOS service integration, SMB, Grafana, Jolt transforms for JSON data,
greater efficiency when merging larger datasets with MergeContent.

We released NiFi 2.0.0 milestone 3 (M3) on May 16 2024. M3 included more
than
400 JIRAs with better integrations to AWS, Azure, OpenSearch and SMB.  The
new
UI is now available and considered feature complete but the old UI remains
available.

## Community Health:
Community health remains strong and growing.

JIRA and Mailing list activity remains consistent.

The slack community grew by another 150 participants in the general channel
alone, increasing from 3,133 to 3,284 since the last report.

We continue to produce frequent releases in the NiFi community and enjoy
rather active vote participation even when it takes a few release candidates
to get it right.

We are seeing 30 or more unique contributors monthly with on average several
commits per day landing and that is just the core nifi codebase.  Code
activity for NiFi this calendar year appears higher than any previous year
when reviewing Github activity data.

Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)

[Joe Witt]

+1 binding

full clean build w contrib
full suite of integration tests including python/etc..  (see below)
built docker image and deployed over an existing set of volumes showing
upgrades well from previous releases
updated release notes a bit to focus more on end user perspective

Test error during integration tests suggesting potentially unstable test.
Ran fine on second try
[ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.657 s <<< FAILURE! -- in
org.apache.nifi.processors.standard.util.TestServerSFTPTransfer
[ERROR]
org.apache.nifi.processors.standard.util.TestServerSFTPTransfer.testGetListingWithBatchSize
-- Time elapsed: 0.019 s <<< ERROR!
org.apache.nifi.processors.standard.socket.ClientConnectException: SSH
Client connection failed [127.0.0.1:51893]
at
org.apache.nifi.processors.standard.ssh.StandardSSHClientProvider.getClient(StandardSSHClientProvider.java:116)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getSFTPClient(SFTPTransfer.java:631)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:327)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:289)
at
org.apache.nifi.processors.standard.util.TestServerSFTPTransfer.testGetListingWithBatchSize(TestServerSFTPTransfer.java:225)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: net.schmizz.sshj.transport.TransportException: Timeout expired:
3 MILLISECONDS
at
net.schmizz.sshj.transport.TransportException$1.chain(TransportException.java:33)
at
net.schmizz.sshj.transport.TransportException$1.chain(TransportException.java:27)
at net.schmizz.concurrent.Promise.retrieve(Promise.java:139)
at net.schmizz.concurrent.Event.await(Event.java:105)
at
net.schmizz.sshj.transport.KeyExchanger.waitForDone(KeyExchanger.java:172)
at net.schmizz.sshj.transport.KeyExchanger.startKex(KeyExchanger.java:160)
at net.schmizz.sshj.transport.TransportImpl.doKex(TransportImpl.java:254)
at net.schmizz.sshj.SSHClient.doKex(SSHClient.java:824)
at net.schmizz.sshj.SSHClient.onConnect(SSHClient.java:812)
at net.schmizz.sshj.SocketClient.connect(SocketClient.java:69)
at
org.apache.nifi.processors.standard.ssh.StandardSSHClientProvider.getClient(StandardSSHClientProvider.java:113)
... 7 more
Caused by: java.util.concurrent.TimeoutException: Timeout expired: 3
MILLISECONDS
... 16 more

On Sat, Jun 29, 2024 at 7:01 AM Pierre Villard 
wrote:

> +1, binding
>
> Went through the usual steps, deployed a 3-nodes secured cluster in GCP
> with NiFi Registry and confirmed everything is working as expected.
>
> Thanks for taking care of this release David!
>
> Pierre
>
> Le ven. 28 juin 2024 à 22:22, Dan S  a écrit :
>
> > +1 (non-binding)
> > Ran through Release Candidate Verification
> > Verified signatures and hashes
> > Built with:
> >
> > Maven home: /opt/apache-maven-3.9.6
> > Java version: 21.0.3, vendor: Red Hat, Inc., runtime:
> > /usr/lib/jvm/java-21-openjdk-21.0.3.0.9-1.el8.x86_64
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "4.18.0-513.24.1.el8_9.x86_64", arch: "amd64",
> > family: "unix"
> >
> > Exercised new feature NIFI-13304 SplitExcel
> > Discovered bug NIFI-13466
> >  and
> > created task to research NIFI-13467
> > 
> >
> > Thanks for RM'ing David and all your hard work to get to this point!
> >
> > On Fri, Jun 28, 2024 at 1:16 PM Matt Burgess 
> wrote:
> >
> > > +1 (binding) Ran through release helper and tried various flows,
> verified
> > > versioning against a NiFi Registry (not the Git-backed one)
> > >
> > > Apache Maven 3.9.7 (8b094c9513efc1b9ce2d952b3b9c8eaedaf8cbf0)
> > > Java version: 21, vendor: Oracle Corporation, runtime:
> > > /Users/mburgess/.sdkman/candidates/java/21-oracle
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac"
> > >
> > > Thanks for RM'ing David!
> > >
> > > On Fri, Jun 28, 2024 at 9:40 AM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > > > Team,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > > NiFi 2.0.0-M4.
> > > >
> > > > Please review the following guide for how to verify a release
> candidate
> > > > build:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > >
> > > > The source being voted on the and the convenience binaries are
> > > > available on the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4
> > > >
> > > > The build artifacts are available on the Apache Nexus Repository:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachenifi-1275
> > > >
> > >

Re: Python extensions in their own repo...

[Joe Witt]

Great work David - thanks!

On Wed, Jun 26, 2024 at 2:36 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> Thanks to collaboration with Joe Witt, the new nifi-python-extensions
> repository [1] is now populated with the initial set of Python
> Processors.
>
> The repository includes a standard GitHub workflow for pull request
> validation that checks license headers and Python code formatting.
>
> The project uses Hatch [2] to run code formatting as well as build
> source and binary distribution packages.
>
> The source distribution and binary wheel packages both contain the
> Python Processors, which can be placed into an Apache NiFi 2
> installation.
>
> The source distribution archive will provide a suitable release
> candidate file when we are ready to release a version of
> nifi-python-extensions.
>
> In Jira, there is a new python-extensions-2.0.0 version to target for
> features and fixes.
>
> There is certainly more room for documentation and improvement, but
> this should provide a reasonable foundation for decoupled Python
> Processor development efforts.
>
> Regards,
> David Handermann
>
> [1] https://github.com/apache/nifi-python-extensions
> [2] https://hatch.pypa.io
>
> On Sat, Jun 22, 2024 at 9:06 AM David Handermann
>  wrote:
> >
> > Joe,
> >
> > Thanks for raising the discussion, and thanks to everyone for the
> feedback thus far. This tracks our previous discussion on the topic [1].
> >
> > I am also strongly in favor of separating out extensions into their own
> repositories for many of the reasons already mentioned. Starting with a
> single dedicated repository named nifi-python-extensions should be a good
> opportunity to prove out the concept. I agree considering the Java
> extensions is more involved, and I think should consider that separately.
> >
> > I would be glad to handle the initial work of creating the new
> repository and setting up the initial build structure. I am familiar with
> the work necessary to publish to the PyPI repository, and that could
> provide an optional distribution channel. We would still make the source
> distribution available through standard Apache channels, following standard
> project policies. Based on the current structure, it should be
> straightforward to download an archive of the Python extensions and expand
> that for those who want to have them as part of their NiFi installation.
> >
> > Once the initial repository is in place, that last initial step would be
> a pull request to remove the Python extensions from the main repository.
> >
> > I can proceed along these lines, unless any substantive objections come
> up, and the pull request process will also provide opportunity for
> additional consideration and review.
> >
> > Regards,
> > David Handermann
> >
> > [1] https://lists.apache.org/thread/nok561sg1dzw3zrott06gkl34hdjxbb3
> >
> > On Fri, Jun 21, 2024, 9:14 PM Marton Szasz  wrote:
> >>
> >> Hi Joe, Arpad and all,
> >>
> >> I'm strongly in favor of moving all Python components to a separate
> >> repository. It could be called apache/nifi-python-extensions or
> >> -components, and contain all Python components that this community
> >> maintains. I would prefer that over a separate repo for each extension,
> >> because it seems easier to keep track of all components maintained by
> >> the community if they are in the same repo, than if they were separate.
> >>
> >> Since MiNiFi C++ implemented a large subset of the same Python API, I
> >> think it makes our lives easier if we share the code, and keep the
> >> Python components in their own dedicated location. As NiFi and MiNiFi
> >> C++ both approach their next major version, and we commit to a stable
> >> Python API, I expect it to become easier to maintain the Python
> >> components separately, targeting this stable API, and we could align the
> >> release frequency with the maintenance needs of the Python components.
> >>
> >> I'm neutral of whether to package them with convenience binaries or
> >> leave that up to the user. Hopefully we can come up with a user-friendly
> >> way to install them if they're not included. I wouldn't include them in
> >> the source tarballs.
> >>
> >> I would keep the Java components separate from Python components.
> >> Whether that's in the NiFi repo or somewhere else, both are fine with
> me.
> >>
> >> Regarding introducing breaking changes: on the NiFi side, unit tests
> >> should co

Re: Python extensions in their own repo...

[Joe Witt]

"I would suggest starting with
moving the Python ones to a dedicated repo, let's have a workflow
established and polished there, might follow with some Java ones in case it
works well."

Yeah kinda where my head is too

On Fri, Jun 21, 2024 at 2:07 PM Arpad Boda  wrote:

> Joe,
>
> Interesting thoughts, I see a lot of pros and cons. Let me list the most
> important ones of both:
> +cves in extensions doesn't make nifi "vulnerable" automatically as they
> live in a different repo.
> +the responsibility of being up-to-date is being moved to the maintainers
> of the given extension, same applies for the stability of the tests
> covering that extension
>
> -easier to introduce breaking changes accidentally: a breaking change might
> go through and get committed. Especially in case of Java extensions, they
> python api is pretty thin (yet!). Only an extension developer will find it,
> most probably not immediately, when things already depend on the breaking
> change and it gets very difficult to make the right call in this case
> -might lose some extensions as they get even less maintained than they are
> now
>
> Overall I have no strong opinion either ways, I would suggest starting with
> moving the Python ones to a dedicated repo, let's have a workflow
> established and polished there, might follow with some Java ones in case it
> works well.
>
> Cheers,
> Arpad
>
> On Friday, June 21, 2024, Joe Witt  wrote:
>
> > Team,
> >
> > For the longest time we had all these Java based extensions and it was
> > often inconvenient for them to live within the codebase.  Indeed it makes
> > the builds crazy long and it delays getting new components out.  We had a
> > lot of work to do for this to be convenient and perhaps we still have
> gaps
> > remaining.
> >
> > Now we have these Python components.  I am not confident we really want
> > these in the codebase for similar but even more important reasons.  The
> > python components have similar issues when it comes to Licensing and
> Notice
> > recognition.  They have their own rapid vulnerability tracking.  Our
> > current tooling doesn't make tracking that very easy.
> >
> > I'm concerned about where the Python ones are heading in terms of
> > maintainability but also generally for the builds as well with the Java
> > ones.  Is it time to move to a repo for the Java extensions and its own
> > project/group name and versioning?  Same for Python extensions?
> >
> > This lets them evolve on their own schedule.  It does bring up an
> > interesting challenge as it relates to a convenience binary.  The ideal
> > state is extensions are released and shipped independent of the nifi
> > application.  But we'd need to make that really nice/easy for the users.
> >
> > We have a lot going on so maybe still not time to tackle this.  Curious
> to
> > hear thoughts
> >
> > Thanks
> >
>

Python extensions in their own repo...

[Joe Witt]

Team,

For the longest time we had all these Java based extensions and it was
often inconvenient for them to live within the codebase.  Indeed it makes
the builds crazy long and it delays getting new components out.  We had a
lot of work to do for this to be convenient and perhaps we still have gaps
remaining.

Now we have these Python components.  I am not confident we really want
these in the codebase for similar but even more important reasons.  The
python components have similar issues when it comes to Licensing and Notice
recognition.  They have their own rapid vulnerability tracking.  Our
current tooling doesn't make tracking that very easy.

I'm concerned about where the Python ones are heading in terms of
maintainability but also generally for the builds as well with the Java
ones.  Is it time to move to a repo for the Java extensions and its own
project/group name and versioning?  Same for Python extensions?

This lets them evolve on their own schedule.  It does bring up an
interesting challenge as it relates to a convenience binary.  The ideal
state is extensions are released and shipped independent of the nifi
application.  But we'd need to make that really nice/easy for the users.

We have a lot going on so maybe still not time to tackle this.  Curious to
hear thoughts

Thanks

Re: Support Passing in FF Attribute to Flow File create methods

[Joe Witt]

Jim

Ok - understood.  The trade-off that appears to be getting made to my eyes
looks like less expressive code.  If it were substantially valuable in
terms of cpu, heap, io then I think it makes the case more compelling.
I've not validated this with true testing but my intuition for this example
is that it would be barely measurable.  If you have different benchmark
results to share that would be interesting to learn more about.

Thanks

On Mon, Jun 17, 2024 at 8:23 AM Jim Steinebrey 
wrote:

> Joe,
> Thanks for your detailed response and the separate email about moving NiFi
> forward.
> My intent in this PR was to decrease memory allocation of a FF and thereby
> decrease memory that needed to be gc’ed.
> Given your feedback in this email thread and some other PR feedback I got
> this morning,
> I have concluded this PR (https://github.com/apache/nifi/pull/8968)
> to avoid a single FF allocation is not worth the complication of the API
> and applying in so many places,
> so I am going to close the PR in a little while if no one minds.
>
> Thanks,
> Jim
>
>
> > On Jun 15, 2024, at 12:59 PM, Joe Witt  wrote:
> >
> > Jim,
> >
> > Missing from this proposal is an explanation of the core 'why' or benefit
> > this provides.  I could see two potential ones based on the comment and
> > this seems related to recent patterns of code changes in this similar
> > vein.
> >
> > 1. We avoid a temporary java object representing the flow file.
> > 2. It could reduce lines of code.
> >
> > If these are the only potential benefits I don't find them compelling.
> > The cost in a memory/garbage collection sense I would expect is nearly
> > immeasurably minor.  While it would be great if it meant some
> considerable
> > improvement in code required or readability I dont see that in this case.
> > From the example processor change shown for GenerateTableFetch the code
> to
> > me looks less readable and less expressive.
> >
> > So then there are other things we'd consider which we cannot see just
> from
> > the PR.
> >
> > 1. Does this make the system run measurably faster such that it reduces
> CPU
> > usage, lowers GC?
> > 2. Does this drive down memory usage measurably/etc..?
> > 3. Does this make our codebase in general more maintainable/etc..?
> >
> > If you could share more on what led you to feel this is a helpful
> addition
> > that information would be great for the community to know in considering
> > this change.
> >
> > As it is right now I am not supportive.  However, I want to be clear
> about
> > a couple things as while the feedback above is negative relative to the
> > proposal...
> >
> > It is excellent that you're looking into things and considering
> > opportunities to improve.
> >
> > It is excellent that you didn't take it so far but instead had an idea,
> > wrote it down in such a way others can review, and offer feedback.
> >
> > This PR and the related ones prior to it make me think there is value in
> a
> > thread that outlines things people can do who are truly motivated to
> really
> > help drive things forward.  I'll share some of those thoughts in another
> > thread.
> >
> > The above is of course just my view.  Will keep an eye for others'
> inputs.
> >
> > Thanks
> >
> >
> >
> > On Sat, Jun 15, 2024 at 9:24 AM Jim Steinebrey 
> > wrote:
> >
> >> Hi,
> >>
> >> https://github.com/apache/nifi/pull/8968
> >>
> >> This PR is a proposal to see if the PMC agrees with this API change.
> >> I have not written any new unit tests for this PR yet until I get buy-in
> >> on it.
> >>
> >> I added two new create methods to the API of ProcessSession to allow the
> >> option of
> >> passing in ff attributes to be added to the flow file being created.
> This
> >> is not
> >> a breaking change because the original create methods are still there.
> >>
> >> ProcessSession
> >> FlowFile create(FlowFile parent, final Map attributes);
> >> FlowFile create(final Map attributes);
> >>
> >> Changing a core API is a very significant change but I hope people will
> >> see it
> >> as worthwhile to allow us to to avoid creating an extra FlowFile object
> in
> >> many
> >> places that FlowFiles are created. Not all places set attributes right
> >> after \
> >> FF creation, but very many of them do and could benefit.
> >>
> >> There are 150+ places where these new methods can be used and I only
> >> changed the
> >> GenerateTableFetch processor to call them so you how the new methods are
> >> used.
> >> I expect using these new create methods has the potential to
> >> avoid noticeable transient memory allocation like the earlier API
> addition
> >> of
> >> SessionContext.isAutoTerminated did.
> >>
> >> Also if this PR is approved, myself and others can change processors to
> >> call these
> >> new methods in a future PRs (not as part of this PR).
> >>
> >> Regards,
> >> Jim Steinebrey
>
>

Thoughts on various ways to help drive NiFi and the community forward

[Joe Witt]

Team,

We are seeing an increase in the unique contributors on a monthly basis to
NiFi and a clear and steady increase in the overall contribution rate over
the past 1-2 years in particular.  This is awesome but we do have important
problems to cover and we need to make it easier to help newer contributors
know where to jump in and add the most value.

For sure people will contribute various things just because they have a
particular interest such as some new extension, etc..  But from a community
direction perspective there are things we need to continue to improve upon
and we want to help make the path to committership and PMC membership more
clear as a function of adding value to the community.

So not necessarily ordered, here are some key things I've observed that
need more attention and are among the most valuable things for the health
of the nifi system and community.

Dependency Management

This is a hugely important area as we have a massive amount of dependencies
and this work is just plain not that fun.  But if we want to present a
beautiful landscape we need to pull weeds.  We need to reduce dependencies
and keep things recent.  This is largely accomplished today by a very small
group of people.  We have to increase the contribution level here or
perhaps the automation level.  We have tooling that is helping now but
still needs more attention.

Vulnerability Management particular as it relates to dependencies

Related to dependency management but broader.  This one at times requires
us to do more than simply maintain dependencies.  It sometimes forces us to
change libraries and sometimes even makes maintaining
compatibility/configuration complicated.  These are among the most critical
and most difficult tasks to take on.  This is largely done today by an
extremely small group of people.

Maintenance of extensions

We have a ton of components in NiFi.  Obviously not all of them really do
require active maintenance but there is often a contribute and forget model
involved.  For us to keep these broad range of components under community
care they need active maintenance.  I already mentioned the dependency
factor above but the broader point here is that APIs/versions of systems
evolve.  We need to be staying up to date.  There are systems like Kafka,
Elastic, Mongo, Relational Databases, etc.. that we should always be
supporting the latest on.  These are among the most wildly popular sources
and destinations.  We do tend to fall behind here and this is powerfully
useful for people to contribute to.  These are the things our users
actually use NiFi for!

Maintenance of things other than core nifi

If you look at components ancillary to NiFi they receive far less attention
and effort trending toward them going away.  The NiFi registry is a great
example of this as NiFi MiNiFi Java for instance.  These are things which
are useful and indeed there is a decent user base in both cases.  But they
require more active maintenance than they get.  We need to evolve when it
comes to security methods offered for authn/authz and that takes focus.

Performance analysis and improvements targeted as a result

This is done very rarely as far as I can tell outside a couple of people
who I'm sure we could all name in a few seconds.  The codebase is large and
Java and other parts evolve.  Continually emphasizing faster and more
efficient operations is vital to NiFi's continued growth in the
community at large.  People who spend time running NiFi in a measured
manner with realistic flows and doing things like profiling for CPU,
memory, etc.. and finding bottlenecks and offering solutions - absolutely
can be high impact and valuable.  Even if not at a point where a confident
contribution can be made, the very act of doing this level of evaluation
and flagging concerns is valuable.

Improvements related to tests both unit and integration - but also for
users of NiFi

As part of our push to NiFi 2.0 we have been running these tests far more
often and more complete than ever before.  Github Actions now pretty much
does run all the things.  We've deleted a wild amount of junk tests that
would always fail or were spurious or required some complex local
access/setup and therefore nobody ever ran them.  We need to always improve
coverage but we want to make tests faster/more efficient and more
leveraging of things like test containers would be highly valuable.
Importantly though here I also mean to highlight we need to do a lot more
for users of NiFi that want to create automated tests for their flows.  We
should make it easy for them to provide a flow definition or perhaps a DSL
for their flow and write tests which would leverage actual nifi to validate
a given input results in an expected output.  This is something that would
add a ton of value for our user base but also lead into some really
powerful testing we can do against our own framework.

Development of blogs/videos/conference presentations/social media posts

Buildi

Re: Support Passing in FF Attribute to Flow File create methods

[Joe Witt]

Jim,

Missing from this proposal is an explanation of the core 'why' or benefit
this provides.  I could see two potential ones based on the comment and
this seems related to recent patterns of code changes in this similar
vein.

1. We avoid a temporary java object representing the flow file.
2. It could reduce lines of code.

If these are the only potential benefits I don't find them compelling.
The cost in a memory/garbage collection sense I would expect is nearly
immeasurably minor.  While it would be great if it meant some considerable
improvement in code required or readability I dont see that in this case.
>From the example processor change shown for GenerateTableFetch the code to
me looks less readable and less expressive.

So then there are other things we'd consider which we cannot see just from
the PR.

1. Does this make the system run measurably faster such that it reduces CPU
usage, lowers GC?
2. Does this drive down memory usage measurably/etc..?
3. Does this make our codebase in general more maintainable/etc..?

If you could share more on what led you to feel this is a helpful addition
that information would be great for the community to know in considering
this change.

As it is right now I am not supportive.  However, I want to be clear about
a couple things as while the feedback above is negative relative to the
proposal...

It is excellent that you're looking into things and considering
opportunities to improve.

It is excellent that you didn't take it so far but instead had an idea,
wrote it down in such a way others can review, and offer feedback.

This PR and the related ones prior to it make me think there is value in a
thread that outlines things people can do who are truly motivated to really
help drive things forward.  I'll share some of those thoughts in another
thread.

The above is of course just my view.  Will keep an eye for others' inputs.

Thanks



On Sat, Jun 15, 2024 at 9:24 AM Jim Steinebrey 
wrote:

> Hi,
>
> https://github.com/apache/nifi/pull/8968
>
> This PR is a proposal to see if the PMC agrees with this API change.
> I have not written any new unit tests for this PR yet until I get buy-in
> on it.
>
> I added two new create methods to the API of ProcessSession to allow the
> option of
> passing in ff attributes to be added to the flow file being created. This
> is not
> a breaking change because the original create methods are still there.
>
> ProcessSession
> FlowFile create(FlowFile parent, final Map attributes);
> FlowFile create(final Map attributes);
>
> Changing a core API is a very significant change but I hope people will
> see it
> as worthwhile to allow us to to avoid creating an extra FlowFile object in
> many
> places that FlowFiles are created. Not all places set attributes right
> after \
> FF creation, but very many of them do and could benefit.
>
> There are 150+ places where these new methods can be used and I only
> changed the
> GenerateTableFetch processor to call them so you how the new methods are
> used.
> I expect using these new create methods has the potential to
> avoid noticeable transient memory allocation like the earlier API addition
> of
> SessionContext.isAutoTerminated did.
>
> Also if this PR is approved, myself and others can change processors to
> call these
> new methods in a future PRs (not as part of this PR).
>
> Regards,
> Jim Steinebrey

Re: Drop unused features from MiNiFi C++

[Joe Witt]

Marton

I strongly support this. The path needs to be maintainable and keeping
things around that dont get the love create many challenges.

Thanks

On Tue, Jun 11, 2024 at 7:48 AM Marton Szasz  wrote:

> Hi community,
>
> We're considering to drop certain features from MiNiFi C++ that seem to
> be unused, and not worth maintaining. If you would like to keep some of
> them, please share your opinion and your use case for them!
>
> To be removed:
> 1. CoAP C2: not used anywhere as far as I'm aware, and not well
> maintained. C2 over HTTP would remain unchanged.
> 2. nanofi: This was an initiative around 2019-2020 to rewrite about half
> of minifi in a C library form for integration to other software. There
> was no development on it since early 2020, and I'm not aware of any users.
> 3. pcap extension: not well tested or maintained, probably not used by
> anyone
> 4. usb camera extension: same as pcap
> 5. sensors extension: same as pcap
> 6. openwsman extension: same as pcap
>
> As we progress towards the new major versions, now is our best
> opportunity to remove unused code, so the codebase becomes easier to
> maintain, and this hopefully unlocks new feature possibilities. Let me
> know what you think!
>
> Thanks,
> Marton
>
>

Re: [DISCUSS] Progress Update on new UI

[Joe Witt]

Thanks Matt - this path makes sense to me.  I've been using the new ui
exclusively for the past couple of months and now with the improvements and
gaps covered I think it is the perfect time to go to it exclusively.  We
give at least one milestone release.  We gather feedback/fix things.  Then
we launch 2.0!

Big thanks to all those who contributed to the new UI both in coding as
well as feedback.

One other thing worth tackling over time is updating documentation imagery
to leverage the new UI but that is something a large group of people can
contribute to and much easier when the new UI is what is in use by default.

Thanks

On Thu, May 30, 2024 at 11:09 AM Scott Aslan  wrote:

> Outstanding! The new UI is fun, fast, and fantastic. Gathering feedback
> from the community will be invaluable, and the best way to do this is by
> having them actively use the new UI.
>
> On Thu, May 30, 2024 at 12:37 PM Matt Gilman 
> wrote:
>
> > Team,
> >
> > About a month ago a [DISCUSS] thread was sent regarding the UI updates
> and
> > requesting that any future contributions target both old and new UIs.
> Since
> > then there has been substantial progress as we continue to close the gap
> on
> > feature parity with the old UI. Only a few features remain [1] [2] [3]
> [4].
> >
> > The updated UI was included in the most recent 2.0.0-M3 release [5]
> > alongside the old UI. Since then we've received lots of helpful feedback
> > and all reported bugs and differences from the existing UI have been
> > addressed. A new JIRA has been filed [6] to remove the old UI and install
> > the new UI as the default. This will be a great step forward to be able
> to
> > move away from some frameworks that are no longer officially supported
> [7].
> > When 2.0.0-M4 is released it will only contain the new UI. Until then
> > please continue to provide any feedback.
> >
> > Additional work has begun that will establish a good path forward on
> > modernizing the remaining UIs within NiFi (documentation, data viewers,
> > custom UIs, etc) that also have outdated/unsupported dependencies. As we
> > continue to make progress on the remaining features, work will begin to
> > start modernizing these other UIs.
> >
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-13316
> > [2] https://issues.apache.org/jira/browse/NIFI-13317
> > [3] https://issues.apache.org/jira/browse/NIFI-13318
> > [4] https://issues.apache.org/jira/browse/NIFI-13319
> > [5]
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
> > [6] https://issues.apache.org/jira/browse/NIFI-13313
> > [7] https://issues.apache.org/jira/browse/NIFI-11481
> >
>

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 2.0.0

[Joe Witt]

+1 binding

On Thu, May 23, 2024 at 10:33 AM Matt Burgess  wrote:

> +1 binding, verified the commit hash, applied the patch to NiFi main and
> verified the NAR Maven plugin works as expected.
>
> Thanks for RM'ing Pierre!
>
> On Wed, May 22, 2024 at 7:31 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hello Apache NiFi Community,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > NAR Maven Plugin 2.0.0. This new release is intended to be used with NiFi
> > 2+.
> >
> > The source being voted upon, including signatures, digests, etc. can be
> > found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-2.0.0/
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
> >
> > Please note that when trying this release, some changes are expected on
> the
> > NiFi side as proposed in the PR:
> https://github.com/apache/nifi/pull/8860
> >
> > The Git tag is nifi-maven-2.0.0-RC1
> > The Git commit ID is d0de49a34a1197988eaa007d28e6eb20ab6d701f
> >
> >
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=d0de49a34a1197988eaa007d28e6eb20ab6d701f
> >
> > Checksums of nifi-nar-maven-plugin-2.0.0-source-release.zip:
> > SHA256: ae889c843a82c4823fca0bf5fedf275dbb812c2b11e25ae26b0bd4ad7d26ccae
> > SHA512:
> >
> >
> c9afefb093d8b9568012d6697f302eb4a138f0a65a7045eaeb1bff551fb4bff23c805294700ca20984a3493eb61fe80a39d8b5b393afc85cf3f6ab1a28df831c
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pvillard.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 5 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354745
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion2.0.0
> >
> > The vote will be open for 72 hours. Please download the release
> > candidate and evaluate the necessary items including checking hashes,
> > signatures, build from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-nar-maven-plugin-2.0.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because …
> >
>

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC4)

[Joe Witt]

+1 binding.

Source builds now on osx.  usual checks for license/notice good.  Checks on
signature/hashes/latest commit all good.  Wasn't able to 'make docker'
successfully but likely a local config issue.

On Thu, May 16, 2024 at 11:10 AM Pierre Villard 
wrote:

> +1 (binding)
>
> Went through the usual steps
> Built on MacOS X.
> Tested a basic flow tailing a local file, doing some transformations and
> sending data with InvokeHTTP to a NiFi instance with FlowFile format.
>
> Thanks team for taking care of this release and the exciting upcoming 1.0.
>
> Le mer. 15 mai 2024 à 17:30, Ferenc Gerlits  a écrit
> :
>
> > +1 (non-binding)
> >
> > verified signature and hashes
> > installed the Windows MSI and ran the service on Windows
> > compiled on Fedora 40 and ran ExecuteScript with lua
> >
> > It mostly worked as expected. I found two minor issues:
> > https://issues.apache.org/jira/browse/MINIFICPP-2374,
> > https://issues.apache.org/jira/browse/MINIFICPP-2375, but neither of
> > these is a showstopper, we'll fix them in the next release.
> >
> > Thank you for RMing!
> > Ferenc
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0-M3 (RC1)

[Joe Witt]

+1 binding.

Verified full clean build with contrib check and all unit tests pass.
Ran entire integration/system test suite including python components.  All
pass.
Built and deployed maven docker image.  All good.
Ran a flow which leverages both Java and Python based extensions.  All good.
Ran dependency update reports.  Looks good but did file [1]
Ran dependency-check vuln reporting and do note a series of findings which
look new and need review/follow-up.

[1] https://issues.apache.org/jira/browse/NIFI-13227

Thanks David for RM.

Thanks everyone.  We've had 30 different contributors in the past month and
have a new surge in commits higher than we've seen since 2016!

On Tue, May 14, 2024 at 4:48 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M3.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1269
>
> Git Tag: nifi-2.0.0-M3-RC1
> Git Commit ID: f2215c6522a5571189290760c55f0317f8562cbd
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/f2215c6522a5571189290760c55f0317f8562cbd
>
> Checksums of nifi-2.0.0-M3-source-release.zip
>
> SHA256: d471a0a9e4e04faf13bbe13c7d83be6f8002fba598bf0968a3c1b339802a185a
> SHA512:
> cd0b8bbd3fe4ea7ebe8cdac6ac8a7afa97fd7b6a521c2cbcb2c0cdc94899b652bf3726c8fe432e16f44a9dc81907414bbb42e03113f0cd9fb51aa1de9cd727a7
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 411
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M3
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [discuss] removal of nifi-external module in 2.x/deprecate in 1.x

[Joe Witt]

Thanks Otto and David - I dont plan to solicit user input as the issue is
more of what the developers/maintainers are seeing.  For anything we have
there is/was at least someone using it.  But that has to be matched with
someone maintaining it as well.  If there are pockets of use/maintenance
those can remain on-going but not within the apache project itself until we
re-establish critical mass for that standalone kafka connect capability.
Given the great work ongoing right now for the Kafka processors I think we
keep our focus there.

Will put the PR up.

Thanks

On Tue, May 7, 2024 at 1:40 PM David Handermann 
wrote:

> Joe,
>
> Thanks for highlighting the Kafka Connect external module. I concur
> with your evaluation, I have not observed community usage or
> questions, so it seems best to remove from the main branch.
>
> Regards,
> David Handermann
>
> On Tue, May 7, 2024 at 2:18 PM Otto Fowler 
> wrote:
> >
> > I would say nuke it, as you are saying.  I would also say you should
> cross
> > post this to users@ as they will have relevant input
> >
> >
> > On May 7, 2024 at 1:32:11 PM, Joe Witt (joew...@apache.org) wrote:
> >
> > Team,
> >
> > The only component remaining is kafka-connect in the nifi-external
> module.
> > As noted in [1] it does not see much active maintenance and I never see
> in
> > the apache community any questions/usage related either. Kafka is
> > currently on 3.7.0 and this component is Kafka 2.6 for example.
> >
> > We should not carry this forward in the 2.x line. To be clear Kafka
> > remains a super important component to interact with and our Kafka
> > components within NiFi do see maintenance including an active PR under
> > review right now. But this is our last 'nifi-external' remaining and is
> > not maintained.
> >
> > I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x
> and
> > deprecation on 1x but wanted to flag this here in case there is some
> > substantial usage and/or maintenance/update coming soon. Even if desire
> > remains to keep it alive that should be considered to occur in another
> repo
> > or by another group that leverages it - but please let me know if there
> is
> > apache community usage/mx I'm not aware of.
> >
> > Thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-13171
> >
> > Thanks
> > Joe
>

Re: Azure lib issue in 1.26

[Joe Witt]

Eduardo

Yes this was found/fixed today[1]

Sorry for the trouble

[1] https://issues.apache.org/jira/browse/NIFI-13181

Thanks
Joe

On Wed, May 8, 2024 at 6:59 PM Eduardo Fontes 
wrote:

> Hi!
>
> Someone with errors like that em Azure components in Nifi 1.26?
>
>  java.lang.NoSuchMethodError:
> 'com.microsoft.aad.msal4j.AbstractApplicationBase$Builder
>
> com.microsoft.aad.msal4j.ConfidentialClientApplication$Builder.logPii(boolean)'
>
> I see thar msal lib was updated since 1.25.
>
> Best regards.
> Eduardo Fontes
>

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)

[Joe Witt]

 pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 15.3.0.0.1.1708646388
volume: /
location: /
install-time: 1710790868
sw_vers
ProductName: macOS
ProductVersion: 14.4.1
BuildVersion: 23E224

Thanks
Joe


On Wed, May 8, 2024 at 8:15 AM Gábor Gyimesi  wrote:

> For checking the xcode version you can use the following command:
> $ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
> And to check the macos version use the following command:
> $ sw_vers
>
> On Wed, 8 May 2024 at 16:35, Gábor Gyimesi  wrote:
>
> > The build output should be visible on the stdout and stderr channels so
> > the output should be visible above the python output. If it is hidden for
> > some reason when you use the python bootstrap, you could use the cmake
> > commands to rerun the build with the default extensions:
> > $ mkdir build && cd build && cmake -G Ninja .. && cmake --build .
> --target
> > package
> >
> > BR,
> > Gabor
> >
> > On Wed, 8 May 2024 at 16:23, Joe Witt  wrote:
> >
> >> Gabor - thanks.  Can you share a specific command and/or file I can look
> >> at
> >> which would have the good info?
> >>
> >> On Wed, May 8, 2024 at 7:00 AM Gábor Gyimesi 
> >> wrote:
> >>
> >> > Hi Team,
> >> >
> >> > In light of the issues found on the MacOS platform I am canceling RC3
> >> and
> >> > will create RC4 once the issues are resolved.
> >> >
> >> > Two of the runtime issues have already been addressed by these 2 PRs:
> >> > https://github.com/apache/nifi-minifi-cpp/pull/1782
> >> > https://github.com/apache/nifi-minifi-cpp/pull/1783
> >> >
> >> > The investigation of the build issues on the latest MacOS version is
> >> still
> >> > ongoing.
> >> >
> >> > @Joe can you confirm that your build issue is on the MacOS version
> >> 14.4.1
> >> > with xcode 15.3 or that it is due to the flex issue Adam mentioned?
> >> > Otherwise please share the issue so we could check that one out as
> well.
> >> >
> >> > Thanks,
> >> > Gabor
> >> >
> >> > On Wed, 8 May 2024 at 15:40, Adam Debreceni 
> >> wrote:
> >> >
> >> > > -1 (non-binding)
> >> > >
> >> > > Verified signatures/hashes, compiled on macos 14.2.1 with xcode 15.1
> >> and
> >> > > deployed simple flow through c2 protocol,
> >> > > but I have found the following on macos:
> >> > >   - getting errors running nifi python processors, due to python
> >> > searching
> >> > > for minifi_native.so
> >> > >   - minifi.plist is missing from the package making it impossible to
> >> > > install the agent as a service
> >> > >
> >> > > (We also have to be mindful about the flex package as either we use
> >> the
> >> > > system flex or the brew flex
> >> > > but for the latter both the PATH and the CMAKE_CXX_FLAGS should be
> set
> >> > > accordingly)
> >> > >
> >> > > We also found that it does not compile on the newest macos 14.4.1
> with
> >> > > xcode 15.3.
> >> > >
> >> > > On Wed, May 8, 2024 at 1:15 PM Martin Zink 
> >> > wrote:
> >> > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > -Verified signature, hashes and git commit hash
> >> > > > -Built it from source with default extensions on RockyLinux 9
> >> (aarch64
> >> > > > and x86_64), macOS 14.3.1 (M1) and Windows 10 (x86_64) using the
> new
> >> > > > python based bootstrap
> >> > > > -Ran through a couple of C2 initiated flows
> >> > > >
> >> > > > Everything worked as expected,
> >> > > >
> >> > > >
> >> > > > thanks Gabor for RMing great work,
> >> > > > Martin
> >> > > >
> >> > > >
> >> > > > On Wed, May 8, 2024 at 9:18 AM Gábor Gyimesi <
> lordga...@apache.org>
> >> > > wrote:
> >> > > > >
> >> > > > > Hi Joe,
> >> > > > >
> >> > > > > Could you check the ninja output for the build error? The output
> >> you
> >> > > sent
> >> &g

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)

[Joe Witt]

Gabor - thanks.  Can you share a specific command and/or file I can look at
which would have the good info?

On Wed, May 8, 2024 at 7:00 AM Gábor Gyimesi  wrote:

> Hi Team,
>
> In light of the issues found on the MacOS platform I am canceling RC3 and
> will create RC4 once the issues are resolved.
>
> Two of the runtime issues have already been addressed by these 2 PRs:
> https://github.com/apache/nifi-minifi-cpp/pull/1782
> https://github.com/apache/nifi-minifi-cpp/pull/1783
>
> The investigation of the build issues on the latest MacOS version is still
> ongoing.
>
> @Joe can you confirm that your build issue is on the MacOS version 14.4.1
> with xcode 15.3 or that it is due to the flex issue Adam mentioned?
> Otherwise please share the issue so we could check that one out as well.
>
> Thanks,
> Gabor
>
> On Wed, 8 May 2024 at 15:40, Adam Debreceni  wrote:
>
> > -1 (non-binding)
> >
> > Verified signatures/hashes, compiled on macos 14.2.1 with xcode 15.1 and
> > deployed simple flow through c2 protocol,
> > but I have found the following on macos:
> >   - getting errors running nifi python processors, due to python
> searching
> > for minifi_native.so
> >   - minifi.plist is missing from the package making it impossible to
> > install the agent as a service
> >
> > (We also have to be mindful about the flex package as either we use the
> > system flex or the brew flex
> > but for the latter both the PATH and the CMAKE_CXX_FLAGS should be set
> > accordingly)
> >
> > We also found that it does not compile on the newest macos 14.4.1 with
> > xcode 15.3.
> >
> > On Wed, May 8, 2024 at 1:15 PM Martin Zink 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > -Verified signature, hashes and git commit hash
> > > -Built it from source with default extensions on RockyLinux 9 (aarch64
> > > and x86_64), macOS 14.3.1 (M1) and Windows 10 (x86_64) using the new
> > > python based bootstrap
> > > -Ran through a couple of C2 initiated flows
> > >
> > > Everything worked as expected,
> > >
> > >
> > > thanks Gabor for RMing great work,
> > > Martin
> > >
> > >
> > > On Wed, May 8, 2024 at 9:18 AM Gábor Gyimesi 
> > wrote:
> > > >
> > > > Hi Joe,
> > > >
> > > > Could you check the ninja output for the build error? The output you
> > sent
> > > > is only the bootstrap script's traceback of the build step's failure,
> > but
> > > > there should probably be an error in the build output as well
> starting
> > > with
> > > > "error: ".
> > > >
> > > > BR,
> > > > Gabor
> > > >
> > > > On Wed, 8 May 2024 at 02:26, Joe Witt  wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Sigs/Hashes are good.
> > > > >
> > > > > Building source on macos I'm getting a build failure.  The only
> > option
> > > > > selected was to skip tests otherwise left as defaults.  Am
> following
> > > the
> > > > > readme and release guide.
> > > > >
> > > > > ninja: build stopped: subcommand failed.
> > > > > Build was unsuccessful
> > > > > Traceback (most recent call last):
> > > > >   File
> > > > >
> > > > >
> > >
> >
> "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/main.py",
> > > > > line 62, in 
> > > > > main_menu(minifi_options, package_manager)
> > > > >   File
> > > > >
> > > > >
> > >
> >
> "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py",
> > > > > line 87, in main_menu
> > > > > done =
> > > main_menu_options[main_menu_prompt["sub_menu"]](minifi_options,
> > > > > package_manager)
> > > > >
> > > > >
> > > > >
> > >
> >
> 
> > > > >   File
> > > > >
> > > > >
> > >
> >
> "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py",
> > > > > line 60, in do_one_click_build
> > > > > assert do_build(minifi_options, packag

Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)

[Joe Witt]

Hello

Sigs/Hashes are good.

Building source on macos I'm getting a build failure.  The only option
selected was to skip tests otherwise left as defaults.  Am following the
readme and release guide.

ninja: build stopped: subcommand failed.
Build was unsuccessful
Traceback (most recent call last):
  File
"/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/main.py",
line 62, in 
main_menu(minifi_options, package_manager)
  File
"/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py",
line 87, in main_menu
done = main_menu_options[main_menu_prompt["sub_menu"]](minifi_options,
package_manager)

 

  File
"/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py",
line 60, in do_one_click_build
assert do_build(minifi_options, package_manager)


So then I select to not use ninja and run the one click build again.
It seems to run for much longer then the same output prints again.

Thanks
Joe

On Tue, May 7, 2024 at 9:45 AM Arpad Boda  wrote:

> +1 (binding)
>
> Built on ubuntu, verified signature and checksums. All tests passed,
> started and ran a simple flow with no issues.
>
> Nice job, team!
>
> On Tue, May 7, 2024 at 11:35 AM Gábor Gyimesi 
> wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > MiNiFi C++ 0.99.0.
> >
> > The source tarball, some binary builds, plus signatures and digests can
> be
> > found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.99.0/
> >
> > The Git tag is minifi-cpp-0.99.0-RC3
> > The Git commit ID is d3d205f3c8a8376bbad4c8a47758d585d57c653b
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=d3d205f3c8a8376bbad4c8a47758d585d57c653b
> >
> > Checksums of nifi-minifi-cpp-0.99.0-source.tar.gz:
> > SHA256: c00ff309776720b295cb52ab12c1c3d11182af4f19ed0a379d58f96b16841059
> > SHA512:
> >
> >
> d1eda2c19da8f0e073cf460bf889c8b4a1b893de4b245dc0147d1c79545d5298c74bedf585afd7091dc9474c923fc2a7075a81a6397bedce310e5838e3afa898
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/lordgamez.asc
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173087303
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 121 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520&version=12353618
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65145325#ReleaseNotesMiNiFi(C++)-Versioncpp-0.99.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-minifi-cpp-0.99.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: [discuss] identifying key components needed updates which require deeper review/consideration

[Joe Witt]

Awesome Chris and David - both of those are done and dusted!  Amazing.

Matt; Thanks yes for Hive it should just be simplifying/updating the
Iceberg dependency on Hive to the latest or even removing if not
necessary.  Iceberg itself can be updated but my understanding is the 1.5.1
release has a problem and there will soon be a 1.5.2 we should wait for
(someone else had a PR for that and backed it out for now).  The flyway one
is hopefully easy similar to the QuestDB one.

We have some other dependencies in need to attention which I've not created
JIRAs for:
1. Mongo DB.  We're on 4.11 and the latest is 5.1
2. SNMP4J.  We're on 2.8/2.7 and the latest are 3.8 for the primary lib and
agent.
3. Redis.  We're on 3.10 and the latest is 5.1
4. Camel Salesforce Lib.  We're on 3.14 and the latest is 4.5
5. Fastcsv.  We're on 2.2.2 and the latest is 3.1.0

There are a handful of others but these are the top of mind ones in
scanning the results.  For those wanting to see this list on your own in
the root of nifi.git run:

mvn -T 1.0C versions:dependency-updates-aggregate-report
-DonlyUpgradable=true

Then open target/site/dependency-updates-aggregate-report.html

Thanks
Joe


On Tue, May 7, 2024 at 2:05 PM David Handermann 
wrote:

> Joe,
>
> Thanks for highlighting these areas to be updated. I was able to
> address the QuestDB update with a couple minor adjustments, and
> submitted a PR [1].
>
> Regards,
> David Handermann
>
> [1] https://github.com/apache/nifi/pull/8773
>
> On Tue, May 7, 2024 at 2:58 PM Chris Sampson
>  wrote:
> >
> > I’ll pick up the Elasticsearch dependency upgrade.
> >
> > I’ll try to test it again OpenSearch as well, if for no reason other
> than to prove either way whether the current NiFi components will work with
> the AWS service, although I may not get to do that and instead testing will
> focus on Elasticsearch 8.x instances (and the existing Integration Tests
> within NiFi).
> >
> >
> > Cheers,
> >
> > ---
> > Chris Sampson
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > > On 7 May 2024, at 20:54, Joe Witt  wrote:
> > >
> > > Yeah sorry - what I wrote for the Hive one was a bit nonsensical.  I
> tried
> > > to clarify it better in the JIRA.
> > >
> > > Thanks
> > >
> > > On Tue, May 7, 2024 at 12:38 PM Matt Burgess 
> wrote:
> > >
> > >> For [1]  Hive 3 was removed in
> > >> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the
> Hive
> > >> dependency from Apache Iceberg we probably need a release from them
> that
> > >> brings in Hive 4 dependencies? Or do you suggest we bring the Hive
> > >> components back as long as we update the version?
> > >>
> > >> I'll take the Flyway one in the meantime, database integrations are my
> > >> blessing and my curse :) If that goes more smoothly than I expect I
> can
> > >> grab the QuestDB one too, will update the Jiras as appropriate.
> > >>
> > >> Regards,
> > >> Matt
> > >>
> > >> On Tue, May 7, 2024 at 1:56 PM Joe Witt  wrote:
> > >>
> > >>> Team,
> > >>>
> > >>> We have made massive strides going from thousands of out of date
> > >> libraries
> > >>> to dozens in the 2.x line vs any prior releases.  And we can/should
> stay
> > >> on
> > >>> top of these better going forward.  A few key bits that are
> outstanding
> > >>> with no obvious immediate work underway are as follows.  If anyone is
> > >>> working on these, willing to work on these, please volunteer and let
> us
> > >>> know.  For components that we cannot find active engagement on we
> should
> > >>> consider removal though these aren't great examples as they're indeed
> > >>> common usage components.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/NIFI-13173
> > >>> Hive 4 recently came out but Hive also gives us some of the most
> > >>> complicated POM/library mangling and oldest dependencies to
> > >> override/etc..
> > >>> Putting serious detailed focus here is very valuable from a
> maintenance
> > >>> point of view.  While existing usage of Hive 3 likely remains for
> some
> > >>> time, those users are like NiFi 1.x users anyway.  In any case if
> there
> > >> are
> > >>> motivated developers interested to see our Hive components be healthy
> > >&

Re: [discuss] identifying key components needed updates which require deeper review/consideration

[Joe Witt]

Yeah sorry - what I wrote for the Hive one was a bit nonsensical.  I tried
to clarify it better in the JIRA.

Thanks

On Tue, May 7, 2024 at 12:38 PM Matt Burgess  wrote:

> For [1]  Hive 3 was removed in
> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive
> dependency from Apache Iceberg we probably need a release from them that
> brings in Hive 4 dependencies? Or do you suggest we bring the Hive
> components back as long as we update the version?
>
> I'll take the Flyway one in the meantime, database integrations are my
> blessing and my curse :) If that goes more smoothly than I expect I can
> grab the QuestDB one too, will update the Jiras as appropriate.
>
> Regards,
> Matt
>
> On Tue, May 7, 2024 at 1:56 PM Joe Witt  wrote:
>
> > Team,
> >
> > We have made massive strides going from thousands of out of date
> libraries
> > to dozens in the 2.x line vs any prior releases.  And we can/should stay
> on
> > top of these better going forward.  A few key bits that are outstanding
> > with no obvious immediate work underway are as follows.  If anyone is
> > working on these, willing to work on these, please volunteer and let us
> > know.  For components that we cannot find active engagement on we should
> > consider removal though these aren't great examples as they're indeed
> > common usage components.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-13173
> > Hive 4 recently came out but Hive also gives us some of the most
> > complicated POM/library mangling and oldest dependencies to
> override/etc..
> > Putting serious detailed focus here is very valuable from a maintenance
> > point of view.  While existing usage of Hive 3 likely remains for some
> > time, those users are like NiFi 1.x users anyway.  In any case if there
> are
> > motivated developers interested to see our Hive components be healthy
> > please do dive in on this.
> >
> > [2] https://issues.apache.org/jira/browse/NIFI-13169
> > The NiFI registry is a major release behind Flyway DB.  We need to stay
> > current given the importance of this component.
> >
> > [3] https://issues.apache.org/jira/browse/NIFI-13168
> > We are a few incrementals behind and a minor release behind QuestDB.  I
> > tried bumping the incremental but it broke tests.
> >
> > [4] https://issues.apache.org/jira/browse/NIFI-11686
> > We are a full major release behind and several incremental/minors as
> well.
> > Apparently there is some concern related to the community changes in
> > Elastic vs OpenSearch/etc.. but regardless we need to stay active in
> > maintaining this component.  We really should get to the latest main.
> >
> > Thanks to anyone who can help tackle these and make progress both in
> terms
> > of doing the upgrade/update work, validating tests, fixing licenses, but
> > also reviews.
> >
> > Thanks
> > Joe
> >
>

[discuss] identifying key components needed updates which require deeper review/consideration

[Joe Witt]

Team,

We have made massive strides going from thousands of out of date libraries
to dozens in the 2.x line vs any prior releases.  And we can/should stay on
top of these better going forward.  A few key bits that are outstanding
with no obvious immediate work underway are as follows.  If anyone is
working on these, willing to work on these, please volunteer and let us
know.  For components that we cannot find active engagement on we should
consider removal though these aren't great examples as they're indeed
common usage components.

[1] https://issues.apache.org/jira/browse/NIFI-13173
Hive 4 recently came out but Hive also gives us some of the most
complicated POM/library mangling and oldest dependencies to override/etc..
Putting serious detailed focus here is very valuable from a maintenance
point of view.  While existing usage of Hive 3 likely remains for some
time, those users are like NiFi 1.x users anyway.  In any case if there are
motivated developers interested to see our Hive components be healthy
please do dive in on this.

[2] https://issues.apache.org/jira/browse/NIFI-13169
The NiFI registry is a major release behind Flyway DB.  We need to stay
current given the importance of this component.

[3] https://issues.apache.org/jira/browse/NIFI-13168
We are a few incrementals behind and a minor release behind QuestDB.  I
tried bumping the incremental but it broke tests.

[4] https://issues.apache.org/jira/browse/NIFI-11686
We are a full major release behind and several incremental/minors as well.
Apparently there is some concern related to the community changes in
Elastic vs OpenSearch/etc.. but regardless we need to stay active in
maintaining this component.  We really should get to the latest main.

Thanks to anyone who can help tackle these and make progress both in terms
of doing the upgrade/update work, validating tests, fixing licenses, but
also reviews.

Thanks
Joe

Re: ...not the most recent version of this FlowFile within this session...

[Joe Witt]

Russ

FlowFile result = session.*write*( flowfile, new StreamCallback()
   session.*read*( flowfile, new InputStreamCallback()
   session.*removeAllAttributes*( result, keys );
   session.*putAllAttributes*( result, attributes );
   session.*transfer*( result, SUCCESS );

The session.read passes in the same original flowfile reference that you
send in to the write.  I'm guessing that is not what you intended.  You
probably want

session.read(result, newInputStream


Thanks
Joe

On Tue, May 7, 2024 at 10:38 AM Russell Bateman 
wrote:

> In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.
>
> There are peculiar reasons (that we are trying to fix) that inhibit us
> from moving forward as all of our customer machines are running 1.1.2.
> (Don't shoot me, I'm not DevOps, but just the guy who writes custom
> processors.)
>
> I have custom processor code that, built as noted using 1.13.2, runs
> _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on
> 1.1.2, one processor fails to process a very simple flowfile with the
> error message below.
>
> I'm wondering if there's a dance I can do to tweak one or more of what I
> suspect are the calls where this is happening (in onTrigger()):
>
> FlowFile result = session.*write*( flowfile, new StreamCallback()
>session.*read*( flowfile, new InputStreamCallback()
>session.*removeAllAttributes*( result, keys );
>session.*putAllAttributes*( result, attributes );
>session.*transfer*( result, SUCCESS );
>
>
> Error message:
>
> 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
> failed to process session due to
> org.apache.nifi.processor.exception.FlowFileHandlingException: \
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1,
> container=default, section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923]): \
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1,
> container=default, section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923])
>
> 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928]
> Processor Administratively Yielded for 1 sec due to processing failure
>
>
> I know this is old stuff and I'm prepared be told that I'm out of luck
> or that the solution is unfeasible or would take too much effort to dig
> through. I'm just hoping that someone remembers something quick off the
> top of the head and can give advice.
>
> Thanks,
>
> Russ
>
>
>
>
>
>

[discuss] removal of nifi-external module in 2.x/deprecate in 1.x

[Joe Witt]

Team,

The only component remaining is kafka-connect in the nifi-external module.
As noted in [1] it does not see much active maintenance and I never see in
the apache community any questions/usage related either.  Kafka is
currently on 3.7.0 and this component is Kafka 2.6 for example.

We should not carry this forward in the 2.x line.  To be clear Kafka
remains a super important component to interact with and our Kafka
components within NiFi do see maintenance including an active PR under
review right now.  But this is our last 'nifi-external' remaining and is
not maintained.

I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x and
deprecation on 1x but wanted to flag this here in case there is some
substantial usage and/or maintenance/update coming soon.  Even if desire
remains to keep it alive that should be considered to occur in another repo
or by another group that leverages it - but please let me know if there is
apache community usage/mx I'm not aware of.

Thoughts?

[1] https://issues.apache.org/jira/browse/NIFI-13171

Thanks
Joe

Re: [VOTE] Release Apache NiFi 1.26.0 (RC1)

[Joe Witt]

+1 binding.

Thanks Pierre!

On Mon, May 6, 2024 at 8:08 AM Krisztina Zsihovszki 
wrote:

> +1 (non-binding)
>
> - Verified signatures and hashes
> - Ran build using Maven 3.8.5 on macOS 13.5 with 8.0.392-zulu
> - Contrib check, unit tests passed
>
> Verified these tickets:
>
> [NIFI-12677] - ExcelReader documentation gives an example of a non-existent
> strategy
> [NIFI-12960] - Support reading password-protected files in ExcelReader
>
>
> Thanks for RM'ing, Pierre!
>
> Krisztina
>
> On Mon, May 6, 2024 at 4:55 PM Michael Moser  wrote:
>
> > +1 (binding)
> >
> > verified signature, hashes, commit ID
> > had a glance at the README, LICENSE, NOTICE files
> >
> > built on:
> > Apache Maven 3.9.6 Maven home:
> > /home/mwmoser/.m2/wrapper/dists/apache-maven-3.9.6/a53741d1
> > Java version: 1.8.0_412, vendor: Red Hat, Inc., runtime:
> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.el8.x86_64/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "4.18.0-513.24.1.el8_9.x86_64", arch: "amd64",
> > family: "unix"
> >
> > Ran some simple flows and performed some operations against NiFi
> Registry.
> >
> > Many thanks to Pierre for being RM.
> >
> > -- Mike
> >
> >
> > On Mon, May 6, 2024 at 10:25 AM Peter Turcsanyi 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Verified signatures and hashes.
> > > Built NiFi on Ubuntu 22.04 with Java 8 (Temurin 1.8.0_412+8) and Java
> 11
> > > (Temurin 11.0.23+9), Maven 3.9.6 with empty local repo.
> > >
> > > Ran flows for validating:
> > >  - NIFI-12837 Add DFS setting to smb processors
> > >  - NIFI-12732 ListS3 should reset its tracking state after
> configuration
> > > change
> > >  - NIFI-12936 ListGCSBucket should reset its tracking state after
> > > configuration change
> > >  - NIFI-12928 Add Simple Write strategy in PutAzureDataLakeStorage
> > >
> > > Thanks for RMing Pierre!
> > >
> > > Regards,
> > > Peter Turcsanyi
> > >
> > > On Mon, May 6, 2024 at 4:09 PM Chris Sampson
> > >  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Ran through the release helper. Verified signatures and hashes.
> > > >
> > > > Full clean build of RC with Java openjdk 11.0.23 2024-04-16 on macOS
> > > > Sonoma 14.4.1 using the Maven Wrapper 3.9.6.
> > > >
> > > > Ran a selection of processors/flows, primarily focussed on
> > Elasticsearch
> > > > integration and HTTP Request/Response.
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > ---
> > > > Chris Sampson
> > > > IT Consultant
> > > > chris.samp...@naimuri.com
> > > >
> > > >
> > > > > On 6 May 2024, at 14:31, David Handermann <
> > exceptionfact...@apache.org
> > > >
> > > > wrote:
> > > > >
> > > > > +1 binding
> > > > >
> > > > > - Verified signatures and hashes
> > > > > - Ran build using Maven Wrapper 3.9.6
> > > > > - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-382 x86_64
> > > > >
> > > > > Thanks Pierre!
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > On Fri, May 3, 2024 at 7:47 AM Pierre Villard
> > > > >  wrote:
> > > > >>
> > > > >> Team,
> > > > >>
> > > > >> I am pleased to be calling this vote for the source release of
> > Apache
> > > > NiFi
> > > > >> 1.26.0.
> > > > >>
> > > > >> Please review the following guide for how to verify a release
> > > candidate
> > > > >> build:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > > >>
> > > > >> The source being voted on the and the convenience binaries are
> > > > available on
> > > > >> the Apache Distribution Repository:
> > > > >>
> > > > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.26.0
> > > > >>
> > > > >> The build artifacts are available on the Apache Nexus Repository:
> > > > >>
> > > > >>
> > https://repository.apache.org/content/repositories/orgapachenifi-1265
> > > > >>
> > > > >> Git Tag: nifi-1.26.0-RC1
> > > > >> Git Commit ID: 338083f6850b61cd2651e41bbd73b3cb5330d98c
> > > > >> GitHub Commit Link:
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/nifi/commit/338083f6850b61cd2651e41bbd73b3cb5330d98c
> > > > >>
> > > > >> Checksums of nifi-1.26.0-source-release.zip
> > > > >>
> > > > >> SHA256:
> > > 75ea201c12bb99672b1f075c9520b5f7df8b24e033ec47121848b351a0d47790
> > > > >> SHA512:
> > > > >>
> > > >
> > >
> >
> 5b754d899ce8cff900d5871d44c2fda9224e69fe8a0fe9a7121f3359ed8881300e4d4d0b2fe5befea276e0495c7ebbed04cc307c18527aa7a1cea25a923a
> > > > >>
> > > > >> Release artifacts are signed with the following key:
> > > > >>
> > > > >> https://people.apache.org/keys/committer/pvillard.asc
> > > > >>
> > > > >> KEYS file is available on the Apache Distribution Repository:
> > > > >>
> > > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > > >>
> > > > >> Issues resolved for this version: 128
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354185
> > > > >>
> > > > >> Release note highlights can be found on the proje

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Joe Witt]

Matt

You are referring to NIFI-12998 and it causing rebasing to be necessary for
outstanding PRs?

The rebase should be quite straight forward for most cases.

If you have any questions Im happy to help.

 I rebased a tremendous number of times to keep up with main changing while
the PR took several weeks to create and test and validate against previous
builds.  Also rebased repeatedly during the review and feedback stage.  As
a result I got to practice my rebase skills about 20 times across 50 or
more commits to main during that time.

On your pr branch

git fetch —all
git rebase upstream/main
git status
fix any conflicts
git add .
git rebase —continue

If any conflicts it would likely be for new modules created in the pr which
would just be moving from a path in nifi-nar-bundles to
nifi-extension-bundles.

If you have a pr in mind you have a question on for the rebase let me know.

Thanks
Joe

On Wed, May 1, 2024 at 9:51 PM Matt Burgess  wrote:

> One thing that was mentioned was an included Jira for  "providing a clearer
> distinction between framework and
> extensions," This involved moving extensions to a new module and for
> community folks this is causing problems with any
> new improvements in-flight before that. Seems like it needed more
> announcement than an upcoming RC due to the impact?
>
> On Mon, Apr 29, 2024 at 10:22 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Team,
> >
> > We are getting closer to ready for a release candidate build. Based on
> > some conversations with those working on the user interface, there are
> > still a couple more items to complete this week. Getting a solid build
> > of the new UI into this next version will be very helpful, so getting
> > a few more of those issues resolved should put things in a good
> > position.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Apr 24, 2024 at 8:32 AM Joe Witt  wrote:
> > >
> > > Also agreed there.  Profile should be there to exclude perhaps but
> > include
> > > should be default.
> > >
> > > On Wed, Apr 24, 2024 at 6:30 AM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > > > Thanks for the replies thus far!
> > > >
> > > > With the goal of including the new UI, it seems like a good time to
> > > > move the module from an optional profile to part of the standard
> > > > build. That would make the release candidate build and review process
> > > > simpler, not requiring the optional build profile.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman  >
> > > > wrote:
> > > > >
> > > > > I agree. Including the updated UI and getting some feedback would
> be
> > > > great.
> > > > >
> > > > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt 
> wrote:
> > > > >
> > > > > > Id be a big fan of including the new UI.
> > > > > >
> > > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard <
> > > > > > pierre.villard...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks David, I'm happy to take care of a 1.26 release at the
> > same
> > > > time.
> > > > > > >
> > > > > > > Regarding the M3 release, should the new UI be included and
> > > > instructions
> > > > > > > given on how to access it to start collecting feedback? I'm
> > under the
> > > > > > > impression that almost all of the work has been done, no?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pierre
> > > > > > >
> > > > > > > Le mar. 23 avr. 2024, 02:03, David Handermann <
> > > > > > exceptionfact...@apache.org
> > > > > > > >
> > > > > > > a écrit :
> > > > > > >
> > > > > > > > Team,
> > > > > > > >
> > > > > > > > We are approaching 300 Jira issues resolved for version
> > 2.0.0-M3
> > > > [1],
> > > > > > > > including a number of important dependency upgrades and
> Python
> > > > > > > > framework improvements.
> > > > > > > >
> > > > > > > > There are several open pul

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Joe Witt]

Also agreed there.  Profile should be there to exclude perhaps but include
should be default.

On Wed, Apr 24, 2024 at 6:30 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Thanks for the replies thus far!
>
> With the goal of including the new UI, it seems like a good time to
> move the module from an optional profile to part of the standard
> build. That would make the release candidate build and review process
> simpler, not requiring the optional build profile.
>
> Regards,
> David Handermann
>
> On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman 
> wrote:
> >
> > I agree. Including the updated UI and getting some feedback would be
> great.
> >
> > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt  wrote:
> >
> > > Id be a big fan of including the new UI.
> > >
> > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard <
> > > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > > > Thanks David, I'm happy to take care of a 1.26 release at the same
> time.
> > > >
> > > > Regarding the M3 release, should the new UI be included and
> instructions
> > > > given on how to access it to start collecting feedback? I'm under the
> > > > impression that almost all of the work has been done, no?
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > Le mar. 23 avr. 2024, 02:03, David Handermann <
> > > exceptionfact...@apache.org
> > > > >
> > > > a écrit :
> > > >
> > > > > Team,
> > > > >
> > > > > We are approaching 300 Jira issues resolved for version 2.0.0-M3
> [1],
> > > > > including a number of important dependency upgrades and Python
> > > > > framework improvements.
> > > > >
> > > > > There are several open pull requests that are getting close to
> > > > > completion, which should be possible to include in an upcoming
> release
> > > > > version. There is also a significant pull request [2] for
> NIFI-12998
> > > > > [3] that includes some important changes to module directory
> > > > > hierarchy, providing a clearer distinction between framework and
> > > > > extensions, and implementing significant cleanup of dependency
> > > > > declarations across the repository.
> > > > >
> > > > > With these changes in view, we should be ready for another
> milestone
> > > > > release soon after merging the above pull request.
> > > > >
> > > > > Another milestone release seems to be the best course of action for
> > > > > now, providing the opportunity for additional review and testing
> > > > > before a full 2.0.0 release version.
> > > > >
> > > > > I would be glad to handle Release Manager responsibilities for
> > > > > 2.0.0-M3 when things are ready.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3
> > > > > [2] https://github.com/apache/nifi/pull/8677
> > > > > [3] https://issues.apache.org/jira/browse/NIFI-12998
> > > > >
> > > >
> > >
>

Re: Desire to view more than 100 items within a queue

[Joe Witt]

Edward

Moved those cc'd to bcc including yourself.  To really send/receive notes
you'll want to subscribe to the mailing list.

We have to effectively draw the line somewhere at how much data is
retrieved and the click to content mechanism supported isn't meant to be
exhaustive as-is.  We can/should add some pagination but when you consider
most queues are actually changing/evolving so fast it didn't make sense.
For the pattern you use which is effectively a dead-letter queue though
your ask makes perfect sense.

Thanks

On Tue, Apr 23, 2024 at 2:38 PM Edward Wang
 wrote:

> To whom it may concern,
>
> We are using NiFi to pass FlowFiles into ExecuteStreamCommand processors.
> Occasionally, running the scripts using that processor results in errors
> that route into the nonzero status relationship.
> We are pointing the nonzero status relationships to non-functioning
> processors, so that they will remain within the connection indefinitely and
> we can track them.
>
> However, whenever we list the contents of a connection, we can only see
> the details of 100 FlowFiles at a time, despite the connection holding more
> than 100.
> This is the case using the web interface and through calling the API.
>
> I was wondering if anyone had experience or any ideas regarding using list
> queue to show all items.
>
> Thank you.
>
> Sincerely,
> Edward
>

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Joe Witt]

Id be a big fan of including the new UI.

On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard 
wrote:

> Thanks David, I'm happy to take care of a 1.26 release at the same time.
>
> Regarding the M3 release, should the new UI be included and instructions
> given on how to access it to start collecting feedback? I'm under the
> impression that almost all of the work has been done, no?
>
> Thanks,
> Pierre
>
> Le mar. 23 avr. 2024, 02:03, David Handermann  >
> a écrit :
>
> > Team,
> >
> > We are approaching 300 Jira issues resolved for version 2.0.0-M3 [1],
> > including a number of important dependency upgrades and Python
> > framework improvements.
> >
> > There are several open pull requests that are getting close to
> > completion, which should be possible to include in an upcoming release
> > version. There is also a significant pull request [2] for NIFI-12998
> > [3] that includes some important changes to module directory
> > hierarchy, providing a clearer distinction between framework and
> > extensions, and implementing significant cleanup of dependency
> > declarations across the repository.
> >
> > With these changes in view, we should be ready for another milestone
> > release soon after merging the above pull request.
> >
> > Another milestone release seems to be the best course of action for
> > now, providing the opportunity for additional review and testing
> > before a full 2.0.0 release version.
> >
> > I would be glad to handle Release Manager responsibilities for
> > 2.0.0-M3 when things are ready.
> >
> > Regards,
> > David Handermann
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3
> > [2] https://github.com/apache/nifi/pull/8677
> > [3] https://issues.apache.org/jira/browse/NIFI-12998
> >
>

Re: Support Nifi Install on Debian Linux

[Joe Witt]

Hello

NiFi 2.x line requires a minimum of Java 21.

Thanks

On Thu, Apr 18, 2024 at 10:42 PM Fábio Augusto Pássari 
wrote:

> Dear support,
>
> I am installing nifi-2.0.0-M2 on linux ubuntu but it does not initialize
> the Nifi service.
>
> *ls /usr/lib/jvm*
> default-java   java-11-openjdk-amd64 java-8-openjdk-amd64
> java-1.11.0-openjdk-amd64  java-17-openjdk-amd64 openjdk-11
> java-1.17.0-openjdk-amd64  java-1.8.0-openjdk-amd64
> java-1.18.0-openjdk-amd64  java-18-openjdk-amd64
>
> *echo $JAVA_HOME*
> /usr/lib/jvm/java-11-openjdk-amd64
>
> *bin/nifi.sh start*
>
> Java home: /usr/lib/jvm/java-11-openjdk-amd64
> NiFi home: /opt/nifi-2.0.0-M2
>
> Bootstrap Config File: /opt/nifi-2.0.0-M2/conf/bootstrap.conf
>
> Erro: ocorreu LinkageError ao carregar a classe principal
> org.apache.nifi.bootstrap.RunNiFi
> java.lang.UnsupportedClassVersionError:
> org/apache/nifi/bootstrap/RunNiFi has been compiled by a more recent
> version of the Java Runtime (class file version 65.0), this version of the
> Java Runtime only recognizes class file versions up to 55.0
>
> Can you help me?
> --
> Fábio Augusto Pássari
>

Re: Is there a need for the ConvertAvroToJson processor in 2.x?

[Joe Witt]

The best list is here

https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features

But that is a great point.  We generally want to drop any ConvertXtoY bits
we have that aren't well covered by Records.

Thanks

On Wed, Apr 10, 2024 at 1:07 PM Dan S  wrote:

> I guess I am wondering why it was not included under NIFI-11172
> .
>
> On Wed, Apr 10, 2024 at 12:01 PM Mike Thomsen 
> wrote:
>
> > IIRC, the plan was to remove those ConvertToX processors in 2.X
> >
> > On Wed, Apr 10, 2024 at 11:36 AM Dan S  wrote:
> >
> > > I noticed that the ConvertAvroToJson processor is not marked as
> > deprecated
> > > in 2.x. Is there something it offers which cannot be done with the
> > > ConvertRecord processor configured with an AvroReader and
> > > JsonRecordSetWriter?
> > >
> >
>

Apr 24 - Apache NiFi board report

[Joe Witt]

NiFi Community

Thanks as always for the continued progress!



## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago)
There are currently 66 committers and 36 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Csaba Bejan on 2023-10-25.
- Daniel Stieglitz was added as committer on 2024-02-09.
- Mark Bathori was added as committer on 2024-02-09

## Project Activity:
We released Apache  NiFi 2.0.0 milestone 2 January 29th with more than
200 completed JIRAs with new features.  The release has seen quick adoption
and found a variety of bugs and improvements.  To that end the community
is approaching a release of NiFi 2.0.0-M3 with already more than 235
resolved
JIRAs.  We anticipate a release for that soon as there are major
improvements
to the stability of the much anticipated first class integration with Python
components which has been highly valuable to NiFi's emerging role in data
pipelines to feed data to Generative AI applications.  We also released
Apache NiFi 1.25.0 with more than 100 JIRAs at the end of January and have
more than 100 already resolved on the upcoming 1.26.0 line.

## Community Health:
Consistent with past reports community health remains strong and growing.
Mailing list activity remains consistent and we saw a 30% increase on the
dev list in total emails.  The slack community continues to grow with
general hosting 3,133 users at the time of this report. A highly
desirable pattern of engagement is present where it isn't just committers
and PMC engaging to answer questions.  A new venture backed startup in
the US has formed as well which has led to an increase in traffic and
discussion regarding Apache NiFi.

Re: is there a trick to running Nifi on AWS

[Joe Witt]

Mark

I believe you will need to tell NiFi you want it to listen on more than the
localhost/loopback address.

nifi.web.https.host=localhost

Is a default in nifi.properties for instance.

Def take a look through the admin/install guide as well.

Thanks

On Thu, Mar 28, 2024 at 1:32 PM Mark Woodcock 
wrote:

> Howdy,
>
> Cranked up an EC2 instance.
> Installed Java 11.
> set up JAVA_HOME
> Downloaded Nifi 1.25.0
> unzipped Nifi
> set a nifi.sensitive.properties.key
> (https.port is default 8443)
>
> bin/nifi.sh start
>
> But, I can't even seem to access the most basic bit of the UI:
>
> curl -vvvk https://54.91.56.55:8443
> *   Trying 54.91.56.55:8443...
> * connect to 54.91.56.55 port 8443 failed: Connection refused
> * Failed to connect to 54.91.56.55 port 8443 after 17 ms: Connection
> refused
> * Closing connection 0
> curl: (7) Failed to connect to 54.91.56.55 port 8443 after 17 ms:
> Connection refused
>
> I have no doubt, I'm doing something astonishingly dumb.  Would someone be
> kind enough to point it out?
>
> thx,
>
> mew
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

 > > > >>>>>>
> > > > >>> > > > > >>>>>> Regards,
> > > > >>> > > > > >>>>>> David Handermann
> > > > >>> > > > > >>>>>>
> > > > >>> > > > > >>>>>> [1]
> > https://github.com/apache/cassandra-java-driver/
> > > > >>> > > > > >>>>>>
> > > > >>> > > > > >>>>>> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen <
> > > > >>> > > > > >> mikerthom...@gmail.com
> > > > >>> > > > > >>>>
> > > > >>> > > > > >>>>>> wrote:
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> Matt,
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> I got that. My point was that the Java changes
> > > appear to
> > > > >>> be a
> > > > >>> > > one
> > > > >>> > > > > >>> time
> > > > >>> > > > > >>>>>>> thing that DataStax did to make a better driver
> > with
> > > a
> > > > >>> much
> > > > >>> > > more
> > > > >>> > > > > >>>>>>> future-proof API. Since Scylla tracks them as
> > > closely as
> > > > >>> > > > > >> possible, I
> > > > >>> > > > > >>>>>>> suspect that we don't need to plan for a bunch of
> > > > >>> abstraction
> > > > >>> > > to
> > > > >>> > > > > >>> isolate
> > > > >>> > > > > >>>>>>> Java changes.
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>> On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> > > > >>> > > > > >>>>>> steven.mati...@gmail.com>
> > > > >>> > > > > >>>>>>> wrote:
> > > > >>> > > > > >>>>>>>
> > > > >>> > > > > >>>>>>>> That was kinda where i got stuck and fell out on
> > my
> > > > >>> > > branch/jira.
> > > > >>> > > > > >>>>>> Mike and
> > > > >>> > > > > >>>>>>>> I wanted to make a new controller service ,
> > without
> > > > >>> backward
> > > > >>> > > > > >>>>>> compatibility;
> > > > >>> > > > > >>>>>>>> and remove the duplicate driver/connection
> > > properties
> > > > >>> found
> > > > >>> > in
> > > > >>> > > > > >>> some
> > > > >>> > > > > >>>>>> of the
> > > > >>> > > > > >>>>>>>> processors.
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>>> I agree taking out all old stuff and making new
> > > > >>> controller
> > > > >>> > > > > >> service
> > > > >>> > > > > >>>>>> makes
> > > > >>> > > > > >>>>>>>> most sense.  4.x and 5.x should be mostly
> > backwards
> > > > >>> > compatible
> > > > >>> > > > > >> to
> > > > >>> > > > > >>>>>> 2&3.x
> > > > >>> > > > > >>>>>>>> with how it’s used within current processors.
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > > >>>>>>>>
> > > > >>> > > > 

Re: Nifi PutSFTP Processor Dot-Rename Error

[Joe Witt]

Hello

The image didnt come through

Please share nifi version and all settings for the component.

Thanks

On Wed, Mar 20, 2024 at 6:54 PM Prasad Patwardhan
 wrote:

> Dear Nifi Developer Support Team,
>
>
> I am writing to request assistance regarding an error encountered in the
> PutSFTP processor of our Apache NiFi data flow. Below are the details of
> the error:
>
>
> Error:
> ERROR [Timer-Driven Process Thread-223]
> o.a.nifi.processors.standard.PutSFTP
> PutSFTP[id=0a8613a0-018e-1000-ec12-8485106ac6fe] Unable to transfer
> StandardFlowFileRecord[uuid=f07ce855-1a60-45bb-845b-c56f77b8965d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1710445580505-1, container=default,
> section=1], offset=98313,
> length=990],offset=0,name=20240209-203419_II0224SE45_894298_822585_313.tar.gz,size=990]
> to remote host nfs1
>
> due to org.apache.nifi.processor.exception.ProcessException: IOException
> thrown
>
> from PutSFTP[id=0a8613a0-018e-1000-ec12-8485106ac6fe]:
> java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure: java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure; routing to failure
>
> java.io.IOException: Failed to rename dot-file to
> /remote-path/20240209-203419_II0224SE45_894298_822585_313.tar.gz due to 4:
> Failure
>
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:787)
>
> at
> org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:134)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2692)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2660)
>
> at
> org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:126)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
>
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> Caused by: net.schmizz.sshj.sftp.SFTPException: Failure
>
> at net.schmizz.sshj.sftp.Response.error(Response.java:140)
>
> at net.schmizz.sshj.sftp.Response.ensureStatusIs(Response.java:133)
>
> at
> net.schmizz.sshj.sftp.Response.ensureStatusPacketIsOK(Response.java:125)
>
> at net.schmizz.sshj.sftp.SFTPEngine.rename(SFTPEngine.java:250)
>
> at net.schmizz.sshj.sftp.SFTPClient.rename(SFTPClient.java:124)
>
> at net.schmizz.sshj.sftp.SFTPClient.rename(SFTPClient.java:119)
>
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:783)
>
> ... 15 common frames omitted
>
>
> This error occurs when attempting to transfer a file to the remote host
> server that we have on running on Centos7 server called nfs1, specifically
> during the renaming process with the .LCK extension.
>
> In the Nifi PutSFTP processor we have the following settings marked as the
> following to rename the files to .LCK files prior to the completion of the
> transfer which will be renamed on transfer completion.
>
>
> The file does successfully transfer to folder that we have created on our
> Centos7 server however the error shown above pops up in the Nifi UI and the
> Nifi logs.
>
>
> [image: Screenshot 2024-03-20 at 1.36.19 PM.png]
>
> In order to ensure that it was not a permissions/certificate issue, I have
> used the cert to sftp commands in my terminal to the host machine and
> create, delete, and rename files that are inside the target directory.
>
>
> Could you please investigate this issue and provide guidance on how to
> resolve it? If additional information or logs are required, please let me
> know, and I will be happy to provide them.
>
> Thank you for your assistance!
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

Matt

I dont know a ton about Cassandra but when I looked at client/driver notes
for 4+ it said it was compatible all the way back to 3.x.   Not sure what
that means but it surely seems worth exploring.  Also I dont know if the
4.x drivers get rid of the vulnerable bits.

Thanks

On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess  wrote:

> At the very least we should upgrade to Cassandra 3.11.6:
> https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
>
> On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess  wrote:
>
> > If the community agrees to get rid of Cassandra 3 that'll save me effort
> > on the refactor after I add Cassandra 4 :) Otherwise those
> > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> > would not be included in the convenience binary.
> >
> > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
> >
> >> Mike, Matt,
> >>
> >> Happy to hear you both have active efforts or are interested in doing
> so.
> >> Can you help me understand more specifically what that means for the
> >> current set of components?
> >>
> >> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> >> implies the current set of dependencies would remain too right?
> >>
> >> Is the current set of components we have ones we want to retain?  We
> >> certainly need Cassandra components - but are the ones we have now the
> >> right ones?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> >> wrote:
> >>
> >> > I'm actively working this, I pushed my branch up in case anyone wants
> to
> >> > take a look [1]. The idea is to abstract the Cassandra API "up a
> couple
> >> > levels" and provide implementations for Cassandra 3, 4, and eventually
> >> 5.
> >> > For JDBC-like interfaces this is a PITA because of the API (Statement,
> >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we
> >> can
> >> > find a common pattern for abstracting the third-party library
> >> > implementation and API from the NiFi component (Processor,
> >> > ControllerService, etc.) API. I think we're doing something similar
> for
> >> > Kafka?
> >> >
> >> > Regards,
> >> > Matt
> >> >
> >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> >> >
> >> >
> >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> >> > wrote:
> >> >
> >> > > That’s been on my todo list for a little while but things kept
> coming
> >> up.
> >> > > I think I could get started on that now. Based on my initial
> research
> >> it
> >> > > appears that scylla uses the exact same api as datastax so
> supporting
> >> > both
> >> > > in a cql bundle should theoretically be fairly easy.
> >> > >
> >> > >
> >> > > Sent from my iPhone
> >> > >
> >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> >> > > >
> >> > > > Team,
> >> > > >
> >> > > > Cassandra remains a really important system to be able to send
> data
> >> to.
> >> > > > However, it seems like we've not maintained these well.  We have
> >> what
> >> > > > appears to be at least a full generation behind on client versions
> >> (we
> >> > > are
> >> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> >> > > shortly).
> >> > > >
> >> > > > We have components to send data, query data, and use Cassandra as
> a
> >> > cache
> >> > > > store.  We have older mechanisms for json/avro and publish
> >> mechanisms
> >> > for
> >> > > > records.
> >> > > >
> >> > > > The libraries we do have depend on outdated versions of Guava and
> >> > result
> >> > > in
> >> > > > many CVE hits.
> >> > > >
> >> > > > I am inclined to think we should deprecate the 1.x components and
> >> > remove
> >> > > > them as-is from the 2.x line.  Then re-introduce them with record
> >> only
> >> > > > interfaces and built against the latest stable
> >> > > Cassandra/Datastax/ScyllaDB
> >> > > > interfaces.
> >> > > >
> >> > > > I'd love to hear thoughts from those closer to this space both as
> a
> >> > user
> >> > > > and developer so we can make good next steps.
> >> > > >
> >> > > > Thanks
> >> > >
> >> >
> >>
> >
>

Re: [discuss] What to do with the Cassandra components

[Joe Witt]

Mike, Matt,

Happy to hear you both have active efforts or are interested in doing so.
Can you help me understand more specifically what that means for the
current set of components?

The CVE hits are concerning and long standing.  Supporting Cassandra 3
implies the current set of dependencies would remain too right?

Is the current set of components we have ones we want to retain?  We
certainly need Cassandra components - but are the ones we have now the
right ones?

Thanks
Joe

On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess  wrote:

> I'm actively working this, I pushed my branch up in case anyone wants to
> take a look [1]. The idea is to abstract the Cassandra API "up a couple
> levels" and provide implementations for Cassandra 3, 4, and eventually 5.
> For JDBC-like interfaces this is a PITA because of the API (Statement,
> PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
> find a common pattern for abstracting the third-party library
> implementation and API from the NiFi component (Processor,
> ControllerService, etc.) API. I think we're doing something similar for
> Kafka?
>
> Regards,
> Matt
>
> [1] https://github.com/mattyb149/nifi/tree/cassy4
>
>
> On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> wrote:
>
> > That’s been on my todo list for a little while but things kept coming up.
> > I think I could get started on that now. Based on my initial research it
> > appears that scylla uses the exact same api as datastax so supporting
> both
> > in a cql bundle should theoretically be fairly easy.
> >
> >
> > Sent from my iPhone
> >
> > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> > >
> > > Team,
> > >
> > > Cassandra remains a really important system to be able to send data to.
> > > However, it seems like we've not maintained these well.  We have what
> > > appears to be at least a full generation behind on client versions (we
> > are
> > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > shortly).
> > >
> > > We have components to send data, query data, and use Cassandra as a
> cache
> > > store.  We have older mechanisms for json/avro and publish mechanisms
> for
> > > records.
> > >
> > > The libraries we do have depend on outdated versions of Guava and
> result
> > in
> > > many CVE hits.
> > >
> > > I am inclined to think we should deprecate the 1.x components and
> remove
> > > them as-is from the 2.x line.  Then re-introduce them with record only
> > > interfaces and built against the latest stable
> > Cassandra/Datastax/ScyllaDB
> > > interfaces.
> > >
> > > I'd love to hear thoughts from those closer to this space both as a
> user
> > > and developer so we can make good next steps.
> > >
> > > Thanks
> >
>

[discuss] What to do with the Cassandra components

[Joe Witt]

Team,

Cassandra remains a really important system to be able to send data to.
However, it seems like we've not maintained these well.  We have what
appears to be at least a full generation behind on client versions (we are
on 3x vs 4x which is the latest stable with 5x apparently coming shortly).

We have components to send data, query data, and use Cassandra as a cache
store.  We have older mechanisms for json/avro and publish mechanisms for
records.

The libraries we do have depend on outdated versions of Guava and result in
many CVE hits.

I am inclined to think we should deprecate the 1.x components and remove
them as-is from the 2.x line.  Then re-introduce them with record only
interfaces and built against the latest stable Cassandra/Datastax/ScyllaDB
interfaces.

I'd love to hear thoughts from those closer to this space both as a user
and developer so we can make good next steps.

Thanks

Re: website: PMC members also listed as committers?

[Joe Witt]

For anyone interested in improving the logic here is where the page lives:

https://github.com/apache/nifi-site/blob/main/content/community/_index.md

It utilizes this directive/entry such as
{{< project-members project="nifi" >}}

That is powered by a short-code in hugo terms (I'm told) which can be found
here
https://github.com/apache/nifi-site/blob/main/themes/nifi/layouts/shortcodes/project-members.html

That short-code leverages information which is found in
https://github.com/apache/nifi-site/blob/main/config.toml
on lines 31 and 32.

Perhaps the solution is to improve the logic of the short-code such that it
allows a second argument such as 'exclude-project' which might look like
{{< project-members project="nifi" exclude-project="nifi-pmc">}}

Then update the short code such that if the second argument is provided we
remove names in the 'nifi' list that are also in the 'nifi-pmc' list.

Not sure what mechanisms/etc.. are available but this should give anyone
motivated good context to work with.

Thanks

On Wed, Feb 21, 2024 at 1:18 PM Joe Witt  wrote:

> Marton
>
> I wasn't sure why so I checked the source [1] and found a very nice
> improvement which is the page is now dynamically generated reflecting our
> actual membership/roster in ASF records.  So people don't need to mess with
> this anymore which is great.
>
> It is duplicative between PMC and Committership listing and ideally we
> only show the delta since being a PMC member implies committership.  I'm
> sure someone could improve that logic to only list the committers.
>
> [1]
> https://github.com/apache/nifi-site/blob/main/content/community/_index.md
>
> On Wed, Feb 21, 2024 at 10:17 AM Marton Szasz  wrote:
>
>> Hi all,
>>
>> I just noticed that on the people page of the nifi website, PMC members
>> are also listed under committers. This seems to be a recent change,
>> maybe since the redesign?
>>
>> Old website without name duplication:
>>
>> https://web.archive.org/web/20231208050808/https://nifi.apache.org/people.html
>> New website with name duplication: https://nifi.apache.org/community/
>>
>> Is this change intentional? If so, why? I find it a bit confusing.
>>
>> Thanks,
>> Marton
>>
>>

Re: website: PMC members also listed as committers?

[Joe Witt]

Marton

I wasn't sure why so I checked the source [1] and found a very nice
improvement which is the page is now dynamically generated reflecting our
actual membership/roster in ASF records.  So people don't need to mess with
this anymore which is great.

It is duplicative between PMC and Committership listing and ideally we only
show the delta since being a PMC member implies committership.  I'm sure
someone could improve that logic to only list the committers.

[1]
https://github.com/apache/nifi-site/blob/main/content/community/_index.md

On Wed, Feb 21, 2024 at 10:17 AM Marton Szasz  wrote:

> Hi all,
>
> I just noticed that on the people page of the nifi website, PMC members
> are also listed under committers. This seems to be a recent change,
> maybe since the redesign?
>
> Old website without name duplication:
>
> https://web.archive.org/web/20231208050808/https://nifi.apache.org/people.html
> New website with name duplication: https://nifi.apache.org/community/
>
> Is this change intentional? If so, why? I find it a bit confusing.
>
> Thanks,
> Marton
>
>

Re: [DISCUSS] New Project Repository for Python Extensions?

[Joe Witt]

David

Good and important thread for NiFi.  Some quick thoughts on this more
generally.  Perhaps we ...

0. Have a repo for the 'nifi-api'.  Any true extension should be against a
specific version of the nifi api only.  This should change far less
frequently than the other bits.  Probably should review that this is true
from history.  We probably also have to include things that are
'effectively' part of the true API we need to honor which includes
expression language, records, and hopefully relatively few other things.
1. Have the mono repo like we do now for the nifi framework, nifi
application, minifi, registry and such.  And notably the nifi-assembly
remains here.
2. Have a repo for 'nifi-java-extensions' and we move everything there
which is meant as an extension component against the nifi-api specifically
(not the framework things like provenance/etc. - frankly anything that is
related to the 'nifi-framework-api' is NOT a true extension and doesn't
belong in this 'nifi-java-extension' repo)
3. Have a repo for 'nifi-python-extensions'

We can release 0 whenever we need to improve the nifi-api and then 1-3 at
the same time like we do today.  Or we can when needed release them
independently like fixing specific bugs or vulnerabilities/etc..

Worth really digging in and understanding what this could/should look like
to give ourselves cleaner air going forward.

Thanks
Joe

On Thu, Feb 1, 2024 at 1:03 PM Gábor Gyimesi  wrote:

> Hi David,
>
> MiNiFi C++ had had a Python API before, using Python's stable C API, but
> the processors had a different, simpler format like this following example:
>
> https://github.com/apache/nifi-minifi-cpp/blob/main/extensions/python/pythonprocessors/examples/GaussianDistributionWithNumpy.py
>
> Our goal was to be able to support NiFi's Python API, with the ease of only
> copying the Python processor file to MiNiFi C++'s configured python
> processor path, and use them in the flow config the same way as the
> original MiNiFi C++ style python processors.
>
> There is already an open PR for supporting this for NiFi's
> FlowFileTransform processor types (as of now MiNiFi C++ does not support
> record based flow file processing):
> https://github.com/apache/nifi-minifi-cpp/pull/1712 and also an open PR
> for
> supporting virtual environments:
> https://github.com/apache/nifi-minifi-cpp/pull/1721 as previously MiNiFi
> C++ only supported system installed Python packages. The implementation
> uses the same C API bindings as before, importing NiFi's nifiapi adapted to
> MiNiFi C++'s python API to be able to use NiFi's Python processors.
>
> There are still a few limitations due to the differences between NiFi and
> MiNiFi C++ implementations which are listed here:
>
> https://github.com/apache/nifi-minifi-cpp/blob/d27430260c8c35dac52011bdb31b22b36e10539d/extensions/python/PYTHON.md
> Some of these limitations are being addressed by these jira tickets in this
> epic: https://issues.apache.org/jira/browse/MINIFICPP-2272
>
> I tested all the available Python processors (aside from RecordTransform
> processors) of NiFi 2.0.0-M2 and they seem to be working with MiNiFi C++
> with these PRs, so it looks promising.
>
> Regards,
> Gabor Gyimesi
>
>
>
> On Thu, 1 Feb 2024 at 19:03, David Handermann  >
> wrote:
>
> > Hi Gabor,
> >
> > Thanks for the reply.
> >
> > It is helpful to know about the progress of Python Processor support
> > in MiNiFi C++. Is the goal to support the same NiFi Python API as
> > implemented for NiFi itself?
> >
> > The goal of a separate repository for Python extensions would be to
> > keep it self-contained for testing and releasing. From that
> > perspective, it would have a dependency on a declared version of the
> > NiFi Python API, and would include automated build workflows for
> > testing.
> >
> > For the NiFi framework components, there would still need to be
> > internal components that support testing implementations of Python
> > APIs, but the Python Extensions repository would have its own
> > decoupled set of tests.
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Feb 1, 2024 at 11:05 AM Gábor Gyimesi 
> wrote:
> > >
> > > Hi David,
> > >
> > > Currently we are in the process of implementing support for the NiFi
> > python
> > > processors in MiNiFi C++. Probably in the next open source release this
> > > feature will be available, so the available NiFi Python processors will
> > be
> > > usable in MiNiFi C++ as well. I think this idea would help with the
> > > collaboration of supporting these processors in both Java and C++
> > projects.
> > > It would certainly make release verification easier to be able to
> > > concentrate only on the Python processors if they are released
> > separately.
> > >
> > > My concern is how would the automatic testing and verification would
> work
> > > in this scenario? Would all the testing of the Python processors be
> moved
> > > to the new repository and would be tested there separately, with both
> > NiFi
> > > 

Re: UpdateAttribute Failure Relationship

[Joe Witt]

Lucas

The change proposed is specific to the original intent, implementation,
evolution, and bridging to new intent for certain cases of that specific
processor.

I remain supportive of processors using the capabiltities of the api and
framework to best meet the user needs.  “failure” is just a string. The
meaning of that depends on a processor and what it does.

Thanks

On Fri, Feb 9, 2024 at 5:04 PM Lucas Ottersbach 
wrote:

> I think that's a good approach which actually addresses the underlying
> issue. Thank you Joe, Mark and all others.
>
> As far as I know the default last resort behaviour of rollback + yield,
> that a lot processors exhibit, is due to them being based on
> AbstractProcessor.
>
> Does it make sense to incorporate the outlined approach into
> AbstractProcessor instead of UpdateAttribute?
> This way, other processors can undergo the same (opt-in) behavioural change
> without having to re-implement it on a per processor basis.
> Every processor in question only would need to add the property declared by
> AbstractProcessor to its list of properties. Processors that do not include
> the property wouldn't support to configure the behaviour and thus default
> to the current behaviour instead.
>
> I think that would provide a good middle ground, both for users that want a
> explicit failure relationship and for those that rather want a simpler flow
> and ensure those kinds of errors won't happen another way.
>
> What do you think?
>
> Lucas
>
> Joe Witt  schrieb am Sa., 10. Feb. 2024, 00:06:
>
> > Lots of good commentary and great focus on minimizing impact to the users
> > while fixing what is admittedly not the most desired behavior for some
> > cases as it relates to the very popular UpdateAttribute.
> >
> > We do not want to enforce that all processors have failure relationships.
> > Presumably that notion is specific to processors which take an input
> > flowfile.  Even then though we have many examples where having a failure
> > relationship does not add value.  A few examples such as DistributeLoad,
> > DuplicateFlowFile and others which have concepts such as 'unmatched'
> etc..
> > Certainly most things can and should but we dont need a strict policy
> > here.  We should still let people building processors be thoughtful about
> > what is best.
> >
> > UpdateAttribute specifically... There is history to why it works the way
> it
> > does.  Things changed and it didn't evolve or couldn't because its use
> was
> > so widespread and we didnt want to create too much pain for users.  But
> > because of 2.0 and some improvements like the ability to code up
> migration
> > behaviors on a per extension basis we can work our way out of this
> without
> > causing pain for the users.
> >
> > For NiFi 1.x it should stay as is.
> > For NiFi 2.x a solution is outlined in NIFI-6344.  It reads:
> > Given the new capabilities for migrating configs in NiFi 2.0 we can fix
> > this.
> >
> > Add a property to UpdateAttribute that is 'Failure Strategy' and the
> > options are 'rollback' or 'route to failure'. If that property is set
> with
> > rollback it behaves like it does now and I recommend that remain the
> > default. If that property is set to 'route to failure' then we add a
> > relationship which needs to be set which is of course called 'failure'.
> For
> > flows being migrated from a version before this behavior was available
> to a
> > version that has this capability we just set the value of this parameter
> to
> > our default.
> >
> > This lets existing flows migrate over just fine. It lets us give users a
> > failure path for the cases they want one. It lets us keep the vast
> majority
> > of flows and uses of this where failure is not relevant stay clean. And
> it
> > handles migration.
> >
> > The processor needs to be updated to catch the exceptions and then follow
> > this logic. Today it just lets it fly to the framework which causes the
> > processor to yield and penalizes the flowfile for the default time. When
> > now catching the problem we should just avoid yielding and instead
> penalize
> > the specific offending flowfile which lets everything else operate super
> > fast.
> >
> > Thanks to Mark Payne for the chat on this.
> >
> > This can be done at any time by anyone that wants to take it on.  It is
> not
> > a blocker for nifi 2.x.  The migration capabilities give us really nice
> > options for many cases we've hit over the years going forward.
> >
> > Than

Re: UpdateAttribute Failure Relationship

[Joe Witt]

Lots of good commentary and great focus on minimizing impact to the users
while fixing what is admittedly not the most desired behavior for some
cases as it relates to the very popular UpdateAttribute.

We do not want to enforce that all processors have failure relationships.
Presumably that notion is specific to processors which take an input
flowfile.  Even then though we have many examples where having a failure
relationship does not add value.  A few examples such as DistributeLoad,
DuplicateFlowFile and others which have concepts such as 'unmatched' etc..
Certainly most things can and should but we dont need a strict policy
here.  We should still let people building processors be thoughtful about
what is best.

UpdateAttribute specifically... There is history to why it works the way it
does.  Things changed and it didn't evolve or couldn't because its use was
so widespread and we didnt want to create too much pain for users.  But
because of 2.0 and some improvements like the ability to code up migration
behaviors on a per extension basis we can work our way out of this without
causing pain for the users.

For NiFi 1.x it should stay as is.
For NiFi 2.x a solution is outlined in NIFI-6344.  It reads:
Given the new capabilities for migrating configs in NiFi 2.0 we can fix
this.

Add a property to UpdateAttribute that is 'Failure Strategy' and the
options are 'rollback' or 'route to failure'. If that property is set with
rollback it behaves like it does now and I recommend that remain the
default. If that property is set to 'route to failure' then we add a
relationship which needs to be set which is of course called 'failure'. For
flows being migrated from a version before this behavior was available to a
version that has this capability we just set the value of this parameter to
our default.

This lets existing flows migrate over just fine. It lets us give users a
failure path for the cases they want one. It lets us keep the vast majority
of flows and uses of this where failure is not relevant stay clean. And it
handles migration.

The processor needs to be updated to catch the exceptions and then follow
this logic. Today it just lets it fly to the framework which causes the
processor to yield and penalizes the flowfile for the default time. When
now catching the problem we should just avoid yielding and instead penalize
the specific offending flowfile which lets everything else operate super
fast.

Thanks to Mark Payne for the chat on this.

This can be done at any time by anyone that wants to take it on.  It is not
a blocker for nifi 2.x.  The migration capabilities give us really nice
options for many cases we've hit over the years going forward.

Thanks


On Fri, Feb 9, 2024 at 2:53 PM Adam Taft  wrote:

> I mean, this really speaks to the principal that (in my humble opinion) All
> Processors Shalt Have Failure Relationships as best practice.
>
> So I think the decision is really, how much pain do we want to impose on
> NiFi 2.0 adopters. UpdateAttribute and RouteOnAttribute are used literally
> everywhere (especially on large installations), and it would be quite the
> chore(!) to update these if we were to make a non-compatible change for
> 2.0.  But 2.0 is also a very logical time/place to make such backwards
> breaking changes to UpdateAttribute, RouteOnAttribute and/or other
> processors that are missing failure relationships.
>
> I just fundamentally have never liked the lack of control for flowfiles
> getting requeued if a processor exception is left uncaught. You are almost
> always just going to repeat the failure condition, it's rarely useful to
> retry without some sort of deliberate flow manager action. The enhancements
> to relationships (to enable retries, backoffs, etc.) even reinforces my
> point of view that requeuing is almost always wrong.
>
> So I think the decision here is how much pain is this going to cause for
> NiFi 2.0 adoption? And if it's too much pain, then we should do something
> in a backwards compatible kind of way (like offering a dynamic failure
> relationship for those target processors). And if it's not too much pain,
> just add failure relationships directly to the processors and rip the
> bandaid off.
>
> A solution like a "try/catch" option to the expression language is not that
> exciting to me. I do believe that adoption will be hindered if we make a
> breaking change because of the high usage of Update and Route processors.
> And so 'm concluding that a dynamic relationship approach, leaving the EL
> as designed, is the best path forward. But I'm hoping there's even a more
> clever solution out there waiting for someone to bring forward.
>
> I do believe whatever we do should be a 2.0 thing. Let's not consider any
> changes on this for 1.x (unless it somehow enables better adoption).
>
> /Adam
>
> On Fri, Feb 9, 2024 at 1:27 PM u...@moosheimer.com 
> wrote:
>
> > Yes, that certainly involves a lot of effort.
> > I wonder whether it's a good idea to fix a p

Re: NiFi 1.25.0 - Bootstrap Notification Services are deprecated - no replacement?

[Joe Witt]

Martin

Correct. They are deprecated in the 1.x line (still usable) and are removed
in the 2.x line.  With more and more deployments happening in things like
K8S environments there are far better ways of offering these which reduce
the burden on NiFi itself and the dependency management it represents.
Similarly in traditional linux style service installs there are other
options that monitor services better that can be used as well.  We're
trying to strike a better balance than we have in the past of
building/providing all the things ourselves vs helping/integrating with
common/popular services as we go forward.

Thanks

On Thu, Feb 1, 2024 at 11:05 AM Martin Fong  wrote:

> We get the following WARN from the deprecated log and soon will be gone.
> So this feature will be gone and no replacement?
>
> 2024-02-01 11:37:44,048 WARN [main]
> deprecation.org.apache.nifi.bootstrap.RunNiFi Bootstrap Notification
> Services are deprecated [email-notification]
> org.apache.nifi.deprecation.log.DeprecationException: Reference Class
> [org.apache.nifi.bootstrap.RunNiFi] ClassLoader
> [jdk.internal.loader.ClassLoaders$AppClassLoader@55054057]
> at
> org.apache.nifi.deprecation.log.StandardDeprecationLogger.getExtendedArguments(StandardDeprecationLogger.java:63)
> at
> org.apache.nifi.deprecation.log.StandardDeprecationLogger.warn(StandardDeprecationLogger.java:54)
> at
> org.apache.nifi.bootstrap.RunNiFi.registerNotificationServices(RunNiFi.java:422)
> at org.apache.nifi.bootstrap.RunNiFi.loadServices(RunNiFi.java:410)
> at org.apache.nifi.bootstrap.RunNiFi.(RunNiFi.java:177)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:294)
>
> We have followed this in the past and still using it:
> https://pierrevillard.com/2017/05/11/monitoring-nifi-bootstrap-notifier/
>
> Please advise,
> Martin Fong
> Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
> Technology Services Division, Technology Infrastructure Services
> City of Toronto
> 703 Don Mills Road, 2nd Floor
> Toronto, ON
> M3C 3N3
> Tel:   416-397-7565
> e-mail: martin.f...@toronto.ca
>
> This e-mail message is confidential and subject to copyright. Any
> unauthorized use or disclosure is prohibited. If you have received this
> email and are not the intended recipient, please advise and delete it.
> Thank you.
>
>

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)

[Joe Witt]

+1 binding

Thanks again David

On Fri, Jan 26, 2024 at 1:46 PM Bryan Bende  wrote:

> +1 binding
>
> Ran through release helper
> Ran simple flow
> Saved flows to registry, verified DB persistence is fixed
>
> Thanks David!
>
> On Fri, Jan 26, 2024 at 1:33 PM Robert Fellows 
> wrote:
> >
> > +1 non-binding
> >
> > verified hashes
> > build with java 21
> > ran simple flow
> > built with new UI
> >
> >
> > On Fri, Jan 26, 2024 at 12:11 PM Nissim Shiman  >
> > wrote:
> >
> > >  +1 (non-binding)
> > >
> > > verified source release sha256/512 checksums
> > >
> > > successfully ran build using:
> > > Apache Maven 3.9.6
> > > Java 21 2023-09-19 LTS
> > > linux kernel 3.10.0-1160
> > >
> > > Ran various simple flows successfully.  Migrated registry from 1.24 and
> > > 2.0.0-M1 to this RC successfully.  (i.e. and tested importing PGs,
> > > committing new versions of PGs to registry successfully, both from
> nifi and
> > > filesystem).  Verified flows were persisted in registry after registry
> > > restart as well.
> > >
> > > Noticed non-blocking issue where reporting tasks' link to referenced
> > > controller service is not completely working.  Created NIFI-12678 for
> it.
> > >
> > > Tested:
> > > NIFI-11389 Controller Services's link to referencing Controller
> Services
> > > components not always working
> > > NIFI-12387 Flow Configuration History can record a Comments change for
> > > Controller Service when no changes have been made
> > >
> > > NIFI-12394 when importing versioned flow with component that migrates
> > > properties, controller service reference is invalid
> > > NIFI-12666 Correct NiFi Registry Data Source Configuration
> > >
> > > Thank you David and team for the upcoming release!
> > >
> > > Nissim Shiman
> > >
> > > On Friday, January 26, 2024 at 02:30:45 AM UTC, David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > >  Team,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi 2.0.0-M2.
> > >
> > > Please review the following guide for how to verify a release candidate
> > > build:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > >
> > > The source being voted on and the convenience binaries are available
> > > on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
> > >
> > > The build artifacts are available on the Apache Nexus Repository:
> > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1264
> > >
> > > Git Tag: nifi-2.0.0-M2-RC4
> > > Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> > > GitHub Commit Link:
> > >
> > >
> https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> > >
> > > Checksums of nifi-2.0.0-M2-source-release.zip
> > >
> > > SHA256:
> 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b
> > > SHA512:
> > >
> e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799
> > >
> > > Release artifacts are signed with the following key:
> > >
> > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > >
> > > KEYS file is available on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > Issues resolved for this version: 214
> > >
> > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861
> > >
> > > Release note highlights can be found on the project wiki:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
> > >
> > > The vote will be open for 72 hours.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> > > Then please vote:
> > >
> > > [] +1 Release this package as nifi-2.0.0-M2
> > > [] +0 no opinion
> > > [] -1 Do not release this package because...
> > >
> >
> >
> >
> > --
> > ---
> > Rob Fellows
>

Re: [VOTE] Release Apache NiFi 1.25.0 (RC1)

[Joe Witt]

+1 binding.

Full clean build with contrib check and the usual checks works out.  Ran
simple local flow and checks out as well.  Checked latest commit.

Thanks Pierre!

On Thu, Jan 25, 2024 at 11:53 AM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.25.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.25.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1263
>
> Git Tag: nifi-1.25.0-RC1
> Git Commit ID: 6ecc398d3f92425447e43242af4992757e25b3c5
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/6ecc398d3f92425447e43242af4992757e25b3c5
>
> Checksums of nifi-1.25.0-source-release.zip
>
> SHA256: 6a14b35bf5beb4ae3fcf76df8d09676e61c6bc309a97dc6785eae84b7cbaef78
> SHA512:
>
> 1ffb2586ce9591ce2c5aa39fdec43a89ffd29081a31d51dc95dd953cb7f94584d0a0171bb1ba7c9495f431aec3770d324dbabb319b01bb6fdce5a0a00426fffa
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 103
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353860
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.25.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.25.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC3)

[Joe Witt]

+1 binding

All the usual checks.  Build/Integration Tests/Contrib check all clean. Ran
local flow on osx.  Built/deployed in docker and was able to
reattach/continue from a flow four months old like nothing happened.  Very
slick.

Also built with the new UI included and that is coming along super nicely!

Thanks everyone and as always David for RC efforts!

On Thu, Jan 25, 2024 at 10:10 AM Gábor Gyimesi  wrote:

> +1 non-binding
>
> - Verified hashes and signatures
> - Successfully built NiFi with contrib-check in the following environment:
>   - Ubuntu 22.04 6.5.0-14-generic
>   - java 21 2023-09-19 LTS
> Java(TM) SE Runtime Environment (build 21+35-LTS-2513)
> Java HotSpot(TM) 64-Bit Server VM (build 21+35-LTS-2513, mixed mode,
> sharing)
>   - Apache Maven 3.6.3
> - Designed and ran a flow with S3 upload successfully
> - Built a containerized version of NiFi and tested site to site use cases
> and http communication between NiFi and MiNiFi C++
>
> Thanks for RMing David!
>
> Regards,
> Gabor
>
>
>
> On Thu, 25 Jan 2024 at 15:46, Robert Fellows 
> wrote:
>
> > +1 non-binding
> >
> > *verified hashes*
> >
> > *built with:*
> > openjdk version "21.0.1" 2023-10-17 LTS
> > OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
> > OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed
> mode,
> > sharing)
> >
> > General usage of the application looks good. Added a couple of the new
> > processors and services without issue.
> >
> >
> > On Thu, Jan 25, 2024 at 8:11 AM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Team,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi 2.0.0-M2.
> > >
> > > Please review the following guide for how to verify a release candidate
> > > build:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > >
> > > The source being voted on and the convenience binaries are available
> > > on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
> > >
> > > The build artifacts are available on the Apache Nexus Repository:
> > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1262
> > >
> > > Git Tag: nifi-2.0.0-M2-RC3
> > > Git Commit ID: 439ac5f596fe90d2591376caec1499ed86abfd6b
> > > GitHub Commit Link:
> > >
> > >
> >
> https://github.com/apache/nifi/commit/439ac5f596fe90d2591376caec1499ed86abfd6b
> > >
> > > Checksums of nifi-2.0.0-M2-source-release.zip
> > >
> > > SHA256:
> 0637647ba7b6b8f0743cf00d95dd3e99250f2737ae9583daee511477cad22847
> > > SHA512:
> > >
> >
> 56d6c196ec64884c56a65278f7dd32cc62f1595ff5b2ec37403dcd54b7757a9ac70aa75a50ee6f61237e37b486d32b1ebbf95a4e38a2ea5dc7f694c779d5c6c4
> > >
> > > Release artifacts are signed with the following key:
> > >
> > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > >
> > > KEYS file is available on the Apache Distribution Repository:
> > >
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > Issues resolved for this version: 207
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861
> > >
> > > Release note highlights can be found on the project wiki:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
> > >
> > > The vote will be open for 72 hours.
> > >
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> > > Then please vote:
> > >
> > > [] +1 Release this package as nifi-2.0.0-M2
> > > [] +0 no opinion
> > > [] -1 Do not release this package because...
> > >
> >
> >
> > --
> > ---
> > Rob Fellows
> >
>

Re: [CANCEL][VOTE] Release Apache NiFi 2.0.0-M2 (RC1)

[Joe Witt]

; could
> > > >> not be resolved: org.apache.nifi:nifi-py4j-bridge:jar:2.0.0-M2
> > (absent),
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 (absent):
> Could
> > not
> > > >> find artifact org.apache.nifi:nifi-py4j-bridge:jar:2.0.0-M2 in
> > central (
> > > >> https://repo.maven.apache.org/maven2) -> [Help 1]
> > > >> [ERROR] Failed to execute goal on project nifi-opentelemetry-nar:
> > Could
> > > >> not resolve dependencies for project
> > > >> org.apache.nifi:nifi-opentelemetry-nar:nar:2.0.0-M2: The following
> > > >> artifacts could not be resolved:
> > > >> org.apache.nifi:nifi-opentelemetry-processors:jar:2.0.0-M2 (absent),
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 (absent):
> Could
> > not
> > > >> find artifact
> > org.apache.nifi:nifi-opentelemetry-processors:jar:2.0.0-M2 in
> > > >> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> > > >> [ERROR] Failed to execute goal on project nifi-jolt-nar: Could not
> > > >> resolve dependencies for project
> > > >> org.apache.nifi:nifi-jolt-nar:nar:2.0.0-M2: The following artifacts
> > could
> > > >> not be resolved:
> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2
> > > >> (absent), org.apache.nifi:nifi-jolt-processors:jar:2.0.0-M2
> (absent):
> > > >> org.apache.nifi:nifi-standard-shared-nar:nar:2.0.0-M2 was not found
> in
> > > >> https://repo.maven.apache.org/maven2 during a previous attempt.
> This
> > > >> failure was cached in the local repository and resolution is not
> > > >> reattempted until the update interval of central has elapsed or
> > updates are
> > > >> forced -> [Help 1]
> > > >
> > > >
> > > > without an option to resume. Please advise. Thanks!
> > > >
> > > >
> > > >
> > > > On Mon, Jan 22, 2024 at 10:25 AM David Handermann <
> > > > exceptionfact...@apache.org> wrote:
> > > >
> > > >> RC1 for 2.0.0-M2 is cancelled.
> > > >>
> > > >> On Mon, Jan 22, 2024 at 9:23 AM David Handermann
> > > >>  wrote:
> > > >> >
> > > >> > Joe,
> > > >> >
> > > >> > Thanks for highlighting the build issue related to the optional
> > > >> > profile for the new UI. In the interest of maintaining consistency
> > in
> > > >> > the source release, I agree this issue is worth resolving.
> > > >> >
> > > >> > A new version of the json-path library was also released over the
> > > >> > weekend, resolving CVE-2023-51074. Although this does not appear
> to
> > > >> > have a direct impact on NiFi usage, it would be helpful to include
> > as
> > > >> > well.
> > > >> >
> > > >> > I am cancelling RC1 and I will follow up with an RC2 build after
> > > >> > getting the json-path 2.9.0 upgrade reviewed and merged.
> > > >> >
> > > >> > Regards,
> > > >> > David Handermann
> > > >> >
> > > >> > On Mon, Jan 22, 2024 at 9:15 AM Joe Witt 
> > wrote:
> > > >> > >
> > > >> > > -1
> > > >> > >
> > > >> > > David,
> > > >> > >
> > > >> > > First of all thanks for continuing to plow through the magic of
> > > >> release
> > > >> > > management.
> > > >> > >
> > > >> > > I noticed the static check failed in github and it was due to a
> > > >> snapshot
> > > >> > > reference. I then tried a local build of the RC and it failed
> for
> > me
> > > >> for
> > > >> > > the same reason.  The difference here is that I was wanting to
> > > >> include the
> > > >> > > new UI so it could also be evaluated in the RC.
> > > >> > >
> > > >> > > I think this is worth resolving and doing and RC2 for but if you
> > > >> disagree
> > > >> > > that is fair as well.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Sat, Jan 20, 2024 at 9:3

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC1)

[Joe Witt]

-1

David,

First of all thanks for continuing to plow through the magic of release
management.

I noticed the static check failed in github and it was due to a snapshot
reference. I then tried a local build of the RC and it failed for me for
the same reason.  The difference here is that I was wanting to include the
new UI so it could also be evaluated in the RC.

I think this is worth resolving and doing and RC2 for but if you disagree
that is fair as well.

Thanks

On Sat, Jan 20, 2024 at 9:36 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M2.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1254
>
> Git Tag: nifi-2.0.0-M2-RC1
> Git Commit ID: b462c7051d004be70fba34f2795bd5c682cd1124
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/b462c7051d004be70fba34f2795bd5c682cd1124
>
> Checksums of nifi-2.0.0-M2-source-release.zip
>
> SHA256: c42655212abacef6f901f2a43ad7f43d3172c631652c517d822a36321199d411
> SHA512:
> b1a4d6790f85719a46f321f64cfc436da459795f03315a2f578940e266aa98a3d4a6d50d5be83583e7ac7adbf4a10f6050be314acc62a27fd247d082100718f4
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 197
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M2
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [DISCUSS] Preparing for NiFi 2.0.0-M2

[Joe Witt]

Wonderful progress. Definitely time and an M2 seems like a good idea for
the reasons noted.

Happy to help take RM if you get squeezed on time.

Thanks

On Mon, Jan 15, 2024 at 12:46 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> We have had some great feedback and many improvements since the
> release of NiFi 2.0.0-M1 at the end of November. With over 160 [1]
> Jira issues already resolved for the next iteration, we are in a good
> position to prepare for another milestone release version.
>
> The main branch now includes significant framework dependency upgrades
> such as Spring 6 and Jetty 12, along with several new components and
> other bug fixes. Considering the significant number of changes to
> framework libraries already in place, it would be useful to release a
> second milestone version before a full general availability release.
>
> There is a pull request for NIFI-9458 [2] that includes impactful
> lower-level changes to date and time parsing and formatting, which is
> part of the NiFi 2.0 Release Goals [3]. Following the review and
> incorporation of these changes, I would be glad to handle Release
> Manager responsibilities for a NiFi 2.0.0-M2 version, ideally at some
> point this week.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12353861
> [2] https://github.com/apache/nifi/pull/8248
> [3]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
>

board report for Jan 2024

[Joe Witt]

Team,

Congrats again on all the great work.  Thanks to everyone involved.  Here
is our board report for Jan 2024.

## Description:
The mission of NiFi is the creation and maintenance of software related to
providing an easy to use, powerful, and reliable system to process and
distribute data.

Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
integrate with and leverage the command and control of NiFi. There are both
Java and C++ implementations.

Apache NiFi Registry is a centralized registry for key configuration items
including flow versions, assets, and extensions for Apache NiFi and Apache
MiNiFi.

Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
NiFi classloader isolation model.

Apache NiFi Flow Design System is a theme-able set of high quality UI
components and utilities for use across the various Apache NiFi web
applications in order to provide a more consistent user experience.

## Project Status:
Current project status: Ongoing. High.
Issues for the board: None.

## Membership Data:
Apache NiFi was founded 2015-07-14 (8 years ago)
There are currently 64 committers and 36 PMC members in this project.
The Committer-to-PMC ratio is roughly 8:5.

Community changes, past quarter:
- Csaba Bejan was added to the PMC on 2023-10-25
- No new committers.  Last addition was Timea Barna on 2023-08-02.

## Project Activity:
The community conducted its first major (semantic versioning) milestone 1
release of Apache NiFi 2.0.0 M1 in November of 2023.  The first new major
release since 2016!  This release is a total game changer for NiFi.  In
addition to the more than 900 JIRAs included which comprise of features, bug
fixes, and security related changes this release makes major steps forward.
The NiFi 2.0 means we are up to date with and based on Java 21 and the
latest
of many great projects like Apache Maven, Spring, Jetty, and many more.
Users
can now write legit NiFi components in Python and enjoy a first class
integrated experience in NiFi as if they were written in Java. We eliminated
many defunct code areas, improved and clarified APIs, eliminated poorly
maintained code, and much more.  We anticipate having a full blown 2.0
production release soon though we already know of several production users
of
the 2.0 M1 release now.

We also kept the momentum up with the NiFi 1.x line releasing, also in
November, Apache NiFi 1.24.0.  Again this comes with improvements, security
changes, and bug fixes.

We have also at long last finally launched a revamped website for
nifi.apache.org.

## Community Health:
Community health remains strong and growing.  Our mailing list activity
as measured by the dev list dropped by 17% but overall activity remained
busy.  The dev list drop likely relates to the Holidays and the activity
on Slack.  PR activity, reviews, and merges remain highly active with more
than 31 authors contributing code in this quarter.  We added another 150
or so net Slack users this quarter now sitting at 2980 in the general
channel. We continue to ensure discussions that should turn into JIRAs or
other documented mailing lists do get sent to those mechanisms. We continue
to see plenty of commercial activity around NiFi in the form of vendor
material, blogs, social media posts, etc..

Re: New Project Website Design Staged for Deployment

[Joe Witt]

Super appreciative David and James.  A good step forward to refresh and
retool and grow from.

Thanks!

On Wed, Jan 3, 2024 at 8:39 PM David Handermann 
wrote:

> Team,
>
> Thanks to background work from several designers, and significant
> recent effort from James Mingardi-Elliott [1], we have a refreshed
> project website design deployed to staging and ready for production.
>
> The combined set of changes can be reviewed in the following pull request:
>
> https://github.com/apache/nifi-site/pull/82
>
> The staging site provides a functional way to view the new design,
> based on the main-staging branch [2] of the nifi-site repository.
>
> https://nifi.staged.apache.org
>
> The new design includes a modernized home page and streamlined
> navigation, with a focus on making the most popular pages quickly
> accessible.
>
> There is room for improvement in areas like generated project
> documentation, but the primary goal of this new design is to provide a
> fresh foundation for future improvements.
>
> The pull request provides an opportunity for correcting any notable
> problems, with the goal of pushing the new version to production early
> next week.
>
> Regards,
> David Handermann
>
> [1] https://github.com/james-elliott
> [2] https://github.com/apache/nifi-site/tree/main-staging
>

Re: Error in the expression-language-guide page.

[Joe Witt]

Looks like Alex is planning to address it
https://issues.apache.org/jira/browse/NIFI-12535

Thanks


On Thu, Dec 21, 2023 at 8:40 AM Dan S  wrote:

> Can you please add a little more detail which examples you are referring to
> and what the errors are?
>
> On Wed, Dec 20, 2023 at 11:32 PM A S  wrote:
>
> > Hi,
> > There are errors on the page in Table 13  14. Also, the tables are put
> > with same name 'PadLeft'.
> >
> > Thanks,
> > Ankura
> >
>

Re: NiFi SSO Login Using OIDC

[Joe Witt]

Hello Lourdes

It is extremely hard to find someone to respond meaningfully to questions
related to a NiFi release that is so many years back.  There have been
thousands of JIRAs to fix/improve/change the way such components behave.
If you can please get updated to something more recent I think you'll find
getting help from the community easier.

Thanks

On Mon, Nov 6, 2023 at 6:58 AM Lourdes Ursela Carmen Kuskanto
 wrote:

> Dear NiFi Developer Team,
>
> I want to enable SSO login in NiFi version 1.4 in my Compute Engine in GCP
> using OIDC. I want to allowed all user from my Azure Active Directory group
> that i have made before to be able to login to NiFi using their account as
> registered. But i can't seem to make it work.
>
> I have followed this step :
> https://github.com/benkelly/NiFi-Authentication-with-Azure-Active-Directory-Setup-Guide
>
> This is my configuration:
>
> nifi.properties
>
> # Core Properties #
> nifi.flow.configuration.file=./conf/flow.xml.gz
> nifi.flow.configuration.archive.enabled=true
> nifi.flow.configuration.archive.dir=./conf/archive/
> nifi.flow.configuration.archive.max.time=30 days
> nifi.flow.configuration.archive.max.storage=500 MB
> nifi.flow.configuration.archive.max.count=
> nifi.flowcontroller.autoResumeState=true
> nifi.flowcontroller.graceful.shutdown.period=10 sec
> nifi.flowservice.writedelay.interval=500 ms
> nifi.administrative.yield.duration=30 sec
> # If a component has no work to do (is "bored"), how long should we wait
> before checking again for work?
> nifi.bored.yield.duration=10 millis
> nifi.queue.backpressure.count=1
> nifi.queue.backpressure.size=1 GB
>
> nifi.authorizer.configuration.file=./conf/authorizers.xml
>
> nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
> nifi.templates.directory=./conf/templates
> nifi.ui.banner.text=
> nifi.ui.autorefresh.interval=30 sec
> nifi.nar.library.directory=./lib
> nifi.nar.library.autoload.directory=./extensions
> nifi.nar.working.directory=./work/nar/
> nifi.documentation.working.directory=./work/docs/components
>
> 
> # State Management #
> 
> nifi.state.management.configuration.file=./conf/state-management.xml
> # The ID of the local state provider
> nifi.state.management.provider.local=local-provider
> # The ID of the cluster-wide state provider. This will be ignored if NiFi
> is not clustered but must be populat>
> nifi.state.management.provider.cluster=zk-provider
> # Specifies whether or not this instance of NiFi should run an embedded
> ZooKeeper server
> nifi.state.management.embedded.zookeeper.start=false
> # Properties file that provides the ZooKeeper properties to use if
> 
>
> nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
>
>
> # H2 Settings
> nifi.database.directory=./database_repository
> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>
> # FlowFile Repository
>
> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>
> nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
> nifi.flowfile.repository.directory=./flowfile_repository
> nifi.flowfile.repository.checkpoint.interval=20 secs
> nifi.flowfile.repository.always.sync=false
> nifi.flowfile.repository.encryption.key.provider.implementation=
> nifi.flowfile.repository.encryption.key.provider.location=
> nifi.flowfile.repository.encryption.key.provider.password=
> nifi.flowfile.repository.encryption.key.id=
> nifi.flowfile.repository.encryption.key=
> nifi.flowfile.repository.retain.orphaned.flowfiles=true
>
>
> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
> nifi.queue.swap.threshold=2
>
> # Content Repository
>
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=1 MB
> nifi.content.repository.directory.default=./content_repository
> nifi.content.repository.archive.max.retention.period=7 days
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=../nifi-content-viewer/
> nifi.content.repository.encryption.key.provider.implementation=
> nifi.content.repository.encryption.key.provider.location=
> nifi.content.repository.encryption.key.provider.password=
> nifi.content.repository.encryption.key.id=
> nifi.content.repository.encryption.key=
>
> # Provenance Repository Properties
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> nifi.provenance.repository.encryption.key.provider.implementation=
> nifi.provenance.repository.encryption.key.provider.location=
> nifi.provenance.repository.encryption.key.provider.password=
> nifi.provenance.repository.encryption.key.id=
> nifi.provenance.repository.encryption.key=
>
> # P

Re: Docker container for 2.0.0-M1

[Joe Witt]

Hello

Yeah pls do submit a PR.

Thanks

On Wed, Nov 29, 2023 at 9:34 PM Matthew Hawkins  wrote:

> Hi Joe,
>
> I have it running against Bellsoft's v21 alpine-musl base fine, with python
> enabled. Happy to submit the Dockerfile if you want. In a nutshell,
>
> apk add --no-cache --no-interactive --no-progress coreutils unzip curl jq
> xmlstarlet procps-ng python3 py3-pip py3-virtualenv ripgrep
>
> And other than obviously Alpine not Debian, Busybox wants adduser/addgroup
> not the util-linux variants. I moved the apk call to the scripts section so
> only one apk update is needed, and deleted the apt cache cleanup. I added
> the ripgrep package to grok logs.
>
> I initially missed the VOLUME line in the dockerfile. Unfortunately I can't
> seem to get the -v flag to work to use local folders instead of anonymous
> volumes. I appreciate it's probably done for performance but it would be
> nice if they had sensible names. Host is Win11 with docker desktop and
> wsl2. For brevity, I'm adding a -v
> c:\users\me\nifi\conf:/opt/nifi/nifi-current/conf format line for each of
> the exported volumes to the docker run invocation (wrote a small cmd to
> launch as I'm not typing that monstrosity out daily by hand)
> I know I can manually docker volume some nice named ones and alter the
> VOLUME line accordingly. Should a docker compose be used instead? Because
> this is all feeling hackish.
>
> KR,
>
> On Wed, 29 Nov 2023, 17:24 Joe Witt,  wrote:
>
> > Matt
> >
> > Alpine is still desirable but it was about availability of Alpine w Java
> 21
> > at that time.  I think Alpine also required some different command and
> such
> > in the dockerfile.
> >
> > As far as the repo volumes they can be mapped as you wish now.  It is
> how I
> > keep updating my own flow deployments I test with when I want to retain
> > state across deployment.  Are you not able to map them?
> >
> > Thanks
> >
> >
> >
> > On Tue, Nov 28, 2023 at 9:41 PM Matthew Hawkins 
> > wrote:
> >
> > > Two Q's regarding the docker container;
> > >
> > > 1. Why the Debian version instead of Alpine? Was it compatibility with
> > > external stuff? I can confirm 2.0.0-M1 works fine with the alpine
> version
> > > of liberica-jdk on at least basic flows, with python enabled. It'll
> save
> > > disk space. I love Debian but Alpine is king of containers!
> > >
> > > 2. If we shift the repository folders out of NIFI_HOME per the regular
> > best
> > > practice doco, then a persistent  docker volume could easily be used to
> > > house these, making upgrades painless. I guess the flow.json, keystores
> > and
> > > configs could move too.
> > >
> > > Kind regards,
> > > Matt
> > >
> >
>

Re: Docker container for 2.0.0-M1

[Joe Witt]

Matt

Alpine is still desirable but it was about availability of Alpine w Java 21
at that time.  I think Alpine also required some different command and such
in the dockerfile.

As far as the repo volumes they can be mapped as you wish now.  It is how I
keep updating my own flow deployments I test with when I want to retain
state across deployment.  Are you not able to map them?

Thanks



On Tue, Nov 28, 2023 at 9:41 PM Matthew Hawkins  wrote:

> Two Q's regarding the docker container;
>
> 1. Why the Debian version instead of Alpine? Was it compatibility with
> external stuff? I can confirm 2.0.0-M1 works fine with the alpine version
> of liberica-jdk on at least basic flows, with python enabled. It'll save
> disk space. I love Debian but Alpine is king of containers!
>
> 2. If we shift the repository folders out of NIFI_HOME per the regular best
> practice doco, then a persistent  docker volume could easily be used to
> house these, making upgrades painless. I guess the flow.json, keystores and
> configs could move too.
>
> Kind regards,
> Matt
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC5)

[Joe Witt]

+1 binding

All the usual build checks.  Fired up NiFi.  Amazing to see the 1.x line
only 1.1GB now!

Thanks Pierre for the great RM work on a tough release cycle and to all for
continued help with votes!

Thanks
Joe


On Thu, Nov 23, 2023 at 9:36 AM Gábor Gyimesi  wrote:

> +1 (non-binding)
>
> Went through the same verification steps as per the previous RC, and LGTM:
>
> - Verified binary hashes and signatures
> - Successfully built NiFi with contrib-check using:
>   - Apache Maven 3.6.3
>   - openjdk version "11.0.20.1" 2023-08-24
>   - Ubuntu 22.04 6.2.0-36-generic
> - Designed and ran some simple flows successfully
> - Tested minifi-c2 server heartbeats and flow update successfully with
> MiNiFi C++
> - Tested communication between MiNiFi C++ and NiFi using InvokeHTTP -
> ListenHTTP processors and using Site 2 Site functionality
> with RemoteProcessGroup
>
> Thanks Pierre!
>
> BR,
> Gabor
>
> On Thu, 23 Nov 2023 at 16:14, Pierre Villard 
> wrote:
>
> > Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.24.0.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are available
> on
> > the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1251
> >
> > Git Tag: nifi-1.24.0-RC5
> > Git Commit ID: 5241f434829ca46a26a475600ef7c00e1e271e37
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/5241f434829ca46a26a475600ef7c00e1e271e37
> >
> > Checksums of nifi-1.24.0-source-release.zip
> >
> > SHA256: 715eb61cdc017a5f7ba4d845ae962fdf83d389829db2a8948be14f99f2cc8272
> > SHA512:
> >
> >
> 574147002b905ef64447edec0c7308f4fc3a97b3c7f01edca05464b2b418bcd3922f956d093736014443eec88ceba36af81df37398c5fe23a1288235b79d7306
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/pvillard.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 285
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353443
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [] +1 Release this package as nifi-1.24.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC6)

[Joe Witt]

+1 binding

Checked all the things.  Fired it up.  Looks great.

Huge thanks to you David for continuing through this challenging release as
an RM.  And also to all who keep participating in this vote.  NiFi 2.0 is
super important to the future of NiFi and this milestone release is a great
step forward.

Thanks

On Thu, Nov 23, 2023 at 2:21 AM Gábor Gyimesi  wrote:

> +1 (non-binding)
>
> Went through the same verification process as per the previous RC, LGTM:
>
> - Verified binary hashes and signatures
> - Successfully built NiFi with contrib-check using:
>   - Apache Maven 3.6.3
>   - Java 21 2023-09-19 LTS
>   - Ubuntu 22.04 6.2.0-36-generic
> - Designed and ran some simple flows successfully
> - Tested minifi-c2 server successfully with MiNiFi C++ (using a draft of
> the change conforming the new header requirements
> https://issues.apache.org/jira/browse/MINIFICPP-2244)
> - Built and started docker container successfully from
> nifi-docker/dockerhub/Dockerfile
>
> Thanks David!
>
> Best Regards,
> Gabor
>
> On Thu, 23 Nov 2023 at 03:49, David Handermann <
> exceptionfact...@apache.org>
> wrote:
>
> > Team,
> >
> > Thank you for your patience and continued support in reviewing these
> > release candidates.
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi 2.0.0-M1.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are
> > available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1250
> >
> > Git Tag: nifi-2.0.0-M1-RC6
> > Git Commit ID: 49fa0d86746f544e294b4ba04b2795d426ba0271
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/49fa0d86746f544e294b4ba04b2795d426ba0271
> >
> > Checksums of nifi-2.0.0-M1-source-release.zip
> >
> > SHA256: 1a0271c35f585a8e49c5e578422a8e3910fbfc50069a7c875bc6cec5a056eb91
> > SHA512:
> >
> 99e824a0bce109bde84218b914487fd44ba344d5ae9d8fbed9982b41dc2ee7e57b6dd30550c17a4868d43afd900705c48ec48d5e9e63a1b4130c79858f64963c
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/exceptionfactory.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 959
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12339599
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> > Then please vote:
> >
> > [] +1 Release this package as nifi-2.0.0-M1
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC3)

[Joe Witt]

+1 binding

Full clean build with contrib check.  All integration tests.  All clear.
Sigs/hashes good.
Build and deploy docker and run containers with python and friends all
good.  Though it is 1.7GB so we'll want to improve.
Build and run the convenience binary and all checks out.   Only 870MB now!
So awesome.
Run various flows/processors including custom python processors. All good.

This release is packed and a great step toward a full blown 2.0 release.

Thanks every and thanks David!

Joe

On Sat, Nov 18, 2023 at 5:45 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M1.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1245
>
> Git Tag: nifi-2.0.0-M1-RC3
> Git Commit ID: 5f7e637082d3ab2c45ce3d10e3bc31344e7581da
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/5f7e637082d3ab2c45ce3d10e3bc31344e7581da
>
> Checksums of nifi-2.0.0-M1-source-release.zip
>
> SHA256: d620aec234e14aaf0ca1da58b3c465b23e7a97a7ab4af3e6eb199f56509aaf39
> SHA512:
> 32713d4039104ae163201638933e6d7d35a681611c40353fa1707514d0bf3ea6af6627f54d2b4a4263933242293abe20755803cd5b4c0d1de655ccc978bd98b1
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 945
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12339599
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M1
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)

[Joe Witt]

+1 binding

Thanks everyone.  Thanks Pierre!

On Fri, Nov 17, 2023 at 1:39 PM Nissim Shiman 
wrote:

>  +1 (non-binding)
>
> verified source release sha256/512 checksums
>
> successfully ran build using:
> Apache Maven 3.9.5
> Java openjdk version: 1.8.0_382
> linux kernel 3.10.0-1160
>
> Ran various simple flows successfully.  Migrated registry from 1.23.2 to
> this RC successfully.  (i.e. and tested importing PGs, committing new
> versions of PGs to registry successfully.)
>
> Tested/verified:
> NIFI-11782 NPE when adding Snippet with label into Process Group
> NIFI-12084 Logging by process group is not workin on 1.x support branch
> (thank you to all those who worked on this and its parent ticket,
> NIFI-3065.  This new granular level of logging is really nice)
>
> While testing, noticed/filed non holdup issues:
> NIFI-12387 Flow Configuration History can record a Comments change for
> Controller Service when no changes have been made
> NIFI-12388 Process Group log file suffix can become out of sync with its
> name when PG was copied from a version controlled PG
>
> Thank you Pierre and team for the upcoming release!
>
> Nissim Shiman
>
>
> On Friday, November 17, 2023 at 08:10:20 PM UTC, Dan S <
> dsti...@gmail.com> wrote:
>
>  +1 (non-binding)
>
>   - Went through the Release Candidate Verification
>   <
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
>   - Verified signatures and hashes
>   - Built on CentOS Linux 7
>   - Java version Oracle 17.0.7
>   - Maven version  3.9.5
>   - Executed simple flow to exercise NIFI-11197
>   
>   - Confirmed fix for NIFI-12165
>   
>   - Confirmed fix for NIFI-11959
>   
>
> Thanks for RM'ing Pierre!
>
> On Fri, Nov 17, 2023 at 1:01 PM Marton Szasz  wrote:
>
> > +1 (binding)
> >
> > Compiled and ran with Java 17 on Ubuntu 22.04, executing a simple flow.
> >
> > On Fri, Nov 17, 2023 at 6:59 PM Matt Burgess 
> wrote:
> > >
> > > +1 (binding)
> > >
> > > Ran through release helper, ran NiFi standalone with Java 8, tested
> > > various flows and components.  Thanks for RM'ing Pierre!
> > >
> > > On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard
> > >  wrote:
> > > >
> > > > Team,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > > 1.24.0.
> > > >
> > > > Please review the following guide for how to verify a release
> candidate
> > > > build:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > >
> > > > The source being voted on the and the convenience binaries are
> > available on
> > > > the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
> > > >
> > > > The build artifacts are available on the Apache Nexus Repository:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachenifi-1241
> > > >
> > > > Git Tag: nifi-1.24.0-RC3
> > > > Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > > GitHub Commit Link:
> > > >
> >
> https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91
> > > >
> > > > Checksums of nifi-1.24.0-source-release.zip
> > > >
> > > > SHA256:
> > 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a
> > > > SHA512:
> > > >
> >
> 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8
> > > >
> > > > Release artifacts are signed with the following key:
> > > >
> > > > https://people.apache.org/keys/committer/pvillard.asc
> > > >
> > > > KEYS file is available on the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > Issues resolved for this version: 280
> > > >
> > > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353443
> > > >
> > > > Release note highlights can be found on the project wiki:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
> > > >
> > > > The vote will be open for 72 hours.
> > > >
> > > > Please download the release candidate and evaluate the necessary
> items
> > > > including checking hashes, signatures, build from source, and test.
> > Then
> > > > please vote:
> > > >
> > > > [] +1 Release this package as nifi-1.24.0
> > > > [] +0 no opinion
> > > > [] -1 Do not release this package because...
> >
>

Re: Removing JRuby?

[Joe Witt]

Team,

Coming back to this as I've been reviewing all the great output of OWASP
that we're getting daily now (thanks David) as well as some findings from a
nice vendor tool by JFrog.  And what is popping up left and right are
issues in terms of vulnerabilities tied to Jython-standalone.  It is very
clear this component needs to go and is not actively maintained.  It is
also as noted tied to Python 2 which is eol.  I'll file a JIRA to deprecate
in 1.x and remove in 2.x.

The discussion about Javascript/Ruby, etc.. I think the best course going
forward is a specific component appropriate to each desired mechanism is
introduced in 2.x as interest/effort dictates.  The native Python support
will eliminate the need for Jython obviously so not worried about that
coming back.

Thanks

On Tue, Nov 7, 2023 at 5:17 AM Mike Thomsen  wrote:

> Matt,
>
> I don't use JRuby, but the reason it raised my eyebrow a little was that
> AFAIK JRuby is the complete opposite of Jython in terms of enterprise
> readiness and parity with its C-based counterpart.
>
> Edward,
>
> The problem with Lua here is that the Java versions of Lua aren't being
> maintained. I think the most recent release on any branch of them was 3
> years ago. They're basically dead in the water as projects.
>
> On Mon, Nov 6, 2023 at 2:57 PM Edward Armes 
> wrote:
>
> > While I do like the ExecuteScript processors as they are great at digging
> > you out of a hole. The performance of them isn't that great.
> >
> > That being said I would ve careful about dropping Lua support as there
> is a
> > growing list of software that supports end user/administrator extensions
> > via Lua for those that don't want to have to re-compile software
> > themselves. On the other hand given that Jython doesn't yet have a
> Python 3
> > implementation it could be argued dropping Jython support is a must given
> > that the Python 2.x line is basically end of life.
> >
> > Now I wonder if its worth re-factoring the ExecuteScript processors to be
> > per language implementations inheriting from a common base like a few
> > processors do already.
> >
> > Edward
> >
> > On Mon, 6 Nov 2023, 16:24 Matt Burgess,  wrote:
> >
> > > I believe it is because in both ExecuteScript and ExecuteGroovyScript
> > > you can do "regular" groovy, but EGS has extra built-ins such as easy
> > > access to controller services, Groovy SQL stuff, etc, and we could
> > > keep building it out. But IMO we'd have to port the rest of the
> > > scripted components (ScriptedReader/Writer, etc.) over to the Groovy
> > > bundle and make sure there's a drop-in replacement in the Python stuff
> > > before we'd want to deprecate the scripted bundle.
> > >
> > > On the JRuby front, is that something you use actively? This question
> > > is for you and the entire community of course.
> > >
> > > Regards,
> > > Matt
> > >
> > > On Mon, Nov 6, 2023 at 7:12 AM Mike Thomsen 
> > > wrote:
> > > >
> > > > If we deprecate ExecuteScript, I think we need to have groovyx be
> ready
> > > to
> > > > function as a drop-in replacement if it's not there already.
> > > >
> > > > On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > IIRC the removal of these engines was mostly due to lack of use or
> at
> > > > > least the perception thereof. If JRuby is being used by the
> community
> > > > > actively, I'm happy to revisit that discussion. Luaj's JSR-223
> > > > > interface left something to be desired, but JRuby just needed a
> > system
> > > > > variable set or something like that.
> > > > >
> > > > > For the groovyx bundle, because it is Groovy-specific I tend to
> think
> > > > > we could make better use of that than ExecuteScript, especially if
> we
> > > > > do get rid of all the engines. We have a Groovy-specific
> processor, a
> > > > > "real" Python SDK, and no more Nashorn. Perhaps we move all the
> > > > > scripted components to the Groovy bundle, although I believe some
> > > > > folks still make use of Jython for these. Of course if we reinstate
> > > > > JRuby for ExecuteScript it's probably best to keep things the way
> > they
> > > > > are, or create a jruby bundle. The original scripting bundle was
> > > > > aiming to support several engines, but if it turns out only one or
> > two
> > > > > will be useful, it may not be worth shoehorning all that JSR-223
> > logic
> > > > > when engine-specific components could be simpler, more easily
> > > > > maintained, and allow for the idioms of the language to be better
> > used
> > > > > (as is done in the groovyx bundle).
> > > > >
> > > > > Just my two cents, looking forward to everyone's thoughts!
> > > > >
> > > > > - Matt
> > > > >
> > > > > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/NIFI-11646
> > > > > >
> > > > > > I get the removal of Lua, but not the removal of JRuby. It's a
> > clean
> > > > > > reimplementation of Ruby native to the JV

Re: [DISCUSS] Release Timing for NiFi 2.0.0 M1 and 1.24.0

[Joe Witt]

Very cool - thanks David and Pierre for tackling RM duties!

On Mon, Nov 13, 2023 at 11:59 AM Pierre Villard 
wrote:

> Thanks David, sounds good, will get started on the 1.24 release soon.
>
> Le lun. 13 nov. 2023 à 19:37, David Handermann <
> exceptionfact...@apache.org>
> a écrit :
>
> > Thanks for replies Pierre and Mark!
> >
> > With several pull requests merged last week addressing the Python
> > concerns Mark raised, I believe we are ready to proceed with the
> > release candidate process.
> >
> > Based on a conversation Pierre, we should be able to collaborate on
> > preparing candidate builds for both 2.0.0-M1 and 1.24.0.
> >
> > We will need to make a few website adjustments for the 2.0.0-M1
> > version in terms of documentation and downloads, but it should be
> > possible to handle that with some alternative navigation links for
> > now.
> >
> > I plan on starting the release candidate build process for 2.0.0-M1 soon.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Nov 8, 2023 at 5:53 PM Mark Payne  wrote:
> > >
> > > Hey Pierre,
> > >
> > > I ran into a couple of issues (they appear to be timing issues) around
> > the Python stuff on startup. I think we need to hold off on the release
> > until these are resolved but I think I should have a PR up some time
> > tomorrow hopefully.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > >
> > > > On Nov 8, 2023, at 10:30 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > Hey David,
> > > >
> > > > With all of the recent changes we merged, I believe we are in the
> right
> > > > spot to start the process with 1.24 and 2.0M1 releases.
> > > >
> > > > Happy to help with this,
> > > > Pierre
> > > >
> > > > Le ven. 3 nov. 2023 à 22:46, David Handermann <
> > exceptionfact...@apache.org>
> > > > a écrit :
> > > >
> > > >> Team,
> > > >>
> > > >> We have made great progress on a large number of features and fixes
> > > >> for NiFi 2.0.0. The Jira version page for 2.0.0 [1] lists almost 900
> > > >> issues! At the same time, we have 250 issues tagged for version
> 1.24.0
> > > >> [2].
> > > >>
> > > >> Although there is still more work to do for a full release of
> 2.0.0, I
> > > >> believe we are at a good point to release a milestone version.
> > > >> Releasing 1.24.0 at the same time will also provide the latest
> > > >> upgrades, plus new features and deprecations that will help prepare
> > > >> for upgrading to 2.0.0.
> > > >>
> > > >> We have informally referred to the initial release as a milestone
> > > >> version, so it seems like officially naming it 2.0.0-M1 would be the
> > > >> most straightforward option.
> > > >>
> > > >> There are a couple more open issues that I believe we can wrap up in
> > > >> the next few days, but otherwise we should be able to target early
> > > >> next week for preparing a release build.
> > > >>
> > > >> With the goal of coordinating a release of multiple versions, I
> would
> > > >> be glad to handle release manager duties for one of the versions.
> > > >>
> > > >> Regards,
> > > >> David Handermann
> > > >>
> > > >> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > >> [2] https://issues.apache.org/jira/projects/NIFI/versions/12353443
> > > >>
> > >
> >
>