Re: [VOTE] Release Apache NiFi 2.0.0 (RC2)

[Matt Burgess]

+1 (binding)

Ran through release helper, tested various flows and versioning with the
Github Registry Client, ran into [1] but I'll try to knock that out shortly
(not a blocker as it's documentation). We should also improve the
documentation for the Registry clients in general, such as the requirement
for a repo, a personal access token (PAT) and at least one branch to be
created in Github before the Github Registry Client can be used.

Thanks for RM'ing David, and to the whole community on all the hard work
and awesome new features!

[1] https://issues.apache.org/jira/browse/NIFI-13230

On Tue, Oct 29, 2024 at 11:26 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1294/
>
> Git Tag: nifi-2.0.0-RC2
> Git Commit ID: 2f13b609bdb77cb985aa35e8b66b4f04274d7c59
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/2f13b609bdb77cb985aa35e8b66b4f04274d7c59
>
> Checksums of nifi-2.0.0-source-release.zip
>
> SHA512:
> deff4c81842759ef14651874a1b3204f75d0287a578174505b409ab2869704ed77527a5e262dd030bb7a5d1531a8d55105b015444c5802f55caf80439fe11220
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 358
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354889
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: Request for maintainers on extensions with libraries that are dated/EOL

[Matt Burgess]

For Redis I wrote up [1] and put up a PR [2] in case we do agree to upgrade
to the latest version. I tested all Redis-backed components with no errors
or changes in behavior

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-13944
[2] https://github.com/apache/nifi/pull/9465


On Mon, Oct 28, 2024 at 3:13 PM Matt Burgess  wrote:

> I added [1] and a PR [2] to upgrade the easy ones. I commented on the
> Salesforce version and we should decide if we want to update the Mongo
> components to 8, and redis to 6.4+
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-13943
> [2] https://github.com/apache/nifi/pull/9464
>
> On Mon, Oct 28, 2024 at 1:36 PM Joe Witt  wrote:
>
>> Team,
>>
>> These need help from maintainers to keep them updated or they risk
>> requiring removal.
>>
>> Problematically Dated libraries we have shown as:
>> package-name currVersion latestVersionAvail
>>
>> com.dropbox.core 5.4.6 7.0.0
>>  One year old
>>
>> org.apache.camel/salesforce 3.14.10 4.8.1
>>  Already EOL for two years.
>>
>> org.mongodb 4.11.4 5.2.0
>>Compatible with Mongo 6 and 7 which are still supported. (all older are
>> EOL)
>>Not compatible with Mongo 8 which is latest.
>>
>> org.questdb 7.4.2 8.1.4
>>
>> org.snmp4j 2.8.18 3.8.2
>>
>> redis.clients 3.10.0 5.2.0
>>  Compatible with up to 6.2 which is EOL on Feb 2025
>>  Not compatible with 6.4 or 7.x which are latest
>>
>> Thanks
>> Joe
>>
>

Re: Request for maintainers on extensions with libraries that are dated/EOL

[Matt Burgess]

I added [1] and a PR [2] to upgrade the easy ones. I commented on the
Salesforce version and we should decide if we want to update the Mongo
components to 8, and redis to 6.4+

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-13943
[2] https://github.com/apache/nifi/pull/9464

On Mon, Oct 28, 2024 at 1:36 PM Joe Witt  wrote:

> Team,
>
> These need help from maintainers to keep them updated or they risk
> requiring removal.
>
> Problematically Dated libraries we have shown as:
> package-name currVersion latestVersionAvail
>
> com.dropbox.core 5.4.6 7.0.0
>  One year old
>
> org.apache.camel/salesforce 3.14.10 4.8.1
>  Already EOL for two years.
>
> org.mongodb 4.11.4 5.2.0
>Compatible with Mongo 6 and 7 which are still supported. (all older are
> EOL)
>Not compatible with Mongo 8 which is latest.
>
> org.questdb 7.4.2 8.1.4
>
> org.snmp4j 2.8.18 3.8.2
>
> redis.clients 3.10.0 5.2.0
>  Compatible with up to 6.2 which is EOL on Feb 2025
>  Not compatible with 6.4 or 7.x which are latest
>
> Thanks
> Joe
>

Re: [VOTE] Release Apache NiFi 1.28.0 (RC1)

[Matt Burgess]

+1 (binding) Ran through release helper, tested various flows using Java 8.

Thanks for RM'ing Ferenc!

On Tue, Oct 22, 2024 at 6:46 AM Ferenc Kis  wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.28.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.28.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1289
>
> Git Tag: nifi-1.28.0-RC1
> Git Commit ID: 8ecf23e77c8ca828a77f3b84554ed3347d8f7fa2
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/8ecf23e77c8ca828a77f3b84554ed3347d8f7fa2
>
> Checksums of nifi-1.28.0-source-release.zip
>
> SHA256: 96ddd83ee11f6dd0889ff2f4b4112487f021b2b3f0573d7c0eeff40672620e93
> SHA512:
>
> 22f0051b5e4a41b913e36f1fa2fabe6871471b0a4a51f3673b2fcc453382cd5d6eb132f7a6686ecdc065c295a2cfaf8199a653b822837f73a23016b6ae4bd143
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/briansolo1985.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 63
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354883
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.28.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.28.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: ExecuteSQL Run schedule - No FlowFile to route to failure

[Matt Burgess]

If there is no inbound connection then there will be no FlowFile routed to
failure, so you'd have to use GenerateFlowFile upstream to get an output
FlowFile. It is odd that a timeout exception is not a
SQLTransientException, IMO it should be routed to retry not failure but
that's for PostgreSQL to specify.

Regards,
Matt

On Sun, Oct 20, 2024 at 6:43 AM u...@moosheimer.com 
wrote:

> Dear NiFi experts,
>
> I use NiFi 1.26 and in my flow I use (a lot of) ExecuteSQL, which are
> started via cron.
>
> If a timeout error occurs, then I get the bulletin
> “*No FlowFile to route to failure*: org.postgresql.util.PSQLException:
> ERROR: canceling statement due to statement timeout”.
>
> And unfortunately no FlowFile is routed to failure.
>
> Didn't this problem exist many years ago?
> Unfortunately, I can't remember how it was solved. Is it better to use
> GenerateFlowFile instead of Cron?
> Or is the error possibly back again?
> Or am I missing something completely different?
>
> Couldn't find anything in the docs either. Maybe someone can tell me
> what the best way is?
>
> Thanks!
> Uwe
>

Re: [DISCUSS] Release Timing for NiFi 1.28.0

[Matt Burgess]

+1 I was informed that fix was backported already under a separate Jira and
PR, so I'm good with going forward.

On Thu, Oct 17, 2024 at 9:47 AM Matt Burgess  wrote:

> As with 2.0.0, if we can backport [1] into 1.28 I think we should.
>
> Thanks Ferenc!
>
> [1] https://github.com/apache/nifi/pull/9273
>
> On Thu, Oct 17, 2024 at 8:11 AM Ferenc Kis 
> wrote:
>
>> Hi Team,
>>
>> all of the remaining tickets are now merged, so we can proceed with the
>> release.
>> https://issues.apache.org/jira/projects/NIFI/versions/12354883
>>
>> Please let me know if there are still any tickets in progress which could
>> be blockers for this release.
>>
>> I plan to start the release candidate process before the end of this
>> week, if there are no objections.
>>
>> Regards
>> Ferenc
>>
>> On Thu, Oct 10, 2024 at 10:47 PM David Handermann <
>> exceptionfact...@apache.org> wrote:
>>
>> > Ferenc,
>> >
>> > Thanks for the reply! I will continue reviewing that soon to get
>> > things ready for the release.
>> >
>> > Regards,
>> > David Handermann
>> >
>> > On Thu, Oct 10, 2024 at 1:51 PM Ferenc Kis 
>> > wrote:
>> > >
>> > > Thanks David!
>> > >
>> > > We are still waiting for
>> > https://issues.apache.org/jira/browse/NIFI-13744,
>> > > it seems to be important to get into 1.28
>> > > Once it is merged and backported to 1.28, I plan to start to release
>> > process
>> > >
>> > > Regards
>> > > Ferenc
>> > >
>> > > On Thu, Oct 10, 2024 at 3:16 PM David Handermann <
>> > > exceptionfact...@apache.org> wrote:
>> > >
>> > > > Ferenc,
>> > > >
>> > > > Thanks for summarizing the status of key issues.
>> > > >
>> > > > With the dependency upgrades addressed and the other noted pull
>> > > > requests merged, the support branch seems to be in a good position
>> for
>> > > > a release candidate build.
>> > > >
>> > > > Regards,
>> > > > David Handermann
>> > > >
>> > > > On Fri, Oct 4, 2024 at 6:52 AM Ferenc Kis 
>> > wrote:
>> > > > >
>> > > > > Hi Team,
>> > > > >
>> > > > > let's recap on the current status.
>> > > > >
>> > > > > * These tickets are still in progress, but seem to get close to
>> merge
>> > > > > https://issues.apache.org/jira/browse/NIFI-13744 -
>> > > > > https://github.com/apache/nifi/pull/9285
>> > > > > https://issues.apache.org/jira/browse/NIFI-13776 -
>> > > > > https://github.com/apache/nifi/pull/9286
>> > > > >
>> > > > > * As discussed I went through the dependencies, and updated as
>> much
>> > as I
>> > > > > could.
>> > > > > Details are on the Jira, but in a nutshell: went through the
>> commits
>> > from
>> > > > > the 2024 year which contain dependency upgrades. These commits are
>> > from
>> > > > the
>> > > > > 1.26 and the 1.27 release period, so the list should reflect well
>> > which
>> > > > > dependencies are usually upgraded from release to release.
>> > > > > Unfortunately not all dependency upgrades were feasible due to
>> > various
>> > > > > reasons. If I remember correctly this was the reason why those
>> > components
>> > > > > were deprecated and removed from the NiFi 2.x line.
>> > > > > *This PR needs to be reviewed.*
>> > > > > https://issues.apache.org/jira/browse/NIFI-13836 -
>> > > > > https://github.com/apache/nifi/pull/9340
>> > > > >
>> > > > > * As requested I created a Jira for backporting NIFI-13529. The
>> > commit
>> > > > > couldn't be simply cherry-picked to support/1.x due to major
>> > refactorings
>> > > > > on the 2.x line.
>> > > > > *This PR needs to be reviewed.*
>> > > > > https://issues.apache.org/jira/browse/NIFI-13837 -
>> > > > > https://github.com/apache/nifi/pull/9341
>> > > > >
>> > > > > Regards
>> > > > > Ferenc
>> > > > >
>&

Re: [DISCUSS] Release Timing for NiFi 1.28.0

[Matt Burgess]

As with 2.0.0, if we can backport [1] into 1.28 I think we should.

Thanks Ferenc!

[1] https://github.com/apache/nifi/pull/9273

On Thu, Oct 17, 2024 at 8:11 AM Ferenc Kis  wrote:

> Hi Team,
>
> all of the remaining tickets are now merged, so we can proceed with the
> release.
> https://issues.apache.org/jira/projects/NIFI/versions/12354883
>
> Please let me know if there are still any tickets in progress which could
> be blockers for this release.
>
> I plan to start the release candidate process before the end of this
> week, if there are no objections.
>
> Regards
> Ferenc
>
> On Thu, Oct 10, 2024 at 10:47 PM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Ferenc,
> >
> > Thanks for the reply! I will continue reviewing that soon to get
> > things ready for the release.
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Oct 10, 2024 at 1:51 PM Ferenc Kis 
> > wrote:
> > >
> > > Thanks David!
> > >
> > > We are still waiting for
> > https://issues.apache.org/jira/browse/NIFI-13744,
> > > it seems to be important to get into 1.28
> > > Once it is merged and backported to 1.28, I plan to start to release
> > process
> > >
> > > Regards
> > > Ferenc
> > >
> > > On Thu, Oct 10, 2024 at 3:16 PM David Handermann <
> > > exceptionfact...@apache.org> wrote:
> > >
> > > > Ferenc,
> > > >
> > > > Thanks for summarizing the status of key issues.
> > > >
> > > > With the dependency upgrades addressed and the other noted pull
> > > > requests merged, the support branch seems to be in a good position
> for
> > > > a release candidate build.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Fri, Oct 4, 2024 at 6:52 AM Ferenc Kis 
> > wrote:
> > > > >
> > > > > Hi Team,
> > > > >
> > > > > let's recap on the current status.
> > > > >
> > > > > * These tickets are still in progress, but seem to get close to
> merge
> > > > > https://issues.apache.org/jira/browse/NIFI-13744 -
> > > > > https://github.com/apache/nifi/pull/9285
> > > > > https://issues.apache.org/jira/browse/NIFI-13776 -
> > > > > https://github.com/apache/nifi/pull/9286
> > > > >
> > > > > * As discussed I went through the dependencies, and updated as much
> > as I
> > > > > could.
> > > > > Details are on the Jira, but in a nutshell: went through the
> commits
> > from
> > > > > the 2024 year which contain dependency upgrades. These commits are
> > from
> > > > the
> > > > > 1.26 and the 1.27 release period, so the list should reflect well
> > which
> > > > > dependencies are usually upgraded from release to release.
> > > > > Unfortunately not all dependency upgrades were feasible due to
> > various
> > > > > reasons. If I remember correctly this was the reason why those
> > components
> > > > > were deprecated and removed from the NiFi 2.x line.
> > > > > *This PR needs to be reviewed.*
> > > > > https://issues.apache.org/jira/browse/NIFI-13836 -
> > > > > https://github.com/apache/nifi/pull/9340
> > > > >
> > > > > * As requested I created a Jira for backporting NIFI-13529. The
> > commit
> > > > > couldn't be simply cherry-picked to support/1.x due to major
> > refactorings
> > > > > on the 2.x line.
> > > > > *This PR needs to be reviewed.*
> > > > > https://issues.apache.org/jira/browse/NIFI-13837 -
> > > > > https://github.com/apache/nifi/pull/9341
> > > > >
> > > > > Regards
> > > > > Ferenc
> > > > >
> > > > > On Thu, Oct 3, 2024 at 10:31 AM Mathew Kiprop <
> > mathewkipro...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Is this issue[1] backported to the 1.x line and will it be
> > available in
> > > > > > 1.28.0? I haven't seen it tracked here [2]
> > > > > >
> > > > > > [1] https://github.com/apache/nifi/pull/9066
> > > > > > [2]
> https://issues.apache.org/jira/projects/NIFI/versions/12354883
> > > > > >
> > > > > > On Mon, Sep 30, 2024 at 9:33 PM David Handermann <
> > > > > > exceptionfact...@apache.org> wrote:
> > > > > >
> > > > > > > Ferenc,
> > > > > > >
> > > > > > > Thanks for reviewing the dependency upgrades.
> > > > > > >
> > > > > > > Reviewing the issues assigned to release version 1.28.0 [1]
> > there are
> > > > > > > several new Processors, and additional features, so that seems
> > the
> > > > > > > logical next step for a new version.
> > > > > > >
> > > > > > > We are also approaching readiness for the 2.0.0 release, but I
> > will
> > > > > > > start a separate thread on that when ready.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > [1]
> > https://issues.apache.org/jira/projects/NIFI/versions/12354883
> > > > > > >
> > > > > > > On Mon, Sep 30, 2024 at 6:26 AM Ferenc Kis <
> > briansolo1...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Joe,
> > > > > > > >
> > > > > > > > Thanks for the heads up. I agree with the importance of
> > dependency
> > > > > > > upgrades.
> > > > > > > >
> > > > > > > > I will go through the list of dependencies, and update
> > whichever is
> > > > > >

Re: [DISCUSS] Preparing Release of Apache NiFi 2.0.0

[Matt Burgess]

+1 although it would be nice to get [1] in, and I took a shot at [2] but I
can't figure out how the "front page" docs are generated, I searched for
the DIVs I saw in the Dev Console but the IDs didn't show up in my search
of NiFi and nifi-site.

Thanks David!

[1] https://github.com/apache/nifi/pull/9273
[2] https://issues.apache.org/jira/browse/NIFI-13761

On Thu, Oct 17, 2024 at 9:27 AM Michael Moser  wrote:

> +1 Definitely feels like we're in a good place for 2.0.0
> -- Mike
>
>
> On Thu, Oct 17, 2024 at 8:41 AM Robert Fellows 
> wrote:
>
> > +1
> > Thanks David!
> >
> > On Thu, Oct 17, 2024 at 8:08 AM Gábor Gyimesi 
> wrote:
> >
> > > +1
> > >
> > > Thanks David!
> > >
> > > Regards,
> > > Gabor
> > >
> > > On Thu, 17 Oct 2024 at 14:05, Ferenc Kis 
> > wrote:
> > > >
> > > > +1
> > > >
> > > > Thanks David
> > > >
> > > > On Thu, Oct 17, 2024 at 1:34 PM Márk Báthori  >
> > > wrote:
> > > >
> > > > > +1
> > > > > Thanks David!
> > > > >
> > > > > Peter Turcsanyi  ezt írta (időpont: 2024.
> okt.
> > > 17.,
> > > > > Cs, 13:03):
> > > > >
> > > > > > +1 Thanks David and everyone
> > > > > >
> > > > > > On Wed, Oct 16, 2024 at 8:15 PM Kevin Doran 
> > > wrote:
> > > > > >
> > > > > > >  +1 :)
> > > > > > >
> > > > > > > On Oct 16, 2024 at 14:11:45, Pierre Villard <
> > > > > pierre.villard...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Still a few things that would be "nice to have" for parity
> with
> > > 1.x
> > > > > but
> > > > > > > > definitely time to get the first release out!
> > > > > > > >
> > > > > > > > Thanks for taking care of this David!
> > > > > > > >
> > > > > > > > Pierre
> > > > > > > >
> > > > > > > > Le mer. 16 oct. 2024 à 19:36, Matt Gilman <
> > > matt.c.gil...@gmail.com>
> > > > > a
> > > > > > > > écrit :
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks for RMing David!!
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Oct 16, 2024 at 12:46 PM Lucas Ottersbach <
> > > > > > > >
> > > > > > > > lucas.ottersb...@gmail.com> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > > +1
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > Thanks David. And to everyone who contributed. Looking
> > forward
> > > to
> > > > > the
> > > > > > > >
> > > > > > > > > release.
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > David Handermann  schrieb am
> > > Mi., 16.
> > > > > > > Okt.
> > > > > > > >
> > > > > > > > > 2024, 17:59:
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > > Team,
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > It has been almost one year since we released Milestone 1
> > of
> > > NiFi
> > > > > > > >
> > > > > > > > > > 2.0.0. Over the four milestone releases, we have resolved
> > > almost
> > > > > > 2000
> > > > > > > >
> > > > > > > > > > Jira issues, added numerous capabilities, and removed
> large
> > > > > amounts
> > > > > > > of
> > > > > > > >
> > > > > > > > > > technical debt. Although there is always more that could
> be
> > > done,
> > > > > > we
> > > > > > > >
> > > > > > > > > > have reached the point where we should release a stable
> > 2.0.0
> > > > > > > version.
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > The current release version [1] has almost 300 Jira
> issues
> > > > > > resolved.
> > > > > > > >
> > > > > > > > > > With the release of NiFi API 2.0.0 and the NiFi NAR Maven
> > > Plugin
> > > > > > > >
> > > > > > > > > > 2.1.0, we are in a solid position to go forward with a
> > 2.0.0
> > > > > > version.
> > > > > > > >
> > > > > > > > > > Although there are bound to be additional areas we want
> to
> > > > > improve,
> > > > > > > we
> > > > > > > >
> > > > > > > > > > should consider those after a general availability
> release.
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > With that background, I would be glad to handle Release
> > > Manager
> > > > > > > >
> > > > > > > > > > responsibilities for this version. Unless there are any
> > > critical
> > > > > > > >
> > > > > > > > > > blockers, I plan to start the release candidate process
> > > before
> > > > > the
> > > > > > > end
> > > > > > > >
> > > > > > > > > > of this week. That should allow time for a few more items
> > to
> > > be
> > > > > > > >
> > > > > > > > > > completed.
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > >
> > > > > > > > > > David Handermann
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > [1]
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354889
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > >

Re: Jira contributor access

[Matt Burgess]

Nicolae,

I have added you as a Contributor to the NiFi Jira projects.

Regards,
Matt

On Mon, Oct 14, 2024 at 9:21 AM nicolae puica 
wrote:

> Hi there,
>
> I’d like to request the Jira contributor access since I’ve created my
> first jira issue . My
> username is nicolae93. Thanks in advance!
>
> Best Regards,
> Nicolae

Re: Oct 24 Board report for Apache NiFi

[Matt Burgess]

Thanks for preparing and submitting this Joe! Glad to be part of such a
vibrant community!

On Mon, Oct 7, 2024 at 6:31 PM Joe Witt  wrote:

> Team
>
> Below is what I've submitted to the board for this quarter.  Thanks again!
>
> Joe
>
>
> ---
>
> ## Description:
> The mission of NiFi is the creation and maintenance of software related to
> providing an easy to use, powerful, and reliable system to process and
> distribute data.
>
> Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
> integrate with and leverage the command and control of NiFi. There are both
> Java and C++ implementations.
>
> Apache NiFi Registry is a centralized registry for key configuration items
> including flow versions, assets, and extensions for Apache NiFi and Apache
> MiNiFi.
>
> Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
> NiFi classloader isolation model.
>
> Apache NiFi Flow Design System is a theme-able set of high quality UI
> components and utilities for use across the various Apache NiFi web
> applications in order to provide a more consistent user experience.
>
> ## Project Status:
> Current project status: Ongoing. High. Issues for the board: None.
>
> ## Membership Data:
> Apache NiFi was founded 2015-07-14 (9 years ago) There are currently 67
> committers and 37 PMC members in this project. The Committer-to-PMC ratio
> is
> roughly 9:5.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Gabor Gyimesi on 2024-06-26.
> - No new committers. Last additional was Lehel Boer on 2024-06-16.
>
> ## Project Activity:
> The big push remains focused on making Apache NiFi 2.0.0 happen.  We've
> completed four huge milestone releases to that and it looks likely our next
> release will be officially NiFi 2.0.0 and for GA usage.
>
> Toward making NiFi 2.0.0 official we've broken out into its own release the
> NiFi API which helps codify our core contract of backwards compatibility
> and
> points of extension for the project. We released the Apache NiFi API 2.0.0
> on
> September 23, 2024.
>
> To leverage packaging NARs that understand the split of the NiFi API vs the
> core framework version now we also have the latest NiFi Nar Maven Plugin
> 2.1.0
> as of Sep 26th.
>
> MiNiFi CPP continues to see JIRAs and PR activity as well.
>
> ## Community Health:
> JIRA and Mailing list activity remains consistent.
>
> The slack community grew by another 160 participants in the general channel
> alone increasing from 3,284 in our previous report and consistent with
> quarter
> over quarter growth for some time.  Slack is generally quite busy with
> threads
> from people needing general user help or questions, development ideas, and
> review requests.
>
> As noted above we've split out the NiFi API.  Part of this was so we can
> introduce a NiFI Improvement Process for changes to our core API.  This
> process was inspired by the tremendous growth the communities of Kafka and
> Airflow have produced and these communities have healthy discussions on how
> to
> introduce changes to critical sections.
>
> We continue to produce frequent releases in the NiFi community and enjoy
> rather active vote participation.  Releases we generate are substantial
> along
> all key dimensions of feature, improvements, bug fixes, and emphasis on
> security.  The NiFi
> 2.0 release line is among the most vulnerability free we've ever had when
> it
> comes to dependency management.
>
> Mailing lists remain quite active despite the continued increase of
> activity
> in Slack.
>

Re: Build issue on nifi-frontend goal package-web-ui

[Matt Burgess]

Mine's not hanging, just seems to have leftover processes running

On Fri, Oct 11, 2024 at 5:00 PM Matt Gilman  wrote:

> Matt,
>
> Your build is hanging too?
>
> The daemon should be disabled when running a maven build following this
> [1].
>
> [1] https://github.com/apache/nifi/pull/8953
>
> On Fri, Oct 11, 2024 at 4:47 PM Matt Burgess  wrote:
>
> > I'm seeing the same:
> >
> >  503 96990 1   0 11:13AM ?? 0:00.46
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/daemon/server/start.js
> >   503 96991 96990   0 11:13AM ?? 0:00.19
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > /var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-0.sock
> >   503 96992 96990   0 11:13AM ?? 0:00.31
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > /var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-1.sock
> >   503 96993 96990   0 11:13AM ?? 0:00.15
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > /var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-2.sock
> >   503 96994 96990   0 11:13AM ?? 0:00.13
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > /var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-3.sock
> >   503 96995 96990   0 11:13AM ?? 0:00.13
> > /Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
> >
> >
> /Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > /var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-4.sock
> >
> > Is this related to my previous post about a task still running after
> > build[1]?
> >
> > Thank you,
> > Matt
> >
> > [1] https://lists.apache.org/thread/oh76b5b6bcb96p5pqrmmrgfmlm1dl4pq
> >
> > On Fri, Oct 11, 2024 at 3:44 PM Robert Fellows 
> > wrote:
> >
> > > Mike,
> > > Maybe we should try to see if you can run just the ui build in
> isolation.
> > > I'd recommend jumping into the frontend directory "nifi/nifi-frontend"
> > and
> > > running the maven command from there.
> > >
> > > On Fri, Oct 11, 2024 at 3:23 PM Matt Gilman 
> > > wrote:
> > >
> > > > What happened when you ran the reset command (or manually killed
> those)
> > > and
> > > > rebuilt?
> > > >
> > > > On Fri, Oct 11, 2024 at 1:56 PM Mike Thomsen  >
> > > > wrote:
> > > >
> > > > > Here's the ps:
> > > > >
> > > > >   501 70566 70565   0  1:51PM ?? 0:00.58
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-0.sock
> > > > >   501 70567 70565   0  1:51PM ?? 0:00.57
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-1.sock
> > > > >   501 70568 70565   0  1:51PM ?? 0:00.36
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
&

Re: Build issue on nifi-frontend goal package-web-ui

[Matt Burgess]

I'm seeing the same:

 503 96990 1   0 11:13AM ?? 0:00.46
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/daemon/server/start.js
  503 96991 96990   0 11:13AM ?? 0:00.19
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
/var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-0.sock
  503 96992 96990   0 11:13AM ?? 0:00.31
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
/var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-1.sock
  503 96993 96990   0 11:13AM ?? 0:00.15
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
/var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-2.sock
  503 96994 96990   0 11:13AM ?? 0:00.13
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
/var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-3.sock
  503 96995 96990   0 11:13AM ?? 0:00.13
/Users/mburgess/.nvm/versions/node/v22.9.0/bin/node
/Users/mburgess/git/nifi/nifi-frontend/src/main/frontend/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
/var/folders/yr/587684352xl10tnymhbp7d1rgq/T/plugin96990-4.sock

Is this related to my previous post about a task still running after
build[1]?

Thank you,
Matt

[1] https://lists.apache.org/thread/oh76b5b6bcb96p5pqrmmrgfmlm1dl4pq

On Fri, Oct 11, 2024 at 3:44 PM Robert Fellows 
wrote:

> Mike,
> Maybe we should try to see if you can run just the ui build in isolation.
> I'd recommend jumping into the frontend directory "nifi/nifi-frontend" and
> running the maven command from there.
>
> On Fri, Oct 11, 2024 at 3:23 PM Matt Gilman 
> wrote:
>
> > What happened when you ran the reset command (or manually killed those)
> and
> > rebuilt?
> >
> > On Fri, Oct 11, 2024 at 1:56 PM Mike Thomsen 
> > wrote:
> >
> > > Here's the ps:
> > >
> > >   501 70566 70565   0  1:51PM ?? 0:00.58
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-0.sock
> > >   501 70567 70565   0  1:51PM ?? 0:00.57
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-1.sock
> > >   501 70568 70565   0  1:51PM ?? 0:00.36
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-2.sock
> > >   501 70569 70565   0  1:51PM ?? 0:00.35
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-3.sock
> > >   501 70570 70565   0  1:51PM ?? 0:00.34
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node/node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker
> > > /var/folders/c8/pqwtjgbn4sz4733mh3t45174gn/T/plugin70565-4.sock
> > >   501 70542 70409   0  1:51PM ttys0010:00.70 npm exec env-cmd -f
> > > .build.env nx run-many -t build --parallel=6 --verbose=true
> > >   501 70562 70542   0  1:51PM ttys0010:00.15 node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/.bin/env-cmd
> > > -f .build.env nx run-many -t build --parallel=6 --verbose=true
> > >   501 70565 70562   0  1:51PM ttys0010:00.59 node
> > >
> > >
> >
> /Users/mikethomsen/workspace/nifi/nifi-frontend/target/frontend-working-directory/node_modules/.bin/nx
> > > run-many -t build --parallel=6 --

[DISCUSS] Add VerifiableRegistryClient interface to verify Registry Client implementations

[Matt Burgess]

Team,

I recently submitted a NiFi Improvement Proposal (NIP) [1] to add a
VerifiableRegistryClient interface to verify Registry Client
implementations. AFAIK Registry clients didn't exist when the rest of the
Verifiable component interfaces were added, but I think it would be a good
idea to verify Registry connections before trying things like starting
version control.

I invite some discussion about [1] and would like to initiate a vote
starting Oct. 11 since it is new and fairly straightforward, if there are
no objections / concerns. For larger-scope NIPs in the future it may be
necessary to allow more discussion (say, a week?) before voting, but I'm
happy to discuss that as well.

Eager to hear thoughts about [1] and the NIP process!

Thank you,
Matt

[1] https://issues.apache.org/jira/browse/NIP-1

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 2.1.0 (RC1)

[Matt Burgess]

+1 (binding) Verified successful build of the plugin and NiFi using main
(with the new version).

Thanks for RM'ing David!

On Mon, Sep 23, 2024 at 3:03 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 2.1.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification+for+NiFi+NAR+Maven+Plugin
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-2.1.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1283/
>
> Git Tag: nifi-maven-2.1.0-RC1
> Git Commit ID: 33591a2b3cc2134218f6a56ed3fe4a5eb2e3fdad
> GitHub Commit Link:
>
> https://github.com/apache/nifi-maven/commit/33591a2b3cc2134218f6a56ed3fe4a5eb2e3fdad
>
> Checksums of nifi-nar-maven-plugin-2.1.0-source-release.zip
>
> SHA512:
> 098eeecc7e9217d5676320688c405eae26db502eb86098b2aafb7a02f5c9651da31558affec9804efbb387835d625d899a588b6f38ca059c1a82769b05e0e785
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 1
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12355125
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion2.1.0
>
> The vote will be open for 48 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-nar-maven-plugin-2.1.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi API 2.0.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified successful build

Apache Maven 3.9.9
]Java version: 21.0.4, vendor: Eclipse Adoptium, runtime:
/Users/mburgess/.sdkman/candidates/java/21.0.4-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.6.1", arch: "aarch64", family: "mac"

Thanks for RM'ing David!

On Thu, Sep 19, 2024 at 2:43 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi API 2.0.0.
>
> This is the first release of nifi-api as a separate module, following
> the recent vote in favor:
>
> https://lists.apache.org/thread/p6d7qsyto8x8xqzr6scp3fp33lrbb82c
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification+for+NiFi+API
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-api-2.0.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1282/
>
> Git Tag: nifi-api-2.0.0-RC1
> Git Commit ID: 0a171b92813959cc4d84727b7b5c74c763fd4292
> GitHub Commit Link:
>
> https://github.com/apache/nifi-api/commit/0a171b92813959cc4d84727b7b5c74c763fd4292
>
> Checksums of nifi-api-2.0.0-source-release.zip
>
> SHA512:
> fe9790322797f80283c22b803ff8e8dbe17385e2e81349299a6b8d05be820a746d06dd75ea1ae26c9144f5e5f7c5ff349cc9681e5f4ab283ea91001411d74cbf
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 1
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12355111
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiAPIVersion2.0.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-api-2.0.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: Facing Issue with Ni-Fi

[Matt Burgess]

Raja,

To what system are you migrating? If it is a straight DB migration, you
should be able to use ExecuteSQL -> PutDatabaseRecord. Just make sure your
Fetch Size property on ExecuteSQL is not set to the default of zero or the
PostgreSQL driver will try to fetch the whole thing into memory. For 15 GB
you could probably use 1 as the Fetch Size, if that seems slow then
1000 might work better. You can also set Max Rows Per FlowFile to match
Fetch Size, then each "chunk" will be handled individually by
PutDatabaseRecord, rather than building a single 15 GB FlowFile to send
"downstream". The downside is that if some FlowFiles succeed and others
fail, you'll need to do something with the failed ones and eventually route
them back to PutDatabaseRecord in the hopes they will pass the second time.

Regards,
Matt

Re: Flow comparison

[Matt Burgess]

Thank you for the explanation, makes sense now. But with only 11 components
in the flow, should it take 30 seconds?

On Tue, Jul 23, 2024 at 3:01 PM Mark Payne  wrote:

> Matt,
>
> Yes, this is normal. On startup, the flow is empty. NiFi then compares the
> empty flow to the flow.json.gz, and these are the differences it found.
> It is a bit confusing, admittedly, but it is functioning as intended.
>
> Thanks
> -Mark
>
>
> > On Jul 23, 2024, at 2:56 PM, Matt Burgess  wrote:
> >
> > I recently noticed this on NiFi startup of a standalone instance even
> > though I hadn't changed the flow in the meantime:
> >
> > 2024-07-23 14:54:29,559 INFO [main] o.a.n.c.s.StandardProcessScheduler
> > Enabling
> >
> StandardControllerServiceNode[service=JsonRecordSetWriter[id=e09a1886-0190-1000-cf4b-219dcc5cd740],
> > name=ControllerJsonRecordSetWriter, active=false]
> > 2024-07-23 14:54:59,566 INFO [main] o.a.n.c.s.StandardProcessScheduler
> > Enabling
> >
> StandardControllerServiceNode[service=LoggingRecordSink[id=e099d0c8-0190-1000-6c6e-3ced65b37c79],
> > name=LoggingRecordSink, active=false]
> > 2024-07-23 14:54:59,610 INFO [main]
> > o.a.n.f.s.StandardVersionedComponentSynchronizer Updating
> >
> StandardProcessGroup[identifier=e093123a-0190-1000-99e5-7d0a58c73b11,name=NiFi
> > Flow] to org.apache.nifi.flow.VersionedExternalFlow@2db2b708; there are
> 11
> > differences to take into account:
> > Processor with ID e09541c5-0190-1000-e56c-055331154880 exists in Proposed
> > Flow but not in Currently Loaded Flow
> > Connection with ID e0949308-0190-1000-6687-2d75cb063321 exists in
> Proposed
> > Flow but not in Currently Loaded Flow
> > Controller Service with ID e0964148-0190-1000-7bd7-dafc1e1e9366 exists in
> > Proposed Flow but not in Currently Loaded Flow
> > Processor with ID e094680b-0190-1000-2126-47be3080ac93 exists in Proposed
> > Flow but not in Currently Loaded Flow
> > Connection with ID e093d3cc-0190-1000-37ac-6dcdae031f1a exists in
> Proposed
> > Flow but not in Currently Loaded Flow
> > Connection with ID e094778f-0190-1000-7719-38608aba3d77 exists in
> Proposed
> > Flow but not in Currently Loaded Flow
> > Processor with ID e093a7c2-0190-1000-25a4-da6588eda8dc exists in Proposed
> > Flow but not in Currently Loaded Flow
> > Connection with ID e0957da4-0190-1000-43d4-31cd10da78f4 exists in
> Proposed
> > Flow but not in Currently Loaded Flow
> > Funnel with ID e09483f6-0190-1000-2ee8-887387914ec3 exists in Proposed
> Flow
> > but not in Currently Loaded Flow
> > Processor with ID e093e767-0190-1000-d8f3-2e1aefe682bb exists in Proposed
> > Flow but not in Currently Loaded Flow
> > Controller Service with ID e0965f87-0190-1000-af75-55f211b2b672 exists in
> > Proposed Flow but not in Currently Loaded Flow
> >
> > Looks like it took 30 seconds to do the comparison and found differences
> in
> > the proposed vs current flow. Is this expected?
> >
> > Thanks,
> > Matt
>
>

Flow comparison

[Matt Burgess]

I recently noticed this on NiFi startup of a standalone instance even
though I hadn't changed the flow in the meantime:

2024-07-23 14:54:29,559 INFO [main] o.a.n.c.s.StandardProcessScheduler
Enabling
StandardControllerServiceNode[service=JsonRecordSetWriter[id=e09a1886-0190-1000-cf4b-219dcc5cd740],
name=ControllerJsonRecordSetWriter, active=false]
2024-07-23 14:54:59,566 INFO [main] o.a.n.c.s.StandardProcessScheduler
Enabling
StandardControllerServiceNode[service=LoggingRecordSink[id=e099d0c8-0190-1000-6c6e-3ced65b37c79],
name=LoggingRecordSink, active=false]
2024-07-23 14:54:59,610 INFO [main]
o.a.n.f.s.StandardVersionedComponentSynchronizer Updating
StandardProcessGroup[identifier=e093123a-0190-1000-99e5-7d0a58c73b11,name=NiFi
Flow] to org.apache.nifi.flow.VersionedExternalFlow@2db2b708; there are 11
differences to take into account:
Processor with ID e09541c5-0190-1000-e56c-055331154880 exists in Proposed
Flow but not in Currently Loaded Flow
Connection with ID e0949308-0190-1000-6687-2d75cb063321 exists in Proposed
Flow but not in Currently Loaded Flow
Controller Service with ID e0964148-0190-1000-7bd7-dafc1e1e9366 exists in
Proposed Flow but not in Currently Loaded Flow
Processor with ID e094680b-0190-1000-2126-47be3080ac93 exists in Proposed
Flow but not in Currently Loaded Flow
Connection with ID e093d3cc-0190-1000-37ac-6dcdae031f1a exists in Proposed
Flow but not in Currently Loaded Flow
Connection with ID e094778f-0190-1000-7719-38608aba3d77 exists in Proposed
Flow but not in Currently Loaded Flow
Processor with ID e093a7c2-0190-1000-25a4-da6588eda8dc exists in Proposed
Flow but not in Currently Loaded Flow
Connection with ID e0957da4-0190-1000-43d4-31cd10da78f4 exists in Proposed
Flow but not in Currently Loaded Flow
Funnel with ID e09483f6-0190-1000-2ee8-887387914ec3 exists in Proposed Flow
but not in Currently Loaded Flow
Processor with ID e093e767-0190-1000-d8f3-2e1aefe682bb exists in Proposed
Flow but not in Currently Loaded Flow
Controller Service with ID e0965f87-0190-1000-af75-55f211b2b672 exists in
Proposed Flow but not in Currently Loaded Flow

Looks like it took 30 seconds to do the comparison and found differences in
the proposed vs current flow. Is this expected?

Thanks,
Matt

Re: [DISCUSS] 1.x Maintenance

[Matt Burgess]

gt; > > needs
> > > > > > is
> > > > > > > > there anybody available and interested to do that work?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Sun, Jul 7, 2024 at 5:42 AM Pierre Villard <
> > > > > > > pierre.villard...@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Given all the things that are being removed in 2.x without
> even
> > > > > > having
> > > > > > > a
> > > > > > > > > single discussion about it (PR opened and merged within 48
> > > hours),
> > > > > we
> > > > > > > > > should not expect users (especially very large ones) to
> move
> > > from
> > > > > 1.x
> > > > > > > to
> > > > > > > > > 2.x any time soon. Things that have been removed over the
> last
> > > few
> > > > > > days
> > > > > > > > are
> > > > > > > > > going to be extremely impactful and will require users to
> do a
> > > > > > > > significant
> > > > > > > > > amount of work / re-architecturing to move to 2.x. It's a
> bit
> > > sad
> > > > > to
> > > > > > > see
> > > > > > > > so
> > > > > > > > > many things being done without any discussion in the
> community.
> > > > > > > > >
> > > > > > > > > Le ven. 5 juil. 2024 à 20:11, Michael Moser <
> > > moser...@gmail.com> a
> > > > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > My opinion is that the NiFi 1.x line is already in
> > > maintenance
> > > > > and
> > > > > > > > > doesn't
> > > > > > > > > > need new features or APIs, but bug fixes are still
> important
> > > to
> > > > > get
> > > > > > > in,
> > > > > > > > > and
> > > > > > > > > > security fixes are still critical.
> > > > > > > > > >
> > > > > > > > > > That said, this is an open source community-driven
> project,
> > > so if
> > > > > > you
> > > > > > > > can
> > > > > > > > > > find contributors and committers that want to put in
> extra
> > > work
> > > > > to
> > > > > > > > > support
> > > > > > > > > > 1.x, then why not?  I'm sure many of us have seen
> something
> > > new
> > > > > in
> > > > > > > NiFi
> > > > > > > > > and
> > > > > > > > > > thought "I wouldn't have done that".  But others in the
> > > community
> > > > > > > > thought
> > > > > > > > > > it was a good idea, so why not?
> > > > > > > > > >
> > > > > > > > > > I don't really agree with the argument that putting too
> much
> > > work
> > > > > > > into
> > > > > > > > > 1.x
> > > > > > > > > > will delay people's transition to 2.0.  People will
> > > transition to
> > > > > > 2.0
> > > > > > > > on
> > > > > > > > > > their own schedule, with myriad reasons for that
> schedule.
> > > As a
> > > > > > > > > community,
> > > > > > > > > > we can simply make recommendations and provide solutions
> that
> > > > > help
> > > > > > > > with a
> > > > > > > > > > preferred direction.
> > > > > > > > > >
> > > > > > > > > > Kind regards,
> > > > > > > > > > -- Mike
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 3, 2024 at 1:27 PM Joe Witt <
> joe.w...@gmai

Re: [DISCUSS] 1.x Maintenance

[Matt Burgess]

> > > support any versions prior.
> > > > > > > >
> > > > > > > > NiFi 1.x line is built on Java 8 and requires either Java 8,
> > 11,
> > > or
> > > > > 17.
> > > > > > > >
> > > > > > > > The dates you mention which extend to 2030/etc.. for a
> > particular
> > > > > > > > distributor of JVMs is a function of what sort of support
> that
> > > > > > commercial
> > > > > > > > vendor is making available to customers and it is different
> > from
> > > > what
> > > > > > > that
> > > > > > > > means for the public generally.
> > > > > > > >
> > > > > > > > Notably though it is also a very different concern from
> > critical
> > > > > > > > dependencies we have and how they evolve. For example we rely
> > on
> > > > > things
> > > > > > > > like Spring and Jetty and so on.  And these all choose to
> adopt
> > > > > > language
> > > > > > > > features or discontinue key lines which tie to certain Java
> > > > versions,
> > > > > > > > etc..  We can't control whether they choose to move to new
> > major
> > > > > lines
> > > > > > > when
> > > > > > > > they fix vulnerabilities or whether they backport, etc..  We
> > held
> > > > the
> > > > > > > line
> > > > > > > > on Java 8 support far longer than I imagined we could but the
> > > > current
> > > > > > > > confluence of events/scenarios means we made the jump all the
> > way
> > > > to
> > > > > > Java
> > > > > > > > 21 (an LTS) release with the hopes that we can set a good
> clock
> > > for
> > > > > > > people
> > > > > > > > to work with for some time.
> > > > > > > >
> > > > > > > > The Java 1.x line will continue to be available for users to
> > > > download
> > > > > > and
> > > > > > > > will still see general maintenance to the extent there are
> > > > > contributors
> > > > > > > > available to do that.  You also have access to vendor
> supported
> > > > paths
> > > > > > to
> > > > > > > > consider there which is similar to the stated 'Extended
> Support
> > > > > > > > Availability' of Java.  But they too will have to navigate
> the
> > > > > > complexity
> > > > > > > > of the reported vulnerabilities.
> > > > > > > >
> > > > > > > > My experience in the enterprise customer space is that
> > tolerance
> > > > for
> > > > > > > > components which contain reported vulnerabilities is far far
> > > lower
> > > > > than
> > > > > > > > ever before.  This in turn means the focus shifts to giving
> > > people
> > > > > > strong
> > > > > > > > upgrade paths to consider.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Wed, Jul 3, 2024 at 10:10 AM Maucieri, Judith <
> > > copos...@ctc.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If I may:  One large obstacle to "our" shift to v2.x is the
> > > > absence
> > > > > > of
> > > > > > > > > Java 8 support (unless I overlooked updates to the plan
> > stated
> > > in
> > > > > > > > November
> > > > > > > > > 2022 during release of version 1.19.0, Java 8 only remains
> in
> > > > NiFi
> > > > > > 1.x
> > > > > > > > > releases, which you hinted at remaining accurate).
> > > > > > > > >
> > > > > > > > > I make this statement in support of multiple scenarios
> where
> > we
> > > > > > employ
> > > > > > > > > Apache NiFi.  C

Re: [VOTE] Release Apache NiFi 1.27.0 (RC2)

[Matt Burgess]

+1 (binding) ran through the release helper and tested various flows
including NIFI-13422.

Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256)
Java version: 1.8.0_372, vendor: Azul Systems, Inc., runtime:
/Users/mburgess/.sdkman/candidates/java/8.0.372-zulu/zulu-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac"

Thanks for RM'ing Pierre!

On Wed, Jul 3, 2024 at 9:01 AM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.27.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1276
>
> Git Tag: nifi-1.27.0-RC2
> Git Commit ID: e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18
>
> Checksums of nifi-1.27.0-source-release.zip
>
> SHA256: 0ac01d54eeceb4a4f4863880bf67dfa71e6a585036c5caf0546c592bf55ced48
> SHA512:
>
> 675c7750752bf3092061c9eaac39a975955e9bf881e6bee3124c5842738d8c8626b6a551f33ef7a678018bd83e0323c1f0f0d79101255494d8ca91be7fc750f5
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 37
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.27.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

[DISCUSS] 1.x Maintenance

[Matt Burgess]

There have been some ongoing discussions [1,2] about what to bring back for
PRs to 1.x vs trying to push forward with 2.x. There are of course great
points from everyone. On the 2.x front, namely that 2.x has many
improvements not just to components but the framework and UI as well, and
that we've seen less contributions to the maintenance on the 1.x line. On
the 1.x front that community members need to support 1.x for their users
for the time being.

I'm opening this discussion thread so we can collect everyone's thoughts so
the PMC can make a sensible recommendation/decision moving forward. For
myself, I think all bug PRs should be backported where prudent/possible,
and even new features (although that can be a difficult backport due to
moving the extensions in [3], but I recommend a separate PR against 1.x,
hopefully a simple copy-paste if there are no Java 21-specific code issues,
but please run contrib-check against Java 8 first).

I disagree with the "atrophy" sentiment from [2], a mature release line
normally experiences less contributions because it is mostly stable in its
own right. Guiding users to 2.x IMO is the right call but many of us have
to deal with the fact that it doesn't usually happen overnight, especially
without a GA release.

I opened this discussion with my opinions but if I may humbly speak for the
PMC, we need your voice, and we look forward to all of your thoughts,
opinions, and questions.

Thank you,
Matty B

[1] https://issues.apache.org/jira/browse/NIFI-13196
[2] https://github.com/apache/nifi/pull/9018
[3] https://issues.apache.org/jira/browse/NIFI-12998

Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)

[Matt Burgess]

+1 (binding) Ran through release helper and tried various flows, verified
versioning against a NiFi Registry (not the Git-backed one)

Apache Maven 3.9.7 (8b094c9513efc1b9ce2d952b3b9c8eaedaf8cbf0)
Java version: 21, vendor: Oracle Corporation, runtime:
/Users/mburgess/.sdkman/candidates/java/21-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac"

Thanks for RM'ing David!

On Fri, Jun 28, 2024 at 9:40 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M4.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1275
>
> Git Tag: nifi-2.0.0-M4-RC1
> Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d
>
> Checksums of nifi-2.0.0-M4-source-release.zip
>
> SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6
> SHA512:
> d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 153
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354678
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M4
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.27.0 (RC1)

[Matt Burgess]

I'm seeing SHA256 and SHA512 mismatches, also my automated RC script
noticed your GPG key is expired.

On Thu, Jun 27, 2024 at 5:11 PM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.27.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1272
>
> Git Tag: nifi-1.27.0-RC1
> Git Commit ID: bd86d1f07fe8d0f45b72e91e925ee85ffa7ed143
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/bd86d1f07fe8d0f45b72e91e925ee85ffa7ed143
>
> Checksums of nifi-1.27.0-source-release.zip
>
> SHA256: 7df2933b366aff8e1dc3abe9bafb3e8891186ca937d34d646952d43db624d4e2
> SHA512:
>
> 36c0f107b5a98388c2fb2c48d24f4df9980d9685a9b31d08c0a7a6a47b54de556ff368a8c622a5011dab4448274fc16f6e18c5d29368a81d003baa7c7382bd4d
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 35
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.27.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: processors authoring advice

[Matt Burgess]

Wei,

I have looked at adding a processor for calling stored procedures, but the
problem I encountered was how to allow the user to specify input and/or
output parameters to stored procedures and functions. You can use
ExecuteSQL to execute a stored procedure with no input/output parameters as
you have suggested in option 2. We could allow user-defined properties but
since we're limited to key/value pairs we can't represent the parameter
name, the value, and the type (input or output or inout).

Perhaps we could do something like having properties "Input parameters",
"Output Parameters", and "Inout Parameters" that accept a comma-delimited
list of parameter names along with user-defined properties where you would
again specify the parameter name but the value would be the optional value
you want to put in for an input or inout parameter. This doesn't seem like
a good user experience to me but perhaps the user would be willing to
accept the awkward configuration if they have the capability for calling
any kind of stored procedure or function. Our existing processors already
support multiple ResultSets so we can reuse that code.

I will give this some more thought but would welcome any and all opinions
on the matter.

Thank you,
Matt

On Thu, Jun 13, 2024 at 12:13 PM Zhong Wei  wrote:

> Hi, I should apologize and beg your pardon in advance for my informal
> English expression since English is not my home language.
>
> I am a developer and IT maintainer of a college. I am trying to use Nifi
> to transfer data between databases. The power of NiFi surprise me. I want
> to use it as my main tool.
>
> However, for one of my case, I have not found good solution, even after
> talking  with ChatGPT4, and some digging on internet.
>
> My case is:
>
> Mutiple RDBMS => NiFi(extract) => meta database(transformer) => NiFi(load)
> => data warehouse.
>
> Why I need meta database(it's also my centerized transformer)? because
> there are many circumstances that attaching, translating, converting or
> removing columns from large dataset are needed, and those transforming
> operations rely on bunch of meta tables in meta database.
>
> So, I think that in my batch transforming scenario, no any other tool is
> more powerful than stored procedures or functions in meta database. So I
> hope hungrily NiFi to provide the flow ability like below:
>
>
>   1.
> Source RDBMS => ExecuteSQLRecord => NewNifiProcessor(Call
> StoredProcedure/Function with Records and fetch transformed records back)
> =>PutDatabaseRecord => data warehouse
>
> or
>
>   2.
> Srouce RDBMS => ExecuteSQLRecord => (1)ExecuteSQL(create temp table on
> meta database) =>(2)ExecuteSQL(insert records into temp table)=>
> (3)ExecuteSQL(call storedprocedure/function and fetch records back) =>
> PutDatabseRecord => data warehouse
>
> in the 2nd flow, exactly the same opening database connection instance
> need to be guaranteed during the three steps (1)(2)(3) in any
> circumstance(i.e multiple flows share one database pool controller
> service), in order to enclose both three steps in one database connection
> session. however, I don't know currently how to ensure the same session
> across processors, so my solution is :
>
> ... insert records into fixed table => transform records in
> function/stored procedure => truncate the fixed table => return records
> =>...
>
> that can work, but is a bit odd.
>
> Lastly, in my opinion, for batch transforming dataset, stored procedure or
> function in database is more powerful and maintainable than any other
> solution. so, I think more processors centering on utilizing this ability
> should be built.
>
> best regards
>
> Wei.
> 2024/6/13
>

Re: SplitPCAP processor bug with large files and proposed fix

[Matt Burgess]

Jack,

For 1, check one of the existing Split processors such as SplitRecord for
the "fragment.*" attribute pattern, it should be consistent with that
behavior and code.

For 2, if an invalid packet should "spoil the bunch" so to speak, you can
use session.rollback() then transfer the original FlowFile to the failure
relationship and commit that.

Regards,
Matt

On Mon, Jun 24, 2024 at 10:03 AM Jack Hinton 
wrote:

> Hi all,
>
> I've been doing some further testing with the SplitPCAP processor, and
> I've found that with file sizes larger than around 3GB it tends to error
> out with the response that some packet or other in the main PCAP file is
> invalid. I've been unable to determine precisely why an invalid packet
> error is returned rather than a framework-generated error, but I have found
> that if any resultant flowfiles are transferred as soon as they're split
> from the original then this issue no longer occurs.
>
> To remedy this, I propose that flowfiles should be transferred in
> configurably-sized batches during the process of splitting the main file
> rather than being collated and sent after processing is complete. This will
> also have the effect that the amount of RAM that is dedicated to the task
> of splitting PCAPs can be determined by the user rather than by how big the
> original PCAP file is. There are some issues with this, however:
>
>
>   1.
> 'Split' processors mark every resultant flowfile with a 'number X of Y'
> attribute that means the resultant flowfiles can't be sent off until it's
> known how many there are in total.
>   2.
> As the 'split' flowfiles would be transferred as they're created, if there
> is an invalid packet later in the original PCAP then a situation could
> arise where flowfiles are transferred both to the 'split' relationship and
> the 'failure' relationship.
>
> Does anyone have any thoughts on how to address those problems?
>
> Thanks,
>
> Jack Hinton
>

Re: Thoughts on various ways to help drive NiFi and the community forward

[Matt Burgess]

Mathew, certainly! I've been meaning to update the blog too. Email me at
mattyb...@apache.org and we'll get 'em up there :)

Thanks,
Matt

On Sat, Jun 15, 2024 at 2:46 PM Mathew Kiprop 
wrote:

> +1 to all subjects addressed Joe.
>
> I have been planning to improve developer advocacy around scripting
> components. The groovy scripts have been one of my best tools to build
> flows in NiFi (Rest API development , complex data structures
> transformation, controlled flows scheduling…..). I’ll take this opportunity
> to ask Mat Burges whether I can contribute some of my work to
> http://funnifi.blogspot.com/?m=1.
>
> Mathew
>
> On Sat, 15 Jun 2024 at 21:03, Joe Witt  wrote:
>
> > Team,
> >
> > We are seeing an increase in the unique contributors on a monthly basis
> to
> > NiFi and a clear and steady increase in the overall contribution rate
> over
> > the past 1-2 years in particular.  This is awesome but we do have
> important
> > problems to cover and we need to make it easier to help newer
> contributors
> > know where to jump in and add the most value.
> >
> > For sure people will contribute various things just because they have a
> > particular interest such as some new extension, etc..  But from a
> community
> > direction perspective there are things we need to continue to improve
> upon
> > and we want to help make the path to committership and PMC membership
> more
> > clear as a function of adding value to the community.
> >
> > So not necessarily ordered, here are some key things I've observed that
> > need more attention and are among the most valuable things for the health
> > of the nifi system and community.
> >
> > Dependency Management
> >
> > This is a hugely important area as we have a massive amount of
> dependencies
> > and this work is just plain not that fun.  But if we want to present a
> > beautiful landscape we need to pull weeds.  We need to reduce
> dependencies
> > and keep things recent.  This is largely accomplished today by a very
> small
> > group of people.  We have to increase the contribution level here or
> > perhaps the automation level.  We have tooling that is helping now but
> > still needs more attention.
> >
> > Vulnerability Management particular as it relates to dependencies
> >
> > Related to dependency management but broader.  This one at times requires
> > us to do more than simply maintain dependencies.  It sometimes forces us
> to
> > change libraries and sometimes even makes maintaining
> > compatibility/configuration complicated.  These are among the most
> critical
> > and most difficult tasks to take on.  This is largely done today by an
> > extremely small group of people.
> >
> > Maintenance of extensions
> >
> > We have a ton of components in NiFi.  Obviously not all of them really do
> > require active maintenance but there is often a contribute and forget
> model
> > involved.  For us to keep these broad range of components under community
> > care they need active maintenance.  I already mentioned the dependency
> > factor above but the broader point here is that APIs/versions of systems
> > evolve.  We need to be staying up to date.  There are systems like Kafka,
> > Elastic, Mongo, Relational Databases, etc.. that we should always be
> > supporting the latest on.  These are among the most wildly popular
> sources
> > and destinations.  We do tend to fall behind here and this is powerfully
> > useful for people to contribute to.  These are the things our users
> > actually use NiFi for!
> >
> > Maintenance of things other than core nifi
> >
> > If you look at components ancillary to NiFi they receive far less
> attention
> > and effort trending toward them going away.  The NiFi registry is a great
> > example of this as NiFi MiNiFi Java for instance.  These are things which
> > are useful and indeed there is a decent user base in both cases.  But
> they
> > require more active maintenance than they get.  We need to evolve when it
> > comes to security methods offered for authn/authz and that takes focus.
> >
> > Performance analysis and improvements targeted as a result
> >
> > This is done very rarely as far as I can tell outside a couple of people
> > who I'm sure we could all name in a few seconds.  The codebase is large
> and
> > Java and other parts evolve.  Continually emphasizing faster and more
> > efficient operations is vital to NiFi's continued growth in the
> > community at large.  People who spend time running NiFi in a measured
> > manner with realistic flows and doing things like profiling for CPU,
> > memory, etc.. and finding bottlenecks and offering solutions - absolutely
> > can be high impact and valuable.  Even if not at a point where a
> confident
> > contribution can be made, the very act of doing this level of evaluation
> > and flagging concerns is valuable.
> >
> > Improvements related to tests both unit and integration - but also for
> > users of NiFi
> >
> > As part of our push to NiFi 2.0 we have been running these tests

Unfinished process

[Matt Burgess]

I just built from the latest main and haven't run it yet but I still have a
running process many minutes later:

  503 19756 1   0  6:12PM ?? 0:02.10
/Users/mburgess/git/nifi/nifi-frontend/target/frontend-working-directory/node/node
/Users/mburgess/git/nifi/nifi-frontend/target/frontend-working-directory/node_modules/nx/src/daemon/server/start.js
  503 32716  8641   0  7:23PM ttys0010:00.00 grep -i nifi

Is there something I need to do?

Thanks,
Matt

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 2.0.0

[Matt Burgess]

+1 binding, verified the commit hash, applied the patch to NiFi main and
verified the NAR Maven plugin works as expected.

Thanks for RM'ing Pierre!

On Wed, May 22, 2024 at 7:31 AM Pierre Villard 
wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> NAR Maven Plugin 2.0.0. This new release is intended to be used with NiFi
> 2+.
>
> The source being voted upon, including signatures, digests, etc. can be
> found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-2.0.0/
>
> A helpful reminder on how the release candidate verification process works:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
>
> Please note that when trying this release, some changes are expected on the
> NiFi side as proposed in the PR: https://github.com/apache/nifi/pull/8860
>
> The Git tag is nifi-maven-2.0.0-RC1
> The Git commit ID is d0de49a34a1197988eaa007d28e6eb20ab6d701f
>
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=d0de49a34a1197988eaa007d28e6eb20ab6d701f
>
> Checksums of nifi-nar-maven-plugin-2.0.0-source-release.zip:
> SHA256: ae889c843a82c4823fca0bf5fedf275dbb812c2b11e25ae26b0bd4ad7d26ccae
> SHA512:
>
> c9afefb093d8b9568012d6697f302eb4a138f0a65a7045eaeb1bff551fb4bff23c805294700ca20984a3493eb61fe80a39d8b5b393afc85cf3f6ab1a28df831c
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 5 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354745
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion2.0.0
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-2.0.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because …
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin

[Matt Burgess]

+1 for a 1.6.0 release

On Mon, May 20, 2024 at 1:25 PM Bryan Bende  wrote:

> Thanks Pierre. I agree it would be good to kick out a release. I would
> lean towards 1.6.0 since the commits seem to be improvements rather
> than bug fixes for a patch.
>
> On Mon, May 20, 2024 at 1:08 PM Pierre Villard
>  wrote:
> >
> > Hi all,
> >
> > We just merged a couple of improvements for the NiFi NAR Maven Plugin and
> > another improvement has been merged some time ago for which the community
> > expressed interest [1]. I think it could be a good time to release a new
> > version, either as 1.5.2 or as 1.6.0. Once we have a new release, there
> > will be a need for updating the NiFi code base and I have a PR ready for
> > that [2].
> >
> > I'm happy to help with the release process.
> >
> > Do you agree it is time for a new release or do you see additional
> changes
> > we should make?
> >
> > Thanks,
> > Pierre
> >
> > [1] https://github.com/apache/nifi-maven/pull/35
> > [2] https://issues.apache.org/jira/browse/NIFI-13267
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Yeah as I was going through it and creating all those delegates I thought
about trying to define a higher-level API but that's about where I stopped.
I'd definitely be interested to see what you've got.

Thanks,
Matt

On Mon, May 20, 2024 at 12:50 PM Mike Thomsen 
wrote:

> Matt,
>
> IIRC in your PR you had abstractions on the basic APIs. I think the right
> approach would be to make the driver controller services provide an API for
> querying so the processors and other services never have to work with the
> api or an abstraction of it.
>
> I have a lot of progress on such an an approach I can post to my repo if
> you're interested.
>
> On Fri, May 17, 2024 at 7:31 AM Matt Burgess  wrote:
>
> > Now that the Couchbase PR is up I can continue my work on this if
> > everyone's ok with the approach.
> >
> > On Fri, May 17, 2024 at 5:30 AM Pierre Villard <
> > pierre.villard...@gmail.com>
> > wrote:
> >
> > > Hey guys, what's the latest on this?
> > >
> > > Le sam. 23 mars 2024 à 01:49, Mike Thomsen  a
> > > écrit :
> > >
> > > > Fair enough, Joe.
> > > >
> > > > Matt,
> > > >
> > > > I poked around your branch a little this evening. I agree with you
> and
> > > > David 100% now on the need for some sort of abstraction. I think the
> > > Graph
> > > > Bundle's model could serve as a good starting point for how to
> approach
> > > the
> > > > problem. The client drivers in that bundle do the heavy lifting of
> > doing
> > > > the querying and passing the results back via a callback system to
> the
> > > > processors that call them to ensure that the processors don't know
> > > anything
> > > > other than they're getting back a Map of result data each iteration.
> > > > Thoughts?
> > > >
> > > > On Fri, Mar 22, 2024 at 2:48 PM Joe Witt  wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > The bundles we include cannot have libs with know vulns and that
> > last a
> > > > > very long time.  That is a more pressing blocker.
> > > > >
> > > > > As noted top of thread we all recognize the importance of being
> able
> > to
> > > > > integrate with Cassandra but including that must come with active
> mx
> > > > > especially as it relates to vulns.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Fri, Mar 22, 2024 at 11:42 AM Mike Thomsen <
> > mikerthom...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > The scope tag was probably copy pasta. You raise a valid point
> > about
> > > > the
> > > > > > processor dependencies that completely slipped my mind. That
> said,
> > I
> > > > > think
> > > > > > it's premature to remove the cassandra bundle until we have a
> > working
> > > > > > replacement. I would consider that support a blocker for 2.X.
> > > > > >
> > > > > > On Fri, Mar 22, 2024 at 12:00 PM Matt Burgess <
> mattyb...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > David beat me to it :) IMO the only NAR that should have any
> > > > > dependencies
> > > > > > > on Cassandra is the services NAR, not the processors or
> services
> > > API.
> > > > > > >
> > > > > > > On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
> > > > > > > exceptionfact...@apache.org> wrote:
> > > > > > >
> > > > > > > > Mike,
> > > > > > > >
> > > > > > > > Thanks for sharing the branch, it is helpful to have that as
> a
> > > > > > > > reference example. Have you been able to exercise any of that
> > > > > approach
> > > > > > > > at runtime?
> > > > > > > >
> > > > > > > > Based on what is there right now, attempting to mark the
> > DataStax
> > > > > > > > java-driver-core as provided does not look like it will work.
> > It
> > > > may
> > > > > > > > pass unit tests, but runtime NAR class loading requires that
> >

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Now that the Couchbase PR is up I can continue my work on this if
everyone's ok with the approach.

On Fri, May 17, 2024 at 5:30 AM Pierre Villard 
wrote:

> Hey guys, what's the latest on this?
>
> Le sam. 23 mars 2024 à 01:49, Mike Thomsen  a
> écrit :
>
> > Fair enough, Joe.
> >
> > Matt,
> >
> > I poked around your branch a little this evening. I agree with you and
> > David 100% now on the need for some sort of abstraction. I think the
> Graph
> > Bundle's model could serve as a good starting point for how to approach
> the
> > problem. The client drivers in that bundle do the heavy lifting of doing
> > the querying and passing the results back via a callback system to the
> > processors that call them to ensure that the processors don't know
> anything
> > other than they're getting back a Map of result data each iteration.
> > Thoughts?
> >
> > On Fri, Mar 22, 2024 at 2:48 PM Joe Witt  wrote:
> >
> > > Mike,
> > >
> > > The bundles we include cannot have libs with know vulns and that last a
> > > very long time.  That is a more pressing blocker.
> > >
> > > As noted top of thread we all recognize the importance of being able to
> > > integrate with Cassandra but including that must come with active mx
> > > especially as it relates to vulns.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Mar 22, 2024 at 11:42 AM Mike Thomsen 
> > > wrote:
> > >
> > > > The scope tag was probably copy pasta. You raise a valid point about
> > the
> > > > processor dependencies that completely slipped my mind. That said, I
> > > think
> > > > it's premature to remove the cassandra bundle until we have a working
> > > > replacement. I would consider that support a blocker for 2.X.
> > > >
> > > > On Fri, Mar 22, 2024 at 12:00 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > David beat me to it :) IMO the only NAR that should have any
> > > dependencies
> > > > > on Cassandra is the services NAR, not the processors or services
> API.
> > > > >
> > > > > On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
> > > > > exceptionfact...@apache.org> wrote:
> > > > >
> > > > > > Mike,
> > > > > >
> > > > > > Thanks for sharing the branch, it is helpful to have that as a
> > > > > > reference example. Have you been able to exercise any of that
> > > approach
> > > > > > at runtime?
> > > > > >
> > > > > > Based on what is there right now, attempting to mark the DataStax
> > > > > > java-driver-core as provided does not look like it will work. It
> > may
> > > > > > pass unit tests, but runtime NAR class loading requires that
> > classes
> > > > > > be available in the same NAR, or in a parent NAR. That means when
> > > NiFi
> > > > > > tries to load the Controller Service interface, it must have
> access
> > > to
> > > > > > a version of the relevant Cassandra driver classes. By marking
> the
> > > > > > dependency as provided, it will not be available in the API NAR,
> > and
> > > > > > thus not available when loading the service interface. Including
> it
> > > in
> > > > > > the API NAR won't work either, because it conflicts with the
> > ScyllaDB
> > > > > > java-driver-core in the implementation NAR.
> > > > > >
> > > > > > This is the reason Matt and I highlighted for providing a layer
> of
> > > > > > abstraction at the Controller Service API level.
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > >
> > > > > > On Fri, Mar 22, 2024 at 8:13 AM Mike Thomsen <
> > mikerthom...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Work so far:
> > https://github.com/MikeThomsen/nifi/tree/cql-changes
> > > > > > >
> > > > > > > On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen <
> > > mikerthom...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Matt/David

Re: [VOTE] Release Apache NiFi 2.0.0-M3 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified various flows. Thanks for RM'ing David!

On Mon, May 13, 2024 at 11:48 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M3.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1269
>
> Git Tag: nifi-2.0.0-M3-RC1
> Git Commit ID: f2215c6522a5571189290760c55f0317f8562cbd
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/f2215c6522a5571189290760c55f0317f8562cbd
>
> Checksums of nifi-2.0.0-M3-source-release.zip
>
> SHA256: d471a0a9e4e04faf13bbe13c7d83be6f8002fba598bf0968a3c1b339802a185a
> SHA512:
> cd0b8bbd3fe4ea7ebe8cdac6ac8a7afa97fd7b6a521c2cbcb2c0cdc94899b652bf3726c8fe432e16f44a9dc81907414bbb42e03113f0cd9fb51aa1de9cd727a7
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 411
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M3
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: Inquiry Regarding Missing Documentation for selectHive3QL in nifi-hive3-nar for NiFi 1.25.0

[Matt Burgess]

Jing,

Because SelectHive3QL (and the nifi-hive3-nar in general) is not included
with the convenience binary for Apache NiFi, the documentation for it is
not published on the website. If you downloaded and installed the NAR into
your instance, the documentation is available by dragging a SelectHive3QL
instance onto the canvas, right-clicking on it and choosing "View usage" or
"View documentation".

Regards,
Matt


On Wed, May 15, 2024 at 12:10 AM yetong gao  wrote:

> Dear NiFi Development Team,
>
> I hope this message finds you well. I am currently working with Apache NiFi
> 1.25.0 and it's already operational in our data processing pipeline. In my
> research, I've noticed a scarcity of documentation regarding the
> selectHive3QL processor, specifically on the official NiFi documentation
> website.
>
> I'd greatly appreciate any information you could provide on why the
> selectHive3QL processor's documentation is not available on the NiFi
> website, or if there are plans to include it in the future.
>
> I believe comprehensive documentation for each processor in NiFi is
> essential for users to maximise the potential of this powerful platform.
>
> I look forward to your response. Thank you in advance.
>
> Best Regard,
> Jing
>

Re: [discuss] identifying key components needed updates which require deeper review/consideration

[Matt Burgess]

For [1]  Hive 3 was removed in
https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive
dependency from Apache Iceberg we probably need a release from them that
brings in Hive 4 dependencies? Or do you suggest we bring the Hive
components back as long as we update the version?

I'll take the Flyway one in the meantime, database integrations are my
blessing and my curse :) If that goes more smoothly than I expect I can
grab the QuestDB one too, will update the Jiras as appropriate.

Regards,
Matt

On Tue, May 7, 2024 at 1:56 PM Joe Witt  wrote:

> Team,
>
> We have made massive strides going from thousands of out of date libraries
> to dozens in the 2.x line vs any prior releases.  And we can/should stay on
> top of these better going forward.  A few key bits that are outstanding
> with no obvious immediate work underway are as follows.  If anyone is
> working on these, willing to work on these, please volunteer and let us
> know.  For components that we cannot find active engagement on we should
> consider removal though these aren't great examples as they're indeed
> common usage components.
>
> [1] https://issues.apache.org/jira/browse/NIFI-13173
> Hive 4 recently came out but Hive also gives us some of the most
> complicated POM/library mangling and oldest dependencies to override/etc..
> Putting serious detailed focus here is very valuable from a maintenance
> point of view.  While existing usage of Hive 3 likely remains for some
> time, those users are like NiFi 1.x users anyway.  In any case if there are
> motivated developers interested to see our Hive components be healthy
> please do dive in on this.
>
> [2] https://issues.apache.org/jira/browse/NIFI-13169
> The NiFI registry is a major release behind Flyway DB.  We need to stay
> current given the importance of this component.
>
> [3] https://issues.apache.org/jira/browse/NIFI-13168
> We are a few incrementals behind and a minor release behind QuestDB.  I
> tried bumping the incremental but it broke tests.
>
> [4] https://issues.apache.org/jira/browse/NIFI-11686
> We are a full major release behind and several incremental/minors as well.
> Apparently there is some concern related to the community changes in
> Elastic vs OpenSearch/etc.. but regardless we need to stay active in
> maintaining this component.  We really should get to the latest main.
>
> Thanks to anyone who can help tackle these and make progress both in terms
> of doing the upgrade/update work, validating tests, fixing licenses, but
> also reviews.
>
> Thanks
> Joe
>

Re: [VOTE] Release Apache NiFi 1.26.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified various flows for fixes including
https://issues.apache.org/jira/browse/NIFI-13121

Thanks for RM'ing Pierre!

On Fri, May 3, 2024 at 8:47 AM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.26.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.26.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1265
>
> Git Tag: nifi-1.26.0-RC1
> Git Commit ID: 338083f6850b61cd2651e41bbd73b3cb5330d98c
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/338083f6850b61cd2651e41bbd73b3cb5330d98c
>
> Checksums of nifi-1.26.0-source-release.zip
>
> SHA256: 75ea201c12bb99672b1f075c9520b5f7df8b24e033ec47121848b351a0d47790
> SHA512:
>
> 5b754d899ce8cff900d5871d44c2fda9224e69fe8a0fe9a7121f3359ed8881300e4d4d0b2fe5befea276e0495c7ebbed04cc307c18527aa7a1cea25a923a
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 128
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354185
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.26.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.26.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.26.0 (RC1)

[Matt Burgess]

You may be running into [1], can you upgrade your JDK?

Regards,
Matt

[1] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8219013


On Fri, May 3, 2024 at 2:54 PM Dan S  wrote:

> II am encountering the exception below when building with Java 8
> My environment per maven -version
>
> Maven home: /opt/apache-maven-3.9.6
> Java version: 1.8.0_412, vendor: Red Hat, Inc., runtime:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-2.el8.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.18.0-513.24.1.el8_9.x86_64", arch: "amd64",
> family: "unix"
>
>
> org.apache.nifi.security.util.crypto.HashServiceTest.testShouldHashEmptyValue
> -- Time elapsed: 0.043 s <<< ERROR!
> java.lang.IllegalArgumentException:
> java.security.NoSuchAlgorithmException: SHA3-224 MessageDigest not
> available
>
>
> Please advise
>
>
>
>
>
> On Fri, May 3, 2024 at 5:52 PM Nissim Shiman 
> wrote:
>
> >  +1 (non-binding)
> >
> > verified source release sha256/512 checksums
> >
> > successfully ran build using:
> > Apache Maven 3.9.6
> > Java openjdk version: 1.8.0_402
> > linux kernel 3.10.0-1160
> >
> > Ran various simple flows successfully.
> >
> > Tested against out of the box/empty registry, (i.e. created bucket,
> > imported PG. made new version of PG, imported new version of PG back to
> > graph)
> >
> > Migrated existing/populated registry from 1.24.0 to this RC successfully.
> > (i.e. and tested importing PGs, committing new versions of PGs to
> registry
> > successfully, importing updated PG to graph)
> >
> > Tested/verified:
> > NIFI-12858 - Processor change history on property hover
> >
> > Thank you Pierre and team for 1.26.0!
> >
> > Nissim Shiman
> >
> > On Friday, May 3, 2024 at 12:47:45 PM UTC, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> >
> >  Team,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.26.0.
> >
> > Please review the following guide for how to verify a release candidate
> > build:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> >
> > The source being voted on the and the convenience binaries are available
> on
> > the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.26.0
> >
> > The build artifacts are available on the Apache Nexus Repository:
> >
> > https://repository.apache.org/content/repositories/orgapachenifi-1265
> >
> > Git Tag: nifi-1.26.0-RC1
> > Git Commit ID: 338083f6850b61cd2651e41bbd73b3cb5330d98c
> > GitHub Commit Link:
> >
> >
> https://github.com/apache/nifi/commit/338083f6850b61cd2651e41bbd73b3cb5330d98c
> >
> > Checksums of nifi-1.26.0-source-release.zip
> >
> > SHA256: 75ea201c12bb99672b1f075c9520b5f7df8b24e033ec47121848b351a0d47790
> > SHA512:
> >
> >
> 5b754d899ce8cff900d5871d44c2fda9224e69fe8a0fe9a7121f3359ed8881300e4d4d0b2fe5befea276e0495c7ebbed04cc307c18527aa7a1cea25a923a
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/pvillard.asc
> >
> > KEYS file is available on the Apache Distribution Repository:
> >
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > Issues resolved for this version: 128
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354185
> >
> > Release note highlights can be found on the project wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.26.0
> >
> > The vote will be open for 72 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [] +1 Release this package as nifi-1.26.0
> > [] +0 no opinion
> > [] -1 Do not release this package because...
> >
>

Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3

[Matt Burgess]

One thing that was mentioned was an included Jira for  "providing a clearer
distinction between framework and
extensions," This involved moving extensions to a new module and for
community folks this is causing problems with any
new improvements in-flight before that. Seems like it needed more
announcement than an upcoming RC due to the impact?

On Mon, Apr 29, 2024 at 10:22 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> We are getting closer to ready for a release candidate build. Based on
> some conversations with those working on the user interface, there are
> still a couple more items to complete this week. Getting a solid build
> of the new UI into this next version will be very helpful, so getting
> a few more of those issues resolved should put things in a good
> position.
>
> Regards,
> David Handermann
>
> On Wed, Apr 24, 2024 at 8:32 AM Joe Witt  wrote:
> >
> > Also agreed there.  Profile should be there to exclude perhaps but
> include
> > should be default.
> >
> > On Wed, Apr 24, 2024 at 6:30 AM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Thanks for the replies thus far!
> > >
> > > With the goal of including the new UI, it seems like a good time to
> > > move the module from an optional profile to part of the standard
> > > build. That would make the release candidate build and review process
> > > simpler, not requiring the optional build profile.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman 
> > > wrote:
> > > >
> > > > I agree. Including the updated UI and getting some feedback would be
> > > great.
> > > >
> > > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt  wrote:
> > > >
> > > > > Id be a big fan of including the new UI.
> > > > >
> > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard <
> > > > > pierre.villard...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks David, I'm happy to take care of a 1.26 release at the
> same
> > > time.
> > > > > >
> > > > > > Regarding the M3 release, should the new UI be included and
> > > instructions
> > > > > > given on how to access it to start collecting feedback? I'm
> under the
> > > > > > impression that almost all of the work has been done, no?
> > > > > >
> > > > > > Thanks,
> > > > > > Pierre
> > > > > >
> > > > > > Le mar. 23 avr. 2024, 02:03, David Handermann <
> > > > > exceptionfact...@apache.org
> > > > > > >
> > > > > > a écrit :
> > > > > >
> > > > > > > Team,
> > > > > > >
> > > > > > > We are approaching 300 Jira issues resolved for version
> 2.0.0-M3
> > > [1],
> > > > > > > including a number of important dependency upgrades and Python
> > > > > > > framework improvements.
> > > > > > >
> > > > > > > There are several open pull requests that are getting close to
> > > > > > > completion, which should be possible to include in an upcoming
> > > release
> > > > > > > version. There is also a significant pull request [2] for
> > > NIFI-12998
> > > > > > > [3] that includes some important changes to module directory
> > > > > > > hierarchy, providing a clearer distinction between framework
> and
> > > > > > > extensions, and implementing significant cleanup of dependency
> > > > > > > declarations across the repository.
> > > > > > >
> > > > > > > With these changes in view, we should be ready for another
> > > milestone
> > > > > > > release soon after merging the above pull request.
> > > > > > >
> > > > > > > Another milestone release seems to be the best course of
> action for
> > > > > > > now, providing the opportunity for additional review and
> testing
> > > > > > > before a full 2.0.0 release version.
> > > > > > >
> > > > > > > I would be glad to handle Release Manager responsibilities for
> > > > > > > 2.0.0-M3 when things are ready.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3
> > > > > > > [2] https://github.com/apache/nifi/pull/8677
> > > > > > > [3] https://issues.apache.org/jira/browse/NIFI-12998
> > > > > > >
> > > > > >
> > > > >
> > >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

David beat me to it :) IMO the only NAR that should have any dependencies
on Cassandra is the services NAR, not the processors or services API.

On Fri, Mar 22, 2024 at 11:10 AM David Handermann <
exceptionfact...@apache.org> wrote:

> Mike,
>
> Thanks for sharing the branch, it is helpful to have that as a
> reference example. Have you been able to exercise any of that approach
> at runtime?
>
> Based on what is there right now, attempting to mark the DataStax
> java-driver-core as provided does not look like it will work. It may
> pass unit tests, but runtime NAR class loading requires that classes
> be available in the same NAR, or in a parent NAR. That means when NiFi
> tries to load the Controller Service interface, it must have access to
> a version of the relevant Cassandra driver classes. By marking the
> dependency as provided, it will not be available in the API NAR, and
> thus not available when loading the service interface. Including it in
> the API NAR won't work either, because it conflicts with the ScyllaDB
> java-driver-core in the implementation NAR.
>
> This is the reason Matt and I highlighted for providing a layer of
> abstraction at the Controller Service API level.
>
> Regards,
> David Handermann
>
> On Fri, Mar 22, 2024 at 8:13 AM Mike Thomsen 
> wrote:
> >
> > Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes
> >
> > On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen 
> wrote:
> >
> > > Matt/David,
> > >
> > > By this evening, I should be at a point where I can share my branch. It
> > > should be far enough along that y'all can see what I mean about how
> most of
> > > the changes really weren't that complicated. My sense is that if we
> > > collaborate on it, we can probably get it ready for a PR within a week
> or
> > > two.
> > >
> > > It would probably be a good idea to plan to revisit the Cassandra DMC's
> > > design and make it more flexible.
> > >
> > > One nice thing about the new DataStax driver is that it supports
> > > configuration by a very detailed configuration file format, so we can
> give
> > > users that option + combine it with EL/parameters (I envision an option
> > > where the user puts EL in the file, we load the file, preprocess the
> EL and
> > > load that into the driver)
> > >
> > > On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen 
> > > wrote:
> > >
> > >> If it were that simple, they would probably have just gone with that
> > >> solution. That said, the API is functionally vendor agnostic at this
> point
> > >> at the Java API level. So I see no need to add abstraction above
> that. I've
> > >> got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few
> pain
> > >> points where I'm having to dig deep into the docs to make progress,
> but so
> > >> far, so good.
> > >>
> > >> On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess 
> > >> wrote:
> > >>
> > >>> It would be interesting to see if you exclude the Scylla API JAR
> from the
> > >>> Scylla implementation and instead include DataStax's, if that works.
> > >>> However I'm still leaning towards a vendor-agnostic API.
> > >>>
> > >>> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > At first glance, the package names look identical to me:
> > >>> >
> > >>> >
> https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
> > >>> >
> > >>> > So I see no reason to not take them at their word that it's drop-in
> > >>> >
> > >>> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
> > >>> > exceptionfact...@apache.org> wrote:
> > >>> >
> > >>> > > Mike,
> > >>> > >
> > >>> > > One important thing to mention about the DataStax vs ScyllaDB
> driver
> > >>> > > is that the Maven coordinates are different, and managing the
> > >>> > > dependencies correctly will make or break the implementation.
> > >>> > >
> > >>> > > In other words, if it is possible to use the DataStax 4 core JAR
> in
> > >>> > > the Controller Service API, but use the ScyllaDB 3 query JAR in
> the
> > >>> > > Scyl

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

It would be interesting to see if you exclude the Scylla API JAR from the
Scylla implementation and instead include DataStax's, if that works.
However I'm still leaning towards a vendor-agnostic API.

On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen 
wrote:

> At first glance, the package names look identical to me:
>
> https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html
>
> So I see no reason to not take them at their word that it's drop-in
>
> On Wed, Mar 20, 2024 at 11:04 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> > Mike,
> >
> > One important thing to mention about the DataStax vs ScyllaDB driver
> > is that the Maven coordinates are different, and managing the
> > dependencies correctly will make or break the implementation.
> >
> > In other words, if it is possible to use the DataStax 4 core JAR in
> > the Controller Service API, but use the ScyllaDB 3 query JAR in the
> > ScyllaDB implementation, then that could avoid the need for additional
> > abstraction. Without taking a closer look, however, I would be
> > surprised if this worked.
> >
> > Although ScyllaDB highlights their forked driver as a drop-in
> > replacement for the DataStax version, and maintains the same Java
> > package names, there is a difference between a complete replacement
> > and a shared API JAR. Without a common API JAR, that both
> > implementations can use, it will be necessary to provide an
> > abstraction in NiFi that avoids depending on either library at the
> > Controller Service API level.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen 
> > wrote:
> > >
> > > Matt/David,
> > >
> > > I got some time this morning to take a crack at directly migrating it
> > over
> > > to the DataStax 4.17 driver. Definitely got a lot of work to do, but so
> > far
> > > I haven't hit any real snags. This is a branch that reverts the commit
> to
> > > remove the cassandra bundle and reuses the existing features as a
> > > foundation. From what I'm seeing so far (feels like I'm about 25% done)
> > it
> > > should be doable to reuse the existing bundle, but rename it to the
> "CQL
> > > Bundle" and just add a second controller service for Scylla that is
> > > otherwise 100% the same codewise.
> > >
> > > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen 
> > wrote:
> > >
> > > > A cursory look at the Cassandra 5 stuff didn’t indicate any
> > > > incompatibility. So yeah, I think we are likely pretty safe to use
> the
> > 4.17
> > > > driver
> > > > Sent from my iPhone
> > > >
> > > > > On Mar 19, 2024, at 3:35 PM, Matt Burgess 
> > wrote:
> > > > >
> > > > > Is it likely now (due to the refactor) that we will simply be able
> > to
> > > > > upgrade the driver when Cassandra 5 is GA? Also does anyone use
> > Netflix's
> > > > > Astyanax [1]?
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java
> > > > >
> > > > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen <
> > mikerthom...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Realistically, I think we are only likely to see two drivers:
> > > > >>
> > > > >> * DataStax
> > > > >> * ScyllaDB
> > > > >>
> > > > >> The latter makes a selling point of being a binary compatible,
> > drop-in
> > > > >> replacement for the former.
> > > > >>
> > > > >> That's why I don't see a need to have an abstraction layer per
> se. I
> > > > think
> > > > >> we only need "DataStaxConnectionProviderImpl" and
> > > > >> "ScyllaDBConnectionProviderImpl" with the difference being which
> > jar is
> > > > >> imported by maven.
> > > > >>
> > > > >> On Tue, Mar 19, 2024 at 2:59 PM David Handermann <
> > > > >> exceptionfact...@apache.org> wrote:
> > > > >>
> > > > >>> Mike,
> > > > >>>
> > > > >>> Thanks for the reply and clarification.
> > > > >>>
> > > &g

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

/github.com/apache/cassandra-java-driver/
> > > >>
> > > >> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen <
> mikerthom...@gmail.com
> > >
> > > >> wrote:
> > > >> >
> > > >> > Matt,
> > > >> >
> > > >> > I got that. My point was that the Java changes appear to be a one
> > time
> > > >> > thing that DataStax did to make a better driver with a much more
> > > >> > future-proof API. Since Scylla tracks them as closely as
> possible, I
> > > >> > suspect that we don't need to plan for a bunch of abstraction to
> > isolate
> > > >> > Java changes.
> > > >> >
> > > >> > On Tue, Mar 19, 2024 at 11:07 AM Steven Matison <
> > > >> steven.mati...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > That was kinda where i got stuck and fell out on my branch/jira.
> > > >> Mike and
> > > >> > > I wanted to make a new controller service , without backward
> > > >> compatibility;
> > > >> > > and remove the duplicate driver/connection properties found in
> > some
> > > >> of the
> > > >> > > processors.
> > > >> > >
> > > >> > > I agree taking out all old stuff and making new controller
> service
> > > >> makes
> > > >> > > most sense.  4.x and 5.x should be mostly backwards compatible
> to
> > > >> 2&3.x
> > > >> > > with how it’s used within current processors.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess <
> > mattyb...@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > The abstraction is to isolate Java API changes, not protocol
> > > >> > > compatibility
> > > >> > > > Changing to the java-driver comes with a number of changes to
> > the
> > > >> code
> > > >> > > (see
> > > >> > > > Steven's and my branches), if we can abstract that API it
> should
> > > >> lead to
> > > >> > > > more maintainable code in the future by not having to change
> any
> > > >> > > > processors, just the controller service implementation.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen <
> > > >> mikerthom...@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >>
> >
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
> > > >> > > > >
> > > >> > > > > Directly quoting Scylla docs here:
> > > >> > > > >
> > > >> > > > > > The Scylla Java Driver is a drop-in replacement for the
> > > >> DataStax Java
> > > >> > > > > Driver. As such, no code changes are needed to use this
> > driver.
> > > >> > > > >
> > > >> > > > > On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen <
> > > >> mikerthom...@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Matt,
> > > >> > > > > >
> > > >> > > > > > I don't think we need to really "abstract above" the
> drivers
> > > >> because
> > > >> > > > the
> > > >> > > > > > Java DataStax driver appears to support 4.X all the way
> > back to
> > > >> 2.X,
> > > >> > > as
> > > >> > > > > > well as the enterprise versions from DataStax
> > > >> > > > > >
> > > >> > > > > >
> > > >> https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
> > > >> > > > > >
> > > >> > &

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

The abstraction is to isolate Java API changes, not protocol compatibility.
Changing to the java-driver comes with a number of changes to the code (see
Steven's and my branches), if we can abstract that API it should lead to
more maintainable code in the future by not having to change any
processors, just the controller service implementation.


On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen 
wrote:

>
> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html
>
> Directly quoting Scylla docs here:
>
> > The Scylla Java Driver is a drop-in replacement for the DataStax Java
> Driver. As such, no code changes are needed to use this driver.
>
> On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen 
> wrote:
>
> > Matt,
> >
> > I don't think we need to really "abstract above" the drivers because the
> > Java DataStax driver appears to support 4.X all the way back to 2.X, as
> > well as the enterprise versions from DataStax
> >
> > https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html
> >
> > Similar situation with Scylla. When I looked at the driver, it appeared
> to
> > copy verbatim the entire public API of that driver. So I think before we
> > dive into abstractions, it's worth doing a bit more validation of these
> > details. IMHO, this might be a much lighter lift than anticipated.
> >
> >
> > On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess 
> wrote:
> >
> >> Totally agree, that's what my branch does (see link in previous email).
> >> The
> >> more I work with it, the more I think I can abstract it further from
> their
> >> JDBC-like API but I started with a bunch of delegate classes then I
> figure
> >> I'll see where I can consolidate to more abstract concepts. If I don't
> >> have
> >> to support Cassandra 3 with the new API, so much the better.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
> >> exceptionfact...@apache.org> wrote:
> >>
> >> > Matt et al,
> >> >
> >> > It is good to see the background effort on moving Cassandra
> >> > capabilities in a supportable direction.
> >> >
> >> > I think new Cassandra components will require a significant departure
> >> > from current Controller Service abstractions. Right now, the existing
> >> > service interface does not provide a clean abstraction from the
> >> > Cassandra library, which is part of the reason for the current
> >> > coupling to the legacy driver version.
> >> >
> >> > Following up from Joe's comments, it seems like the cleanest way
> >> > forward is to deprecate the current bundle on the 1.x branch, and
> >> > remove the current bundle from the main branch. That will provide a
> >> > clean slate for new Service and Processor implementations, without
> >> > concern for uncertain compatibility questions.
> >> >
> >> > Regards,
> >> > David Handermann
> >> >
> >> > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess 
> >> wrote:
> >> > >
> >> > > What do y'all think about removing the individual connection
> >> properties
> >> > > from the Cassandra processors for NiFi 2.0 and requiring a
> >> > > CassandraSessionProvider instead? I think we started doing that
> >> elsewhere
> >> > > (Elasticsearch maybe?), I noticed duplicate code in the
> >> > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep
> >> those
> >> > > properties I can refactor them into a utility class.
> >> > >
> >> > > Thanks,
> >> > > Matt
> >> > >
> >> > >
> >> > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison <
> >> steven.mati...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > I got through quite a bit of work to enable 4.x…
> >> > > >
> >> > > > The 3.x pieces that were not backwards compatible is very edge use
> >> > case and
> >> > > > could have been done slightly differently but with work around.
> >> > > >
> >> > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> >> > > >
> >> > > >
> >> > > >
> >> > > &g

Re: DB Dynamic Connection

[Matt Burgess]

Eduardo,

I believe if you have a Parameter Context set on a process group then those
values will be available when you evaluate Expression Language anywhere in
the processor / controller code, as long as your properties are referring
to the parameters and not variables.

Regards,
Matt

On Sun, Mar 17, 2024 at 9:38 PM Eduardo Fontes 
wrote:

> Hi Matt,
>
> I encountered some issues while attempting to implement what I had in mind.
> The main obstacle is that I'm unsure how to pass a "Parameter Context"
> parameter name as a variable, as the evaluation of P.C. occurs during the
> startup phase of the processor/controller. I require this functionality to
> transmit sensitive values, such as passwords, without exposing them in
> flowfile properties as plain text. My initial idea was to obtain a
> parameter context from within a controller code, which is invoked from
> within the onTrigger function of a processor.
>
> Perhaps it would be better if the Parameter Context acted as Azure Key
> Vault or AWS Secrets Manager!
>
> Any thoughts?
>
> PS.: My plan B is to implement it using a Scripted Processor that: 1) get
> values for db connection from a secret vault in the cloud; 2) make a DB
> connection; 3) Read the data and send to processor relationship; 4) close
> connection.
>
> On Fri, Mar 15, 2024 at 5:23 PM Matt Burgess  wrote:
>
> > True, but my concern is that you might see performance issues with a new
> > connection each time, especially if the same value(s) come in many times
> in
> > a row (i.e. choosing the same connection config). Having a small cache
> > might afford you some speedups.
> >
> > Regards,
> > Matt
> >
> > On Sun, Mar 10, 2024 at 9:17 AM Eduardo Fontes  >
> > wrote:
> >
> > > Hi Matt,
> > >
> > > I don't think I need a pool or a cache, since DB connection will be
> used
> > > once for an object (table/view). So I think that won't be a problem
> > create
> > > a DB connection, read object and destroy connection, for each object.
> > >
> > > I'll try to implement this using DBCPService Controller Interface.
> > >
> > > Thanks for your consideration.
> > >
> > > Eduardo Fontes
> > >
> > > On Tue, Mar 5, 2024 at 11:10 PM Matt Burgess 
> > wrote:
> > >
> > > > Eduardo,
> > > >
> > > > It doesn't sound like DBCPConnectionPoolLookup will work for you
> > because
> > > of
> > > > all the different connection strings. I don't know if there's a good
> > > reason
> > > > why we couldn't create the BasicDataSource when getConnection() is
> > > called,
> > > > passing in a Map of FlowFile attributes (that's how the Lookup
> version
> > > > works). One issue I do see is with "churn" if we're recreating the
> data
> > > > source each time. At that point it's not pooling connections. I
> suppose
> > > you
> > > > could have an internal cache of data sources but it would have to be
> > > > bounded and/or configurable and have a least-recently-used (LRU)
> > eviction
> > > > strategy.
> > > >
> > > > DBCPService is the name of the controller service interface that the
> > > > database processors use, but that's a misnomer since the API doesn't
> > > > mention pooling specifically. Instead you could have an
> implementation
> > > that
> > > > uses a cache vs a pooling approach. But Apache DBCP does handle a lot
> > of
> > > > the management (validation, eviction, idle timeouts, etc.)  so unless
> > > > there's no way to avoid the potential memory/performance issues (like
> > > > having 50+ controller services in a PG) you could try to wrangle
> > smaller
> > > > pools per data source and cache those if that's ok for your use case.
> > > >
> > > > My two cents,
> > > > Matt
> > > >
> > > > On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes <
> > eduardo.fon...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Everybody!
> > > > >
> > > > > I'm thinking about make a generic ingestor with Apache NiFi but I
> > found
> > > > > some difficulties because of the DataBase Connection Pool
> controller.
> > > It
> > > > > doesn't accept flowfiles parameters for its properties, specially
> > > > > connection string, username and password (for security reasons,
> some
> > > > > sensitive parameter name instead password itself).
> > > > >
> > > > > This is important because, as a generic ingestor, I might have
> > hundreds
> > > > of
> > > > > different connection strings, and I had a lot of problems when I
> > tried
> > > to
> > > > > put 50 DBCP controllers in a Process Group.
> > > > >
> > > > > I wouldn't like to create a flow for each ingestion, but one flow
> for
> > > > each
> > > > > database vendor.
> > > > >
> > > > > Does anyone have any suggestions on how I can achieve this? Would
> it
> > be
> > > > > easy to create a parameterized DBCP controller? (That I could do it
> > > > myself)
> > > > >
> > > > > Best regards.
> > > > >
> > > > > Eduardo Fontes
> > > > > Data Eng / System Analyst Sr.
> > > > >
> > > >
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Sounds like a plan, thanks much!

On Mon, Mar 18, 2024 at 4:49 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Matt,
>
> Thanks, that makes sense on further review of the branch you mentioned
> previously.
>
> It sounds like we have a general consensus on a way forward.
>
> I will proceed with writing up the Jira issues and putting together
> pull requests for deprecation and removal of the existing Cassandra 3
> components. That should put things in a good place to land the new
> capabilities when they are ready. It also resolves the current
> vulnerability findings with the legacy driver, so this approach is
> helpful on several fronts.
>
> Regards,
> David Handermann
>
> On Mon, Mar 18, 2024 at 3:30 PM Matt Burgess  wrote:
> >
> > Totally agree, that's what my branch does (see link in previous email).
> The
> > more I work with it, the more I think I can abstract it further from
> their
> > JDBC-like API but I started with a bunch of delegate classes then I
> figure
> > I'll see where I can consolidate to more abstract concepts. If I don't
> have
> > to support Cassandra 3 with the new API, so much the better.
> >
> > Regards,
> > Matt
> >
> > On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
> > exceptionfact...@apache.org> wrote:
> >
> > > Matt et al,
> > >
> > > It is good to see the background effort on moving Cassandra
> > > capabilities in a supportable direction.
> > >
> > > I think new Cassandra components will require a significant departure
> > > from current Controller Service abstractions. Right now, the existing
> > > service interface does not provide a clean abstraction from the
> > > Cassandra library, which is part of the reason for the current
> > > coupling to the legacy driver version.
> > >
> > > Following up from Joe's comments, it seems like the cleanest way
> > > forward is to deprecate the current bundle on the 1.x branch, and
> > > remove the current bundle from the main branch. That will provide a
> > > clean slate for new Service and Processor implementations, without
> > > concern for uncertain compatibility questions.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess 
> wrote:
> > > >
> > > > What do y'all think about removing the individual connection
> properties
> > > > from the Cassandra processors for NiFi 2.0 and requiring a
> > > > CassandraSessionProvider instead? I think we started doing that
> elsewhere
> > > > (Elasticsearch maybe?), I noticed duplicate code in the
> > > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep
> those
> > > > properties I can refactor them into a utility class.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > >
> > > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison <
> steven.mati...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I got through quite a bit of work to enable 4.x…
> > > > >
> > > > > The 3.x pieces that were not backwards compatible is very edge use
> > > case and
> > > > > could have been done slightly differently but with work around.
> > > > >
> > > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess  >
> > > wrote:
> > > > >
> > > > > > Oops used the wrong email address so if there have been
> responses to
> > > the
> > > > > > Cassandra thread since mine I missed them, my bad!
> > > > > >
> > > > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess <
> mattyb...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I believe the CQL protocol is backwards compatible but the Java
> > > API is
> > > > > > > not. For example "com.datastax.driver.core.Session" is now
> > > > > > > "com.datastax.oss.driver.api.core.session.Session" and there
> is no
> > > more
> > > > > > > "Cluster" class. Might be fairly trivial to fix though, if
&

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Totally agree, that's what my branch does (see link in previous email). The
more I work with it, the more I think I can abstract it further from their
JDBC-like API but I started with a bunch of delegate classes then I figure
I'll see where I can consolidate to more abstract concepts. If I don't have
to support Cassandra 3 with the new API, so much the better.

Regards,
Matt

On Mon, Mar 18, 2024 at 4:14 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Matt et al,
>
> It is good to see the background effort on moving Cassandra
> capabilities in a supportable direction.
>
> I think new Cassandra components will require a significant departure
> from current Controller Service abstractions. Right now, the existing
> service interface does not provide a clean abstraction from the
> Cassandra library, which is part of the reason for the current
> coupling to the legacy driver version.
>
> Following up from Joe's comments, it seems like the cleanest way
> forward is to deprecate the current bundle on the 1.x branch, and
> remove the current bundle from the main branch. That will provide a
> clean slate for new Service and Processor implementations, without
> concern for uncertain compatibility questions.
>
> Regards,
> David Handermann
>
> On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess  wrote:
> >
> > What do y'all think about removing the individual connection properties
> > from the Cassandra processors for NiFi 2.0 and requiring a
> > CassandraSessionProvider instead? I think we started doing that elsewhere
> > (Elasticsearch maybe?), I noticed duplicate code in the
> > CassandraSessionProvider and AbstractCassandraProcessor, if we keep those
> > properties I can refactor them into a utility class.
> >
> > Thanks,
> > Matt
> >
> >
> > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison  >
> > wrote:
> >
> > > I got through quite a bit of work to enable 4.x…
> > >
> > > The 3.x pieces that were not backwards compatible is very edge use
> case and
> > > could have been done slightly differently but with work around.
> > >
> > > https://github.com/steven-matison/nifi/tree/nifi-10120-1
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess 
> wrote:
> > >
> > > > Oops used the wrong email address so if there have been responses to
> the
> > > > Cassandra thread since mine I missed them, my bad!
> > > >
> > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > I believe the CQL protocol is backwards compatible but the Java
> API is
> > > > > not. For example "com.datastax.driver.core.Session" is now
> > > > > "com.datastax.oss.driver.api.core.session.Session" and there is no
> more
> > > > > "Cluster" class. Might be fairly trivial to fix though, if that's
> the
> > > > path
> > > > > of least resistance.
> > > > >
> > > > > On Fri, Mar 15, 2024 at 1:40 PM Joe Witt 
> wrote:
> > > > >
> > > > >> Matt
> > > > >>
> > > > >> I dont know a ton about Cassandra but when I looked at
> client/driver
> > > > notes
> > > > >> for 4+ it said it was compatible all the way back to 3.x.   Not
> sure
> > > > what
> > > > >> that means but it surely seems worth exploring.  Also I dont know
> if
> > > the
> > > > >> 4.x drivers get rid of the vulnerable bits.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess <
> mattyb...@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > At the very least we should upgrade to Cassandra 3.11.6:
> > > > >> >
> > > >
> https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> > > > >> >
> > > > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess <
> mattyb...@apache.org>
> > > > >> wrote:
> > > > >> >
> > > > >> > > If the community agrees to get rid of Cassandra 3 that'll
> save me
> > > > >> effort
> > > > >> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > > > >> > > vulnerabilities would 

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

What do y'all think about removing the individual connection properties
from the Cassandra processors for NiFi 2.0 and requiring a
CassandraSessionProvider instead? I think we started doing that elsewhere
(Elasticsearch maybe?), I noticed duplicate code in the
CassandraSessionProvider and AbstractCassandraProcessor, if we keep those
properties I can refactor them into a utility class.

Thanks,
Matt


On Fri, Mar 15, 2024 at 2:44 PM Steven Matison 
wrote:

> I got through quite a bit of work to enable 4.x…
>
> The 3.x pieces that were not backwards compatible is very edge use case and
> could have been done slightly differently but with work around.
>
> https://github.com/steven-matison/nifi/tree/nifi-10120-1
>
>
>
>
>
>
> On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess  wrote:
>
> > Oops used the wrong email address so if there have been responses to the
> > Cassandra thread since mine I missed them, my bad!
> >
> > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess 
> wrote:
> >
> > > I believe the CQL protocol is backwards compatible but the Java API is
> > > not. For example "com.datastax.driver.core.Session" is now
> > > "com.datastax.oss.driver.api.core.session.Session" and there is no more
> > > "Cluster" class. Might be fairly trivial to fix though, if that's the
> > path
> > > of least resistance.
> > >
> > > On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:
> > >
> > >> Matt
> > >>
> > >> I dont know a ton about Cassandra but when I looked at client/driver
> > notes
> > >> for 4+ it said it was compatible all the way back to 3.x.   Not sure
> > what
> > >> that means but it surely seems worth exploring.  Also I dont know if
> the
> > >> 4.x drivers get rid of the vulnerable bits.
> > >>
> > >> Thanks
> > >>
> > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
> > >> wrote:
> > >>
> > >> > At the very least we should upgrade to Cassandra 3.11.6:
> > >> >
> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> > >> >
> > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
> > >> wrote:
> > >> >
> > >> > > If the community agrees to get rid of Cassandra 3 that'll save me
> > >> effort
> > >> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > >> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR
> > that
> > >> > > would not be included in the convenience binary.
> > >> > >
> > >> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt 
> > wrote:
> > >> > >
> > >> > >> Mike, Matt,
> > >> > >>
> > >> > >> Happy to hear you both have active efforts or are interested in
> > doing
> > >> > so.
> > >> > >> Can you help me understand more specifically what that means for
> > the
> > >> > >> current set of components?
> > >> > >>
> > >> > >> The CVE hits are concerning and long standing.  Supporting
> > Cassandra
> > >> 3
> > >> > >> implies the current set of dependencies would remain too right?
> > >> > >>
> > >> > >> Is the current set of components we have ones we want to retain?
> > We
> > >> > >> certainly need Cassandra components - but are the ones we have
> now
> > >> the
> > >> > >> right ones?
> > >> > >>
> > >> > >> Thanks
> > >> > >> Joe
> > >> > >>
> > >> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess <
> > mattyb...@apache.org>
> > >> > >> wrote:
> > >> > >>
> > >> > >> > I'm actively working this, I pushed my branch up in case anyone
> > >> wants
> > >> > to
> > >> > >> > take a look [1]. The idea is to abstract the Cassandra API "up
> a
> > >> > couple
> > >> > >> > levels" and provide implementations for Cassandra 3, 4, and
> > >> eventually
> > >> > >> 5.
> > >> > >> > For JDBC-like interfaces this is a PITA because of the API
> > >> (Statement,
> > >> > >> > Pr

Re: DB Dynamic Connection

[Matt Burgess]

True, but my concern is that you might see performance issues with a new
connection each time, especially if the same value(s) come in many times in
a row (i.e. choosing the same connection config). Having a small cache
might afford you some speedups.

Regards,
Matt

On Sun, Mar 10, 2024 at 9:17 AM Eduardo Fontes 
wrote:

> Hi Matt,
>
> I don't think I need a pool or a cache, since DB connection will be used
> once for an object (table/view). So I think that won't be a problem create
> a DB connection, read object and destroy connection, for each object.
>
> I'll try to implement this using DBCPService Controller Interface.
>
> Thanks for your consideration.
>
> Eduardo Fontes
>
> On Tue, Mar 5, 2024 at 11:10 PM Matt Burgess  wrote:
>
> > Eduardo,
> >
> > It doesn't sound like DBCPConnectionPoolLookup will work for you because
> of
> > all the different connection strings. I don't know if there's a good
> reason
> > why we couldn't create the BasicDataSource when getConnection() is
> called,
> > passing in a Map of FlowFile attributes (that's how the Lookup version
> > works). One issue I do see is with "churn" if we're recreating the data
> > source each time. At that point it's not pooling connections. I suppose
> you
> > could have an internal cache of data sources but it would have to be
> > bounded and/or configurable and have a least-recently-used (LRU) eviction
> > strategy.
> >
> > DBCPService is the name of the controller service interface that the
> > database processors use, but that's a misnomer since the API doesn't
> > mention pooling specifically. Instead you could have an implementation
> that
> > uses a cache vs a pooling approach. But Apache DBCP does handle a lot of
> > the management (validation, eviction, idle timeouts, etc.)  so unless
> > there's no way to avoid the potential memory/performance issues (like
> > having 50+ controller services in a PG) you could try to wrangle smaller
> > pools per data source and cache those if that's ok for your use case.
> >
> > My two cents,
> > Matt
> >
> > On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes 
> > wrote:
> >
> > > Hi Everybody!
> > >
> > > I'm thinking about make a generic ingestor with Apache NiFi but I found
> > > some difficulties because of the DataBase Connection Pool controller.
> It
> > > doesn't accept flowfiles parameters for its properties, specially
> > > connection string, username and password (for security reasons, some
> > > sensitive parameter name instead password itself).
> > >
> > > This is important because, as a generic ingestor, I might have hundreds
> > of
> > > different connection strings, and I had a lot of problems when I tried
> to
> > > put 50 DBCP controllers in a Process Group.
> > >
> > > I wouldn't like to create a flow for each ingestion, but one flow for
> > each
> > > database vendor.
> > >
> > > Does anyone have any suggestions on how I can achieve this? Would it be
> > > easy to create a parameterized DBCP controller? (That I could do it
> > myself)
> > >
> > > Best regards.
> > >
> > > Eduardo Fontes
> > > Data Eng / System Analyst Sr.
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

Oops used the wrong email address so if there have been responses to the
Cassandra thread since mine I missed them, my bad!

On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess  wrote:

> I believe the CQL protocol is backwards compatible but the Java API is
> not. For example "com.datastax.driver.core.Session" is now
> "com.datastax.oss.driver.api.core.session.Session" and there is no more
> "Cluster" class. Might be fairly trivial to fix though, if that's the path
> of least resistance.
>
> On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:
>
>> Matt
>>
>> I dont know a ton about Cassandra but when I looked at client/driver notes
>> for 4+ it said it was compatible all the way back to 3.x.   Not sure what
>> that means but it surely seems worth exploring.  Also I dont know if the
>> 4.x drivers get rid of the vulnerable bits.
>>
>> Thanks
>>
>> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
>> wrote:
>>
>> > At the very least we should upgrade to Cassandra 3.11.6:
>> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
>> >
>> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
>> wrote:
>> >
>> > > If the community agrees to get rid of Cassandra 3 that'll save me
>> effort
>> > > on the refactor after I add Cassandra 4 :) Otherwise those
>> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
>> > > would not be included in the convenience binary.
>> > >
>> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
>> > >
>> > >> Mike, Matt,
>> > >>
>> > >> Happy to hear you both have active efforts or are interested in doing
>> > so.
>> > >> Can you help me understand more specifically what that means for the
>> > >> current set of components?
>> > >>
>> > >> The CVE hits are concerning and long standing.  Supporting Cassandra
>> 3
>> > >> implies the current set of dependencies would remain too right?
>> > >>
>> > >> Is the current set of components we have ones we want to retain?  We
>> > >> certainly need Cassandra components - but are the ones we have now
>> the
>> > >> right ones?
>> > >>
>> > >> Thanks
>> > >> Joe
>> > >>
>> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
>> > >> wrote:
>> > >>
>> > >> > I'm actively working this, I pushed my branch up in case anyone
>> wants
>> > to
>> > >> > take a look [1]. The idea is to abstract the Cassandra API "up a
>> > couple
>> > >> > levels" and provide implementations for Cassandra 3, 4, and
>> eventually
>> > >> 5.
>> > >> > For JDBC-like interfaces this is a PITA because of the API
>> (Statement,
>> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping
>> we
>> > >> can
>> > >> > find a common pattern for abstracting the third-party library
>> > >> > implementation and API from the NiFi component (Processor,
>> > >> > ControllerService, etc.) API. I think we're doing something similar
>> > for
>> > >> > Kafka?
>> > >> >
>> > >> > Regards,
>> > >> > Matt
>> > >> >
>> > >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
>> > >> >
>> > >> >
>> > >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen <
>> mikerthom...@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > That’s been on my todo list for a little while but things kept
>> > coming
>> > >> up.
>> > >> > > I think I could get started on that now. Based on my initial
>> > research
>> > >> it
>> > >> > > appears that scylla uses the exact same api as datastax so
>> > supporting
>> > >> > both
>> > >> > > in a cql bundle should theoretically be fairly easy.
>> > >> > >
>> > >> > >
>> > >> > > Sent from my iPhone
>> > >> > >
>> > >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt 
>> wrote:
>> > >> > > >
>> > &g

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

I believe the CQL protocol is backwards compatible but the Java API is not.
For example "com.datastax.driver.core.Session" is now
"com.datastax.oss.driver.api.core.session.Session" and there is no more
"Cluster" class. Might be fairly trivial to fix though, if that's the path
of least resistance.

On Fri, Mar 15, 2024 at 1:40 PM Joe Witt  wrote:

> Matt
>
> I dont know a ton about Cassandra but when I looked at client/driver notes
> for 4+ it said it was compatible all the way back to 3.x.   Not sure what
> that means but it surely seems worth exploring.  Also I dont know if the
> 4.x drivers get rid of the vulnerable bits.
>
> Thanks
>
> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess 
> wrote:
>
> > At the very least we should upgrade to Cassandra 3.11.6:
> > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt
> >
> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess 
> wrote:
> >
> > > If the community agrees to get rid of Cassandra 3 that'll save me
> effort
> > > on the refactor after I add Cassandra 4 :) Otherwise those
> > > vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> > > would not be included in the convenience binary.
> > >
> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
> > >
> > >> Mike, Matt,
> > >>
> > >> Happy to hear you both have active efforts or are interested in doing
> > so.
> > >> Can you help me understand more specifically what that means for the
> > >> current set of components?
> > >>
> > >> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> > >> implies the current set of dependencies would remain too right?
> > >>
> > >> Is the current set of components we have ones we want to retain?  We
> > >> certainly need Cassandra components - but are the ones we have now the
> > >> right ones?
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> > >> wrote:
> > >>
> > >> > I'm actively working this, I pushed my branch up in case anyone
> wants
> > to
> > >> > take a look [1]. The idea is to abstract the Cassandra API "up a
> > couple
> > >> > levels" and provide implementations for Cassandra 3, 4, and
> eventually
> > >> 5.
> > >> > For JDBC-like interfaces this is a PITA because of the API
> (Statement,
> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping
> we
> > >> can
> > >> > find a common pattern for abstracting the third-party library
> > >> > implementation and API from the NiFi component (Processor,
> > >> > ControllerService, etc.) API. I think we're doing something similar
> > for
> > >> > Kafka?
> > >> >
> > >> > Regards,
> > >> > Matt
> > >> >
> > >> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> > >> >
> > >> >
> > >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen <
> mikerthom...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > That’s been on my todo list for a little while but things kept
> > coming
> > >> up.
> > >> > > I think I could get started on that now. Based on my initial
> > research
> > >> it
> > >> > > appears that scylla uses the exact same api as datastax so
> > supporting
> > >> > both
> > >> > > in a cql bundle should theoretically be fairly easy.
> > >> > >
> > >> > >
> > >> > > Sent from my iPhone
> > >> > >
> > >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt 
> wrote:
> > >> > > >
> > >> > > > Team,
> > >> > > >
> > >> > > > Cassandra remains a really important system to be able to send
> > data
> > >> to.
> > >> > > > However, it seems like we've not maintained these well.  We have
> > >> what
> > >> > > > appears to be at least a full generation behind on client
> versions
> > >> (we
> > >> > > are
> > >> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > >> > > shortly).
> > >> > > >
> > >> > > > We have components to send data, query data, and use Cassandra
> as
> > a
> > >> > cache
> > >> > > > store.  We have older mechanisms for json/avro and publish
> > >> mechanisms
> > >> > for
> > >> > > > records.
> > >> > > >
> > >> > > > The libraries we do have depend on outdated versions of Guava
> and
> > >> > result
> > >> > > in
> > >> > > > many CVE hits.
> > >> > > >
> > >> > > > I am inclined to think we should deprecate the 1.x components
> and
> > >> > remove
> > >> > > > them as-is from the 2.x line.  Then re-introduce them with
> record
> > >> only
> > >> > > > interfaces and built against the latest stable
> > >> > > Cassandra/Datastax/ScyllaDB
> > >> > > > interfaces.
> > >> > > >
> > >> > > > I'd love to hear thoughts from those closer to this space both
> as
> > a
> > >> > user
> > >> > > > and developer so we can make good next steps.
> > >> > > >
> > >> > > > Thanks
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

At the very least we should upgrade to Cassandra 3.11.6:
https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt

On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess  wrote:

> If the community agrees to get rid of Cassandra 3 that'll save me effort
> on the refactor after I add Cassandra 4 :) Otherwise those
> vulnerabilities would only be in a "new" Cassandra 3 services NAR that
> would not be included in the convenience binary.
>
> On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:
>
>> Mike, Matt,
>>
>> Happy to hear you both have active efforts or are interested in doing so.
>> Can you help me understand more specifically what that means for the
>> current set of components?
>>
>> The CVE hits are concerning and long standing.  Supporting Cassandra 3
>> implies the current set of dependencies would remain too right?
>>
>> Is the current set of components we have ones we want to retain?  We
>> certainly need Cassandra components - but are the ones we have now the
>> right ones?
>>
>> Thanks
>> Joe
>>
>> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
>> wrote:
>>
>> > I'm actively working this, I pushed my branch up in case anyone wants to
>> > take a look [1]. The idea is to abstract the Cassandra API "up a couple
>> > levels" and provide implementations for Cassandra 3, 4, and eventually
>> 5.
>> > For JDBC-like interfaces this is a PITA because of the API (Statement,
>> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we
>> can
>> > find a common pattern for abstracting the third-party library
>> > implementation and API from the NiFi component (Processor,
>> > ControllerService, etc.) API. I think we're doing something similar for
>> > Kafka?
>> >
>> > Regards,
>> > Matt
>> >
>> > [1] https://github.com/mattyb149/nifi/tree/cassy4
>> >
>> >
>> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
>> > wrote:
>> >
>> > > That’s been on my todo list for a little while but things kept coming
>> up.
>> > > I think I could get started on that now. Based on my initial research
>> it
>> > > appears that scylla uses the exact same api as datastax so supporting
>> > both
>> > > in a cql bundle should theoretically be fairly easy.
>> > >
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
>> > > >
>> > > > Team,
>> > > >
>> > > > Cassandra remains a really important system to be able to send data
>> to.
>> > > > However, it seems like we've not maintained these well.  We have
>> what
>> > > > appears to be at least a full generation behind on client versions
>> (we
>> > > are
>> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
>> > > shortly).
>> > > >
>> > > > We have components to send data, query data, and use Cassandra as a
>> > cache
>> > > > store.  We have older mechanisms for json/avro and publish
>> mechanisms
>> > for
>> > > > records.
>> > > >
>> > > > The libraries we do have depend on outdated versions of Guava and
>> > result
>> > > in
>> > > > many CVE hits.
>> > > >
>> > > > I am inclined to think we should deprecate the 1.x components and
>> > remove
>> > > > them as-is from the 2.x line.  Then re-introduce them with record
>> only
>> > > > interfaces and built against the latest stable
>> > > Cassandra/Datastax/ScyllaDB
>> > > > interfaces.
>> > > >
>> > > > I'd love to hear thoughts from those closer to this space both as a
>> > user
>> > > > and developer so we can make good next steps.
>> > > >
>> > > > Thanks
>> > >
>> >
>>
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

If the community agrees to get rid of Cassandra 3 that'll save me effort on
the refactor after I add Cassandra 4 :) Otherwise those
vulnerabilities would only be in a "new" Cassandra 3 services NAR that
would not be included in the convenience binary.

On Fri, Mar 15, 2024 at 1:28 PM Joe Witt  wrote:

> Mike, Matt,
>
> Happy to hear you both have active efforts or are interested in doing so.
> Can you help me understand more specifically what that means for the
> current set of components?
>
> The CVE hits are concerning and long standing.  Supporting Cassandra 3
> implies the current set of dependencies would remain too right?
>
> Is the current set of components we have ones we want to retain?  We
> certainly need Cassandra components - but are the ones we have now the
> right ones?
>
> Thanks
> Joe
>
> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess 
> wrote:
>
> > I'm actively working this, I pushed my branch up in case anyone wants to
> > take a look [1]. The idea is to abstract the Cassandra API "up a couple
> > levels" and provide implementations for Cassandra 3, 4, and eventually 5.
> > For JDBC-like interfaces this is a PITA because of the API (Statement,
> > PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
> > find a common pattern for abstracting the third-party library
> > implementation and API from the NiFi component (Processor,
> > ControllerService, etc.) API. I think we're doing something similar for
> > Kafka?
> >
> > Regards,
> > Matt
> >
> > [1] https://github.com/mattyb149/nifi/tree/cassy4
> >
> >
> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen 
> > wrote:
> >
> > > That’s been on my todo list for a little while but things kept coming
> up.
> > > I think I could get started on that now. Based on my initial research
> it
> > > appears that scylla uses the exact same api as datastax so supporting
> > both
> > > in a cql bundle should theoretically be fairly easy.
> > >
> > >
> > > Sent from my iPhone
> > >
> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> > > >
> > > > Team,
> > > >
> > > > Cassandra remains a really important system to be able to send data
> to.
> > > > However, it seems like we've not maintained these well.  We have what
> > > > appears to be at least a full generation behind on client versions
> (we
> > > are
> > > > on 3x vs 4x which is the latest stable with 5x apparently coming
> > > shortly).
> > > >
> > > > We have components to send data, query data, and use Cassandra as a
> > cache
> > > > store.  We have older mechanisms for json/avro and publish mechanisms
> > for
> > > > records.
> > > >
> > > > The libraries we do have depend on outdated versions of Guava and
> > result
> > > in
> > > > many CVE hits.
> > > >
> > > > I am inclined to think we should deprecate the 1.x components and
> > remove
> > > > them as-is from the 2.x line.  Then re-introduce them with record
> only
> > > > interfaces and built against the latest stable
> > > Cassandra/Datastax/ScyllaDB
> > > > interfaces.
> > > >
> > > > I'd love to hear thoughts from those closer to this space both as a
> > user
> > > > and developer so we can make good next steps.
> > > >
> > > > Thanks
> > >
> >
>

Re: [discuss] What to do with the Cassandra components

[Matt Burgess]

I'm actively working this, I pushed my branch up in case anyone wants to
take a look [1]. The idea is to abstract the Cassandra API "up a couple
levels" and provide implementations for Cassandra 3, 4, and eventually 5.
For JDBC-like interfaces this is a PITA because of the API (Statement,
PreparedStatement, BoundStatement, ResultSet, etc.) but I'm hoping we can
find a common pattern for abstracting the third-party library
implementation and API from the NiFi component (Processor,
ControllerService, etc.) API. I think we're doing something similar for
Kafka?

Regards,
Matt

[1] https://github.com/mattyb149/nifi/tree/cassy4


On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen  wrote:

> That’s been on my todo list for a little while but things kept coming up.
> I think I could get started on that now. Based on my initial research it
> appears that scylla uses the exact same api as datastax so supporting both
> in a cql bundle should theoretically be fairly easy.
>
>
> Sent from my iPhone
>
> > On Mar 14, 2024, at 6:18 PM, Joe Witt  wrote:
> >
> > Team,
> >
> > Cassandra remains a really important system to be able to send data to.
> > However, it seems like we've not maintained these well.  We have what
> > appears to be at least a full generation behind on client versions (we
> are
> > on 3x vs 4x which is the latest stable with 5x apparently coming
> shortly).
> >
> > We have components to send data, query data, and use Cassandra as a cache
> > store.  We have older mechanisms for json/avro and publish mechanisms for
> > records.
> >
> > The libraries we do have depend on outdated versions of Guava and result
> in
> > many CVE hits.
> >
> > I am inclined to think we should deprecate the 1.x components and remove
> > them as-is from the 2.x line.  Then re-introduce them with record only
> > interfaces and built against the latest stable
> Cassandra/Datastax/ScyllaDB
> > interfaces.
> >
> > I'd love to hear thoughts from those closer to this space both as a user
> > and developer so we can make good next steps.
> >
> > Thanks
>

Re: DB Dynamic Connection

[Matt Burgess]

Eduardo,

It doesn't sound like DBCPConnectionPoolLookup will work for you because of
all the different connection strings. I don't know if there's a good reason
why we couldn't create the BasicDataSource when getConnection() is called,
passing in a Map of FlowFile attributes (that's how the Lookup version
works). One issue I do see is with "churn" if we're recreating the data
source each time. At that point it's not pooling connections. I suppose you
could have an internal cache of data sources but it would have to be
bounded and/or configurable and have a least-recently-used (LRU) eviction
strategy.

DBCPService is the name of the controller service interface that the
database processors use, but that's a misnomer since the API doesn't
mention pooling specifically. Instead you could have an implementation that
uses a cache vs a pooling approach. But Apache DBCP does handle a lot of
the management (validation, eviction, idle timeouts, etc.)  so unless
there's no way to avoid the potential memory/performance issues (like
having 50+ controller services in a PG) you could try to wrangle smaller
pools per data source and cache those if that's ok for your use case.

My two cents,
Matt

On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes 
wrote:

> Hi Everybody!
>
> I'm thinking about make a generic ingestor with Apache NiFi but I found
> some difficulties because of the DataBase Connection Pool controller. It
> doesn't accept flowfiles parameters for its properties, specially
> connection string, username and password (for security reasons, some
> sensitive parameter name instead password itself).
>
> This is important because, as a generic ingestor, I might have hundreds of
> different connection strings, and I had a lot of problems when I tried to
> put 50 DBCP controllers in a Process Group.
>
> I wouldn't like to create a flow for each ingestion, but one flow for each
> database vendor.
>
> Does anyone have any suggestions on how I can achieve this? Would it be
> easy to create a parameterized DBCP controller? (That I could do it myself)
>
> Best regards.
>
> Eduardo Fontes
> Data Eng / System Analyst Sr.
>

Re: UpdateAttribute Failure Relationship

[Matt Burgess]

Mike's option #2 seems solid but would take a lot of work and there will
always be inputs we don't account for. I support that work but in code
sometimes we just do a "catch(Throwable)" just so it doesn't blow up. What
about a subjectless "try" or "trycatch" function you can wrap around your
whole expression? If no exception is thrown, the evaluated value will be
returned but if one is thrown, you can provide some alternate value that
you can check downstream. As this is optional it would retain the current
behavior unless you use it, and then it takes the place of all those
ifElse(isXYZValid()) calls we'd need throughout the expression.

Regards,
Matt


On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord  wrote:

> IMO... UpdateAttribute has been around since the beginning of time, I can't
> see adding a failure relationship. At the same time I understand the want
> for such exceptions to be handled more gracefully rather than rolling back
> indefinitely.
> I'd vote in favor of considering Moser's option #2... and being able to
> implement an "if this then that" logic within your flow.
>
> Also just thinking... for every UA failure you have to consider a good
> failure-management strategy, which MIGHT add a lot of noise to the flow.
> Something that might otherwise easily be identified in a downstream
> component and/or database/etc.
>
> My 2 cents **
> Phil
>
>
>
>
>
> On Wed, Feb 7, 2024 at 5:18 PM Adam Taft  wrote:
>
> > Or better, the failure relationship just doesn't even exist until the
> > property "Has Failure Relationship" is set to True.  This involves
> updating
> > UpdateAttribute to have dynamic relationships (the failure relationships
> > appearing on true), which isn't hard to do in processor code.
> >
> > This has the advantage of being backwards compatible for existing users
> and
> > allows the failure relationship to exist for new configurations.
> Obviously
> > the processor would need an update to catch Expression Language
> exceptions
> > and then route conditionally to failure.
> >
> > Just thinking out loud.
> > /Adam
> >
> >
> >
> > On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com 
> > wrote:
> >
> > > Hi Mike,
> > >
> > > How about the option of introducing a new property that decides whether
> > to
> > > route to the 'failure' relationship in the event of an error?
> > > If this property is set to false, then the 'failure' relationship is
> > > automatically set to 'terminate' (since nothing is routed there
> anyway).
> > >
> > > Then everyone can decide whether and where they want to use this new
> > > feature or not.
> > > All other options would still be possible with such a solution.
> > >
> > > -- Uwe
> > >
> > > > Am 07.02.2024 um 22:15 schrieb Michael Moser :
> > > >
> > > > Hi Dan,
> > > >
> > > > This has been discussed in the past, as you found with those two Jira
> > > > tickets.  Personally, I'm still not sure whether a new failure
> > > relationship
> > > > on UpdateAttribute in 2.0 is a good approach.  I have heard from some
> > > > dataflow managers that would not want to go through their entire
> graph
> > > when
> > > > upgrading to 2.0 and update every UpdateAttribute configuration.
> > > >
> > > > I have heard some alternatives to a 'failure' relationship that I
> would
> > > > like to share as options.
> > > >
> > > > 1) Add a new property to UpdateAttribute that controls whether a
> > flowfile
> > > > that causes an expression language exception either yields and rolls
> > > back,
> > > > or silently fails to update the attribute and sends the flowfile to
> > > > success.  I personally don't like this, because the use case for
> > "silent
> > > > failure" seems really like a rarely needed edge case.
> > > >
> > > > 2) Identify all expression language methods that can throw an
> exception
> > > and
> > > > document that fact in the Expression Language Guide (some methods
> > already
> > > > mention they can throw an "exception bulletin").  Then implement new
> > > > expression methods to check if an expression could fail, and use that
> > in
> > > > UpdateAttribute advanced rules.  For example, if the format() and
> > > > formatInstant() methods can fail on a negative number, we create a
> new
> > > > method such as isValidMilliseconds().  This already exists for some
> > > cases,
> > > > such as isJson() which can do a quick check of some value before
> > calling
> > > > jsonPathDelete() on it.
> > > >
> > > > I'm curious to hear more thoughts on this.
> > > >
> > > > -- Mike
> > > >
> > > >
> > > >
> > > >> On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:
> > > >>
> > > >> My team is requesting a failure relationship for UpdateAttribute as
> > > seen in
> > > >> NIFI-5448  and
> > > NIFI-6344
> > > >>  as we are
> > > >> experiencing the same problem where a NIFI Expression Language is
> > > throwing
> > > >> an exception. In the PR for NIFI-5448 it was mentioned this

Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)

[Matt Burgess]

+1 (binding)

Ran full build

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 21, vendor: Oracle Corporation
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.2.1", arch: "aarch64", family: "mac"

Verified various flows. Thanks for RM'ing David!

-Matt

On Thu, Jan 25, 2024 at 9:30 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M2.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1264
>
> Git Tag: nifi-2.0.0-M2-RC4
> Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda
>
> Checksums of nifi-2.0.0-M2-source-release.zip
>
> SHA256: 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b
> SHA512:
> e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 214
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M2
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.25.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran full clean install -Pcontrib-check

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 1.8.0_372, vendor: Azul Systems, Inc.,
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.2.1", arch: "aarch64", family: "mac"

Verified CREATE TABLE for Oracle and byte[] generation in GenerateRecord,
ran various flows. Thanks for RM'ing Pierre!


On Thu, Jan 25, 2024 at 1:53 PM Pierre Villard 
wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.25.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.25.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1263
>
> Git Tag: nifi-1.25.0-RC1
> Git Commit ID: 6ecc398d3f92425447e43242af4992757e25b3c5
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/6ecc398d3f92425447e43242af4992757e25b3c5
>
> Checksums of nifi-1.25.0-source-release.zip
>
> SHA256: 6a14b35bf5beb4ae3fcf76df8d09676e61c6bc309a97dc6785eae84b7cbaef78
> SHA512:
>
> 1ffb2586ce9591ce2c5aa39fdec43a89ffd29081a31d51dc95dd953cb7f94584d0a0171bb1ba7c9495f431aec3770d324dbabb319b01bb6fdce5a0a00426fffa
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 103
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353860
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.25.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.25.0
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.24.0 (RC5)

[Matt Burgess]

+1 (binding)
Ran through release helper (verifying keys and hashes and such) and
tested with a handful of different flows. Thanks for seeing this
through Pierre!

- Matt

On Thu, Nov 23, 2023 at 10:13 AM Pierre Villard
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.24.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1251
>
> Git Tag: nifi-1.24.0-RC5
> Git Commit ID: 5241f434829ca46a26a475600ef7c00e1e271e37
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/5241f434829ca46a26a475600ef7c00e1e271e37
>
> Checksums of nifi-1.24.0-source-release.zip
>
> SHA256: 715eb61cdc017a5f7ba4d845ae962fdf83d389829db2a8948be14f99f2cc8272
> SHA512:
> 574147002b905ef64447edec0c7308f4fc3a97b3c7f01edca05464b2b418bcd3922f956d093736014443eec88ceba36af81df37398c5fe23a1288235b79d7306
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 285
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353443
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.24.0
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, ran NiFi standalone with Java 8, tested
various flows and components.  Thanks for RM'ing Pierre!

On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.24.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1241
>
> Git Tag: nifi-1.24.0-RC3
> Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91
>
> Checksums of nifi-1.24.0-source-release.zip
>
> SHA256: 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a
> SHA512:
> 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/pvillard.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 280
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353443
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [] +1 Release this package as nifi-1.24.0
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: Removing JRuby?

[Matt Burgess]

I believe it is because in both ExecuteScript and ExecuteGroovyScript
you can do "regular" groovy, but EGS has extra built-ins such as easy
access to controller services, Groovy SQL stuff, etc, and we could
keep building it out. But IMO we'd have to port the rest of the
scripted components (ScriptedReader/Writer, etc.) over to the Groovy
bundle and make sure there's a drop-in replacement in the Python stuff
before we'd want to deprecate the scripted bundle.

On the JRuby front, is that something you use actively? This question
is for you and the entire community of course.

Regards,
Matt

On Mon, Nov 6, 2023 at 7:12 AM Mike Thomsen  wrote:
>
> If we deprecate ExecuteScript, I think we need to have groovyx be ready to
> function as a drop-in replacement if it's not there already.
>
> On Sun, Nov 5, 2023 at 9:21 PM Matt Burgess  wrote:
>
> > IIRC the removal of these engines was mostly due to lack of use or at
> > least the perception thereof. If JRuby is being used by the community
> > actively, I'm happy to revisit that discussion. Luaj's JSR-223
> > interface left something to be desired, but JRuby just needed a system
> > variable set or something like that.
> >
> > For the groovyx bundle, because it is Groovy-specific I tend to think
> > we could make better use of that than ExecuteScript, especially if we
> > do get rid of all the engines. We have a Groovy-specific processor, a
> > "real" Python SDK, and no more Nashorn. Perhaps we move all the
> > scripted components to the Groovy bundle, although I believe some
> > folks still make use of Jython for these. Of course if we reinstate
> > JRuby for ExecuteScript it's probably best to keep things the way they
> > are, or create a jruby bundle. The original scripting bundle was
> > aiming to support several engines, but if it turns out only one or two
> > will be useful, it may not be worth shoehorning all that JSR-223 logic
> > when engine-specific components could be simpler, more easily
> > maintained, and allow for the idioms of the language to be better used
> > (as is done in the groovyx bundle).
> >
> > Just my two cents, looking forward to everyone's thoughts!
> >
> > - Matt
> >
> > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen 
> > wrote:
> > >
> > > https://issues.apache.org/jira/browse/NIFI-11646
> > >
> > > I get the removal of Lua, but not the removal of JRuby. It's a clean
> > > reimplementation of Ruby native to the JVM and AFAICT is pound for pound
> > as
> > > actively maintained as Groovy.
> > >
> > > Also, at this point, does it make sense to even keep the groovyx bundle
> > > rather than deprecate it for 2.X?
> >

Re: Removing JRuby?

[Matt Burgess]

IIRC the removal of these engines was mostly due to lack of use or at
least the perception thereof. If JRuby is being used by the community
actively, I'm happy to revisit that discussion. Luaj's JSR-223
interface left something to be desired, but JRuby just needed a system
variable set or something like that.

For the groovyx bundle, because it is Groovy-specific I tend to think
we could make better use of that than ExecuteScript, especially if we
do get rid of all the engines. We have a Groovy-specific processor, a
"real" Python SDK, and no more Nashorn. Perhaps we move all the
scripted components to the Groovy bundle, although I believe some
folks still make use of Jython for these. Of course if we reinstate
JRuby for ExecuteScript it's probably best to keep things the way they
are, or create a jruby bundle. The original scripting bundle was
aiming to support several engines, but if it turns out only one or two
will be useful, it may not be worth shoehorning all that JSR-223 logic
when engine-specific components could be simpler, more easily
maintained, and allow for the idioms of the language to be better used
(as is done in the groovyx bundle).

Just my two cents, looking forward to everyone's thoughts!

- Matt

On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen  wrote:
>
> https://issues.apache.org/jira/browse/NIFI-11646
>
> I get the removal of Lua, but not the removal of JRuby. It's a clean
> reimplementation of Ruby native to the JVM and AFAICT is pound for pound as
> actively maintained as Groovy.
>
> Also, at this point, does it make sense to even keep the groovyx bundle
> rather than deprecate it for 2.X?

Re: Provenance events without FlowFiles?

[Matt Burgess]

I won't speak for Mark but I took that to mean that you pass in a
generic Java (or custom) File object (no matter what or where it
points to) so the ProvenanceReporter can use the appropriate
information from that object (like size) to populate the provenance
event, not any real reference to the file itself. There may also be
security concerns with including information about the source
location: that needs to be determined as well.

On Thu, Oct 26, 2023 at 2:59 PM Pierre Villard
 wrote:
>
> >
> > Rather, I’d say it's an UPLOAD_FILE event. So I’d lean more toward an
> > uploadFile() method on ProvenanceReporter that takes as an argument a
> > `File` (as well as a FlowFile). The size would come from the File itself,
> > and the event would convey the information about the local file that was
> > uploaded - probably in the Event Details.
> >
>
> Would that mean that for the "bytes transferred" graph in Status History,
> we would combine SEND and UPLOAD_FILE events? Because, right now, it's not
> showing anything which is confusing.
>
> Also, I'm not sure about the 'File' object. While we have only the local
> file system as an option today for the File Resource Service, I'd expect
> additional implementations such as implementations for CSPs. So we could
> have the case where PutAzureBlobStorage is used with a FileResourceService
> for Google Cloud Storage for example (in order to improve efficiency of
> data movement between cloud providers) and, in this case, not sure we would
> have a 'File' object. Unless you're talking about a more generic File
> object here and not the object for local file system.
>
> Le jeu. 26 oct. 2023 à 09:16, Matt Burgess  a écrit :
>
> > AFAIK it is fine and appropriate to issue multiple provenance events
> > for a single FlowFile. In the case for PutAzureBlobStorage uploading a
> > file to Azure, it is the incoming FlowFile that triggers the upload.
> > Before reporting a provenance event, attributes are added to the
> > FlowFile, so that "version" of the FlowFile can be the one used to
> > report a SEND event. I have done this to said processor as part of a
> > large refactor/improvement of the provenance capability:
> >
> > session.getProvenanceReporter().send(flowFile,
> > blob.getSnapshotQualifiedUri().toString(), transferMillis,
> > REL_SUCCESS);
> >
> > Having said that, to Mark's point it's probably better to have a
> > separate UPLOAD_FILE event, I can change that in my code.
> >
> > I added a couple like this to similar processors, such as
> > TriggerHiveMetastoreEvent:
> >
> > session.getProvenanceReporter().invokeRemoteProcess(flowFile,
> > hiveMetastoreUrl, REL_SUCCESS);
> >
> > I am still working on this, I need to write up a Jira with a thorough
> > treatment of the material and eventually get a PR up for review.
> >
> > Regards,
> > Matt
> >
> > On Thu, Oct 26, 2023 at 12:02 PM Mark Payne  wrote:
> > >
> > > Lehel,
> > >
> > > I don’t believe we should be trying to create a “Mock FlowFile.” I am ok
> > with an update to the ProvenanceReporter interface. But I don’t think it
> > should accept a “size” parameter. Rather, I think this is a completely
> > different type of event that is occurring. This is not a “send” in that
> > it’s not sending the contents of the FlowFile to a remote system. Rather,
> > I’d say it's an UPLOAD_FILE event. So I’d lean more toward an uploadFile()
> > method on ProvenanceReporter that takes as an argument a `File` (as well as
> > a FlowFile). The size would come from the File itself, and the event would
> > convey the information about the local file that was uploaded - probably in
> > the Event Details.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > > > On Oct 26, 2023, at 10:36 AM, Lehel Boér  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I would like to address a particular scenario that has recently come
> > to my attention regarding the use of the PutAzureBlobStorage processor with
> > the FileResourceService.
> > > >
> > > > When the PutAzureBlobStorage processor is used with the
> > FileResourceService, it currently uploads a file from the user's local
> > filesystem to Azure, but it does not create a FlowFile. Instead, it
> > utilizes the incoming FlowFile solely to send a provenance event. In this
> > case the size of the provenance event is the incoming FlowFile's size
> > instead of the uploaded one.
> >

Re: Provenance events without FlowFiles?

[Matt Burgess]

AFAIK it is fine and appropriate to issue multiple provenance events
for a single FlowFile. In the case for PutAzureBlobStorage uploading a
file to Azure, it is the incoming FlowFile that triggers the upload.
Before reporting a provenance event, attributes are added to the
FlowFile, so that "version" of the FlowFile can be the one used to
report a SEND event. I have done this to said processor as part of a
large refactor/improvement of the provenance capability:

session.getProvenanceReporter().send(flowFile,
blob.getSnapshotQualifiedUri().toString(), transferMillis,
REL_SUCCESS);

Having said that, to Mark's point it's probably better to have a
separate UPLOAD_FILE event, I can change that in my code.

I added a couple like this to similar processors, such as
TriggerHiveMetastoreEvent:

session.getProvenanceReporter().invokeRemoteProcess(flowFile,
hiveMetastoreUrl, REL_SUCCESS);

I am still working on this, I need to write up a Jira with a thorough
treatment of the material and eventually get a PR up for review.

Regards,
Matt

On Thu, Oct 26, 2023 at 12:02 PM Mark Payne  wrote:
>
> Lehel,
>
> I don’t believe we should be trying to create a “Mock FlowFile.” I am ok with 
> an update to the ProvenanceReporter interface. But I don’t think it should 
> accept a “size” parameter. Rather, I think this is a completely different 
> type of event that is occurring. This is not a “send” in that it’s not 
> sending the contents of the FlowFile to a remote system. Rather, I’d say it's 
> an UPLOAD_FILE event. So I’d lean more toward an uploadFile() method on 
> ProvenanceReporter that takes as an argument a `File` (as well as a 
> FlowFile). The size would come from the File itself, and the event would 
> convey the information about the local file that was uploaded - probably in 
> the Event Details.
>
> Thanks
> -Mark
>
>
> > On Oct 26, 2023, at 10:36 AM, Lehel Boér  wrote:
> >
> > Hi everyone,
> >
> > I would like to address a particular scenario that has recently come to my 
> > attention regarding the use of the PutAzureBlobStorage processor with the 
> > FileResourceService.
> >
> > When the PutAzureBlobStorage processor is used with the 
> > FileResourceService, it currently uploads a file from the user's local 
> > filesystem to Azure, but it does not create a FlowFile. Instead, it 
> > utilizes the incoming FlowFile solely to send a provenance event. In this 
> > case the size of the provenance event is the incoming FlowFile's size 
> > instead of the uploaded one.
> >
> > There are potential solutions to address this issue and ensure that the 
> > provenance events are handled effectively. Two main options have been 
> > proposed:
> >
> >
> >  *   Create a Mock FlowFile: A mock FlowFile with a size matching that of 
> > the local file being uploaded could be generated. This mock FlowFile would 
> > serve as the basis for the provenance event, even though its size might not 
> > reflect the actual content.
> >
> >  *   Modify the ProvenanceReporter Interface: Alternatively, we could 
> > introduce a new method in the ProvenanceReporter interface that doesn't 
> > require a FlowFile but instead accepts a "size" parameter as an argument. 
> > This would eliminate the need for a mock FlowFile.
> >
> > The lack of a FlowFile operation in this situation creates a distinct 
> > challenge because provenance events are typically tied to FlowFiles. Still, 
> > it's important to indicate data transmission for monitoring and tracking.
> >
> > While the idea of a "size" parameter for the provenance event seems 
> > preferable, we need to carefully consider its feasibility, potential 
> > complexities, and community acceptance. The FileResourceService already 
> > deviates from NiFi's concept of using FlowFiles to hold payload data, and 
> > we must avoid further complicating the framework unless absolutely 
> > necessary.
> >
> > If you have any insights or suggestions, please feel free to reply to this 
> > email or join the discussion.
> >
> > Best Regards,
> > Lehel
>

Re: JOLTTransformRecord problem

[Matt Burgess]

For some reason I don't have the original thread, I must've
inadvertently deleted it. IIRC your example input was a single JSON
object and I said if that were the case you could use
JoltTransformJSON instead. However if that is NOT the case (which is
your point c above) then you have a couple of options:

1) To continue using JoltTransformJSON with a top-level array you need
to surround your spec with
"*": {  }
and will need to use "[&1]." in front of all the output fields. This
will output the transformation to the same index in the array as it
was in the input.

2) One major difference between JoltTransformJSON and
JoltTransformRecord is that the former reads the entire thing into
memory, where JoltTransformRecord reads one record at a time. So your
current spec should work with JoltTransformRecord, but if you are
still getting the original error, can you provide (or re-provide if
you already did, I can't find the original thread) sample input that
represents the "real" input (not just one JSON object if you'll be
getting multiple records or if the top-level is an array even with
only one object in it), desired output, and the error with full stack
trace? I'm guessing there is an inference error with complex fields,
if you know what the input and output schemas are you can provide them
to the Reader and Writer respectively instead of using "Infer schema".
That should work around any inference issues.
With NiFi 1.23.2 you also have the new ExtractRecordSchema processor,
you can try that before your JoltTransformRecord processor with the
same reader and see what it comes out with as a schema. Then you can
manually alter it to better match your data and use that in the Reader
specified in JoltTransformRecord.

Regards,
Matt

On Tue, Oct 10, 2023 at 4:47 PM Mark Woodcock  wrote:
>
> Chris,
>
> 1) I've upgraded to 1.23.2  (which appears to be the latest and greatest).
>
> 2) I've tested the JoltTransformRecord with
> a) JsonTreeReader w/ InferredSchema
> b) JsonRecordSetWriter w/ InheritsSchema
> c) a GetFile processor which grabs a text file with the various bits of
> test data
>
> It appears that your suspicions are correct:
> i) if I test with just that single record as the entire content of the
> file, the processor is successful.
> ii) if I test with multiple records, none of which have the complicated
> inner field, all is successful.
> c) if I test with multiple records, where at least one has the complicated
> inner field, I get the earlier noted error.
>
> IOW, yep, it only happens with *more* data.
>
> bummer,
>
> mew
>
> On Tue, Oct 10, 2023 at 2:43 PM Chris Sampson
>  wrote:
>
> > Using your example (single JSON Object and Jolt Spec) seems to work fine
> > in both JoltTransformJSON and JoltTransformRecord when run on the current
> > main branch (which is for the upcoming 2.0.0 release).
> >
> > To test, I setup a GenerateFlowFile processor to output the example JSON
> > you gave, then sent that through both of the Jolt processors using a
> > JsonTreeReader with “Inferred Schema”, and a JsonRecordSetWriter that
> > “Inherits Schema” for the Record processor.
> >
> > If you run *just* your example from this email chain through the Jolt
> > processors on the version of NiFi you’re using, do you see the errors you
> > mention, or does that only happen with more data?
> >
> >
> > Cheers,
> >
> > ---
> > Chris Sampson
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > > On 10 Oct 2023, at 15:45, Mark Woodcock 
> > wrote:
> > >
> > > H,
> > >
> > > One small problem:  While JOLTTransformJSON is quite lovely (a) it has a
> > > great "advanced" interface that allows one to test their spec and json
> > > inputs and (b) it actually works for the cases that I noted...it treats
> > the
> > > input a single blob of JSON.  Unfortunately, my input files are
> > collections
> > > of JSON records (which--less the noted problem--JOLTTransformRecord does
> > > quite nicely with)--that's literally how they arrive, not the result of
> > me
> > > formatting them at all.
> > >
> > > Is there a way to get JTJ to treat the input as records?
> > > Does 1.22 or 1.23 have the fix for JTR?
> > >
> > > thx,
> > >
> > > mew
> > >
> > >
> > > On Mon, Oct 9, 2023 at 3:21 PM Mark Woodcock  wrote:
> > >
> > >> confirmed:  version 1.21.
> > >> How recent is the fix?
> > >>
> > >> thx,
> > >>
> > >> mew
> > >>
> > >>
> > >> On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock 
> > wrote:
> > >>
> > >>> Matt,
> > >>>
> > >>> Unfortunately (at home now) the details are all at work at the moment,
> > >>> but I know that I didn't start this work until April (at the
> > earliest), so
> > >>> I'm surely using at least 1.21; is the fix more recent than that?
> >  {If so,
> > >>> perhaps there is a bug.}
> > >>>
> > >>> Fortunately, yea, JSON out is the intent; I need the data to be in that
> > >>> format to set up a subsequent transform to AVRO, so it seems there are
> > two
> > >>> possible ways out (depending on which version I'm run

Re: JOLTTransformRecord problem

[Matt Burgess]

mew,

What version of NiFi are you using? We may have fixed the bug in newer
versions but if you are using the latest, this is bug. We shouldn't
infer CHOICE[STRING,RECORD] when the only entry is a RECORD.

Since you only have one top-level record in your FlowFile you can use
JoltTransformJSON instead as a workaround. If you want JSON output
then it's a solution and not a workaround :) The error here isn't
about JOLT but about JoltTransformRecord using NiFi Records to apply
the transformation to rather than the raw JSON (the latter is what
JoltTransformJSON does).

Regards,
Matt

Re: How to Connect Apache NiFi to PostgreSql and using nifi insert records from csv file to database

[Matt Burgess]

Gennady,

That is a great explanation by Lars, also note that by default GetFile
will remove the CSV from your system. If you want to keep it but only
fetch it once, use ListFile -> FetchFile rather than GetFile.

Regards,
Matt

On Fri, Oct 6, 2023 at 12:44 PM Lars Winderling
 wrote:
>
> Hi Gennady,
>
> you should set up a jdbc connection pool controller service and connect it to 
> your postgres instance. Then you will need a csv record reader controller 
> service, configured with the correct csv format. Eventually, put an 
> ExecuteSQLRecord processor onto the canvas. It should use the connection pool 
> and the csv reader. Check the remaining properties of the processor for more 
> config options. Cf. also 
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html.
> Of course, you will need some processor to get the csv files in the first 
> place. For local files, use GetFile. So that depends on the source of your 
> files. When the correct processor is found, connect it to the sql processor 
> (success relationship).
> When using GetFile, you will need to put a GenerateFlowfile processor in 
> front of it, and connect them. When starting it, it will produce a flowfile. 
> The GetFile proc will obtain the data, the sql processor will insert it into 
> the db.
> Make sure to configure error relationships as well by e.g. terminating them 
> (just to get you started).
> I hope, you will manage to go on from there. The docs are pretty extensive, 
> as is the user mailing list archive.
>
> Best, Lars
>
> On 6 October 2023 18:06:56 CEST, Gennady Kondratov 
>  wrote:
> >How to Connect Apache NiFi to PostgreSql and using nifi insert records from 
> >csv file to database/
> >i'd like to ask you how to get it or where looking for about it
> >

Re: NIFI Jira ticket not updated with link to NIFI PR code in Github

[Matt Burgess]

Dan,

That happens sometimes, it can take a while for the link to be added
to the Jira, but usually it does happen eventually. Sometimes I'll add
the link manually then remove it when the system catches up and adds
the link to the Jira.

Regards,
Matt

On Thu, Sep 7, 2023 at 10:49 AM Dan S  wrote:
>
> I recently submitted a PR for
> https://issues.apache.org/jira/browse/NIFI-11197 at
> https://github.com/apache/nifi/pull/7665 but the Jira ticket has not been
> updated with the link to the PR as it usually is. Please advise. Thanks!

Re: PGVector and Database Driver

[Matt Burgess]

Maybe this [1]? Perhaps you have to call unwrap() yourself in this
case. IIRC you don't have access to the DataSource but you can check
it directly on the connection.

Regards,
Matt

[1] 
https://stackoverflow.com/questions/36986653/cast-java-sql-connection-to-pgconnection

On Thu, Aug 31, 2023 at 8:15 PM u...@moosheimer.com  wrote:
>
> Mark & Matt,
>
> Thanks for the quick help. I really appreciate it.
>
> PGvector.addVectorType(con) returns the following:
> *java.sql.SQLException: Cannot unwrap to org.postgresql.PGConnection*
>
> Could this be a connection pool issue?
>
> Interestingly, I didn't call addVectorType() at all in my test java code
> and it still works?!
> I'll have to check again ... maybe I'm not seeing it correctly anymore.
> It is already 2:05 a.m. here.
>
>
> Regards,
> Uwe
>
>
> java.sql.SQLException: Cannot unwrap to org.postgresql.PGConnection
>
> On 31.08.23 18:53, Matt Burgess wrote:
> > This means the JDBC driver you're using does not support the use of
> > the two-argument setObject() call when the object is a PGVector. Did
> > you register the Vector type by calling:
> >
> > PGvector.addVectorType(conn);
> >
> > The documentation [1] says that the two-argument setObject() should
> > work if you have registered the Vector type.
> >
> > Regards,
> > Matt
> >
> > [1]https://github.com/pgvector/pgvector-java
> >
> > On Thu, Aug 31, 2023 at 12:01 PM Mark Payne  wrote:
> >> Hey Uwe,
> >>
> >> The DBCPConnectionPool returns a java.sql.Connection. From that you’d 
> >> create a Statement. So I’m a little confused when you say that you’ve got 
> >> it working in Pure JDBC but not with NiFi, as the class returned IS pure 
> >> JDBC. Perhaps you can share a code snippet of what you’re doing in the 
> >> “Pure JDBC” route that is working versus what you’re doing in the NiFi 
> >> processor that’s not working?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Aug 31, 2023, at 10:58 AM,u...@moosheimer.com  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am currently writing a processor to write OpenAI embeddings to Postgres.
> >>> I am using DBCPConnectionPool for this.
> >>> I use Maven to integrate PGVector (https://github.com/pgvector/pgvector).
> >>>
> >>> With pure JDBC this works fine. With the database classes from NiFi I get 
> >>> the error:
> >>> *Cannot infer the SQL type to use for an instance of 
> >>> com.pgvector.PGvector. Use setObject() with an explicit Types value to 
> >>> specify the type to use.*
> >>>
> >>> I use -> setObject (5, new PGvector(embeddingArray)).
> >>> embeddingArray is defined as: float[] embeddingArray
> >>>
> >>> Of course I know why I get the error from NiFi and not from the JDBC 
> >>> driver, but unfortunately this knowledge does not help me.
> >>>
> >>> Can anyone tell me what SQLType I need to specify for this?
> >>> I have searched the internet and the NiFi sources on GitHub for several 
> >>> hours now and have found nothing.
> >>>
> >>> One option would be to use native JDBC and ignore the ConnectionPool. But 
> >>> that would be a very bad style in my opinion.
> >>> Perhaps there is a better solution?
> >>>
> >>> Any help, especially from Matt B., is appreciated as I'm at a loss.
> >>> Thanks guys.

Re: PGVector and Database Driver

[Matt Burgess]

This means the JDBC driver you're using does not support the use of
the two-argument setObject() call when the object is a PGVector. Did
you register the Vector type by calling:

PGvector.addVectorType(conn);

The documentation [1] says that the two-argument setObject() should
work if you have registered the Vector type.

Regards,
Matt

[1] https://github.com/pgvector/pgvector-java

On Thu, Aug 31, 2023 at 12:01 PM Mark Payne  wrote:
>
> Hey Uwe,
>
> The DBCPConnectionPool returns a java.sql.Connection. From that you’d create 
> a Statement. So I’m a little confused when you say that you’ve got it working 
> in Pure JDBC but not with NiFi, as the class returned IS pure JDBC. Perhaps 
> you can share a code snippet of what you’re doing in the “Pure JDBC” route 
> that is working versus what you’re doing in the NiFi processor that’s not 
> working?
>
> Thanks
> -Mark
>
>
> > On Aug 31, 2023, at 10:58 AM, u...@moosheimer.com wrote:
> >
> > Hi,
> >
> > I am currently writing a processor to write OpenAI embeddings to Postgres.
> > I am using DBCPConnectionPool for this.
> > I use Maven to integrate PGVector (https://github.com/pgvector/pgvector).
> >
> > With pure JDBC this works fine. With the database classes from NiFi I get 
> > the error:
> > *Cannot infer the SQL type to use for an instance of com.pgvector.PGvector. 
> > Use setObject() with an explicit Types value to specify the type to use.*
> >
> > I use -> setObject (5, new PGvector(embeddingArray)).
> > embeddingArray is defined as: float[] embeddingArray
> >
> > Of course I know why I get the error from NiFi and not from the JDBC 
> > driver, but unfortunately this knowledge does not help me.
> >
> > Can anyone tell me what SQLType I need to specify for this?
> > I have searched the internet and the NiFi sources on GitHub for several 
> > hours now and have found nothing.
> >
> > One option would be to use native JDBC and ignore the ConnectionPool. But 
> > that would be a very bad style in my opinion.
> > Perhaps there is a better solution?
> >
> > Any help, especially from Matt B., is appreciated as I'm at a loss.
> > Thanks guys.
>

Re: Jira contributor access

[Matt Burgess]

Umar,

Welcome to the NiFi project! Please use the Self-Service Portal at
https://selfserve.apache.org/jira-account.html to request access to
Jira per the current Apache rules. A PMC member will review and
approve as prudent.

Thank you,
Matt

On Sun, Aug 27, 2023 at 3:35 PM Umar Hussain  wrote:
>
> Hello NiFi Team,
> I need contributor access to NiFi Jira to assign myself to this ticket (
> https://issues.apache.org/jira/browse/NIFI-11995).
> Please provide me the access.
> Username: umar.hussain
> Email: umarhussain.w...@gmail.com
>
> Thank you
>
> --
> Regards,
>
> Umar Hussain
> Software Engineer
> https://umarhussain.xyz

Re: Apache NiFi compatibility with SQL Server 2008 R2

[Matt Burgess]

Frank,

Yes NiFi is compatible with MSSQL 2008, there's even a specific
database adapter in some processors to generate the correct SQL for it
(it's called "MSSQL2008DatabaseAdapter").

Regards,
Matt

On Thu, Aug 24, 2023 at 11:23 AM Frank Mansilla
 wrote:
>
> Dear Apache NiFi Community,
>
> I am interested in using Apache NiFi to manage data streams involving SQL
> Server 2008 R2.
> Could you confirm if NiFi is compatible with this database engine?
> I would greatly appreciate your guidance on this.
>
> Thanks and regards!
>
> Frank Mansilla
> frankmansilla1...@gmail.com
> Bolivar, Buenos Aires, Argentina

Re: FileMaker and Apache NiFi

[Matt Burgess]

Samuel,

Where do you see that DBCPConnectionPool uses JNDI? As far as I know
it just uses Apache DBCP and JDBC DataSources using the supplied
driver.

Regards,
Matt

On Wed, Aug 23, 2023 at 9:14 AM Namazi, Samuel  wrote:
>
> To whom it may concern,
>
> I am trying to connect my FileMaker database to NiFi as a data source. 
> However that ist not so easily done, as NiFi's DBCPConnectionPool (1.22.0) 
> uses JNDI, but FileMaker does not support the JNDI standard for JDBC. It 
> probably is possible to use a JDBC-ODBC Bridge or create a custom processor 
> and controller to make it work, but those options aren't really considerable 
> for me. Am I overlooking something and there is an easy solution to this ? If 
> there is something else that I can do, I would greatly appreciate your help.
>
> Kind regards
>
> Samuel Namazi

Re: [VOTE] Release Apache NiFi 1.23.2 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper and my usual flows verifying no
regressions. Thanks for RM'ing David!

On Mon, Aug 21, 2023 at 4:16 PM David Handermann
 wrote:
>
> Team,
>
> NOTE: This is a shortened vote window given the narrow scope of changes.
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.23.2.
>
> Please review the following guide for how to verify a release candidate build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.2
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1235
>
> Git Tag: nifi-1.23.2-RC1
> Git Commit ID: dec043e590f26ba2f3594f4f297dcd2b7e565ab7
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/dec043e590f26ba2f3594f4f297dcd2b7e565ab7
>
> Checksums of nifi-1.23.2-source-release.zip
>
> SHA256: cfaa374c383a316de6d9230a0786ebe7dd97c03f8903b36365faa27bf6424fac
> SHA512: 
> a63fa624532aa3f1769013d67499caf48b9771bc3515ffca4daba2dd379722c3b25bea4519b56bc647d4e05db122e7a41519a52a703a0179b431fa881dadb4cf
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved in this version: 1
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353568
>
> Release note highlights can be found on the project wiki:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.2
>
> The vote will be open for 24 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-1.23.2
> [] +0 no opinion
> [] -1 Do not release this package because...

Re: [DISCUSS] Release Apache NiFi 1.23.2 for NIFI-11971

[Matt Burgess]

+1

On Mon, Aug 21, 2023 at 9:58 AM David Handermann
 wrote:
>
> Team,
>
> Thanks for the work on release for NiFi 1.23.1 last week.
>
> As it happens, another important bug surfaced last week with Content
> Repository handling of empty files, described in NIFI-11971 [1]. This
> bug was introduced due to changes for NIFI-11670 [2] released in NiFi
> 1.23.0.
>
> This issue is serious enough that we should release a new patch
> version 1.23.2, limited to this particular fix.
>
> I plan to prepare a release candidate build and send out a
> short-duration vote thread later today.
>
> Regards,
> David Handermann
>
> [1] https://issues.apache.org/jira/browse/NIFI-11971
> [2] https://issues.apache.org/jira/browse/NIFI-11670

Re: JoltTransformJSON EL when using file

[Matt Burgess]

Matthew,

What was your second case trying to use ${now():toNumber()} ? My unit
test evaluated the expression into an integer and it worked (versus
putting the expression in quotes which would make it a string).

Regards,
Matt

On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins  wrote:
>
> Hi Matt,
>
> The tag will be ${firstname}, the spec is correct.
> (well, I also get lastname, but it's unimportant)
>
> Weird thing is this failed similarly with a file based input on the Record
> based jolt processor as well on a completely different system.
>
> Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> source with include-graph,include-media,include-rules,include-sql-reporting
>
> In the second case I was trying to get ${now():toNumber()} into a json
> record using either default or modify-overwrite-beta (tried both). The
> following success processor failed on reading schema as the literal
> ${now...} was a string not a long and the JVM refused the type cast. If I
> insert the transform directly then it still fails with a type cast problem
> from generic Object :/ (that was midnight last night, and I tossed my hands
> in the air and turned the server off)
>
> I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> postgresql. Trying to be a good nifi citizen and use record based
> processing where possible. Since I don't run Hadoop at home, it's all
> manual schemas and using postgres as the perm data store. I'm using json as
> mqtt does it intrinsically and I can jolt transform other data into json
> form and then store it easily in postgres. Well, that was the idea 😁
>
> Kr,
>
> On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:
>
> > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > worked. I noticed you referred to "attrname" in your post but your
> > spec refers to "firstname", is that a typo?
> >
> > Regards,
> > Matt
> >
> > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
> > >
> > > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > > glance at the code seems like it should work, but I'll try to
> > > reproduce it and follow up, thanks for bringing this to our attention!
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > >
> > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins 
> > wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> > into
> > > > the content.
> > > > When putting the spec directly to the processor it works fine. If I
> > have
> > > > the spec in an external file however it seems to put in the literal
> > string
> > > > ${attrname} - ie the EL didn't process.
> > > >
> > > > Have I done something incorrectly or should I be now asking for an
> > account
> > > > to log a bug?
> > > >
> > > > Sample spec run using Chain DSL:
> > > >
> > > > [{
> > > >   "operation": "shift",
> > > >   "spec": {
> > > > "@": "values"
> > > >   },
> > > >   {
> > > > "operation": "default",
> > > > "spec": {
> > > >   "firstname": "${firstname}"
> > > > }
> > > >   }
> > > > ]
> > > >
> > > > Kind regards,
> > > > Matthew
> >

Re: [VOTE] Release Apache NiFi 1.23.1 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified NIFI-11922 [1] and stored flows
in Registry.

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Java version: 17.0.7, vendor: Oracle Corporation, runtime:
/Users/mburgess/.sdkman/candidates/java/17.0.7-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"

Good work team, and thanks for RM'ing David!

[1] https://issues.apache.org/jira/browse/NIFI-11922

On Tue, Aug 15, 2023 at 2:08 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.1.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted upon and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.1/
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1234
>
> Git Tag: nifi-1.23.1-RC1
> Git Commit ID: f739e9dd2d476b3c0df5a806ff1dffd24be52916
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/f739e9dd2d476b3c0df5a806ff1dffd24be52916
>
> Checksums of nifi-1.23.1-source-release.zip:
>
> SHA256: 01f5f67a9f9703232f6fe260f6c1da73f3e3a0764b8e8464f915cfad168278e6
> SHA512:
> f8a010ad5a5dd1f71fe04e5d2bf1c6637d2d0a8a7c580a0ff4dbd76c12c2e5ab4ac43e1f5314f9fca85cebe1606bd5e7ae0a8b62f577ddc68696ebd0155d
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 51 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353541
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.1
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.23.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [DISCUSS] Prepare Apache NiFi 1.23.1 Release?

[Matt Burgess]

+1, thank you David!

On Sat, Aug 12, 2023 at 10:26 AM David Handermann
 wrote:
>
> Team,
>
> Following the release of Apache NiFi 1.23.0, several important bugs have
> been resolved in a few Processors [1], and there have been a handful of
> incremental dependency upgrades [2]. Preparing an incremental patch release
> would provide the opportunity to incorporate these bug fixes and pull in a
> selected number of dependency upgrades.
>
> I am willing to handle the Release Manager responsibilities. If there is a
> positive response to this approach, I can put together a selected list of
> issues next week.
>
> Regards,
> David Handermann
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20issuetype%20%3D%20Bug%20AND%20fixVersion%20%3D%201.24.0
>
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.24.0

Re: JoltTransformJSON EL when using file

[Matt Burgess]

Just to follow up, I added a unit test to put EL in a JOLT spec and it
worked. I noticed you referred to "attrname" in your post but your
spec refers to "firstname", is that a typo?

Regards,
Matt

On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
>
> I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> glance at the code seems like it should work, but I'll try to
> reproduce it and follow up, thanks for bringing this to our attention!
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-4957
>
> On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins  wrote:
> >
> > Hi devs,
> >
> > Using 1.23.0 I have a simple transform adding some flowfile attributes into
> > the content.
> > When putting the spec directly to the processor it works fine. If I have
> > the spec in an external file however it seems to put in the literal string
> > ${attrname} - ie the EL didn't process.
> >
> > Have I done something incorrectly or should I be now asking for an account
> > to log a bug?
> >
> > Sample spec run using Chain DSL:
> >
> > [{
> >   "operation": "shift",
> >   "spec": {
> > "@": "values"
> >   },
> >   {
> > "operation": "default",
> > "spec": {
> >   "firstname": "${firstname}"
> > }
> >   }
> > ]
> >
> > Kind regards,
> > Matthew

Re: JoltTransformJSON EL when using file

[Matt Burgess]

I added file support to JoltTransformJSON under NIFI-4957 [1], a first
glance at the code seems like it should work, but I'll try to
reproduce it and follow up, thanks for bringing this to our attention!

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-4957

On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins  wrote:
>
> Hi devs,
>
> Using 1.23.0 I have a simple transform adding some flowfile attributes into
> the content.
> When putting the spec directly to the processor it works fine. If I have
> the spec in an external file however it seems to put in the literal string
> ${attrname} - ie the EL didn't process.
>
> Have I done something incorrectly or should I be now asking for an account
> to log a bug?
>
> Sample spec run using Chain DSL:
>
> [{
>   "operation": "shift",
>   "spec": {
> "@": "values"
>   },
>   {
> "operation": "default",
> "spec": {
>   "firstname": "${firstname}"
> }
>   }
> ]
>
> Kind regards,
> Matthew

Re: Java 17 features in 2.x

[Matt Burgess]

That's a fair point and I think represents the way we want to go
forward. If I understand correctly, you're saying bug fixes meant for
both lines shouldn't need/use Java 17 features but new capabilities
for 2.0 should encourage the use of Java 17 features when prudent?

Thanks,
Matt

On Mon, Aug 7, 2023 at 8:33 PM Joe Witt  wrote:
>
> The views shared thus far are certainly reasonable but I do want to add
> a different take.
>
> The reason we want to do major releases from time to time is so that we can
> take advantage of leaps in the language and frameworks we rely on.  To that
> end it would seem unfortunate to not start aggressively taking advantage of
> that.  In particular we've been held to Java 8 standards for at least 7
> years.  I would advocate we allow and even encourage usage of Java 17
> features/syntax to help move forward.
>
> The point about backporting is important and I agree the 'easiest' way is
> if changes made to main are backportable.  Then again we don't really need
> everything to be backportable and for sure that will start happening less
> and less.  If we're talking about 'bug fixes' then it probably makes sense
> to prefer for now they avoid Java 17 assuming a given fix should target
> both 2.x and 1.x lines.  But if we're talking about new features or even
> improvements I think we should be free to move on to Java 17
> idioms/benefits.  If a contrib does this then it just won't go to the 1.x
> line.  This atrophy is natural/ok I think and lets us give the 2.x line the
> attention/growth it deserves.
>
> Thanks
>
> On Mon, Aug 7, 2023 at 2:19 PM David Handermann 
> wrote:
>
> > Mike,
> >
> > Thanks for raising the question. Following Matt's comments, I recommend
> > minimizing use of Java 17 features to make it easier to backport changes
> > for now.
> >
> > Incremental adjustments can be done when backporting, but it requires the
> > author and reviewer to pay careful attention since the GitHub automated
> > builds for the main branch run on Java 17.
> >
> > As Matt recommended, the alternative is to provide separate pull requests
> > for the main and support branches.
> >
> > We already have a few Java 11 and 17 references on the main branch for
> > things like List.of(), and most of these are easy to adjust when
> > backporting, but they do require careful attention.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Aug 7, 2023 at 4:04 PM Matt Burgess  wrote:
> >
> > > In my opinion that's ok, but I think it would be helpful if a PR is
> > > going to be backported to support/nifi-1.x that the PR author provides
> > > two PRs, one against main with Java 17 features and one against
> > > support/nifi-1.x with Java 8 features. That being said, allowing Java
> > > 17 features may make maintenance tougher while there's an active 1.x
> > > branch. Maybe we should wait until we only support NiFi 2.x?
> > >
> > > On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen 
> > > wrote:
> > > >
> > > > Since we're standardizing on 17, we're free and clear to use Java 17
> > > > features, right?
> > >
> >

Re: Java 17 features in 2.x

[Matt Burgess]

In my opinion that's ok, but I think it would be helpful if a PR is
going to be backported to support/nifi-1.x that the PR author provides
two PRs, one against main with Java 17 features and one against
support/nifi-1.x with Java 8 features. That being said, allowing Java
17 features may make maintenance tougher while there's an active 1.x
branch. Maybe we should wait until we only support NiFi 2.x?

On Mon, Aug 7, 2023 at 4:59 PM Mike Thomsen  wrote:
>
> Since we're standardizing on 17, we're free and clear to use Java 17
> features, right?

Re: [VOTE] Release Apache NiFi 1.23.0 (RC3)

[Matt Burgess]

+1 (binding)

Ran through release helper, verified H2 migration on Registry,
ExtractRecordSchema, various other flows to ensure no regressions.

Thanks for RM'ing David!

On Tue, Jul 25, 2023 at 5:33 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.0.
>
> Please review the following guide for how to verify a release candidate
> build:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The source being voted upon and the convenience binaries are available on
> the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1233
>
> Git Tag: nifi-1.23.0-RC3
> Git Commit ID: 27a690a30a6ae77c217a47ac0601e85970777ca2
> GitHub Commit Link:
> https://github.com/apache/nifi/commit/27a690a30a6ae77c217a47ac0601e85970777ca2
>
> Checksums of nifi-1.23.0-source-release.zip:
>
> SHA256: 39c97d89804b005cc995c56a87a3df6f68c44ee42114dffe756bbac90a3593cf
> SHA512:
> f256ca731a67435e9883626931abc58f28cda9deb6a7d0a84ed97b78104e43b3b638ee2297d79f92bf5a1e19f62cc78e0a886fa0094593ab34b21d658c59eadd
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 132 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353320
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.23.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.23.0 (RC2)

[Matt Burgess]

+1 (binding), verified hashes, keys, and commit IDs, ran flows to
verify NIFI-11783 [1], NIFI-11807 [2], and verified no regression in
NIFI-11621 [3].

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 17.0.7, vendor: Oracle Corporation, runtime:
/Users/mburgess/.sdkman/candidates/java/17.0.7-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"

Thanks for RM'ing David!

[1] https://issues.apache.org/jira/browse/NIFI-11783
[2] https://issues.apache.org/jira/browse/NIFI-11807
[3] https://issues.apache.org/jira/browse/NIFI-11621

On Thu, Jul 20, 2023 at 8:16 PM David Handermann
 wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.23.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1231
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.23.0-RC2
> The Git commit ID is 2c707ad4e35fa7086024b966f279c8d4da486144
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=2c707ad4e35fa7086024b966f279c8d4da486144
> https://github.com/apache/nifi/commit/2c707ad4e35fa7086024b966f279c8d4da486144
>
> Checksums of nifi-1.23.0-source-release.zip:
> SHA256: 79d8a2cc6f3c2e2b6a31398d2347a0fa5e2849cd47058f6ee7798b231c114b94
> SHA512:
> d80ecab0eb5721ceca1f8840902841fd33ff173ca2c6aff8beefeac6dddc3fe865f2dca9d1e985b339ad75a19ebafffc95d637840957258ae80961bf8db5b0ac
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 115 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353320
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.23.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: Use of attribute uuid and other "native" attributes

[Matt Burgess]

In general I recommend only sending on those attributes that will be
used at some point downstream (unless you have an "original"
relationship that should maintain the original state with respect to
provenance). If you don't know that ahead of time you'll probably need
to send all/most of the attributes just in case.

Are you using session.create() or session.clone()? They both set a new
"uuid" attribute on the created FlowFile, with at least the latter
setting some other attributes as well (see the Developer Guide [1] for
more details).

Regards,
Matt

[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html

On Tue, Jul 18, 2023 at 12:25 PM Russell Bateman  wrote:
>
> I have a custom processor, /SplitHl7v4Resources/, that splits out
> individual FHIR resources (Patients, Observations, Encounters, etc.)
> from great Bundle flowfiles. So, for a given flowfile, it's split into
> hundreds of smaller ones.
>
> When I do this, I leave the existing NiFi attributes as they were on the
> original flowfile.
>
> As I contemplate the uuid attribute, it occurs to me that I should find
> out what its *significance is for provenance and other potential
> debugging/tracing concerns*. I never really look at it, but, if there
> were some kind of melt-down in a production environment, would I care
> that it multiplied across hundreds of flowfiles besided the original one?
>
> Also these two other NiFi attributes remain unchanged:
>
> filename
> path
>
>
> I do garnish each flowfile with many pointed/significant new attributes
> like resource.type that are my own. In my processing, I don't care about
> NiFi's original attributes, but should I?
>
> Thanks,
> Russ

Re: board report for July 2023

[Matt Burgess]

Looks good to me, this project always has impressive things to report
to the board and they have consistently appreciated it as well, it
clearly indicates a healthy and active community. Thanks for this!

On Tue, Jul 4, 2023 at 12:52 PM Joe Witt  wrote:
>
> Team,
>
> Here is the submitted board report.  Thanks again to all for
> continuing to make this a great project and useful tool for people and
> organizations all over the world!
>
> ## Description:
> The mission of NiFi is the creation and maintenance of software related to
> providing an easy to use, powerful, and reliable system to process and
> distribute data.
>
> Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
> integrate with and leverage the command and control of NiFi. There are both
> Java and C++ implementations.
>
> Apache NiFi Registry is a centralized registry for key configuration items
> including flow versions, assets, and extensions for Apache NiFi and Apache
> MiNiFi.
>
> Apache NiFi Nar Maven Plugin is a release artifact used for supporting the
> NiFi classloader isolation model.
>
> Apache NiFi Flow Design System is a theme-able set of high quality UI
> components and utilities for use across the various Apache NiFi web
> applications in order to provide a more consistent user experience.
>
> ## Project Status:
> Current project status: Ongoing. High.
> Issues for the board: None.
>
> ## Membership Data:
> Apache NiFi was founded 2015-07-14 (9 years ago)
> There are currently 63 committers and 34 PMC members in this project.
> The Committer-to-PMC ratio is roughly 2:1.
>
> Community changes, past quarter:
> - Nándor Soma Abonyi was added to the PMC on 2023-06-25
> - No new committers. Last addition was Ferenc Kis on 2023-03-23.
>
> ## Project Activity:
> The NiFi community released NiFi 1.22.0 Jun 2023 and NiFi 1.21.0 in Apr 2023.
> Each release provided new features, stability, and addressed reported
> vulnerabilities within NiFi and dependencies.  Meanwhile the community is
> rapidly approaching a NiFi 2.0 major release already containing more than 466
> changes. The MiNiFiCPP project also released 0.14.0 in April and we've
> released the latest NiFi Nar Maven Plugin version 1.5.1 in June.
>
>
> ## Community Health:
> Community health remains strong and we see high activity across the various
> channels of contribution and engagement including JIRA, Github, Slack, Mailing
> Lists and so on.  Meetups are less common these days but we still see good
> activity in conferences, blog posts, and more. We also continue to enjoy
> excellent engagement for discussions, voting threads, and release candidate
> evaluation.  Slack remains a super popular point of engagement and we've grown
> again this quarter to more than 2700 people in our 'general' slack channel
> alone. The dev mailing list activity was 20% higher this quarter than last
> while the user list is less active.  Much of the user list activity has
> moved to slack as we've reported for a while now but still the user list is
> an effective tool for collaboration with more than 100 emails in this quarter.
> Via JIRA, commits, and our issues list we still see a lot of activity and even
> growth in activity this quarter.  Importantly our rate of PR activity remains
> high with more than 350+ PRs in the quarter again but importantly we have more
> closed than opened showing we're finally turning the corner in this regard.
> Some of the activity levels and commit levels reflect that we're managing
> actively both the 1.x and 2.x release lines now so we do anticipate many of
> these productivity metrics will drop in the coming quarters as we get the 2.0
> release out and stable.

Re: Jira contributor access

[Matt Burgess]

It appears you have an existing Jira account [1], I have added you as
a Contributor to the NiFi projects in Jira. If you cannot log into
your Jira account and cannot reset your password I believe you have to
create a new one via [2].

Regards,
Matt

[1] 
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ZmlCode&selectedTab=jira.user.profile.panels:user-profile-summary-panel
[2] https://selfserve.apache.org/jira-account.html

On Mon, Jul 3, 2023 at 10:02 AM 张美丽  wrote:
>
> Hi,
> I am ZmlCode.
> I report the problem of https://issues.apache.org/jira/browse/NIFI-10847.
> But no one helped me review it for a long time.
>
>
> Recently, after reading the official documents, I know that I need to verify 
> the current account by email first.
> So I came to the certification.
>
>
> My account is ZmlCode and email is zhangmeili_c...@163.com .Please.
> Thanks again to NIFI for this program, I learned a lot from it and helped me 
> grow up a lot.
> I hope I can make a small contribution in the future. May be.
>
>
> Have a nice day!
> Best regards.

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.1

[Matt Burgess]

+1 (binding)

Verified signatures and hashes, built with the system below, built
NiFi NARs and verified the issue I was encountering is fixed. Thanks
for RM'ing Nandor!

Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 11.0.19, vendor: Eclipse Adoptium, runtime:
/Users/mburgess/.sdkman/candidates/java/11.0.19-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4", arch: "aarch64", family: "mac"

On Wed, Jun 14, 2023 at 6:55 PM Nandor Soma Abonyi
 wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi 
> NAR Maven Plugin 1.5.1.
>
> The source being voted upon, including signatures, digests, etc. can be found 
> at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.5.1/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+NAR+Maven+Plugin+release+candidate
>
> The Git tag is nifi-nar-maven-plugin-1.5.1-rc1
> The Git commit ID is 39fc959426ea405df6360969b55ae2adad47e1aa
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=39fc959426ea405df6360969b55ae2adad47e1aa
>
> Checksums of nifi-nar-maven-plugin-1.5.1-source-release.zip:
> SHA256: 0ddc4efbfe504f9ed6477b8c572f2c6e5ba0c953e2e5b063bdfbd1f934eda6bf
> SHA512: 
> a7281c8a3769db37e3491f3a5e54533b5f26bdcad99f8adb1e5609f1de17309aefb3d49eb9231e75a814e42566525b9afe4a11bda2c4ab48e8bab5a298b72b24
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/nsabonyi.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 3 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353009
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.5.1
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because …
>

Re: [DISCUSS] Release NiFi NAR Maven Plugin 1.5.1

[Matt Burgess]

+1 from me, I ran into the issue so would like to have that fix
available.  Thanks for volunteering to RM!

On Mon, Jun 12, 2023 at 5:36 AM Nandor Soma Abonyi
 wrote:
>
> Hello Apache NiFi Community,
>
> I'd like to initiate a discussion about the next release of 
> nifi-nar-maven-plugin.
>
> Currently, nifi-nar-maven-plugin doesn’t use remote repositories, which was
> fixed in [1]. Though it is the only change since the last release, I consider 
> it significant
> enough to suggest releasing nifi-nar-maven-plugin as 1.5.1.
>
> I would be happy to take on RM duties for this release.
>
> Do you agree it is time for a new release?
>
> Thanks,
> Nandor Soma Abonyi
>
> [1] NIFI-11324  NAR Plugin 
> extension docs not correctly resolving overridden versions from system 
> properties
>

Re: [VOTE] Release Apache NiFi 1.22.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through release helper and tested various flows including
verifying NIFI-11621 [1] for nested array inference and NIFI-11653 [2]
for removing H2 support for DBCPConnectionPool. Also tested some
version control capability for NiFi Registry.

Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /Users/mburgess/.sdkman/candidates/maven/current
Java version: 11.0.19, vendor: Eclipse Adoptium, runtime:
/Users/mburgess/.sdkman/candidates/java/11.0.19-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4", arch: "aarch64", family: "mac"

Thanks for RM'ing Joe!

[1] https://issues.apache.org/jira/browse/NIFI-11621
[2] https://issues.apache.org/jira/browse/NIFI-11653

On Tue, Jun 6, 2023 at 5:38 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.22.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1228
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.22.0-RC1
> The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
> https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8
>
> Checksums of nifi-1.22.0-source-release.zip:
> SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743
> SHA512:
> 5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 186 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353069
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.22.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: User-defined ProvenanceReporter

[Matt Burgess]

All,

We've been discussing this in Slack [1], we can continue there and if
anyone is interested and not on our Apache Slack channel, I encourage
you to join [2] but I can also copy-paste into this thread if need be.

Regards,
Matt

[1] https://apachenifi.slack.com/archives/C0L9VCD47/p1682895885163669
[2] 
https://join.slack.com/t/apachenifi/shared_invite/zt-1u8wrsi2b-dE6Rf0lEzdYWgkBaPykceg

On Sun, Apr 30, 2023 at 7:09 PM Yu Chen  wrote:
>
> Hi,
>
> May I ask if there is a method I can use to implement my own
> ProvenanceReporter (
> https://github.com/apache/nifi/blob/main/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java)
> to replace the default StandardProvenanceReporter (
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProvenanceReporter.java)
> ?
>
>
> Thanks
> Yu

Re: Issue upgrading Nifi(1.7.0) & Minifi(0.5.0) to 1.20.0

[Matt Burgess]

Ignasi,

A lot has changed between MiNiFi 0.5.0 and MiNiFi 1.20.0, including changes
to NARs, how they are loaded, etc. For custom NARs I would try to
rebuild them using the same version of NiFi and MiNiFi dependencies. The
latest Apache NiFi release is now 1.21.0 so I would change the NiFi and
MiNiFi dependencies in your custom NARs to 1.21.0. You may find that there
are compilation errors or some issue that wasn't there before, these would
be things introduced between NiFi 1.7.0 and NiFi 1.21.0. I can't think of
any offhand but wanted to mention it in case your NARs no longer
compile/build.

Regards,
Matt


On Fri, Apr 28, 2023 at 5:31 AM Garriga, Ignasi
 wrote:

> Hi all,
>
> We're trying to upgrade versions of Apache Nifi 1.7.0 with Minifi 0.5.0
> to the latest embedded version 1.20.0.
> We have some custom nars that when we add to the project there are
> problems with the Classloader and bundle not found:
>
> ERROR [main] o.a.n.n.StandardExtensionDiscoveringManager Could not
> instantiate class of type 
> org.apache.nifi.processors.standard.JoltTransformJSON
> using ClassLoader for bundle default:unknown:unversioned because the bundle
> could not be found
>
>
> In the following pictures I share with you the docker-compose that we use
> to create an image and all libraries that we use to compile. It can help to
> identify something or give a clue to find the solution.
>
> pm-data-pipeline:
> image: apache/nifi-minifi:1.20.0
> restart: always
> hostname: pm-data-pipeline
> container_name: pm-data-pipeline
> environment:
>   - NIFI_WEB_HTTPS_PORT=9443
> ports:
>   - "8080:8080"
>   - "9443:9443"
> volumes:
>   - 
> ../../../assets/templates/nMonFlow.yml:/opt/minifi/minifi-1.20.0/conf/config.yml
>   - 
> ../../../assets/nifi-invoke-igw-nar-1.6.0.nar:/opt/minifi/minifi-1.20.0/lib/nifi-invoke-igw-nar-1.6.0.nar
>   - 
> ../../../assets/pm-data-pipeline-ssl-context-service-nar-1.0.0.nar:/opt/minifi/minifi-1.20.0/lib/pm-data-pipeline-ssl-context-service-nar-1.0.0.nar
>   - 
> ../../../assets/nifi-amqp-nar-1.20.0.nar:/opt/minifi/minifi-1.20.0/lib/nifi-amqp-nar-1.20.0.nar
>
>
> [image: image.png]
>
>
> Thank you in advance for your support and don't hesitate to contact us if
> you need more information .
>
> Regards,
>
>
> --
> *--*
>
> *Ignasi Garriga Sala* Coagulation Software Developer
>
> Roche Diagnostics SL
> Patient Care POC
> Av. de la Generalitat, 171-173
> E-08174 Sant Cugat del Vallès
>
> M +34 619 048 267
> *ignasi.garr...@roche.com* 
>
> roche.es  | AularioRoche
>  | RocheDialog
>  | Twitter
>  | YouTube 
> | Facebook  | LinkedIn
> 
>
> *Confidencialidad* - Este mensaje ha sido dirigido sólo al destinatario o
> a los destinatarios mencionados y puede contener información confidencial
> y/o de uso restringido a profesionales de la salud, en cuyo caso su
> remisión a terceros que no sean profesionales de la salud no está
> permitida. Si usted tiene dudas respecto a quién puede remitir este mensaje
> y/o la información que contiene, por favor, consulte con su remitente. Si
> usted no debería haber recibido este correo o piensa que el destinatario es
> incorrecto, por favor contacte con el remitente y borre este mensaje. El
> uso no autorizado de la información contenida en este mensaje está
> prohibido.
>
> *Protección de datos* - Roche Diagnostics, S.L.U. (con domicilio en Avda.
> de la Generalitat, 171-173, 08174 Sant Cugat del Vallès, Barcelona) es el
> responsable del tratamiento de sus datos de contacto corporativos con el
> fin de poder mantener relaciones con usted o con la empresa, entidad y
> organización para la cual usted trabaja o colabora. Para obtener más
> información sobre cómo tratamos sus datos y cómo ejercer sus derechos,
> consulte nuestra política de privacidad
> .
>
>

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

[Matt Burgess]

+1 (binding), ran through the release helper and a few flows
(including one with the CaptureChangeMySQL changes) on a Mac with Java
11.

Thanks for RM'ing Joe!

On Mon, Apr 3, 2023 at 8:11 PM Joe Witt  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.21.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1226
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
>
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.21.0-RC2
> The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
>
> Checksums of nifi-1.21.0-source-release.zip:
> SHA256: 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5
> SHA512: 
> 58c10976bc22fb5406d9d1d9b7ca7d90c2dbed99565d00f1172bb33b375e9e068da5003d9dbbd87d2b17626e4e310b999c8581718532934e855c2134be763f04
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 126 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352899
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.21.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [discuss] NiFi support for Hadoop ecosystem components

[Matt Burgess]

As one of the small number of people that fight the battle, I like the
idea of Option 1 (full disclosure: I work for a vendor). From a
community standpoint (I'm on the PMC) I'm not strongly opposed to
Option 2 although I wouldn't want to be the one managing and releasing
the artifacts :) Having said that, unless it remained maintained and
released, I feel like it would just be a component graveyard (or maybe
more like the Apache Attic), in which case it seems unnecessary and
that's why I'm behind Option 1. Interested to hear others' thoughts of
course.

Thanks,
Matt

On Fri, Mar 24, 2023 at 2:07 PM Joe Witt  wrote:
>
> Team,
>
> For the full time NiFi has been in Apache we've built with support for
> various Hadoop ecosystem components like HDFS, Hive, HBase, others,
> and more recently formats/serialization modes like necessary for
> Parquet, Orc, Iceberg, etc..
>
> All of these things however present endless challenges with
> compatibility across different versions (Hive being the most difficult
> by far), vendors (hadoop vendors, cloud vendors, etc..).  And also
> super notably the incredible number of dependencies, dependency
> conflicts, inclusions/exclusions, old log libs, vulnerability updates,
> etc..  And last but certainly not least a big reason why our build has
> grown so much.
>
> We have a couple options:
> 1. Deprecate these components in NiFi 1.x and drop them entirely in
> NiFi 2.x.  Leave this as a problem for vendors to deal with.  NiFi
> users interacting with such components are nearly exclusively doing so
> with vendors anyway.
>
> 2. Remove the components from NiFi main code line and create a
> separate repo for 'nifi-hadoop-extensions'.  We manage those
> independently and release them periodically.  They would be available
> for people to grab the nars if they want to use them.  We include none
> of them in the convenience binary going forward by default.
>
> 3. Change nothing.  Continue to battle with the above listed items.
> This is admittedly a bit of a non-option.  We can't keep spending the
> same time/energy on these we have.  It is a very small number of
> people that fight this battle.
>
> Look forward to hearing thoughts on these options or others we might consider.
>
> Thanks

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.5.0 - RC2

[Matt Burgess]

+1 (binding)

Ran through release helper, built NiFi bundles/NARs using this version
of the plugin, all looks good. Thanks for RM'ing Kevin!

On Tue, Mar 14, 2023 at 5:00 PM Kevin Doran  wrote:
>
> Hello Apache NiFi Community,
>
> Issues with RC1 have been addressed in RC2, and I am pleased to again
> be calling this vote for the source release of Apache NiFi NAR Maven
> Plugin 1.5.0.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1222/org/apache/nifi/nifi-nar-maven-plugin/1.5.0/
>
> The Git tag is nifi-nar-maven-plugin-1.5.0-RC2
> The Git commit ID is 277f9e8998ca76a972c90d8ecf3f771414c86700
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=277f9e8998ca76a972c90d8ecf3f771414c86700
>
> Checksum of nifi-nar-maven-plugin-1.5.0-source-release.zip:
>
> SHA512: 
> a265587f8bfb31359cae2335394d88d76d36ae9806030e38fc9846264f20a4e70d642c4f0c08425b843794aaabaeea416b1104ede26671ffcbe001e1976d4efd
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/kdoran.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 4 issues were closed/resolved for this release:
> https://issues.apache.org/jira/projects/NIFI/versions/12352895
>
> Release note highlights can be found here:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352895
>
> *The vote will be open for 24 hours.* Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.5.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: [VOTE] Release Apache NiFi 1.20.0 (RC1)

[Matt Burgess]

+1 (binding)

Ran through the release helper and a few flows. I did see this as a
race condition for InvokeScriptedProcessor but it fixes itself on the
first validation so I don't think we need to tank the RC as a result,
I wrote up [1] to investigate:

2023-02-07 16:26:39,825 ERROR [NiFi Web Server-32]
o.a.n.p.script.InvokeScriptedProcessor
InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64] Error
adding script engine Groovy
2023-02-07 16:26:39,826 ERROR [NiFi Web Server-32]
o.a.n.p.script.InvokeScriptedProcessor
InvokeScriptedProcessor[id=2bbc053d-8b08-3206-7c47-cc00a08beb64]
Unable to load script: No script runner available
org.apache.nifi.processor.exception.ProcessException: No script runner available
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScript(InvokeScriptedProcessor.java:371)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.reloadScriptBody(InvokeScriptedProcessor.java:326)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.setup(InvokeScriptedProcessor.java:230)
at 
org.apache.nifi.processors.script.InvokeScriptedProcessor.onConfigurationRestored(InvokeScriptedProcessor.java:222)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at 
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:316)
at 
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:93)
at 
org.apache.nifi.controller.StandardProcessorNode.onConfigurationRestored(StandardProcessorNode.java:2115)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.addProcessor(StandardVersionedComponentSynchronizer.java:2417)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeProcessors(StandardVersionedComponentSynchronizer.java:932)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:422)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:260)
at 
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:550)
at 
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:255)
at 
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3972)
at 
org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3952)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:435)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
at 
org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:308)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflecti

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.4.0

[Matt Burgess]

+1 binding, ran through release helper (noticed the tag hadn't been
pushed but is now, thanks!) Built NiFi with 1.4.0-RC1 plugin, verified
all worked successfully. Thanks for RM'ing Kevin!

On Mon, Jan 30, 2023 at 6:19 PM Kevin Doran  wrote:
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi NAR Maven Plugin 1.4.0.
>
> The source being voted upon, including signatures, digests, etc. can
> be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1219/org/apache/nifi/nifi-nar-maven-plugin/1.4.0/
>
> The Git tag is nifi-nar-maven-plugin-1.4.0-RC1
> The Git commit ID is 09d7bb9ff679d0eed9feaa066d2cbdd347a20204
> https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=09d7bb9ff679d0eed9feaa066d2cbdd347a20204
>
> Checksum of nifi-nar-maven-plugin-1.4.0-source-release.zip
>
> SHA512: 
> d48bfde8a5bab8a17b320889297c09efa162a7017db2268e274db626926ff9b58173abab54db4e1a50b5664fc86a9501ce2ef267b3e3b768eb80ef5c929860c9
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/kdoran.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 6 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352124
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.4.0
>
> The vote will be open for 72 hours. Please download the release
> candidate and evaluate the necessary items including checking hashes,
> signatures, build from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...

Re: Line number to identify where CSVReader failed

[Matt Burgess]

I was thinking the same, CSVRecordReader could keep track of the
number of records read and if an exception is thrown during iteration
over reading the records, we can output the number of records read
successfully.

On Thu, Jan 19, 2023 at 3:47 PM Joe Witt  wrote:
>
> Dan,
>
> Seems like our record reader mechanism should offer the concept of tracking
> which record it is on such that this could be logged.  It looks from a
> quick check like we track record count on writing so something similar on
> the interface of the reader could be quite helpful.
>
> Perhaps best to file a JIRA.  Someone else might have a better idea of what
> you can do now.
>
> Thanks
>
> On Thu, Jan 19, 2023 at 1:39 PM Dan S  wrote:
>
> > For both QueryRecord and ValidateRecord when I use a CSVReader on a file
> > which has different delimiters than the rest of the file, the error message
> > logged does not include the line number where the parsing failed.  When
> > looking at the code, I did not see any hooks for getting that information.
> > Is there a way to get the line number so it would be easy to identify
> > which lines would need to be fixed?
> >

Re: Getting a NiFi Jira account

[Matt Burgess]

Kim,

According to the new Jira guidelines [1], please submit the following
information to priv...@nifi.apache.org and the PMC will create an
account for you and add you as a Contributor to the NiFi Jira
projects:

email address
preferred username (N.B. hyphens not allowed)
alternate username (in case the preferred one is already in use)
display name, if it is different from the username

Regards,
Matt

[1] https://infra.apache.org/jira-guidelines.html#who

On Mon, Jan 16, 2023 at 9:18 AM Kim Pedersen  wrote:
>
> Hello,
>
> I want to submit a PR on NiFi at Github. I have been reading the contributor 
> guide, concluding that I need to submit a Jira issue before creating the PR. 
> So I need a NiFi Jira acconut. Can you help me with that or point me in the 
> right direction?
>
> Regards,
> Kim Pedersen

Re: nifi proxy issue + jira account

[Matt Burgess]

Kyle,

Per the new Apache Jira instructions [1], please provide the following
information to priv...@nifi.apache.org  and I can
create an account for you and add you as a
contributor:

- email address
- preferred username (N.B. hyphens not allowed)
- alternate username (in case the preferred one is already in use)
- display name, if it is different from the username


Regards,
Matt

[1] https://infra.apache.org/jira-guidelines.html#who

On Thu, Dec 15, 2022 at 4:43 PM Nguyen, Kyle 
wrote:

> Hi.  Thanks for maintaining the project.
>
>
>
> 1.   Can I get a jira account please?  Would like to request *
> StandardProxyConfigurationService* integration into 
> *StandardOath2AccessTokenProvider
> *and into *QuerySalesforceObject*
>
> 2.   Or perhaps you can help me more directly on this stackoverflow:
> https://stackoverflow.com/questions/74482819/connecting-to-an-external-http-api-behind-a-proxy-from-nifi
>
>
>
> I also tried this ENV VAR in my docker nifi instance.  It doesn’t work.
>
>
>
> JAVA_FLAGS=-Dhttp.nonProxyHosts="localhost,.amazonaws.com"
> -Dhttp.proxyHost=proxy.foo.com -Dhttp.proxyPort=3128 -Dhttp.proxyUser=foo
> -Dhttp.proxyPassword=bar
>
>
>
>
>
> [image: cid:17261f6184a4cff311]
>
> *Kyle Nguyen*
> Corporate Technology, Software Engineer
>
>
>
> *Millennium Management LLC*
>
> 399 Park Avenue  |  New York, NY 10022
>
> 📞 +1.212.708.1366  | 📱 +1.929.837.1788
>
> mlp.com 
>
>
>
>
>
> ##
>
> The information contained in this communication is confidential and
>
> may contain information that is privileged or exempt from disclosure
>
> under applicable law. If you are not a named addressee, please notify
>
> the sender immediately and delete this email from your system.
>
> If you have received this communication, and are not a named
>
> recipient, you are hereby notified that any dissemination,
>
> distribution or copying of this communication is strictly prohibited.
> ##
>
>

Re: Log Cleansing

[Matt Burgess]

Bryan,

Do you have an example of the kind of cleansing you'd like to do? NiFi
has things like GrokReader [1], syslog parsers and such to parse logs,
and the corresponding writers to format the log text the way you want.

Regards,
Matt

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.19.1/org.apache.nifi.grok.GrokReader/index.html

On Wed, Dec 14, 2022 at 2:16 PM Bryan Paul  wrote:
>
> Good Afternoon,
>
> Our company is using NIFI as a log data pipeline.  I know there are other 
> solutions on the market that offer log cleansing as a mechanism to remove 
> noise from logs that are sent to SIEM tools.  Does NIFI have this capability?

Re: [VOTE] Adopt NiFi 2.0 Proposed Release Goals

[Matt Burgess]

+1 (binding)

On Mon, Dec 12, 2022 at 12:02 PM David Handermann
 wrote:
>
> Team,
>
> Following positive feedback on NiFi 2.0 Proposed Release Goals [1] on the
> recent discussion thread [2], I am calling this vote to adopt the following
> as Release Goals for NiFi 2.0:
>
> 1. Remove Java 8 support and require Java 11
> 2. Remove deprecated components
> 3. Remove deprecated component properties
> 4. Remove components integrating with unmaintained services
> 5. Remove compatibility classes and methods
> 6. Remove flow.xml.gz in favor of flow.json.gz
> 7. Remove duplicative features
> 8. Upgrade internal Java API references
> 9. Reorganize standard components
> 10. Implement migration tools for upgrading flows
>
> A positive vote indicates agreement on these goals and the initiation of
> the following actions:
>
> 1. Rename NiFi 2.0 Proposed Release Goals to NiFi 2.0 Release Goals
> 2. Create version 1 branch in Git for subsequent support releases on the
> version 1 series
> 3. Update the current main branch in Git to version 2.0.0-SNAPSHOT
>
> The vote will be open for 72 hours and follow standard procedures for
> release votes.
>
> Please review the linked goals and discussions for background.
>
> [ ] +1 Adopt NiFi 2.0 Release Goals
> [ ] +0 No opinion
> [ ] -1 Do not adopt NiFi 2.0 Release Goals for the following reasons...
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals
> [2] https://lists.apache.org/thread/xo77p9t3xg4k70356xrqbdg4m9sg7sf8