Re: Cannot get build working

[Paul Grey]

Yep, the test is fine, but the means of generating an appropriate large
string can hopefully be improved.


On Mon, Jul 29, 2024 at 2:34 PM Chris Sampson
 wrote:

> I’d probably keep it as a unit test, but use a different way of generating
> a large string - the test is ensuring the JSON parsers handle large input,
> but there’s no real need for it to be randomised, so any method of
> generating a long string would suffice.
>
>
> Cheers,
>
> ---
> Chris Sampson
> IT Consultant
> chris.samp...@naimuri.com
>
>
> > On 29 Jul 2024, at 19:21, Paul Grey  wrote:
> >
> >> I personally think this test belongs in an integration test so users
> > wouldn't have to install haveged.
> >
> > 100% agree, and hopefully no one reads into this email thread any sort of
> > recommendation to incorporate haveged into their environments.
> >
> > I've created NIFI-13595, for someone to have a look at this test.  Thanks
> > for the suggestion about the alternate method, hopefully that will help.
> >
> > Also, thank you for your patience working through the issue!  For weird
> > problems like this, an environment where a problem can be reliably
> > reproduced is very helpful.
> >
> >
> > On Mon, Jul 29, 2024 at 1:36 PM Dan S  wrote:
> >
> >> Paul,
> >> Running haveged helped me finish the build and prevent the unit test
> >>
> >>
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling
> >> from hanging. I did not realize though initially that I had to actually
> >> manually start haveged. I found out that from a user's comment on the
> >> article How to Setup Additional Entropy for Cloud Servers Using Haveged
> >> <
> >>
> https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged
> >>> .
> >> After starting manually with:
> >> sudo service haveged start
> >>
> >> I was able to complete the build.
> >>
> >> Thank you so much for this suggestion and help me getting over this
> hump. I
> >> personally think this test belongs in an integration test so users
> wouldn't
> >> have to install haveged. It is interesting to note in the setup method
> of
> >>
> org.apache.nifi.minifi.commons.service.StandardFlowPropertyEncryptorTest,
> >> the method org.apache.commons.lang3.RandomStringUtils.randomAlphabetic
> is
> >> used and that does not seem to cause an issue.
> >>
> >> On Mon, Jul 29, 2024 at 1:17 PM Paul Grey  wrote:
> >>
> >>>> their entropy-rich environment. However, it is not recommended to use
> >> it
> >>>> in virtual machines, since the very reasons that make them prone to
> >>>> entropy starvation will hinder, if not defeat, HAVEGE (or the quality
> —
> >>>> randomness — of the entropy it will gather).
> >>>
> >>>
> >>> The quality of the randomness does not seem especially important in
> this
> >>> context (a unit test).  Whereas, when it comes to generating
> cryptography
> >>> keys, it is _very_ important.
> >>>
> >>> :)
> >>>
> >>>
> >>> On Mon, Jul 29, 2024 at 1:12 PM Paul Grey  wrote:
> >>>
> >>>>> If it returns anything less than 100-200, you have a problem. Try
> >>>> installing rng-tools, or generating I/O, like large find operations.
> >>>>
> >>>> These alternate suggestions might help.  Right now, we just want to
> >> know
> >>>> if your problem is related to entropy.  Based on the library usage,
> >> this
> >>>> seems like a reasonable candidate to investigate.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jul 29, 2024 at 12:59 PM Paul Grey  wrote:
> >>>>
> >>>>> Sure.  I'm suggesting that to diagnose your situation.  Depletion of
> >>>>> entropy may not be the problem.  Here's another way to evaluate that:
> >>>>>
> >>>>> https://major.io/p/check-available-entropy-in-linux/
> >>>>>
> >>>>> # cat /proc/sys/kernel/random/entropy_avail
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 29, 2024 at 12:21 PM Dan S  wrote:
> >>>>>
> >>>>>> Paul,
> >>>>>> It is interesting you are sugg

Re: Cannot get build working

[Paul Grey]

> I personally think this test belongs in an integration test so users
wouldn't have to install haveged.

100% agree, and hopefully no one reads into this email thread any sort of
recommendation to incorporate haveged into their environments.

I've created NIFI-13595, for someone to have a look at this test.  Thanks
for the suggestion about the alternate method, hopefully that will help.

Also, thank you for your patience working through the issue!  For weird
problems like this, an environment where a problem can be reliably
reproduced is very helpful.


On Mon, Jul 29, 2024 at 1:36 PM Dan S  wrote:

> Paul,
>  Running haveged helped me finish the build and prevent the unit test
>
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling
> from hanging. I did not realize though initially that I had to actually
> manually start haveged. I found out that from a user's comment on the
> article How to Setup Additional Entropy for Cloud Servers Using Haveged
> <
> https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged
> >.
> After starting manually with:
> sudo service haveged start
>
> I was able to complete the build.
>
> Thank you so much for this suggestion and help me getting over this hump. I
> personally think this test belongs in an integration test so users wouldn't
> have to install haveged. It is interesting to note in the setup method of
> org.apache.nifi.minifi.commons.service.StandardFlowPropertyEncryptorTest,
> the method org.apache.commons.lang3.RandomStringUtils.randomAlphabetic is
> used and that does not seem to cause an issue.
>
> On Mon, Jul 29, 2024 at 1:17 PM Paul Grey  wrote:
>
> > > their entropy-rich environment. However, it is not recommended to use
> it
> > > in virtual machines, since the very reasons that make them prone to
> > > entropy starvation will hinder, if not defeat, HAVEGE (or the quality —
> > > randomness — of the entropy it will gather).
> >
> >
> > The quality of the randomness does not seem especially important in this
> > context (a unit test).  Whereas, when it comes to generating cryptography
> > keys, it is _very_ important.
> >
> > :)
> >
> >
> > On Mon, Jul 29, 2024 at 1:12 PM Paul Grey  wrote:
> >
> > > > If it returns anything less than 100-200, you have a problem. Try
> > > installing rng-tools, or generating I/O, like large find operations.
> > >
> > > These alternate suggestions might help.  Right now, we just want to
> know
> > > if your problem is related to entropy.  Based on the library usage,
> this
> > > seems like a reasonable candidate to investigate.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jul 29, 2024 at 12:59 PM Paul Grey  wrote:
> > >
> > >> Sure.  I'm suggesting that to diagnose your situation.  Depletion of
> > >> entropy may not be the problem.  Here's another way to evaluate that:
> > >>
> > >> https://major.io/p/check-available-entropy-in-linux/
> > >>
> > >> # cat /proc/sys/kernel/random/entropy_avail
> > >>
> > >>
> > >> On Mon, Jul 29, 2024 at 12:21 PM Dan S  wrote:
> > >>
> > >>> Paul,
> > >>>  It is interesting you are suggesting haveged as the article you
> > >>> suggested
> > >>> recommends not using on virtual machines
> > >>>
> > >>> haveged is a daemon — derived from the HAVEGE algorithm
> > >>> > <http://www.irisa.fr/caps/projects/hipsor/> — designed to help
> (the
> > >>> > kernel) gather entropy from more sources (than the kernel itself
> > >>> does). It
> > >>> > is common to install it on physical hosts to gather entropy faster
> > from
> > >>> > their entropy-rich environment. However, it is not recommended to
> use
> > >>> it
> > >>> > in virtual machines, since the very reasons that make them prone to
> > >>> > entropy starvation will hinder, if not defeat, HAVEGE (or the
> > quality —
> > >>> > randomness — of the entropy it will gather).
> > >>>
> > >>>
> > >>> On Mon, Jul 29, 2024 at 10:43 AM Paul Grey 
> wrote:
> > >>>
> > >>> > In the past, I've observed application testing infrastructure (on
> > VMs)
> > >>> hang
> > >>> > due to depletion of available entropy for R

Re: Cannot get build working

[Paul Grey]

> their entropy-rich environment. However, it is not recommended to use it
> in virtual machines, since the very reasons that make them prone to
> entropy starvation will hinder, if not defeat, HAVEGE (or the quality —
> randomness — of the entropy it will gather).


The quality of the randomness does not seem especially important in this
context (a unit test).  Whereas, when it comes to generating cryptography
keys, it is _very_ important.

:)


On Mon, Jul 29, 2024 at 1:12 PM Paul Grey  wrote:

> > If it returns anything less than 100-200, you have a problem. Try
> installing rng-tools, or generating I/O, like large find operations.
>
> These alternate suggestions might help.  Right now, we just want to know
> if your problem is related to entropy.  Based on the library usage, this
> seems like a reasonable candidate to investigate.
>
>
>
>
>
> On Mon, Jul 29, 2024 at 12:59 PM Paul Grey  wrote:
>
>> Sure.  I'm suggesting that to diagnose your situation.  Depletion of
>> entropy may not be the problem.  Here's another way to evaluate that:
>>
>> https://major.io/p/check-available-entropy-in-linux/
>>
>> # cat /proc/sys/kernel/random/entropy_avail
>>
>>
>> On Mon, Jul 29, 2024 at 12:21 PM Dan S  wrote:
>>
>>> Paul,
>>>  It is interesting you are suggesting haveged as the article you
>>> suggested
>>> recommends not using on virtual machines
>>>
>>> haveged is a daemon — derived from the HAVEGE algorithm
>>> > <http://www.irisa.fr/caps/projects/hipsor/> — designed to help (the
>>> > kernel) gather entropy from more sources (than the kernel itself
>>> does). It
>>> > is common to install it on physical hosts to gather entropy faster from
>>> > their entropy-rich environment. However, it is not recommended to use
>>> it
>>> > in virtual machines, since the very reasons that make them prone to
>>> > entropy starvation will hinder, if not defeat, HAVEGE (or the quality —
>>> > randomness — of the entropy it will gather).
>>>
>>>
>>> On Mon, Jul 29, 2024 at 10:43 AM Paul Grey  wrote:
>>>
>>> > In the past, I've observed application testing infrastructure (on VMs)
>>> hang
>>> > due to depletion of available entropy for RNG.  VMs seem especially
>>> > susceptible to this.  Our stopgap solution was to use haveged to make
>>> > sufficient entropy available to meet the needs of the test suite.
>>> >
>>> > In this case, a refactor of the test would probably be a better
>>> > alternative.  Your observation on behavior in your environment with
>>> haveged
>>> > will help inform that decision.
>>> >
>>> >
>>> > On Mon, Jul 29, 2024 at 10:14 AM Dan S  wrote:
>>> >
>>> > > Yes I have a VM on an EC2 instance in AWS.
>>> > >
>>> > > On Mon, Jul 29, 2024 at 10:10 AM Paul Grey 
>>> wrote:
>>> > >
>>> > > > Is this by chance a virtual machine you are on?
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/
>>> > > >
>>> > > > TL;DR: try installing haveged
>>> > > >
>>> > > >
>>> > > > On Mon, Jul 29, 2024 at 9:29 AM Dan S  wrote:
>>> > > >
>>> > > > > So I see the problem line is
>>> > > > > at
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
>>> > > > > and that line has
>>> > > > >  final String val = String.format("{\"large\": \"%s\"}",
>>> > > > > RandomStringUtils.randomAlphanumeric(1));
>>> > > > >
>>> > > > > I am not sure though what is making the process hang.
>>> > > > >
>>> > > > > On Mon, Jul 29, 2024 at 9:11 AM Dan S  wrote:
>>> > > > >
>>> > > > > > Thanks Joe for the tip. Here is what I have
>>> > > > > > for org.apache.maven.surefire.booter.ForkedBooter though I am
>>> not
>>> > > sure
>>> > > > > wha

Re: Cannot get build working

[Paul Grey]

> If it returns anything less than 100-200, you have a problem. Try
installing rng-tools, or generating I/O, like large find operations.

These alternate suggestions might help.  Right now, we just want to know if
your problem is related to entropy.  Based on the library usage, this seems
like a reasonable candidate to investigate.





On Mon, Jul 29, 2024 at 12:59 PM Paul Grey  wrote:

> Sure.  I'm suggesting that to diagnose your situation.  Depletion of
> entropy may not be the problem.  Here's another way to evaluate that:
>
> https://major.io/p/check-available-entropy-in-linux/
>
> # cat /proc/sys/kernel/random/entropy_avail
>
>
> On Mon, Jul 29, 2024 at 12:21 PM Dan S  wrote:
>
>> Paul,
>>  It is interesting you are suggesting haveged as the article you suggested
>> recommends not using on virtual machines
>>
>> haveged is a daemon — derived from the HAVEGE algorithm
>> > <http://www.irisa.fr/caps/projects/hipsor/> — designed to help (the
>> > kernel) gather entropy from more sources (than the kernel itself does).
>> It
>> > is common to install it on physical hosts to gather entropy faster from
>> > their entropy-rich environment. However, it is not recommended to use it
>> > in virtual machines, since the very reasons that make them prone to
>> > entropy starvation will hinder, if not defeat, HAVEGE (or the quality —
>> > randomness — of the entropy it will gather).
>>
>>
>> On Mon, Jul 29, 2024 at 10:43 AM Paul Grey  wrote:
>>
>> > In the past, I've observed application testing infrastructure (on VMs)
>> hang
>> > due to depletion of available entropy for RNG.  VMs seem especially
>> > susceptible to this.  Our stopgap solution was to use haveged to make
>> > sufficient entropy available to meet the needs of the test suite.
>> >
>> > In this case, a refactor of the test would probably be a better
>> > alternative.  Your observation on behavior in your environment with
>> haveged
>> > will help inform that decision.
>> >
>> >
>> > On Mon, Jul 29, 2024 at 10:14 AM Dan S  wrote:
>> >
>> > > Yes I have a VM on an EC2 instance in AWS.
>> > >
>> > > On Mon, Jul 29, 2024 at 10:10 AM Paul Grey  wrote:
>> > >
>> > > > Is this by chance a virtual machine you are on?
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/
>> > > >
>> > > > TL;DR: try installing haveged
>> > > >
>> > > >
>> > > > On Mon, Jul 29, 2024 at 9:29 AM Dan S  wrote:
>> > > >
>> > > > > So I see the problem line is
>> > > > > at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
>> > > > > and that line has
>> > > > >  final String val = String.format("{\"large\": \"%s\"}",
>> > > > > RandomStringUtils.randomAlphanumeric(1));
>> > > > >
>> > > > > I am not sure though what is making the process hang.
>> > > > >
>> > > > > On Mon, Jul 29, 2024 at 9:11 AM Dan S  wrote:
>> > > > >
>> > > > > > Thanks Joe for the tip. Here is what I have
>> > > > > > for org.apache.maven.surefire.booter.ForkedBooter though I am
>> not
>> > > sure
>> > > > > what
>> > > > > > to make of it.
>> > > > > >
>> > > > > > "main" #1 [23885] prio=5 os_prio=0 cpu=1694.03ms elapsed=158.14s
>> > > > > > tid=0x7f1070028ec0 nid=23885 runnable  [0x7f1077b78000]
>> > > > > >java.lang.Thread.State: RUNNABLE
>> > > > > > at java.io.FileInputStream.readBytes(java.base@21.0.3
>> > /Native
>> > > > > > Method)
>> > > > > > at java.io.FileInputStream.read(java.base@21.0.3
>> > > > > > /FileInputStream.java:287)
>> > > > > > at java.io.FilterInputStream.read(java.base@21.0.3
>> > > > > > /FilterInputStream.java:119)
>> > > > > > at
>> > > > > >
>> > sun.security.p

Re: Cannot get build working

[Paul Grey]

Sure.  I'm suggesting that to diagnose your situation.  Depletion of
entropy may not be the problem.  Here's another way to evaluate that:

https://major.io/p/check-available-entropy-in-linux/

# cat /proc/sys/kernel/random/entropy_avail


On Mon, Jul 29, 2024 at 12:21 PM Dan S  wrote:

> Paul,
>  It is interesting you are suggesting haveged as the article you suggested
> recommends not using on virtual machines
>
> haveged is a daemon — derived from the HAVEGE algorithm
> > <http://www.irisa.fr/caps/projects/hipsor/> — designed to help (the
> > kernel) gather entropy from more sources (than the kernel itself does).
> It
> > is common to install it on physical hosts to gather entropy faster from
> > their entropy-rich environment. However, it is not recommended to use it
> > in virtual machines, since the very reasons that make them prone to
> > entropy starvation will hinder, if not defeat, HAVEGE (or the quality —
> > randomness — of the entropy it will gather).
>
>
> On Mon, Jul 29, 2024 at 10:43 AM Paul Grey  wrote:
>
> > In the past, I've observed application testing infrastructure (on VMs)
> hang
> > due to depletion of available entropy for RNG.  VMs seem especially
> > susceptible to this.  Our stopgap solution was to use haveged to make
> > sufficient entropy available to meet the needs of the test suite.
> >
> > In this case, a refactor of the test would probably be a better
> > alternative.  Your observation on behavior in your environment with
> haveged
> > will help inform that decision.
> >
> >
> > On Mon, Jul 29, 2024 at 10:14 AM Dan S  wrote:
> >
> > > Yes I have a VM on an EC2 instance in AWS.
> > >
> > > On Mon, Jul 29, 2024 at 10:10 AM Paul Grey  wrote:
> > >
> > > > Is this by chance a virtual machine you are on?
> > > >
> > > >
> > > >
> > >
> >
> https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/
> > > >
> > > > TL;DR: try installing haveged
> > > >
> > > >
> > > > On Mon, Jul 29, 2024 at 9:29 AM Dan S  wrote:
> > > >
> > > > > So I see the problem line is
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
> > > > > and that line has
> > > > >  final String val = String.format("{\"large\": \"%s\"}",
> > > > > RandomStringUtils.randomAlphanumeric(1));
> > > > >
> > > > > I am not sure though what is making the process hang.
> > > > >
> > > > > On Mon, Jul 29, 2024 at 9:11 AM Dan S  wrote:
> > > > >
> > > > > > Thanks Joe for the tip. Here is what I have
> > > > > > for org.apache.maven.surefire.booter.ForkedBooter though I am not
> > > sure
> > > > > what
> > > > > > to make of it.
> > > > > >
> > > > > > "main" #1 [23885] prio=5 os_prio=0 cpu=1694.03ms elapsed=158.14s
> > > > > > tid=0x7f1070028ec0 nid=23885 runnable  [0x7f1077b78000]
> > > > > >java.lang.Thread.State: RUNNABLE
> > > > > > at java.io.FileInputStream.readBytes(java.base@21.0.3
> > /Native
> > > > > > Method)
> > > > > > at java.io.FileInputStream.read(java.base@21.0.3
> > > > > > /FileInputStream.java:287)
> > > > > > at java.io.FilterInputStream.read(java.base@21.0.3
> > > > > > /FilterInputStream.java:119)
> > > > > > at
> > > > > >
> > sun.security.provider.NativePRNG$RandomIO.readFully(java.base@21.0.3
> > > > > > /NativePRNG.java:426)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(java.base@21.0.3
> > > > > > /NativePRNG.java:529)
> > > > > > at
> > > > > >
> > > >
> > sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@21.0.3
> > > > > > /NativePRNG.java:548)
> > > > > > - locked <0x0007107e1a88> (a java.lang.Object)
> > > > > > at
> > > 

Re: Cannot get build working

[Paul Grey]

In the past, I've observed application testing infrastructure (on VMs) hang
due to depletion of available entropy for RNG.  VMs seem especially
susceptible to this.  Our stopgap solution was to use haveged to make
sufficient entropy available to meet the needs of the test suite.

In this case, a refactor of the test would probably be a better
alternative.  Your observation on behavior in your environment with haveged
will help inform that decision.


On Mon, Jul 29, 2024 at 10:14 AM Dan S  wrote:

> Yes I have a VM on an EC2 instance in AWS.
>
> On Mon, Jul 29, 2024 at 10:10 AM Paul Grey  wrote:
>
> > Is this by chance a virtual machine you are on?
> >
> >
> >
> https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/
> >
> > TL;DR: try installing haveged
> >
> >
> > On Mon, Jul 29, 2024 at 9:29 AM Dan S  wrote:
> >
> > > So I see the problem line is
> > > at
> > >
> > >
> >
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
> > > and that line has
> > >  final String val = String.format("{\"large\": \"%s\"}",
> > > RandomStringUtils.randomAlphanumeric(1));
> > >
> > > I am not sure though what is making the process hang.
> > >
> > > On Mon, Jul 29, 2024 at 9:11 AM Dan S  wrote:
> > >
> > > > Thanks Joe for the tip. Here is what I have
> > > > for org.apache.maven.surefire.booter.ForkedBooter though I am not
> sure
> > > what
> > > > to make of it.
> > > >
> > > > "main" #1 [23885] prio=5 os_prio=0 cpu=1694.03ms elapsed=158.14s
> > > > tid=0x7f1070028ec0 nid=23885 runnable  [0x7f1077b78000]
> > > >java.lang.Thread.State: RUNNABLE
> > > > at java.io.FileInputStream.readBytes(java.base@21.0.3/Native
> > > > Method)
> > > > at java.io.FileInputStream.read(java.base@21.0.3
> > > > /FileInputStream.java:287)
> > > > at java.io.FilterInputStream.read(java.base@21.0.3
> > > > /FilterInputStream.java:119)
> > > > at
> > > > sun.security.provider.NativePRNG$RandomIO.readFully(java.base@21.0.3
> > > > /NativePRNG.java:426)
> > > > at
> > > >
> > >
> >
> sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(java.base@21.0.3
> > > > /NativePRNG.java:529)
> > > > at
> > > >
> > sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@21.0.3
> > > > /NativePRNG.java:548)
> > > > - locked <0x0007107e1a88> (a java.lang.Object)
> > > > at
> > > >
> > >
> >
> sun.security.provider.NativePRNG$Blocking.engineNextBytes(java.base@21.0.3
> > > > /NativePRNG.java:270)
> > > > at java.security.SecureRandom.nextBytes(java.base@21.0.3
> > > > /SecureRandom.java:768)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.CachedRandomBits.(CachedRandomBits.java:67)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:290)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:249)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:147)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:107)
> > > > at
> > > >
> > >
> >
> org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(RandomStringUtils.java:402)
> > > > at
> > > >
> > >
> >
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
> > > > at
> > > >
> > >
> >
> java.lang.invoke.LambdaForm$DMH/0x7f0ffc131000.invokeVirtual(java.base@21.0.3
> > > > /LambdaForm$DMH)
> > > > at
> > > >
> > java.lang.invoke.LambdaForm$MH/0x7f0ffc131800.invoke(java.base@21.0.3
> > > > /LambdaForm$MH)
> > > > at
> > > java.lang.inv

Re: Cannot get build working

[Paul Grey]

Is this by chance a virtual machine you are on?

https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/

TL;DR: try installing haveged


On Mon, Jul 29, 2024 at 9:29 AM Dan S  wrote:

> So I see the problem line is
> at
>
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
> and that line has
>  final String val = String.format("{\"large\": \"%s\"}",
> RandomStringUtils.randomAlphanumeric(1));
>
> I am not sure though what is making the process hang.
>
> On Mon, Jul 29, 2024 at 9:11 AM Dan S  wrote:
>
> > Thanks Joe for the tip. Here is what I have
> > for org.apache.maven.surefire.booter.ForkedBooter though I am not sure
> what
> > to make of it.
> >
> > "main" #1 [23885] prio=5 os_prio=0 cpu=1694.03ms elapsed=158.14s
> > tid=0x7f1070028ec0 nid=23885 runnable  [0x7f1077b78000]
> >java.lang.Thread.State: RUNNABLE
> > at java.io.FileInputStream.readBytes(java.base@21.0.3/Native
> > Method)
> > at java.io.FileInputStream.read(java.base@21.0.3
> > /FileInputStream.java:287)
> > at java.io.FilterInputStream.read(java.base@21.0.3
> > /FilterInputStream.java:119)
> > at
> > sun.security.provider.NativePRNG$RandomIO.readFully(java.base@21.0.3
> > /NativePRNG.java:426)
> > at
> >
> sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(java.base@21.0.3
> > /NativePRNG.java:529)
> > at
> > sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@21.0.3
> > /NativePRNG.java:548)
> > - locked <0x0007107e1a88> (a java.lang.Object)
> > at
> >
> sun.security.provider.NativePRNG$Blocking.engineNextBytes(java.base@21.0.3
> > /NativePRNG.java:270)
> > at java.security.SecureRandom.nextBytes(java.base@21.0.3
> > /SecureRandom.java:768)
> > at
> >
> org.apache.commons.lang3.CachedRandomBits.(CachedRandomBits.java:67)
> > at
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:290)
> > at
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:249)
> > at
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:147)
> > at
> >
> org.apache.commons.lang3.RandomStringUtils.random(RandomStringUtils.java:107)
> > at
> >
> org.apache.commons.lang3.RandomStringUtils.randomAlphanumeric(RandomStringUtils.java:402)
> > at
> >
> org.apache.nifi.processors.elasticsearch.PutElasticsearchJsonTest.testLargeInputStringHandling(PutElasticsearchJsonTest.java:471)
> > at
> >
> java.lang.invoke.LambdaForm$DMH/0x7f0ffc131000.invokeVirtual(java.base@21.0.3
> > /LambdaForm$DMH)
> > at
> > java.lang.invoke.LambdaForm$MH/0x7f0ffc131800.invoke(java.base@21.0.3
> > /LambdaForm$MH)
> > at
> java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.3
> > /Invokers$Holder)
> > at
> >
> jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21.0.3
> > /DirectMethodHandleAccessor.java:153)
> > at
> > jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21.0.3
> > /DirectMethodHandleAccessor.java:103)
> > at java.lang.reflect.Method.invoke(java.base@21.0.3
> > /Method.java:580)
> > at
> >
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> > at
> >
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> > at
> >
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> > at
> >
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> > at
> >
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> > at
> >
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> > at
> >
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda/0x7f0ffc0ab128.apply(Unknown
> > Source)
> > at
> >
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> > at
> >
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall$$Lambda/0x7f0ffc0ab538.apply(Unknown
> > Source)
> > at
> >
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> > at
> >
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$$Lambda/0x7f0ffc0de488.apply(Unknown
> > Source)
> > at
> >
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> > at
> >
> org.junit.jupiter.engine.execution.InvocationIntercept

Re: [DISCUSS] 1.x Maintenance

[Paul Grey]

>
> Kafka 2.6 parity concern - Can you share more on what is missing?
> Obviously supporting Kafka users within NiFi is quite important but I think
> we're making the right steps here.  Whatever gap we have now let's identify
> and see who can knock it out.


I cited a few in the PR description [1].


>- The PR as is should support the following authentication strategies:
>PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. Support for additional
>authentication strategies could be handled via follow on JIRAs.
>- Support for the KafkaRecordSink form factor could be handled via
>follow on JIRAs.
>- Support for Kafka "exactly once" semantics could be handled via
>follow on JIRAs.
>- Migration documentation could be handled via follow on JIRAs.
>
> This should not be considered a complete list.  I consider it likely that
there are others that I did not identify.  I did try to capture that
thinking in the PR [2].

Of note, there is a follow on JIRA [3] to flag the authentication features
that were backed in *Kafka_2_6 by processor properties.  There likely
should be other JIRAs.

In my opinion, this upgrade is more involved than the upgrade from
*Kafka*_2_0 to *Kafka*_2_6, from a design perspective, an implementation
perspective, and from the perspective of representing usages in a flow
definition.  This has the potential to cause a lot of problems for the
community.

It is hard for me to weigh all of the factors that should go into the
retain / remove decision for Kafka 2.6, and I understand that I am a guest
at this party.  It does complicate things that the community is trying to
close the book on NiFi 2.0 with a minimum of technical debt. But I'm not
comfortable saying that the known (and unknown) functionality gaps can be
addressed in the timeline I've seen discussed elsewhere for 2.0 GA.  And I
don't think that 2.0 should be held up for this issue.  There is so much
good in the main line, when compared against 1.x.


[1] https://github.com/apache/nifi/pull/8463
[2] https://github.com/apache/nifi/pull/8463#issuecomment-2090939791
[3] https://issues.apache.org/jira/browse/NIFI-12952


On Mon, Jul 8, 2024 at 8:24 AM David Handermann 
wrote:

> All,
>
> The discussion thus far has highlighted some important aspects of
> project maintenance at both general and specific levels. Clearly
> everyone shares a common goal to provide a supported and maintainable
> product. The concerns mentioned reflect the natural tension in any
> kind of technical debt evaluation, so the challenge is to determine
> how to navigate those concerns.
>
> As Joe said, nothing is lost, the source code and previous releases
> remain available for both internal and external use.
>
> Regarding the timeline for handling the removal of particular
> features, it is important to note that nothing has occurred outside of
> standard procedures. Every change has had a Jira issue, a pull
> request, feedback, and signed-off commits. Regarding updates to the
> deprecated features page, that has always occurred after merging, not
> before, because it reflects committed changes, rather than proposed
> changes. I understand the concern Pierre raised regarding the pace of
> some changes, so if contributing members of the PMC see a need to have
> a standard waiting period for Jira issues or pull requests, we should
> consider that on a separate thread. Not having that right now, it
> remains important that both removals and additions provide some sort
> of rationale as background. Sometimes that only requires a sentence or
> two, sometimes more, but that also characterizes the majority of
> changes. We have had past discussions about the need for feature
> proposals, and if it would improve maintainability to consider review
> timelines, active contributors should discuss the options. For now,
> however, raising issues even after merging changes continues to be a
> valid way to handle things that may have been missed.
>
> On repository encryption itself, the conversation thus far rightly
> acknowledged the problems noted in the Jira issue [1] NIFI-13494.
> Removal of repository encryption is one several security-focused
> removals, along with support for historical encryption processors and
> application property protection. Although these removals will
> undoubtedly impact some users, this is an area where we cannot
> maintain extended support at the community level. Encryption features
> include an inherent promise of security. If certain encryption
> features have fundamental flaws, it is better to remove those features
> than to hold on to compatibility. Major version changes are the
> standard way to introduce breaking changes, and that has been one of
> the primary themes for NiFi 2.0. With that understanding in mind,
> there is value in considering future alternatives. If there are any
> unanswered questions about security-focused feature removals, I would
> be glad to provide more background as one who has both introduce

Re: [VOTE] Release Apache NiFi 2.0.0-M3 (RC1)

[Paul Grey]

+1 (non-binding)

- Verified signatures and hashes of [
https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3/] binaries
- Build of source successful (macOS 14 / Zulu21.28+85-CA (build 21+35))
- Configuration of three node dev cluster (Manual Keystore Generation)
- Application startup successful (set-single-user-credentials, dev cluster)
- Verified successful flow of data using known good flows (Kafka,
PublishSlack)
- Light testing of load balance operations
  - Connection / Round-Robin strategy
- Light testing of cluster disconnect / offload / connect operations
  - No FlowFile activity
  - Light FlowFile activity
  - NIFI-13249 logged for transient offload failure



On Mon, May 13, 2024 at 11:49 PM David Handermann <
exceptionfact...@apache.org> wrote:

> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M3.
>
> Please review the following guide for how to verify a release candidate
> build:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on and the convenience binaries are available
> on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1269
>
> Git Tag: nifi-2.0.0-M3-RC1
> Git Commit ID: f2215c6522a5571189290760c55f0317f8562cbd
> GitHub Commit Link:
>
> https://github.com/apache/nifi/commit/f2215c6522a5571189290760c55f0317f8562cbd
>
> Checksums of nifi-2.0.0-M3-source-release.zip
>
> SHA256: d471a0a9e4e04faf13bbe13c7d83be6f8002fba598bf0968a3c1b339802a185a
> SHA512:
> cd0b8bbd3fe4ea7ebe8cdac6ac8a7afa97fd7b6a521c2cbcb2c0cdc94899b652bf3726c8fe432e16f44a9dc81907414bbb42e03113f0cd9fb51aa1de9cd727a7
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/exceptionfactory.asc
>
> KEYS file is available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> Issues resolved for this version: 411
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155
>
> Release note highlights can be found on the project wiki:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [] +1 Release this package as nifi-2.0.0-M3
> [] +0 no opinion
> [] -1 Do not release this package because...
>

Re: NiFi Manual Keystore - Generate cluster node certificate chain

[Paul Grey]

Matthew,

Thanks much for identifying this issue with the documentation.  I see the
same error message when running step 8.  I've created a JIRA and a pull
request to correct the documentation.

https://issues.apache.org/jira/browse/NIFI-12814
https://github.com/apache/nifi/pull/8424

On Fri, Feb 16, 2024 at 10:05 PM Matthew Wilson
 wrote:

> Hi Devs,
> Recently found that in the steps to create Manual Keystore the step 6 for
> "Generate cluster node certificate chain" currently shows as:
> cat ca.cer nifi1.cer >nifi1.chain.cercat ca.cer nifi2.cer >nifi2.chain.cer
>
> However this creates the chain.cer files incorrectly which generates  the
> error :error:05800074:x509 certificate routines:X509_check_private_key:key
> values mismatch:crypto/x509/x509_cmp.c:408:
> When running step 8 "Generate cluster node keystore"
> Correct syntax for step 6 that works is below:
> cat nifi1.cer ca.cer >nifi1.chain.cercat nifi2.cer ca.cer >nifi2.chain.cer
> Reference URL Page:
> https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/walkthroughs.html#manual-keystore
>
> Please review and correct documentation as appropriate.  If you need more,
> information let me know.
>
>
> Kind Regards,
> Matthew WilsonEmail: wilson...@yahoo.com.au
>
> This e-mail may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorised to receive for the recipient), please contact the sender by
> reply e-mail and delete all copies of this message.

Re: [DISCUSS] Deprecate TLS Toolkit for Removal?

[Paul Grey]

I spent a bit of time working on a draft of the needed openssl commands to
generate equivalent keystores / truststores.  Was not able to finish this
draft before I got side tracked.

https://github.com/greyp9/nifi-tidbits/blob/main/cluster/tls-toolkit-via-openssl/cluster-certs.md

If you're still offering, I would be interested to see the equivalent
keytool commands, to compare and contrast.  Maybe both could be
documented...

If we can reduce or eliminate the needed maintenance efforts associated
with TLS Toolkit, this seems worth some one-time effort.


On Sun, Sep 17, 2023 at 1:46 AM Matthew Hawkins  wrote:

> NiFi in certain configurations requires a specific set of x509_v3
> extensions, e.g. when using InvokeHTTP. This is automatically added when
> using TLS Toolkit but all other options relies upon the CA either accepting
> and passing through extensions in the request, or specifically adding them
> in.
>
> You cannot use a stock standard web cert signing authority as they usually
> ignore passed extensions and just sign a server-only cert, which will
> result in a dysfunctional NiFi. ADCS doesn't have a valid default template
> for NiFi and many installations of that don't deviate from default
> templates.
>
> If you don't use certain processors it's certainly easy to just sign a
> server cert and not know anything is amiss. This shouldn't be the default.
> It's 15 sequential Java keytool commands to build a proper CA chain and
> sign a NiFi cert. If TLS Toolkit is being pulled, a script to invoke
> keytool should be included at the same time.
>
> I'm happy to provide a basic solution to this if asked.
>
> I like keytool as it is available on all platforms and uses the right
> formats NiFi requires for keystore and truststore. The open source Keytool
> Explorer is also a nice way to build and manage the stores, not strictly
> necessary but nice to have.
>
>
>
>
> On Thu, 14 Sept 2023, 05:58 Kevin Doran,  wrote:
>
> >  I agree that tis toolkit can probably be removed in 2.0, and I would add
> > that tinycert.org provides another option for teams that need to setup
> > dev/test environments with trusted certificates.
> >
> > Thanks,
> > Kevin
> >
> > On Sep 13, 2023 at 11:46:19, David Handermann <
> exceptionfact...@apache.org
> > >
> > wrote:
> >
> > > Hi Isha,
> > >
> > > Thanks for the helpful reply. I agree that the TLS Toolkit is most
> > > convenient for development and lab deployments, and that's where we
> > > should be able to provide some documentation for alternatives. The
> > > existing Secure Cluster Walkthrough is a helpful reference for TLS
> > > Toolkit usage, so if we updated that to provide similar guidance using
> > > other tools, that would be useful.
> > >
> > > Getting everything right with TLS can be challenging, but when it
> > > comes to project maintenance, it seems better to focus on core
> > > capabilities and not maintain the TLS Toolkit if the primary use case
> > > is for non-production deployments.
> > >
> > > The encrypt-config command is a different question, but a very good
> > > question. It includes functionality that is very specific to NiFi, and
> > > it is also in need of refactoring. The threat model for containerized
> > > deployments may be somewhat different than running directly on
> > > physical hardware. With the need to support various approaches,
> > > however, some type of configuration encryption remains a relevant
> > > concern.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Wed, Sep 13, 2023 at 10:19 AM Isha Lamboo
> > >  wrote:
> > >
> > >
> > > Hi David,
> > >
> > >
> > > My primary use for the TLS toolkit is for lab deployments, mostly
> during
> > > in-house trainings. I will miss the convenience of having a full set of
> > > keystores and truststores ready to go with a single command, but then
> > > again, a few commands in a script should replicate this well enough,
> > > without the need for maintaining the toolkit.
> > >
> > >
> > > I see no obstacles to adopting NiFi 2.0 if the TLS toolkit is phased
> out,
> > > from the perspective of the deployments I manage.
> > >
> > >
> > > On a side note: How relevant is the encrypt config part of the toolkit
> > > still in a mostly containerized world?
> > >
> > >
> > > Regards,
> > >
> > >
> > > Isha
> > >
> > >
> > > -Oorspronkelijk bericht-
> > >
> > > Van: David Handermann 
> > >
> > > Verzonden: woensdag 13 september 2023 15:16
> > >
> > > Aan: dev@nifi.apache.org
> > >
> > > Onderwerp: [DISCUSS] Deprecate TLS Toolkit for Removal?
> > >
> > >
> > > Team,
> > >
> > >
> > > The TLS Toolkit provides a number of useful features for securing NiFi
> > > server communication, but it also presents several maintenance
> concerns.
> > In
> > > light of other available tools, I am raising the question of removing
> the
> > > TLS Toolkit from the repository as part of NiFi 2.0 technical debt
> > > reduction.
> > >
> > >
> > > With the addition of automatic self-signed certificate generat

Re: ListFTP not taking into account french special characters in filename [Nifi 1.21.0]

[Paul Grey]

This issue sounds similar to this one:
- https://issues.apache.org/jira/browse/NIFI-10143

Are you able to share OS/JRE details for the FTP server and for the NiFi
instance?


On Wed, Aug 16, 2023 at 12:37 PM Ravinder Singh <
ravinder.si...@energisme.com> wrote:

> Hello,
>
>
>
> First of all, I would like to thank you for the great work on Nifi.
>
>
>
> Would it be possible for you to help me on the following problem, please ?
>
>
>
> I have upgraded nifi from 1.16.3 to 1.21.0
>
>
>
> Now, the processors ListFTP, FetchFTP, PutFTP won't work with filenames
> that contains french accents like à é etc..
>
>
>
> It worked like a charm on Nifi 1.16.3
>
>
>
> I will be happy to provide all the information that you need to debug the
> problem.
>
>
>
>
>
>
>
>
>
> PS : I created a post on cloudera =>
> https://community.cloudera.com/t5/Support-Questions/ListFTP-not-taking-into-account-french-special-characters-in/td-p/375276
>
>
>
> Thank you very much.
>
>
>
> Ravinder Singh.
>

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

[Paul Grey]

+1 (non-binding)

- Verified signatures and hashes of [
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/] binaries

- Build of source successful (macOS 12 / Azul Java 1.8.0_322)
- Application startup successful
- Verified successful flow of data using known good flows (Kafka, PutSlack)
- verify NIFI-4890 - OIDC refactor (backport) (Okta dev account)

- Application startup successful (set-single-user-credentials, three node
dev cluster)
- Verified successful flow of data using known good flows (Kafka, PutSlack)
- Testing of cluster disconnect / offload / connect operations

- Build of source successful (Debian 10.13 / OpenJDK 1.8.0_362)
- Application startup successful
- Verified successful flow of data using known good flows (PutSlack)

- Captured test failure (Linux) as NIFI-11391 to investigate.

On Thu, Apr 6, 2023 at 8:31 AM Gábor Gyimesi  wrote:

> +1 (non-binding)
>
> Verified signatures and hashes
> Built NiFi on ubuntu 22.04
> Ran a simple flow and a flow integrated with MiNiFi C++ through InvokeHTTP
>
> LGTM
>
> Thanks Joe!
>
> BR,
> Gabor
>
> On Thu, 6 Apr 2023 at 14:15, Martin Zink  wrote:
>
> > +1 (non-binding)
> >
> > Went through the helper guide verified signatures and hashes.
> > I've also verified that integration with the latest MiNiFi C++ works as
> > expected (InvokeHTTP based flows)
> >
> > Everything looks good
> >
> > Thanks Joe for RMing!
> >
> > On Thu, Apr 6, 2023 at 9:47 AM Ferenc Kis 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Went through the helper guide, full clean build with contrib check,
> > > verified signatures and hashes
> > >   Maven home: /Users/fkis/.sdkman/candidates/maven/current
> > >   Java version: 11.0.18, vendor: Eclipse Adoptium, runtime:
> > > /Users/fkis/.sdkman/candidates/java/11.0.18-tem
> > >   Default locale: en_HU, platform encoding: UTF-8
> > >   OS name: "mac os x", version: "13.2.1", arch: "aarch64", family:
> "mac"
> > > - Started NiFi, created a simple flow with ListenHTTP
> > > - Started Nifi Registry, set up integration with NiFi, and saved the
> > > previous flow to NiFi Registry
> > > - Started MiNiFi Java:
> > >   * created a simple GenerateFlowFile -> InvokeHttp flow and pushed to
> > > MiNiFi via C2 protocol
> > >   * managed to push data from MiNiFi to NiFi
> > >
> > > Thanks Joe for RMing!
> > >
> > > Regards
> > > Ferenc
> > >
> > > On Thu, Apr 6, 2023 at 9:06 AM Pierre Villard <
> > pierre.villard...@gmail.com
> > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > Went through the usual steps. Deployed a 3 node secured cluster on
> GCP
> > > with
> > > > a secure NiFi Registry. Tested some flows. All good.
> > > >
> > > > Thanks Joe for taking care of the release!
> > > >
> > > > Le jeu. 6 avr. 2023 à 05:11, Mark Bean  a
> > écrit :
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Verified signatures and hashes.
> > > > > Performed full build with -Pcontrib-check using maven 3.6.3 and
> Java
> > > > > 11.0.18 on Ubuntu 20.04.1.
> > > > > Installed and ran NiFi again using Java 11.0.18. Executed some
> basic
> > > > flows.
> > > > >
> > > > >
> > > > > On Mon, Apr 3, 2023 at 8:11 PM Joe Witt 
> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I am pleased to be calling this vote for the source release of
> > Apache
> > > > > > NiFi 1.21.0.
> > > > > >
> > > > > > The source zip, including signatures, digests, etc. can be found
> > at:
> > > > > >
> > > https://repository.apache.org/content/repositories/orgapachenifi-1226
> > > > > >
> > > > > > The source being voted upon and the convenience binaries can be
> > found
> > > > at:
> > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/
> > > > > >
> > > > > > A helpful reminder on how the release candidate verification
> > process
> > > > > works:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > > > > >
> > > > > > The Git tag is nifi-1.21.0-RC2
> > > > > > The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1
> > > > > >
> > > > > > Checksums of nifi-1.21.0-source-release.zip:
> > > > > > SHA256:
> > > > 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5
> > > > > > SHA512:
> > > > > >
> > > > >
> > > >
> > >
> >
> 58c10976bc22fb5406d9d1d9b7ca7d90c2dbed99565d00f1172bb33b375e9e068da5003d9dbbd87d2b17626e4e310b999c8581718532934e855c2134be763f04
> > > > > >
> > > > > > Release artifacts are signed with the following key:
> > > > > > https://people.apache.org/keys/committer/joewitt.asc
> > > > > >
> > > > > > KEYS file available here:
> > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > > > >
> > > > > > 126 issues were closed/resolved for this release:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/R

Re: NiFi-10751 - CI Tests Not Running

[Paul Grey]

Looks like the PR build ran successfully; I see that the unit tests ran in
the build output.
https://github.com/apache/nifi/actions/runs/3386010384

You may be looking for automation runs here?
https://github.com/shawnweeks/nifi/actions

This is something that is disabled by default, but you can enable it in
configuration for your github account .

On Thu, Nov 3, 2022 at 10:01 AM Shawn Weeks 
wrote:

> I made an update to a pr I was working on and it’s not running the tests
> again for some reason. I haven’t done any PRs since NiFi switch over to
> having Github run them so I might be doing something wrong. Do I need to
> close the PR and recreate?
>
> [image: 6614.png]
>
> NIFI-10751 Add Username to User and Group CRUD by shawnweeks · Pull
> Request #6614 · apache/nifi 
> github.com 
> 
>
> Thanks
> Shawn
>

Re: [VOTE] Release Apache NiFi 1.17.0 (rc1)

[Paul Grey]

+1 (non-binding)

- Verified signatures and hashes of [
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/] binaries
- Build of source successful (macOS 12 / Azul Java 1.8.0_322)
- Application startup successful (set-single-user-credentials, three node
dev cluster)
- Verified successful flow of data using known good flows (Kafka, PutSlack)
- Testing of cluster disconnect / offload / connect operations
- verify NIFI-9996 - ListFile does not list root directory
- spot check of NIFI-10143 - UTF-8 auto-detection in FTP Processors


On Tue, Jul 26, 2022 at 5:39 PM Joe Skora  wrote:

> Good afternoon,
>
> +1 (binding) for release
>
> * completed full build with 1 and 3 threads  environment:
>   - MacOS 12.4 on Intel Core i9
>   - AdoptOpenJDK (build 1.8.0_292-b10)
>   - Maven 3.8.5
>   - Locale: en_US.UTF-8
> * complete helper guide review: signatures, hashes, build, readme, notice,
> license, etc.
> * tested convenience binary by creating a simple flow on the canvas
>
> Regards,
> Joe
>
>
> On Mon, Jul 25, 2022 at 10:58 PM Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.17.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1209
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.17.0-RC1
> > The Git commit ID is cfa99de02ab5a440e94a2bb0044822ae9d0d99e7
> >
> >
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=cfa99de02ab5a440e94a2bb0044822ae9d0d99e7
> >
> > Checksums of nifi-1.17.0-source-release.zip:
> > SHA256: 060ac40bbd6f3bac08f724bf040a68d52f4449d2e529af9c1d61db8932aa7b1b
> > SHA512:
> >
> >
> 97440184d0c34e1f287eba648eb47e5c276df46d6d89c376616939380d95afb6c03b4e4e9ea4170ba62d300637387d532314381225d000d529a6cc5bc4e8b436
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 305 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351438
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.17.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test. Then
> > please vote:
> >
> > [ ] +1 Release this package as nifi-1.17.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 1.16.2 (RC3)

[Paul Grey]

+1 (non-binding)

- Verified signatures and hashes of [
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.2/] binaries
- Build of source successful (macOS 12 / Azul Java 1.8.0_322)
- Application startup successful (set-single-user-credentials, five node
dev cluster)
- Verified successful flow of data using known good flows (Kafka, PutSlack)
- Testing of cluster disconnect / offload / connect operations
- JIRAs NIFI-10053, NIFI-10055 created for minor issues

On Tue, May 24, 2022 at 7:09 PM Nathan Gough  wrote:

> +1 (non-binding)
>
> Tested out 3 node secure cluster and external ZK with a few different
> processors, set up and tested Google OIDC authentication.
>
> Nathan
>
> On Tue, May 24, 2022 at 5:54 PM Mark Payne  wrote:
>
> > +1 (binding)
> >
> > Performed build with Java 8.
> > Ran with Java 8 and Java 11.
> > Tested insecure standalone installation
> > Tested secured (via certificate) clustered installation
> >
> > Did heavy testing against controller services, restarts, trying to
> > introduce corner cases.
> > Validated that NIFI-10001 was addressed.
> >
> > I did encounter a couple of bugs that I created Jiras for but none were
> > critical and none were regressions introduced in this release.
> >
> > Thanks for handling the RM duties again Joe!
> >
> > -Mark
> >
> >
> > > On May 22, 2022, at 11:44 PM, Joe Witt  wrote:
> > >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 1.16.2.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1203
> > >
> > > The source being voted upon and the convenience binaries can be found
> at:
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.2/
> > >
> > > A helpful reminder on how the release candidate verification process
> > works:
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-1.16.2-RC3
> > > The Git commit ID is 06f04958272dafc30ce357c4c4edcaf470050b52
> > >
> >
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=06f04958272dafc30ce357c4c4edcaf470050b52
> > >
> > > Checksums of nifi-1.16.2-source-release.zip:
> > > SHA256:
> 1fecf7d9f6001cc8e58d4a46ece08e141de705bcd227338ba79e9cb574267415
> > > SHA512:
> > >
> >
> 1f4fd4e5e9f24949830a75949b302a67b8826049406ab8296c4b8c99a5a0aa1d211f84f98699b3af6fb41efa305f35a3f85b21d7958dc09c027cc1ed836c169f
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/joewitt.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 34 issues were closed/resolved for this release:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351721
> > >
> > > Release note highlights can be found here:
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.2
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.16.2
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >
> >
>

request Jira contributor access

[Paul Grey]

Jira username: pgrey