Re: Nifi - Content-repo on AWS-EBS volumes

[Gregory M. Foreman]

Mark:

Got it.  Thank you for the help.

Greg

> On Dec 15, 2023, at 4:14 PM, Mark Payne  wrote:
> 
> Greg,
> 
> Whether or not multiple content repos will have any impact depends very much 
> on where your system’s bottleneck is. If your bottleneck is disk I/O, it will 
> absolutely help. If your bottleneck is CPU, it won’t. If, for example, you’re 
> running on bare metal and have 48 cores on your machine and you’re running 
> with spinning disks, you’ll definitely want to use multiple spinning disks. 
> But if you’re running in AWS on a VM that has 4 cores and you’re using gp3 
> EBS volumes, it’s unlikely that multiple content repos will help.
> 
> Thanks
> -Mark
> 
> 
> 
>> On Dec 15, 2023, at 3:25 PM, Gregory M. Foreman 
>>  wrote:
>> 
>> Mark:
>> 
>> I was just discussing multiple content repos on EBS volumes with a 
>> colleague.  I found your post from a long time ago:
>> 
>> https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv
>> 
>> “Re #2: I don't know that i've used any SAN to back my repositories other 
>> than the EBS provided by Amazon EC2. In that environment, I found that 
>> having one or having multiple repos was essentially equivalent.”
>> 
>> Does that statement still hold true today?  Essentially there is no real 
>> performance benefit to having multiple content repos on multiple EBS volumes?
>> 
>> Thanks,
>> Greg
>> 
>> 
>> 
>>> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
>>> 
>>> Hey Phil,
>>> 
>>> NiFi will not spread the content of a single file over multiple partitions. 
>>> It will write the content of FlowFile 1 to content repo 1, then write the 
>>> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
>>> single FlowFile across multiple repos.
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
>>>> 
>>>> 
>>>> Hello Nifi comrades,
>>>> 
>>>> Here's my scenario...
>>>> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
>>>> volumes serving as their repos.  They've split up their content-repos into 
>>>> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
>>>> EBS volume.  My understanding is that the content-claims for a single file 
>>>> can potentially span across more than one of these repos.(correct me if 
>>>> I've lost my mind over the years)
>>>> For instance if you have a 1 MB file, and lets say your 
>>>> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
>>>> split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
>>>> to S3 or something for instance... it needs to be read from each of the 
>>>> volumes.  
>>>> Whereas if it was a single EBS volume for the cont-repo... it would read 
>>>> from the single volume, which I would think would be more performant?  Or 
>>>> does spreading out any IO contention across volumes provide more of a 
>>>> benefit?
>>>> I know there's different levels of EBS volumes... but not factoring that 
>>>> in for right now.
>>>> 
>>>> Appreciate any insight... trying to determine the best configuration.  
>>>> 
>>>> Thanks,
>>>> Phil
>>>> 
>>>> 
>> 
> 

Re: Nifi - Content-repo on AWS-EBS volumes

[Gregory M. Foreman]

Mark:

I was just discussing multiple content repos on EBS volumes with a colleague.  
I found your post from a long time ago:

https://lists.apache.org/thread/nq3mpry0wppzrodmldrcfnxwzp3n1cjv

“Re #2: I don't know that i've used any SAN to back my repositories other than 
the EBS provided by Amazon EC2. In that environment, I found that having one or 
having multiple repos was essentially equivalent.”

Does that statement still hold true today?  Essentially there is no real 
performance benefit to having multiple content repos on multiple EBS volumes?

Thanks,
Greg



> On Dec 11, 2023, at 8:50 PM, Mark Payne  wrote:
> 
> Hey Phil,
> 
> NiFi will not spread the content of a single file over multiple partitions. 
> It will write the content of FlowFile 1 to content repo 1, then write the 
> next FlowFile to repo 2, etc. so it does round-robin but does not spread a 
> single FlowFile across multiple repos.
> 
> Thanks
> -Mark
> 
> Sent from my iPhone
> 
>> On Dec 11, 2023, at 8:45 PM, Phillip Lord  wrote:
>> 
>> 
>> Hello Nifi comrades,
>> 
>> Here's my scenario...
>> Let's say I have a Nifi cluster running on EC2 instances with attached EBS 
>> volumes serving as their repos.  They've split up their content-repos into 
>> three content-repos per node(cont1, cont2, cont3).  Each being a dedicated 
>> EBS volume.  My understanding is that the content-claims for a single file 
>> can potentially span across more than one of these repos.(correct me if I've 
>> lost my mind over the years)
>> For instance if you have a 1 MB file, and lets say your 
>> max.content.claim.size is 100KB, that's 10 - 100KB claims(ish) potentially 
>> split up across the 3 EBS volumes.  So if Nifi is trying to move that file 
>> to S3 or something for instance... it needs to be read from each of the 
>> volumes.  
>> Whereas if it was a single EBS volume for the cont-repo... it would read 
>> from the single volume, which I would think would be more performant?  Or 
>> does spreading out any IO contention across volumes provide more of a 
>> benefit?
>> I know there's different levels of EBS volumes... but not factoring that in 
>> for right now.
>> 
>> Appreciate any insight... trying to determine the best configuration.  
>> 
>> Thanks,
>> Phil
>> 
>> 

Re: [EXTERNAL] Large json string field errors

[Gregory M. Foreman]

Created a jira ticket for this:

https://issues.apache.org/jira/browse/NIFI-12117



> On Sep 19, 2023, at 5:09 PM, Bryan Bende  wrote:
> 
> Yes, even if we don't expose a property and pick a very large default, it is 
> going to require going through each extension one by one, since they all 
> control the creation of their own ObjectMapper. 
> 
> We may be able to create a shared utility module that they could all depend 
> on with a factory to create an ObjectMapper.
> 
> For this particular case it came from JoltTransformJson which uses a jolt 
> class called JsonUtils to call
> final Object inputJson;
> try (final InputStream in = session.read(original)) {
> inputJson = JsonUtils.jsonToObject(in);
> }
> 
> It has a helper to create a util instance with your own ObjectMapper, so we'd 
> have to switch to calling that:
> 
> JsonUtil customJsonUtil( ObjectMapper mapper )
> 
> 
> On Tue, Sep 19, 2023 at 5:05 PM David Handermann  <mailto:exceptionfact...@apache.org>> wrote:
> Changing the default value to an unlimited number would negate the protection 
> that the new restriction provides.
> 
> It should not be necessary to make global changes, but focusing on Processor 
> and Controller Service references would be a good starting point for 
> introducing configurable sizes.
> 
> Regards,
> David Handermann
> 
> On Tue, Sep 19, 2023 at 3:45 PM Chris Sampson  <mailto:chris.samp...@naimuri.com>> wrote:
> Probably worth noting that the Jackson ObjectMapper is used in lots of places 
> across the code base - 182 hits in GitHub on current main branch (some of 
> those are duplicates within classes).
> 
> So it could be a significant change to update and test all places where it's 
> currently used. The string length limit might not be limited to the 
> ObjectMapper either, so that would need checking and working through.
> 
> A more centralised configuration might be worthwhile if possible.
> 
> 
> On Tue, 19 Sept 2023, 20:42 Gregory M. Foreman, 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Defaulting maxStringLength to Integer.MAX_VALUE would be my vote.  It would 
> work as it had previously and as expected.  I do not currently see a use case 
> where my client would want to restrict the size of a json string…using 
> fileSize to route is adequate.
> 
>> On Sep 18, 2023, at 4:53 PM, Bryan Bende > <mailto:bbe...@gmail.com>> wrote:
>> 
>> It looks like Jackson 2.15 introduced a max length for a single string value 
>> inside of the json document which defaults to 20MB.
>> 
>> It can be configured on the ObjectMapper's factory [1] like:
>> 
>> objectMapper.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(4000).build()));
>> 
>> If we want to make this configurable, we'd probably have to expose a 
>> property in any controller-service/processor that does json parsing with an 
>> ObjectMapper.
>> 
>> [1] 
>> https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940
>>  
>> <https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940>
>> On Mon, Sep 18, 2023 at 4:23 PM Gregory M. Foreman 
>> mailto:gfore...@spinnerconsulting.com>> 
>> wrote:
>> Yes
>> 
>> Sent from my iPhone
>> 
>>> On Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users 
>>> mailto:users@nifi.apache.org>> wrote:
>>> 
>>> 
>>> Is this what you are referring to? 20MB string values vs 20 MB document 
>>> size…
>>> 
>>>  
>>> 
>>> https://github.com/FasterXML/jackson-core/issues/1082 
>>> <https://github.com/FasterXML/jackson-core/issues/1082>
>>>  
>>> 
>>>  
>>> 
>>> From: Joe Witt mailto:joe.w...@gmail.com>> 
>>> Sent: Monday, September 18, 2023 2:25 PM
>>> To: users@nifi.apache.org <mailto:users@nifi.apache.org>
>>> Subject: [EXTERNAL] Re: Large json string field errors
>>> 
>>>  
>>> 
>>> EXT email: be mindful of links/attachments.
>>> 
>>> 
>>>  
>>> Greg
>>> 
>>>  
>>> 
>>> Can you share details on how you're seeing this limit?  Is there a 
>>> resulting stack trace or other output for it in the nifi-app.log?  What 
>>> version of NiFi are you on?
>>> 
>>>  
>>> 
>>> Thanks
>>> 
>>>  
>>> 
>>> On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman 
>>> mailto:gfore...@spinnerconsulting.com>> 
>>> wrote:
>>> 
>>> Hello:
>>> 
>>> It appears Jackson recently set a 20mb upper boundary on json string field 
>>> length.  Is there a way to remove/override this limit in Nifi?
>>> 
>>> Thanks,
>>> Greg
>>> 
> 

Re: [EXTERNAL] Large json string field errors

[Gregory M. Foreman]

"Changing the default value to an unlimited number would negate the protection 
that the new restriction provides.”

Yes, requesting to negate it.  If a dev needs to reduce the bounds, then 
provide a mechanism for this in the processors in a future Nifi release.

From:

https://github.com/FasterXML/jackson-core/issues/1014#issuecomment-1581657373

"One takeway for me, too, is that I should have heeded my own "never add a 
feature just for sake of consistency; only implement things that have been 
specifically requested".
That'd have worked well here, addressing just number length & nesting depth -- 
both of which have specific known threats.”

So there is (currently) no security reason for this change.  Worth noting is 
that number length and nesting depth limits have been introduced as well.


> On Sep 19, 2023, at 5:03 PM, David Handermann  
> wrote:
> 
> Changing the default value to an unlimited number would negate the protection 
> that the new restriction provides.
> 
> It should not be necessary to make global changes, but focusing on Processor 
> and Controller Service references would be a good starting point for 
> introducing configurable sizes.
> 
> Regards,
> David Handermann
> 
> On Tue, Sep 19, 2023 at 3:45 PM Chris Sampson  <mailto:chris.samp...@naimuri.com>> wrote:
> Probably worth noting that the Jackson ObjectMapper is used in lots of places 
> across the code base - 182 hits in GitHub on current main branch (some of 
> those are duplicates within classes).
> 
> So it could be a significant change to update and test all places where it's 
> currently used. The string length limit might not be limited to the 
> ObjectMapper either, so that would need checking and working through.
> 
> A more centralised configuration might be worthwhile if possible.
> 
> 
> On Tue, 19 Sept 2023, 20:42 Gregory M. Foreman, 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Defaulting maxStringLength to Integer.MAX_VALUE would be my vote.  It would 
> work as it had previously and as expected.  I do not currently see a use case 
> where my client would want to restrict the size of a json string…using 
> fileSize to route is adequate.
> 
>> On Sep 18, 2023, at 4:53 PM, Bryan Bende > <mailto:bbe...@gmail.com>> wrote:
>> 
>> It looks like Jackson 2.15 introduced a max length for a single string value 
>> inside of the json document which defaults to 20MB.
>> 
>> It can be configured on the ObjectMapper's factory [1] like:
>> 
>> objectMapper.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(4000).build()));
>> 
>> If we want to make this configurable, we'd probably have to expose a 
>> property in any controller-service/processor that does json parsing with an 
>> ObjectMapper.
>> 
>> [1] 
>> https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940
>>  
>> <https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940>
>> On Mon, Sep 18, 2023 at 4:23 PM Gregory M. Foreman 
>> mailto:gfore...@spinnerconsulting.com>> 
>> wrote:
>> Yes
>> 
>> Sent from my iPhone
>> 
>>> On Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users 
>>> mailto:users@nifi.apache.org>> wrote:
>>> 
>>> 
>>> Is this what you are referring to? 20MB string values vs 20 MB document 
>>> size…
>>> 
>>>  
>>> 
>>> https://github.com/FasterXML/jackson-core/issues/1082 
>>> <https://github.com/FasterXML/jackson-core/issues/1082>
>>>  
>>> 
>>>  
>>> 
>>> From: Joe Witt mailto:joe.w...@gmail.com>> 
>>> Sent: Monday, September 18, 2023 2:25 PM
>>> To: users@nifi.apache.org <mailto:users@nifi.apache.org>
>>> Subject: [EXTERNAL] Re: Large json string field errors
>>> 
>>>  
>>> 
>>> EXT email: be mindful of links/attachments.
>>> 
>>> 
>>>  
>>> Greg
>>> 
>>>  
>>> 
>>> Can you share details on how you're seeing this limit?  Is there a 
>>> resulting stack trace or other output for it in the nifi-app.log?  What 
>>> version of NiFi are you on?
>>> 
>>>  
>>> 
>>> Thanks
>>> 
>>>  
>>> 
>>> On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman 
>>> mailto:gfore...@spinnerconsulting.com>> 
>>> wrote:
>>> 
>>> Hello:
>>> 
>>> It appears Jackson recently set a 20mb upper boundary on json string field 
>>> length.  Is there a way to remove/override this limit in Nifi?
>>> 
>>> Thanks,
>>> Greg
>>> 
> 

Re: [EXTERNAL] Large json string field errors

[Gregory M. Foreman]

Defaulting maxStringLength to Integer.MAX_VALUE would be my vote.  It would 
work as it had previously and as expected.  I do not currently see a use case 
where my client would want to restrict the size of a json string…using fileSize 
to route is adequate.

> On Sep 18, 2023, at 4:53 PM, Bryan Bende  wrote:
> 
> It looks like Jackson 2.15 introduced a max length for a single string value 
> inside of the json document which defaults to 20MB.
> 
> It can be configured on the ObjectMapper's factory [1] like:
> 
> objectMapper.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(4000).build()));
> 
> If we want to make this configurable, we'd probably have to expose a property 
> in any controller-service/processor that does json parsing with an 
> ObjectMapper.
> 
> [1] 
> https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940
>  
> <https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940>
> On Mon, Sep 18, 2023 at 4:23 PM Gregory M. Foreman 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Yes
> 
> Sent from my iPhone
> 
>> On Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users 
>> mailto:users@nifi.apache.org>> wrote:
>> 
>> 
>> Is this what you are referring to? 20MB string values vs 20 MB document size…
>> 
>>  
>> 
>> https://github.com/FasterXML/jackson-core/issues/1082 
>> <https://github.com/FasterXML/jackson-core/issues/1082>
>>  
>> 
>>  
>> 
>> From: Joe Witt mailto:joe.w...@gmail.com>> 
>> Sent: Monday, September 18, 2023 2:25 PM
>> To: users@nifi.apache.org <mailto:users@nifi.apache.org>
>> Subject: [EXTERNAL] Re: Large json string field errors
>> 
>>  
>> 
>> EXT email: be mindful of links/attachments.
>> 
>> 
>>  
>> Greg
>> 
>>  
>> 
>> Can you share details on how you're seeing this limit?  Is there a resulting 
>> stack trace or other output for it in the nifi-app.log?  What version of 
>> NiFi are you on?
>> 
>>  
>> 
>> Thanks
>> 
>>  
>> 
>> On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman 
>> mailto:gfore...@spinnerconsulting.com>> 
>> wrote:
>> 
>> Hello:
>> 
>> It appears Jackson recently set a 20mb upper boundary on json string field 
>> length.  Is there a way to remove/override this limit in Nifi?
>> 
>> Thanks,
>> Greg
>> 

Re: [EXTERNAL] Re: Large json string field errors

[Gregory M. Foreman]

YesSent from my iPhoneOn Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users  wrote:







Is this what you are referring to? 20MB string values vs 20 MB document size…
 
https://github.com/FasterXML/jackson-core/issues/1082
 
 
From: Joe Witt  
Sent: Monday, September 18, 2023 2:25 PM
To: users@nifi.apache.org
Subject: [EXTERNAL] Re: Large json string field errors
 











EXT email: be mindful of links/attachments.





 





Greg

 


Can you share details on how you're seeing this limit?  Is there a resulting stack trace or other output for it in the nifi-app.log?  What version of NiFi are you on?


 


Thanks


 


On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman <gfore...@spinnerconsulting.com> wrote:


Hello:

It appears Jackson recently set a 20mb upper boundary on json string field length.  Is there a way to remove/override this limit in Nifi?

Thanks,
Greg

Re: Large json string field errors

[Gregory M. Foreman]

Joe:

It is Nifi 1.23.2.  The error happens when using JoltTransformJSON on a 
flowfile with a string field of >20 mb.  The message is output as an error to 
nifi-app.log and the bulletin.  In this case base64 content is being produced 
and added as a json field.

Here is further context:

https://github.com/FasterXML/jackson-core/issues/1014

Greg

Large json string field errors

[Gregory M. Foreman]

Hello:

It appears Jackson recently set a 20mb upper boundary on json string field 
length.  Is there a way to remove/override this limit in Nifi?

Thanks,
Greg

Re: source build issue - Nifi 1.23.0

[Gregory M. Foreman]

Thanks for confirming Joe.  Created a ticket:

https://issues.apache.org/jira/browse/NIFI-11917 
<https://issues.apache.org/jira/browse/NIFI-11917>

Greg

> On Aug 7, 2023, at 12:17 PM, Joe Witt  wrote:
> 
> Greg
> 
> Yeah seems likely there is a build issue with the RPM profile. I think we 
> just dont have enough people exercising that thing often enough anymore and 
> so when it breaks it is silent.  The only thing we meant to have an RPM for 
> as far as I recall was the nifi assembly itself but perhaps we had an RPM 
> target in the toolkit.  Probably best to file a JIRA from here with as much 
> detail as you can including the output of 'mvn -version'
> 
> Thanks
> 
> On Mon, Aug 7, 2023 at 8:59 AM Gregory M. Foreman 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Hello:
> 
> I am trying to build Nifi 1.23.0 from source using:
> 
> JAVA_OPTS="-Xms128m -Xmx4g"
> MAVEN_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Xms1024m 
> -Xmx3076m -XX:MaxPermSize=256m"
> 
> mvn -T C2.0 --batch-mode -Prpm -DskipTests clean install
> 
> but the following error is thrown:
> 
> [ERROR] Failed to execute goal 
> org.codehaus.mojo:rpm-maven-plugin:2.2.0:attached-rpm (build-bin-rpm) on 
> project nifi-toolkit-assembly: Source location 
> /root/test/nifi-1.23.0/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.23.0-bin/nifi-toolkit-1.23.0/LICENSE
>  does not exist -> [Help 1]
> 
> The path above ends at:
> 
> /root/test2/nifi/nifi-toolkit/nifi-toolkit-assembly/target/
> 
> There is a zip file within it that has not been unzipped:
> 
> nifi-toolkit-1.23.0-bin.zip
> 
> Is this an issue with the build?  Or am I missing some config setting?
> 
> 
> Thanks,
> Greg

source build issue - Nifi 1.23.0

[Gregory M. Foreman]

Hello:

I am trying to build Nifi 1.23.0 from source using:

JAVA_OPTS="-Xms128m -Xmx4g"
MAVEN_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Xms1024m -Xmx3076m 
-XX:MaxPermSize=256m"

mvn -T C2.0 --batch-mode -Prpm -DskipTests clean install

but the following error is thrown:

[ERROR] Failed to execute goal 
org.codehaus.mojo:rpm-maven-plugin:2.2.0:attached-rpm (build-bin-rpm) on 
project nifi-toolkit-assembly: Source location 
/root/test/nifi-1.23.0/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-1.23.0-bin/nifi-toolkit-1.23.0/LICENSE
 does not exist -> [Help 1]

The path above ends at:

/root/test2/nifi/nifi-toolkit/nifi-toolkit-assembly/target/

There is a zip file within it that has not been unzipped:

nifi-toolkit-1.23.0-bin.zip

Is this an issue with the build?  Or am I missing some config setting?


Thanks,
Greg

Re: retry issue

[Gregory M. Foreman]

Mark:

Thank you for the explanation, yes it helps.  I made the changes and everything 
worked with one exception: preserving the original content.  The non-zero 
status will replace the flowfile content with whatever came through the output 
stream, so the original content is lost.  When the file is processed correctly, 
two flowfiles are created for the original and the output stream relationships. 
 Connecting the original to a routeonattribute processor that inspects the 
execution.status code and traps original files that are errors is an option.  
Are there any options besides using the second processor?

Thanks,
Greg

> On Jul 20, 2022, at 9:22 AM, Mark Payne  wrote:
> 
> Greg,
> 
> You wouldn’t want to retry the “original” relationship. The processor has 3 
> relationships: original, output stream, and nonzero status. It should always 
> send the incoming FlowFile to original. So if you retry that relationship 
> you’ll always retry the flowfile, regardless of whether it was successful or 
> not.
> 
> Instead, you should retry the “nonzero status” relationship. If a FlowFIle is 
> routed to this relationship, it will instead be re-queued and processed 
> again. Nothing will go to original in this case, because the processing is 
> atomic.
> 
> But if a FlowFile is routed to the “output stream” relationship (because it’s 
> not retried) the FlowFile will continue on normally. And the original 
> FlowFile will go to ‘original’.
> 
> Additionally, if all retries are completed and it still is routed to “nonzero 
> status” then at that point, the FlowFile will go to “nonzero status” and the 
> original will be transferred to whatever connection (if any) you have for the 
> “original” relationship.
> 
> Does that help?
> 
> Thanks
> -Mark
> 
>> On Jul 20, 2022, at 9:12 AM, Gregory M. Foreman 
>>  wrote:
>> 
>> Hello:
>> 
>> The ExecuteStreamCommand processor has 3 relationships.  To trap processing 
>> failures, I capture flowfiles from the original relationship, inspect the 
>> cmd exit status, and reroute to a RetryFlowFile processor if the exit status 
>> is not 0.  I wanted to see if this could be simplified with the new retry 
>> feature in 1.16.  When I enable retry on the original relationship and 
>> execute a failing cmd, the flowfiles remain penalized in the previous 
>> success queue, but never exit to the original relationship.  Is this 
>> scenario supposed to work as I have it setup?
>> 
>> Thanks,
>> Greg
> 

retry issue

[Gregory M. Foreman]

Hello:

The ExecuteStreamCommand processor has 3 relationships.  To trap processing 
failures, I capture flowfiles from the original relationship, inspect the cmd 
exit status, and reroute to a RetryFlowFile processor if the exit status is not 
0.  I wanted to see if this could be simplified with the new retry feature in 
1.16.  When I enable retry on the original relationship and execute a failing 
cmd, the flowfiles remain penalized in the previous success queue, but never 
exit to the original relationship.  Is this scenario supposed to work as I have 
it setup?

Thanks,
Greg

flow.xml.gz > flow.json.gz migration

[Gregory M. Foreman]

Hello:

My client upgraded to Nifi 1.16.1.  With the original flow.xml.gz and 
autogenerated flow.json.gz in place everything works fine.  When 
nifi.flow.configuration.json.file is set to the json file location and the 
original nifi.flow.configuration.file is commented out and flow.xml.gz removed, 
the server will not start.  Is there any migration guidance for moving from 
flow.xml.gz to flow.json.gz?

Thanks,
Greg

Re: zookeeper 3.5.9 / CVE-2021-44228

[Gregory M. Foreman]

Shawn:

Thank you, we are going to migrate over to 3.8.0.

David:

Thank you for the detailed explanations.

Greg

> On May 2, 2022, at 11:18 AM, David Handermann  
> wrote:
> 
> Greg,
> 
> NiFi 1.15.0 included build configuration updates that excluded all references 
> to Log4j 1 libraries as described in the following Jira issue:
> 
> https://issues.apache.org/jira/browse/NIFI-9283 
> <https://issues.apache.org/jira/browse/NIFI-9283>
> 
> Although previous versions of NiFi included Log4j 1 libraries, NiFi also 
> leveraged hierarchical class-loading and the Log4j to SLF4J bridge library to 
> route runtime Log4j 1 requests to SLF4J and Logback.  The following post 
> covers the details of logging library management in NiFi:
> 
> https://exceptionfactory.com/posts/2021/12/29/managing-logging-libraries-in-apache-nifi/
>  
> <https://exceptionfactory.com/posts/2021/12/29/managing-logging-libraries-in-apache-nifi/>
> 
> With that background, there should be no concerns related to Log4j 1 and 
> recent versions of NiFi.
> 
> As far as ZooKeeper itself, upgrading the client library version is something 
> that will be addressed as part of regular dependency upgrade reviews.
> 
> Regards,
> David Handermann
> 
> On Mon, May 2, 2022 at 9:56 AM Gregory M. Foreman 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Hello:
> 
> Nifi 1.16.1 included upgrading to zookeeper 3.5.9, which uses log4j 1.2.17 
> (NIFI-9955).  My client currently has an external zookeeper 3.5.8 deployed, 
> it uses log4j 1.2.17, and it has been flagged to upgrade due to the log4j 
> CVE.  I originally thought that log4j 1.x versions were not affected, but I 
> may have over-simplified the logic.  Ref: 
> https://www.petefreitag.com/item/926.cfm 
> <https://www.petefreitag.com/item/926.cfm> (no affiliation).  It appears that 
> zookeeper 3.5.9 is going to EOL in June 2022.  Are there plans to upgrade to 
> zookeeper 3.7.0 or later?
> 
> Thanks,
> Greg
> 
> 

zookeeper 3.5.9 / CVE-2021-44228

[Gregory M. Foreman]

Hello:

Nifi 1.16.1 included upgrading to zookeeper 3.5.9, which uses log4j 1.2.17 
(NIFI-9955).  My client currently has an external zookeeper 3.5.8 deployed, it 
uses log4j 1.2.17, and it has been flagged to upgrade due to the log4j CVE.  I 
originally thought that log4j 1.x versions were not affected, but I may have 
over-simplified the logic.  Ref: https://www.petefreitag.com/item/926.cfm 
 (no affiliation).  It appears that 
zookeeper 3.5.9 is going to EOL in June 2022.  Are there plans to upgrade to 
zookeeper 3.7.0 or later?

Thanks,
Greg

Re: Nifi 1.15.1 RPM issue

[Gregory M. Foreman]

Joe:

The Jira ticket is at:

https://issues.apache.org/jira/browse/NIFI-9525

Thanks,
Greg


> On Dec 21, 2021, at 2:17 PM, Joe Witt  wrote:
> 
> Greg
> 
> I dont quite follow what the diff shows but someone needs to actively
> maintain the RPM config as the config for the assembly itself is done.
> This tends to drift over time as the RPM gets much less attention.
> 
> We'll need someone to look into it.  If there isn't a JIRA please do file one.
> 
> Thanks
> 
> On Tue, Dec 21, 2021 at 12:14 PM Gregory M. Foreman
>  wrote:
>> 
>> I did the following:
>> 
>> cd /opt/nifi/nifi-1.15.1/
>> mv lib lib.bak
>> cp -R 
>> ~/sandbox/nifi-1.15.1-build/nifi-1.15.1/nifi-assembly/target/nifi-1.15.1-bin/nifi-1.15.1/lib
>>  .
>> bin/nifi.sh start
>> 
>> and the server started fine.  Any insights into why the libs would differ?  
>> The diffs between the two directories are attached.
>> 
>> Thanks,
>> Greg
>> 
>> 
>> On Dec 21, 2021, at 12:16 PM, Gregory M. Foreman 
>>  wrote:
>> 
>> Hello Edward:
>> 
>> SELinux is running in Permissive mode.
>> 
>> Greg
>> 
>> On Dec 18, 2021, at 8:15 AM, Edward Armes  wrote:
>> 
>> Hi Greg,
>> 
>> Can you confirm when you deploy via the RPM and start Nifi that selinux is 
>> either running in permissive mode or is disabled
>> 
>> Edward
>> 
>> On Fri, 17 Dec 2021, 20:55 Gregory M. Foreman, 
>>  wrote:
>>> 
>>> David:
>>> 
>>> Correct, only the RPM version.  Running nifi.sh start in the regular build 
>>> output directory works fine.
>>> 
>>> Greg
>>> 
>>> 
>>> On Dec 17, 2021, at 3:38 PM, David Handermann  
>>> wrote:
>>> 
>>> Gregory,
>>> 
>>> Thanks for the confirmation. So this issue is specific to the RPM build, 
>>> correct?  NiFi starts correctly using the tar.gz binary?
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> On Fri, Dec 17, 2021 at 2:23 PM Gregory M. Foreman 
>>>  wrote:
>>>> 
>>>> David:
>>>> 
>>>> No modifications were made to the sources or configurations.  I executed 
>>>> nifi.sh start and that was all.
>>>> 
>>>> Thanks,
>>>> Greg
>>>> 
>>>> On Dec 17, 2021, at 3:14 PM, David Handermann 
>>>>  wrote:
>>>> 
>>>> Gregory,
>>>> 
>>>> Thanks for reporting this issue.  Do you have any notification services 
>>>> configured as part of the bootstrap.conf, such as the HTTP notification 
>>>> service?
>>>> 
>>>> Regards,
>>>> David Handermann
>>>> 
>>>> On Fri, Dec 17, 2021 at 1:56 PM Gregory M. Foreman 
>>>>  wrote:
>>>>> 
>>>>> Hello:
>>>>> 
>>>>> I am having trouble with the Nifi 1.15.1 RPM.  The generated code runs 
>>>>> when executed directly, but the RPM deployed version does not run.  I did 
>>>>> note a significant difference between.  Build output is attached.
>>>>> 
>>>>> 
>>>>> 
>>>>> mvn -version
>>>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>>>> Maven home: /opt/maven
>>>>> Java version: 1.8.0_312, vendor: Red Hat, Inc., runtime: 
>>>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>> OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", 
>>>>> family: "unix"
>>>>> 
>>>>> mvn clean install -Prpm -DskipTests
>>>>> 
>>>>> yum localinstall 
>>>>> nifi-assembly/target/rpm/nifi-bin/RPMS/noarch/nifi-1.15.1-1.el7.noarch.rpm
>>>>> 
>>>>> /opt/nifi/nifi-1.15.1/bin/nifi.sh start
>>>>> nifi.sh: JAVA_HOME not set; results may vary
>>>>> 
>>>>> Java home:
>>>>> NiFi home: /opt/nifi/nifi-1.15.1
>>>>> 
>>>>> Bootstrap Config File: /opt/nifi/nifi-1.15.1/conf/bootstrap.conf
>>>>> 
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: 
>>>>> org/apache/nifi/security/util/TlsConfiguration
>>>>>at java.lang.ClassLoader.defineClass1(Native M

Re: Nifi 1.15.1 RPM issue

[Gregory M. Foreman]

I did the following:cd /opt/nifi/nifi-1.15.1/mv lib lib.bakcp -R ~/sandbox/nifi-1.15.1-build/nifi-1.15.1/nifi-assembly/target/nifi-1.15.1-bin/nifi-1.15.1/lib .bin/nifi.sh startand the server started fine.  Any insights into why the libs would differ?  The diffs between the two directories are attached.Thanks,Greg

diff
Description: Binary data
On Dec 21, 2021, at 12:16 PM, Gregory M. Foreman <gfore...@spinnerconsulting.com> wrote:Hello Edward:SELinux is running in Permissive mode.GregOn Dec 18, 2021, at 8:15 AM, Edward Armes <edward.ar...@gmail.com> wrote:Hi Greg,Can you confirm when you deploy via the RPM and start Nifi that selinux is either running in permissive mode or is disabledEdwardOn Fri, 17 Dec 2021, 20:55 Gregory M. Foreman, <gfore...@spinnerconsulting.com> wrote:David:Correct, only the RPM version.  Running nifi.sh start in the regular build output directory works fine.GregOn Dec 17, 2021, at 3:38 PM, David Handermann <exceptionfact...@apache.org> wrote:Gregory,Thanks for the confirmation. So this issue is specific to the RPM build, correct?  NiFi starts correctly using the tar.gz binary?Regards,David HandermannOn Fri, Dec 17, 2021 at 2:23 PM Gregory M. Foreman <gfore...@spinnerconsulting.com> wrote:David:No modifications were made to the sources or configurations.  I executed nifi.sh start and that was all.Thanks,GregOn Dec 17, 2021, at 3:14 PM, David Handermann <exceptionfact...@apache.org> wrote:Gregory,Thanks for reporting this issue.  Do you have any notification services configured as part of the bootstrap.conf, such as the HTTP notification service?Regards,David HandermannOn Fri, Dec 17, 2021 at 1:56 PM Gregory M. Foreman <gfore...@spinnerconsulting.com> wrote:Hello:

I am having trouble with the Nifi 1.15.1 RPM.  The generated code runs when executed directly, but the RPM deployed version does not run.  I did note a significant difference between.  Build output is attached.



mvn -version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /opt/maven
Java version: 1.8.0_312, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", family: "unix"

mvn clean install -Prpm -DskipTests

yum localinstall nifi-assembly/target/rpm/nifi-bin/RPMS/noarch/nifi-1.15.1-1.el7.noarch.rpm

/opt/nifi/nifi-1.15.1/bin/nifi.sh start
nifi.sh: JAVA_HOME not set; results may vary

Java home: 
NiFi home: /opt/nifi/nifi-1.15.1

Bootstrap Config File: /opt/nifi/nifi-1.15.1/conf/bootstrap.conf

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/nifi/security/util/TlsConfiguration
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:124)
        at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1247)
        at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:289)
Caused by: java.lang.ClassNotFoundException: org.apache.nifi.security.util.TlsConfiguration
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 15 more

Re: Nifi 1.15.1 RPM issue

[Gregory M. Foreman]

Hello Edward:

SELinux is running in Permissive mode.

Greg

> On Dec 18, 2021, at 8:15 AM, Edward Armes  wrote:
> 
> Hi Greg,
> 
> Can you confirm when you deploy via the RPM and start Nifi that selinux is 
> either running in permissive mode or is disabled
> 
> Edward
> 
> On Fri, 17 Dec 2021, 20:55 Gregory M. Foreman, 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> David:
> 
> Correct, only the RPM version.  Running nifi.sh start in the regular build 
> output directory works fine.
> 
> Greg
> 
> 
>> On Dec 17, 2021, at 3:38 PM, David Handermann > <mailto:exceptionfact...@apache.org>> wrote:
>> 
>> Gregory,
>> 
>> Thanks for the confirmation. So this issue is specific to the RPM build, 
>> correct?  NiFi starts correctly using the tar.gz binary?
>> 
>> Regards,
>> David Handermann
>> 
>> On Fri, Dec 17, 2021 at 2:23 PM Gregory M. Foreman 
>> mailto:gfore...@spinnerconsulting.com>> 
>> wrote:
>> David:
>> 
>> No modifications were made to the sources or configurations.  I executed 
>> nifi.sh start and that was all.
>> 
>> Thanks,
>> Greg
>> 
>>> On Dec 17, 2021, at 3:14 PM, David Handermann >> <mailto:exceptionfact...@apache.org>> wrote:
>>> 
>>> Gregory,
>>> 
>>> Thanks for reporting this issue.  Do you have any notification services 
>>> configured as part of the bootstrap.conf, such as the HTTP notification 
>>> service?
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> On Fri, Dec 17, 2021 at 1:56 PM Gregory M. Foreman 
>>> mailto:gfore...@spinnerconsulting.com>> 
>>> wrote:
>>> Hello:
>>> 
>>> I am having trouble with the Nifi 1.15.1 RPM.  The generated code runs when 
>>> executed directly, but the RPM deployed version does not run.  I did note a 
>>> significant difference between.  Build output is attached.
>>> 
>>> 
>>> 
>>> mvn -version
>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>> Maven home: /opt/maven
>>> Java version: 1.8.0_312, vendor: Red Hat, Inc., runtime: 
>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", 
>>> family: "unix"
>>> 
>>> mvn clean install -Prpm -DskipTests
>>> 
>>> yum localinstall 
>>> nifi-assembly/target/rpm/nifi-bin/RPMS/noarch/nifi-1.15.1-1.el7.noarch.rpm
>>> 
>>> /opt/nifi/nifi-1.15.1/bin/nifi.sh start
>>> nifi.sh: JAVA_HOME not set; results may vary
>>> 
>>> Java home: 
>>> NiFi home: /opt/nifi/nifi-1.15.1
>>> 
>>> Bootstrap Config File: /opt/nifi/nifi-1.15.1/conf/bootstrap.conf
>>> 
>>> Exception in thread "main" java.lang.NoClassDefFoundError: 
>>> org/apache/nifi/security/util/TlsConfiguration
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
>>> at 
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>> at 
>>> org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:124)
>>> at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1247)
>>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:289)
>>> Caused by: java.lang.ClassNotFoundException: 
>>> org.apache.nifi.security.util.TlsConfiguration
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>> ... 15 more
>>> 
>>> 
>> 
> 

Re: Nifi 1.15.1 RPM issue

[Gregory M. Foreman]

David:

No modifications were made to the sources or configurations.  I executed 
nifi.sh start and that was all.

Thanks,
Greg

> On Dec 17, 2021, at 3:14 PM, David Handermann  
> wrote:
> 
> Gregory,
> 
> Thanks for reporting this issue.  Do you have any notification services 
> configured as part of the bootstrap.conf, such as the HTTP notification 
> service?
> 
> Regards,
> David Handermann
> 
> On Fri, Dec 17, 2021 at 1:56 PM Gregory M. Foreman 
> mailto:gfore...@spinnerconsulting.com>> 
> wrote:
> Hello:
> 
> I am having trouble with the Nifi 1.15.1 RPM.  The generated code runs when 
> executed directly, but the RPM deployed version does not run.  I did note a 
> significant difference between.  Build output is attached.
> 
> 
> 
> mvn -version
> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
> Maven home: /opt/maven
> Java version: 1.8.0_312, vendor: Red Hat, Inc., runtime: 
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", 
> family: "unix"
> 
> mvn clean install -Prpm -DskipTests
> 
> yum localinstall 
> nifi-assembly/target/rpm/nifi-bin/RPMS/noarch/nifi-1.15.1-1.el7.noarch.rpm
> 
> /opt/nifi/nifi-1.15.1/bin/nifi.sh start
> nifi.sh: JAVA_HOME not set; results may vary
> 
> Java home: 
> NiFi home: /opt/nifi/nifi-1.15.1
> 
> Bootstrap Config File: /opt/nifi/nifi-1.15.1/conf/bootstrap.conf
> 
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/nifi/security/util/TlsConfiguration
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at 
> org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:124)
> at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1247)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:289)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.nifi.security.util.TlsConfiguration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 15 more
> 
> 

Re: Nifi 1.15.1 RPM issue

[Gregory M. Foreman]

Correction:

> I did note a significant difference between the file contents in the lib 
> directories.


> On Dec 17, 2021, at 2:55 PM, Gregory M. Foreman 
>  wrote:
> 
> Hello:
> 
> I am having trouble with the Nifi 1.15.1 RPM.  The generated code runs when 
> executed directly, but the RPM deployed version does not run.  I did note a 
> significant difference between.  Build output is attached.
> 
> 
> 
> mvn -version
> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
> Maven home: /opt/maven
> Java version: 1.8.0_312, vendor: Red Hat, Inc., runtime: 
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", 
> family: "unix"
> 
> mvn clean install -Prpm -DskipTests
> 
> yum localinstall 
> nifi-assembly/target/rpm/nifi-bin/RPMS/noarch/nifi-1.15.1-1.el7.noarch.rpm
> 
> /opt/nifi/nifi-1.15.1/bin/nifi.sh start
> nifi.sh: JAVA_HOME not set; results may vary
> 
> Java home: 
> NiFi home: /opt/nifi/nifi-1.15.1
> 
> Bootstrap Config File: /opt/nifi/nifi-1.15.1/conf/bootstrap.conf
> 
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/nifi/security/util/TlsConfiguration
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>   at 
> org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:124)
>   at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1247)
>   at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:289)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.nifi.security.util.TlsConfiguration
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>   ... 15 more
> 
> 
> 

Re: S2S alternate TLS cert

[Gregory M. Foreman]

Mark:

Thank you for clarifying.

Greg

> On Sep 27, 2021, at 1:55 PM, Mark Payne  wrote:
> 
> Greg,
> 
> The short answer is no, at this point, whatever keystore/truststore is 
> configured in nifi.properties is used for site-to-site communications - both 
> incoming and outgoing.
> 
> It would be helpful to allow for specifying a different SSL Context per 
> Remote Process Group, but at this point, it's just not something that's been 
> implemented.
> 
> Thanks
> -Mark
> 
> 
> On 2021/09/22 14:56:39, "Gregory M. Foreman"  
> wrote: 
>> Hello:
>> 
>> I am trying to configure site-to-site on a Nifi 1.12.1 cluster hosted on 
>> Kubernetes.
>> 
>> I would like one TLS cert to be used for communication outside the cluster.
>> 
>> Inbound s2s (cluster acting as s2s server) is handled by an nginx ingress 
>> proxy that presents the cert to clients.  No issues there.
>> 
>> For outbound s2s (cluster nodes acting as clients), the gui does not have an 
>> option to configure an alternate certificate to use.  Is there some way to 
>> provide this?
>> 
>> Thanks,
>> Greg

S2S alternate TLS cert

[Gregory M. Foreman]

Hello:

I am trying to configure site-to-site on a Nifi 1.12.1 cluster hosted on 
Kubernetes.

I would like one TLS cert to be used for communication outside the cluster.

Inbound s2s (cluster acting as s2s server) is handled by an nginx ingress proxy 
that presents the cert to clients.  No issues there.

For outbound s2s (cluster nodes acting as clients), the gui does not have an 
option to configure an alternate certificate to use.  Is there some way to 
provide this?

Thanks,
Greg

JoltTransformJSON Advanced UI issue

[Gregory M. Foreman]

Hello:

I built Nifi 1.12.1 from source and have run into an issue.  The 
JoltTransformJSON advanced GUI responds with a 500 error:

"An error has occurred loading the editor."

Any suggestions on how to resolve?  It is working fine on the docker image with 
OpenJDK 11.

Thanks,
Greg

===

Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /opt/maven
Java version: 11.0.11, vendor: Red Hat, Inc., runtime: 
/usr/lib/jvm/java-11-openjdk-11.0.11.0.9-1.el7_9.x86_64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.25.1.el7.x86_64", arch: "amd64", 
family: "unix"

===

2021-05-26 08:27:04,956 WARN [NiFi Web Server-25] 
o.e.j.s.h.C._nifi_jolt_transform_json_ui_1_12_1 unavailable
java.lang.IllegalArgumentException: null
at 
jersey.repackaged.org.objectweb.asm.ClassReader.(ClassReader.java:171)
at 
jersey.repackaged.org.objectweb.asm.ClassReader.(ClassReader.java:153)
at 
jersey.repackaged.org.objectweb.asm.ClassReader.(ClassReader.java:425)
at 
org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.process(AnnotationAcceptingListener.java:170)
at 
org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:909)
at 
org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:863)
at 
org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:772)
at 
org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1199)
at 
org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1172)
at 
org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1168)
at 
org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:205)
at 
org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:313)
at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390)
at 
org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:313)
at 
org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:282)
at 
org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:335)
at 
org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:178)
at 
org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:370)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at 
org.eclipse.jetty.servlet.ServletHolder$WrapperServlet.init(ServletHolder.java:1281)
at 
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:594)
at 
org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:478)
at 
org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:720)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
...
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-05-26 08:27:04,959 WARN [NiFi Web Server-25] 
org.eclipse.jetty.server.HttpChannel 
/nifi-jolt-transform-json-ui-1.12.1/api/standard/processor/details
javax.servlet.ServletException: javax.servlet.ServletException: 
api@17a1a==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true,async=false
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at 
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
...
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: 
api@17a1a==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true,async=false
at 
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:612)
at 
org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:478)
at 
org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:720)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
... 
at 

Re: JoltTransformRecord producing IllegalTypeConversionException

[Gregory M. Foreman]

Jim:

Thank you for the suggestions.  I adjusted the schema to only null or string 
array and modified the incoming flowfile to cooperate and it worked.

Greg

> On May 6, 2021, at 3:57 PM, Jim Williams  wrote:
> 
> Greg -
> 
> Maybe use the standard Jolt transformation processor (not the record oriented 
> one) on the initial set of JSON to avoid the need to apply a schema at the 
> start. In that processor, use the 'cardinality' transformation with the MANY 
> option so that any "mac" keys with a scalar string are transformed to an 
> array with a single element.
> 
> Then your Avro schema could go like this:
> 
> ...
> {
> "name": "mac",
> "type": [
> "null",
> {"type":"array","items": "string" }
> ]
> }
> ...
> 
> 
> Warm regards,
> 
> Jim Williams | Manager, Site Reliability Engineering
> 
> jwilli...@alertlogic.com <mailto:jwilli...@alertlogic.com> | alertlogic.com 
> <http://alertlogic.com/>
> 
> 
> 
> -Original Message-
> From: Gregory M. Foreman 
> Sent: Thursday, May 6, 2021 1:50 PM
> To: users@nifi.apache.org <mailto:users@nifi.apache.org>
> Subject: JoltTransformRecord producing IllegalTypeConversionException
> 
> Hello:
> 
> I am using Nifi 1.11.4 and ran into an issue.
> 
> JSON Input:
> 
> {
> "mac":["aabb"]
> 
> }
> 
> avro schema defined:
> 
> {
> "type": "record",
> "name": "generalInformation",
> "namespace": "org.apache.nifi",
> "fields": [
> {
> "name": "mac",
> "type": [
> "null",
> "string",
> {"type":"array","items": "string" }
> ]
> }
> ]
> }
> 
> 
> No errors experienced when running the file through ConvertRecord. But when 
> using JoltTransformRecord with:
> 
> [{
> "operation": "shift",
> "spec": {
> "*": "&"
> }
> }]
> 
> 
> I receive the following:
> 
> 
> JoltTransformRecord[id=33f13cf0-fc17-1b9f-9b11-46f210013a8a] Unable to 
> transform 
> StandardFlowFileRecord[uuid=e9905127-b2a2-444a-95bb-baa87523e59d,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1620323949487-603, 
> container=content1, section=603], offset=760454, 
> length=24],offset=0,name=e9905127-b2a2-444a-95bb-baa87523e59d,size=24] due to 
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
> Cannot convert value [[Ljava.lang.Object;@4e63fe90] of type CHOICE[STRING, 
> ARRAY[STRING]] to Map for field mac because the type is not supported: Cannot 
> convert value [[Ljava.lang.Object;@4e63fe90] of type CHOICE[STRING, 
> ARRAY[STRING]] to Map for field mac because the type is not supported
> 
> 
> Is there a way to workaround this problem on 1.11.4? This works fine in Nifi 
> 1.12.1.
> 
> 
> Thanks,
> Greg
> 
> 
> 
> Confidentiality Notice | This email and any included attachments may be 
> privileged, confidential and/or otherwise protected from disclosure. Access 
> to this email by anyone other than the intended recipient is unauthorized. If 
> you believe you have received this email in error, please contact the sender 
> immediately and delete all copies. If you are not the intended recipient, you 
> are notified that disclosing, copying, distributing or taking any action in 
> reliance on the contents of this information is strictly prohibited.
> 
> 
> Disclaimer
> 
> The information contained in this communication from the sender is 
> confidential. It is intended solely for use by the recipient and others 
> authorized to receive it. If you are not the recipient, you are hereby 
> notified that any disclosure, copying, distribution or taking action in 
> relation of the contents of this information is strictly prohibited and may 
> be unlawful.
> 
> This email has been scanned for viruses and malware, and may have been 
> automatically archived by Mimecast, a leader in email security and cyber 
> resilience. Mimecast integrates email defenses with brand protection, 
> security awareness training, web security, compliance and other essential 
> capabilities. Mimecast helps protect large and small organizations from 
> malicious activity, human error and technology failure; and to lead the 
> movement toward building a more resilient world. To find out more, visit our 
> website.
> 

JoltTransformRecord producing IllegalTypeConversionException

[Gregory M. Foreman]

Hello:

I am using Nifi 1.11.4 and ran into an issue.

JSON Input:

{
"mac":["aabb"]
 
}

avro schema defined:

{
"type": "record",
"name": "generalInformation",
"namespace": "org.apache.nifi",
"fields": [
{
"name": "mac",
"type": [
"null", 
"string", 
{"type":"array","items": "string" }
]
}
]
}


No errors experienced when running the file through ConvertRecord.  But when 
using JoltTransformRecord with:

[{
"operation": "shift",
"spec": {
"*": "&"
}
}]


I receive the following:


JoltTransformRecord[id=33f13cf0-fc17-1b9f-9b11-46f210013a8a] Unable to 
transform 
StandardFlowFileRecord[uuid=e9905127-b2a2-444a-95bb-baa87523e59d,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1620323949487-603, container=content1, 
section=603], offset=760454, 
length=24],offset=0,name=e9905127-b2a2-444a-95bb-baa87523e59d,size=24] due to 
org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
Cannot convert value [[Ljava.lang.Object;@4e63fe90] of type CHOICE[STRING, 
ARRAY[STRING]] to Map for field mac because the type is not supported: Cannot 
convert value [[Ljava.lang.Object;@4e63fe90] of type CHOICE[STRING, 
ARRAY[STRING]] to Map for field mac because the type is not supported


Is there a way to workaround this problem on 1.11.4?  This works fine in Nifi 
1.12.1.


Thanks,
Greg

Re: timestamp parsing

[Gregory M. Foreman]

Mark:

Thanks, but it appears that the processor is not available in Nifi 1.11.4.

Would a custom Jolt transform make sense?

Greg


> On Feb 25, 2021, at 10:01 AM, Mark Payne  wrote:
> 
> Greg,
> 
> I would probably ScriptedTransformRecord in this case and script a solution 
> that looks at the value and determines which pattern to use.
> 
> Thanks
> -Mark
> 
>> On Feb 25, 2021, at 9:35 AM, Gregory M. Foreman 
>>  wrote:
>> 
>> Hello,
>> 
>> I am attempting to parse timestamps of various formats/precision within the 
>> same field in json.  Here is an example (using Nifi 1.11.4):
>> 
>> input:
>> 
>> {"cd": "2021-02-24T12:01:01"}
>> {"cd": "2021-02-24T12:01:01Z"}
>> {"cd": "2021-02-25T12:01:01.99"}
>> {"cd": "2021-02-26T12:01:01.999"}
>> 
>> conversion in UpdateRecord:
>> 
>> /cd = format(toDate(/cd, "-MM-dd'T'HH:mm:ss"), 
>> "-MM-dd'T'HH:mm:ss.SSSZZ")
>> 
>> output:
>> 
>> {"cd":"2021-02-24T12:01:01.000+"}
>> {"cd":"2021-02-24T12:01:01.000+"}
>> {"cd":"2021-02-25T12:01:01.000+"} << precision lost
>> {"cd":"2021-02-26T12:01:01.000+"} << precision lost
>> 
>> changing the conversion in UpdateRecord to this:
>> 
>> /cd = format(toDate(/cd, "-MM-dd'T'HH:mm:ss.SSS"), 
>> "-MM-dd'T'HH:mm:ss.SSSZZ")
>> 
>> output:
>> 
>> {"cd":"2021-02-24T12:01:01"}  << skipped
>> {"cd":"2021-02-24T12:01:01Z"} << skipped
>> {"cd":"2021-02-25T12:01:01.099+"} << not sure if this is correct
>> {"cd":"2021-02-26T12:01:01.999+"} << correct
>> 
>> 
>> Is there a way to change the timestamp parsing format on a per-record basis?
>> 
>> Thanks,
>> Greg
>> 
> 

timestamp parsing

[Gregory M. Foreman]

Hello,

I am attempting to parse timestamps of various formats/precision within the 
same field in json.  Here is an example (using Nifi 1.11.4):

input:

{"cd": "2021-02-24T12:01:01"}
{"cd": "2021-02-24T12:01:01Z"}
{"cd": "2021-02-25T12:01:01.99"}
{"cd": "2021-02-26T12:01:01.999"}

conversion in UpdateRecord:

/cd = format(toDate(/cd, "-MM-dd'T'HH:mm:ss"), 
"-MM-dd'T'HH:mm:ss.SSSZZ")

output:

{"cd":"2021-02-24T12:01:01.000+"}
{"cd":"2021-02-24T12:01:01.000+"}
{"cd":"2021-02-25T12:01:01.000+"} << precision lost
{"cd":"2021-02-26T12:01:01.000+"} << precision lost

changing the conversion in UpdateRecord to this:

/cd = format(toDate(/cd, "-MM-dd'T'HH:mm:ss.SSS"), 
"-MM-dd'T'HH:mm:ss.SSSZZ")

output:

{"cd":"2021-02-24T12:01:01"}  << skipped
{"cd":"2021-02-24T12:01:01Z"} << skipped
{"cd":"2021-02-25T12:01:01.099+"} << not sure if this is correct
{"cd":"2021-02-26T12:01:01.999+"} << correct


Is there a way to change the timestamp parsing format on a per-record basis?

Thanks,
Greg