initial NiFi 1.15.3 start-up issue on mongo-uri sensitive property complaint

[Jeremy Taylor]

Greetings,

I am observing a potential return of ticket 7012 in NiFi 1.15.3.

References:

https://issues.apache.org/jira/browse/NIFI-7012

https://stackoverflow.com/questions/64750710/nifi-fails-to-start-after-setting-sensitive-parameter-into-non-sensitive-propert



On initial startup of NiFi 1.15.3 on my project’s flow, NiFi fails to start due 
to the below stacktrace. This relates to a sensitive parameter context property 
for a sensitive field in a MongoDBControllerService. As a result, this appears 
to me as the same error as the referenced defect ticket above. I tried a couple 
different things to get past this error, including one from the stackoverflow 
reference above, but that caused unwanted problems.  The most graceful 
technique I found was to string-replace my prior version of NiFi (1.13.2) with 
1.15.3 within the raw flow XML for at least all the mongo processors’ 
MongoDBControllerServices, This relates to NiFi-packaged mongo processors and 
my project’s open-source mongo processors. However, in attempting this 
string-replace, I admit I may have also string-replaced additional items beyond 
the various MongoDBControllerServices that had 1.13.2 marked to achieve a 
successful start-up of NiFi and successful execution of all mongo processors at 
runtime.  While this change did seem to work, I would like to confirm that it 
is necessary to do this work-around for successful start-up, or am I missing 
something?

Environment:

Mac OS x86

Java:  OpenJDK version 1.8.0_322

NiFi: 1.15.3

MongoDB (mongodb-driver-legacy) drivers for custom processors and NiFi-packaged 
MongoDB processers:  4.3.2, but our prior custom processors had used 3.12.5

MongoDB:  4.2.8





2022-01-25 19:34:02,809 WARN [main] org.apache.nifi.web.server.JettyServer 
Failed to start web server... shutting down.

org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalArgumentException: The property 'mongo-uri' is a sensitive 
property, so it can only reference Parameters that are sensitive.

   at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)

   at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1475)

   at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)

   at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)

   at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)

   at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)

   at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1067)

   at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:5



Regards,

--
Jeremy H. Taylor
Software Developer
ACES Incorporated, an EverWatch Company
http://acesinc.net
http://everwatchsolutions.com

Re: initial NiFi 1.15.3 start-up issue on mongo-uri sensitive property complaint

[Matt Burgess]

Jeremy,

I can't reproduce this on the latest main branch (closest to 1.15.3).
What's weird about that error message is that it says 'mongo-uri' is a
sensitive property, but it is not.  I set up a Parameter Context (PC)
with a non-sensitive parameter "muri", set the PC on the root Process
Group (where the Controller Service (CS) is defined), I didn't get any
errors about sensitivity. Then I restarted NiFi and everything worked
as expected.

Just to see what happens I tried it the other way, with "muri" being
sensitive. I then got the expected error in the UI that the
sensitivity of the property doesn't match the sensitivity of the
parameter. I then stopped NiFi, edited the flow.json, and restarted.
Instead of crashing, the CS was marked invalid with the aforementioned
message about mismatched sensitivity.

Does this only happen with your own Mongo processors? Is the Mongo URI
property sensitive in your component(s)?

Regards,
Matt

RE: NiFi V1.15.2 Conversion to Avro from JSON

[nathan.english]

Thanks, Mark! Appreciate you retaking another look at this!

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: 26 January 2022 00:21
To: users 
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan, the template is helpful. I put up a PR that I believe addresses 
the issues.

Thanks
-Mark



On Jan 25, 2022, at 7:21 AM, 
nathan.engl...@bt.com wrote:

Hey Mark,

Apologies if this is more of a dev mailing list issue. Happy to go across to it 
if needs be.

I've been testing the fix you've implemented for this and noticed a slight 
issue when the default value should be null. It took a little while to work out 
what was going on, but I think I've cracked it. I have attached a flow I've 
been using to test this.

So in AvroTypeUtil on line 630, it's creating the field as it doesn't exist. 
Instead of setting the default value to null, it is set to 
org.apache.avro.JsonProperties$Null, which throws an error when it tries to 
encode as Avro. For example, the Null String & Byte Type generator in the 
attached flow reproduces this.

In addition to this, I did a quick bit of testing with int and long types with 
default values in the Avro Schema set to 0. So, it looks like Long fields in 
the schema have the default values set as Integers? I have reproduced it with 
the attached flow's Int & Long Type generator.

Of course, this isn't an exhaustive list of the possible Avro types, but it 
does seem the changes made to the Avro Util class since v1.12.1 may be causing 
issues? Or at least it feels that way to me?

The only way I see forward is to restore the logic that was previously in place?

I have also created a bug NIFI-9629 [1].

Kind Regards,

Nathan

https://issues.apache.org/jira/browse/NIFI-9629
From: English,N,Nathan,VIR R
Sent: 20 January 2022 19:53
To: users@nifi.apache.org
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Hey Mark,

You beat me to the fix, Thanks for taking your time to look at it! I appreciate 
that one.

Cheers!

Nathan

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, January 19, 2022 6:54:23 PM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate 
with your template. Thanks for including that. Just put up a Pull Request for 
it, as well.

Thanks
-Mark



[1] 
https://issues.apache.org/jira/browse/NIFI-9594




On Jan 18, 2022, at 4:49 AM, 
nathan.engl...@bt.com wrote:

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] 
https://issues.apache.org/jira/browse/NIFI-9335