[
https://issues.apache.org/jira/browse/NIFI-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jul Tomten updated NIFI-8684:
-----------------------------
Description:
I use InvokeScriptedProcessor
I'm trying to read a sensitive property from the process context
before restarting NiFi it was working fine
after restarting NiFi - NiFi fails to startup with the error below
see https://issues.apache.org/jira/browse/NIFI-7012
2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalArgumentException: The property 'Password3' cannot reference
Parameter 'password3' because Sensitive Parameters may only be referenced by
Sensitive Properties.
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
at org.apache.nifi.NiFi.<init>(NiFi.java:159)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:303)
Caused by: java.lang.IllegalArgumentException: The property 'Password3' cannot
reference Parameter 'password3' because Sensitive Parameters may only be
referenced by Sensitive Properties.
at
org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
at
org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
at
org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
at
org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
at
org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
... 43 common frames omitted
in flow.xml.gz
{code:java}
<parameterContext>
<id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
<name>otto_test</name>
<description/>
<parameter>
<name>password</name>
<description/>
<sensitive>true</sensitive>
<value>enc
{61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
</value>
</parameter>
<parameter>
<name>password3</name>
<description/>
<sensitive>true</sensitive>
<value>enc
{6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
</value>
</parameter>
</parameterContext>
{code}
{code:java}
JIRA messed up and inserted lots of bracket {{ and }} all over the place, I
tried to remove them but may have not got them all or to many removed{code}
{code:java}
<processor>
<id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
<name>InvokeScriptedProcessor2</name>
<position x="-1144.0" y="3640.0"/>
<styles/>
<comment/>
<class>org.apache.nifi.processors.script.InvokeScriptedProcessor</class>
<bundle>
<group>org.apache.nifi</group>
<artifact>nifi-scripting-nar</artifact>
<version>1.13.2</version>
</bundle>
<maxConcurrentTasks>1</maxConcurrentTasks>
<schedulingPeriod>0 sec</schedulingPeriod>
<penalizationPeriod>30 sec</penalizationPeriod>
<yieldPeriod>1 sec</yieldPeriod>
<bulletinLevel>WARN</bulletinLevel>
<lossTolerant>false</lossTolerant>
<scheduledState>RUNNING</scheduledState>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<executionNode>ALL</executionNode>
<runDurationNanos>0</runDurationNanos>
<property>
<name>Script Engine</name>
<value>ECMAScript</value>
</property>
<property>
<name>Script File</name>
</property>
<property>
<name>Script Body</name>
<value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
//http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
//http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
///////////////////////////////////////////////////////////
// "imports" go here
////////////////////////////////////////////////////////////
var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
var Processor = Java.type("org.apache.nifi.processor.Processor");
var Relationship = Java.type("org.apache.nifi.processor.Relationship");
var StandardValidators =
Java.type("org.apache.nifi.processor.util.StandardValidators");
var Validator = Java.type("org.apache.nifi.components.Validator");
var HashSet = Java.type("java.util.HashSet");
var LinkedList = Java.type("java.util.LinkedList");
var PropertyDescriptorBuilder =
Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
var log = null;
var REL_SUCCESS = new
Relationship.Builder().name("success").description('FlowFiles that were
successfully processed are routed here').build();
var REL_FAILURE = new
Relationship.Builder().name("failure").description('FlowFiles that were not
successfully processed are routed here').build();
var strPassword2 = "Password2";
var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
.description("Password used to connect")
.required(true)
//.sensitive(false)
//.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build();
var strPassword3 = "Password3";
var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
.description("Password used to connect")
.required(true)
.sensitive(true)}}
// //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(false)
.addValidator(Validator.VALID)
.build();}}function executeScript(session, context, log, REL_SUCCESS,
REL_FAILURE) {
////////////////////////////////////////////////////////////
// your code goes here
////////////////////////////////////////////////////////////
var pwd2 = context.getProperty(strPassword2).getValue();
var pwd3 = context.getProperty(strPassword3).getValue();
var pwd31 =
context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
log.error('pwd2:'+ pwd2);
log.error('pwd3:'+ pwd3);
var flowFile = session.get();
if(flowFile != null) {}}if(flowFile != null) {
// Create a new StreamCallback, passing in a function to define the interface
method
flowFile = session.write(flowFile,
new StreamCallback(function(inputStream, outputStream){
var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8));
outputStream.write("\npwd2:".getBytes());
outputStream.write(pwd2.getBytes());
// outputStream.write("\npwd3:".getBytes());
// outputStream.write(pwd3.getBytes());
// outputStream.write("\n31:".getBytes());
// outputStream.write(pwd31.getBytes());
outputStream.write("\n".getBytes()); }
));
try{
session.transfer(flowFile, REL_SUCCESS)
}catch(e){
log.error('Something went wrong', e)
session.transfer(flowFile, REL_FAILURE)
}
}
function initialize(context){ log = context.logger; }
function getRelationships()
{ var r = new HashSet(); r.add(REL_FAILURE); r.add(REL_SUCCESS); return r; }
function validate(context){ return null; }
function getPropertyDescriptor(name) {
if(name.equals(strPassword2)){
return prop_password2;
} else if(name.equals(strPassword3)){
return prop_password3;
} else{{
return null;
}
}
function onPropertyModified(descriptor, oldValue, newValue) { return null; }
function getPropertyDescriptors(){
var r = new LinkedList();
r.add(prop_password2);
r.add(prop_password3);
return r;
}
function getIdentifier(){ return null; }
function onTrigger(context, sessionFactory) {
var session = sessionFactory.createSession();
try{
executeScript(session, context, log, REL_SUCCESS, REL_FAILURE);
session.commit();
}catch (t) {
log.error("{} failed to process due to {}; rolling back session",
Java.to([this, t], "java.lang.Object[]"));
session.rollback(true);
throw t;
}
}
processor = this;</value>
</property>}}
<property>}}
<name>Module Directory</name>}}
</property>}}
<property>}}
<name>Password2</name>}}
<value>xyz</value>}}
</property>}}
<property>}}
<name>Password3</name>}}
<value>enc
{da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
</value>}}
</property>}}
</processor>}}
{code}
to recover I had to uncomment the code and remove the property: Password3
using a text editor on the flow.xml
Many thanks to mburgess the cookbook author.
I'm new to NiFi so sorry if I've simply made a programming error.
It seems like I'm not the only one with the similar problem.
I try to createa flow that login to provenance repository in a NiFi with LDAP
using the InvokeHttp in a later step. To login I need to fetch a JWT-token and
then call the provenanc repo REST apis using the token. When fetching the token
I want to hide tha password. Maybe there is a better way. I also would like to
store the token in a secure way and not store it as attribute. I use the
DistributedMapCacheServer to cache the token until expiry but maybe there is a
better way.
[https://host:port/nifi-api/access/token] POST
This endpoint only supports form based authentication but if it supported
basic authentcation too it would be simpler becasue then the InvokeHttp
processor could access it and the password would be hidden by sensitive
property. It doesn't help to actually hide the token in the flow. Howdo you
store sensitve values at attribute in the flow? Maybe it is wrong thinking to
do so.
It seems like many are struggling with oAuth2 token based authentication as
well. client credentials grant and password credentials grant and so on.
Havne't found any cookbook for that.
I'm on Ubuntu and java 1.11 sap machine
In Dell Boomi they have [https://ace.c9.io/] as script editor. It's almost a
bug not to have a decent script editor for InvokeScriptedProcessor and the
other script processors. The tiny little NiFi script window is painful to wotk
with and closes when you press enter to create a new line.
was:
I use InvokeScriptedProcessor
I'm trying to read a sensitive property from the process context
before restarting NiFi it was working fine
after restarting NiFi - NiFi fails to startup with the error below
see https://issues.apache.org/jira/browse/NIFI-7012
2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalArgumentException: The property 'Password3' cannot reference
Parameter 'password3' because Sensitive Parameters may only be referenced by
Sensitive Properties.
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
at org.apache.nifi.NiFi.<init>(NiFi.java:159)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:303)
Caused by: java.lang.IllegalArgumentException: The property 'Password3' cannot
reference Parameter 'password3' because Sensitive Parameters may only be
referenced by Sensitive Properties.
at
org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
at
org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
at
org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
at
org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
at
org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
... 43 common frames omitted
in flow.xml.gz
{code:java}
<parameterContext>
<id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
<name>otto_test</name>
<description/>
<parameter>
<name>password</name>
<description/>
<sensitive>true</sensitive>
<value>enc
{61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
</value>
</parameter>
<parameter>
<name>password3</name>
<description/>
<sensitive>true</sensitive>
<value>enc
{6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
</value>
</parameter>
</parameterContext>
{code}
{code:java}
<processor>
<id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
<name>InvokeScriptedProcessor2</name>
<position x="-1144.0" y="3640.0"/>
<styles/>
<comment/>
<class>org.apache.nifi.processors.script.InvokeScriptedProcessor</class>
<bundle>
<group>org.apache.nifi</group>
<artifact>nifi-scripting-nar</artifact>
<version>1.13.2</version>
</bundle>
<maxConcurrentTasks>1</maxConcurrentTasks>
<schedulingPeriod>0 sec</schedulingPeriod>
<penalizationPeriod>30 sec</penalizationPeriod>
<yieldPeriod>1 sec</yieldPeriod>
<bulletinLevel>WARN</bulletinLevel>
<lossTolerant>false</lossTolerant>
<scheduledState>RUNNING</scheduledState>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<executionNode>ALL</executionNode>
<runDurationNanos>0</runDurationNanos>
<property>
<name>Script Engine</name>
<value>ECMAScript</value>
</property>
<property>
<name>Script File</name>
</property>
<property>
<name>Script Body</name>
<value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
//http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
//http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
//https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
///////////////////////////////////////////////////////////
// "imports" go here
////////////////////////////////////////////////////////////
var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
var Processor = Java.type("org.apache.nifi.processor.Processor");
var Relationship = Java.type("org.apache.nifi.processor.Relationship");
var StandardValidators =
Java.type("org.apache.nifi.processor.util.StandardValidators");
var Validator = Java.type("org.apache.nifi.components.Validator");
var HashSet = Java.type("java.util.HashSet");
var LinkedList = Java.type("java.util.LinkedList");
var PropertyDescriptorBuilder =
Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
var log = null;
var REL_SUCCESS = new
Relationship.Builder().name("success").description('FlowFiles that were
successfully processed are routed here').build();
var REL_FAILURE = new
Relationship.Builder().name("failure").description('FlowFiles that were not
successfully processed are routed here').build();
var strPassword2 = "Password2";
var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
.description("Password used to connect")
.required(true)
//.sensitive(false)
//.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build();
var strPassword3 = "Password3";
var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
.description("Password used to connect")
.required(true)
.sensitive(true)}}
// //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.expressionLanguageSupported(false)
.addValidator(Validator.VALID)
.build();}}function executeScript(session, context, log, REL_SUCCESS,
REL_FAILURE) {
////////////////////////////////////////////////////////////
// your code goes here
////////////////////////////////////////////////////////////
var pwd2 = context.getProperty(strPassword2).getValue();
var pwd3 = context.getProperty(strPassword3).getValue();
var pwd31 =
context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
log.error('pwd2:'+ pwd2);
log.error('pwd3:'+ pwd3);
var flowFile = session.get();
if(flowFile != null) {}}if(flowFile != null) {
// Create a new StreamCallback, passing in a function to define the interface
method
flowFile = session.write(flowFile,
new StreamCallback(function(inputStream, outputStream){
var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8));
outputStream.write("\npwd2:".getBytes());
outputStream.write(pwd2.getBytes());
// outputStream.write("\npwd3:".getBytes());
// outputStream.write(pwd3.getBytes());
// outputStream.write("\n31:".getBytes());
// outputStream.write(pwd31.getBytes());
outputStream.write("\n".getBytes()); }
));
try{
session.transfer(flowFile, REL_SUCCESS)
}catch(e){
log.error('Something went wrong', e)
session.transfer(flowFile, REL_FAILURE)
}
}
function initialize(context){ log = context.logger; }
function getRelationships()
{ var r = new HashSet(); r.add(REL_FAILURE); r.add(REL_SUCCESS); return r; }
function validate(context){ return null; }
function getPropertyDescriptor(name) {
if(name.equals(strPassword2)){
return prop_password2; }
else if(name.equals(strPassword3)){
return prop_password3; }
else{{
{eturn null; }
}
function onPropertyModified(descriptor, oldValue, newValue)
{ return null; }
function getPropertyDescriptors()}}{{
{ var r = new LinkedList(); r.add(prop_password2); r.add(prop_password3);
return r; //return null; }
{{function getIdentifier()
{ return null; }
function onTrigger(context, sessionFactory) {}}
var session = sessionFactory.createSession();}}
try}}{{
{ executeScript(session, context, log, REL_SUCCESS, REL_FAILURE);
session.commit(); }
catch (t) {
log.error("{} failed to process due to {}; rolling back session",
Java.to([this, t], "java.lang.Object[]"));}}
session.rollback(true);}}
throw t;}}
}}}
}}}processor = this;</value>
</property>}}
<property>}}
<name>Module Directory</name>}}
</property>}}
<property>}}
<name>Password2</name>}}
<value>xyz</value>}}
</property>}}
<property>}}
<name>Password3</name>}}
<value>enc
{da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
</value>}}
</property>}}
</processor>}}
{code}
to recover I had to uncomment the code and remove the property: Password3
using a text editor on the flow.xml
Many thanks to mburgess the cookbook author.
I'm new to NiFi so sorry if I've simply made a programming error.
It seems like I'm not the only one with the similar problem.
I try to createa flow that login to provenance repository in a NiFi with LDAP
using the InvokeHttp in a later step. To login I need to fetch a JWT-token and
then call the provenanc repo REST apis using the token. When fetching the token
I want to hide tha password. Maybe there is a better way. I also would like to
store the token in a secure way and not store it as attribute. I use the
DistributedMapCacheServer to cache the token until expiry but maybe there is a
better way.
[https://host:port/nifi-api/access/token] POST
This endpoint only supports form based authentication but if it supported
basic authentcation too it would be simpler becasue then the InvokeHttp
processor could access it and the password would be hidden by sensitive
property. It doesn't help to actually hide the token in the flow. Howdo you
store sensitve values at attribute in the flow? Maybe it is wrong thinking to
do so.
It seems like many are struggling with oAuth2 token based authentication as
well. client credentials grant and password credentials grant and so on.
Havne't found any cookbook for that.
I'm on Ubuntu and java 1.11 sap machine
In Dell Boomi they have [https://ace.c9.io/] as script editor. It's almost a
bug not to have a decent script editor for InvokeScriptedProcessor and the
other script processors. The tiny little NiFi script window is painful to wotk
with and closes when you press enter to create a new line.
> sensitive property not working for InvokeScriptedProcessor
> ----------------------------------------------------------
>
> Key: NIFI-8684
> URL: https://issues.apache.org/jira/browse/NIFI-8684
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Reporter: Jul Tomten
> Priority: Major
> Labels: InvokeScriptedProcessor, property, sensitive
>
> I use InvokeScriptedProcessor
> I'm trying to read a sensitive property from the process context
>
> before restarting NiFi it was working fine
> after restarting NiFi - NiFi fails to startup with the error below
> see https://issues.apache.org/jira/browse/NIFI-7012
>
>
> 2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalArgumentException: The property 'Password3' cannot reference
> Parameter 'password3' because Sensitive Parameters may only be referenced by
> Sensitive Properties.
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> at
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:423)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.Server.doStart(Server.java:387)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
> at org.apache.nifi.NiFi.<init>(NiFi.java:159)
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> at org.apache.nifi.NiFi.main(NiFi.java:303)
> Caused by: java.lang.IllegalArgumentException: The property 'Password3'
> cannot reference Parameter 'password3' because Sensitive Parameters may only
> be referenced by Sensitive Properties.
> at
> org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
> at
> org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
> at
> org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
> ... 43 common frames omitted
>
> in flow.xml.gz
>
>
> {code:java}
> <parameterContext>
> <id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
> <name>otto_test</name>
> <description/>
> <parameter>
> <name>password</name>
> <description/>
> <sensitive>true</sensitive>
> <value>enc
> {61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
> </value>
> </parameter>
> <parameter>
> <name>password3</name>
> <description/>
> <sensitive>true</sensitive>
> <value>enc
> {6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
> </value>
> </parameter>
> </parameterContext>
> {code}
>
>
>
>
>
>
> {code:java}
> JIRA messed up and inserted lots of bracket {{ and }} all over the place, I
> tried to remove them but may have not got them all or to many removed{code}
> {code:java}
> <processor>
> <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
> <name>InvokeScriptedProcessor2</name>
> <position x="-1144.0" y="3640.0"/>
> <styles/>
> <comment/>
>
> <class>org.apache.nifi.processors.script.InvokeScriptedProcessor</class>
> <bundle>
> <group>org.apache.nifi</group>
> <artifact>nifi-scripting-nar</artifact>
> <version>1.13.2</version>
> </bundle>
> <maxConcurrentTasks>1</maxConcurrentTasks>
> <schedulingPeriod>0 sec</schedulingPeriod>
> <penalizationPeriod>30 sec</penalizationPeriod>
> <yieldPeriod>1 sec</yieldPeriod>
> <bulletinLevel>WARN</bulletinLevel>
> <lossTolerant>false</lossTolerant>
> <scheduledState>RUNNING</scheduledState>
> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
> <executionNode>ALL</executionNode>
> <runDurationNanos>0</runDurationNanos>
> <property>
> <name>Script Engine</name>
> <value>ECMAScript</value>
> </property>
> <property>
> <name>Script File</name>
> </property>
> <property>
> <name>Script Body</name>
>
> <value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
>
> //http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
>
> //http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
> ///////////////////////////////////////////////////////////
> // "imports" go here
> ////////////////////////////////////////////////////////////
> var StreamCallback =
> Java.type("org.apache.nifi.processor.io.StreamCallback");
> var IOUtils = Java.type("org.apache.commons.io.IOUtils");
> var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
> var Processor = Java.type("org.apache.nifi.processor.Processor");
> var Relationship = Java.type("org.apache.nifi.processor.Relationship");
> var StandardValidators =
> Java.type("org.apache.nifi.processor.util.StandardValidators");
> var Validator = Java.type("org.apache.nifi.components.Validator");
>
> var HashSet = Java.type("java.util.HashSet");
> var LinkedList = Java.type("java.util.LinkedList");
> var PropertyDescriptorBuilder =
> Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
> var log = null;
> var REL_SUCCESS = new
> Relationship.Builder().name("success").description('FlowFiles that were
> successfully processed are routed here').build();
> var REL_FAILURE = new
> Relationship.Builder().name("failure").description('FlowFiles that were not
> successfully processed are routed here').build();
> var strPassword2 = "Password2";
> var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
> .description("Password used to connect")
> .required(true)
> //.sensitive(false)
> //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(true)
> .addValidator(Validator.VALID)
> .build();
> var strPassword3 = "Password3";
> var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
> .description("Password used to connect")
> .required(true)
> .sensitive(true)}}
> // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(false)
> .addValidator(Validator.VALID)
> .build();}}function executeScript(session, context, log, REL_SUCCESS,
> REL_FAILURE) {
> ////////////////////////////////////////////////////////////
> // your code goes here
> ////////////////////////////////////////////////////////////
> var pwd2 = context.getProperty(strPassword2).getValue();
> var pwd3 = context.getProperty(strPassword3).getValue();
> var pwd31 =
> context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
> log.error('pwd2:'+ pwd2);
> log.error('pwd3:'+ pwd3);
> var flowFile = session.get();
> if(flowFile != null) {}}if(flowFile != null) {
> // Create a new StreamCallback, passing in a function to define the
> interface method
> flowFile = session.write(flowFile,
> new StreamCallback(function(inputStream, outputStream){
> var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
> outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8));
>
> outputStream.write("\npwd2:".getBytes());
> outputStream.write(pwd2.getBytes());
> // outputStream.write("\npwd3:".getBytes());
> // outputStream.write(pwd3.getBytes());
> // outputStream.write("\n31:".getBytes());
> // outputStream.write(pwd31.getBytes());
> outputStream.write("\n".getBytes()); }
> ));
>
> try{
> session.transfer(flowFile, REL_SUCCESS)
> }catch(e){
> log.error('Something went wrong', e)
> session.transfer(flowFile, REL_FAILURE)
> }
> }
> function initialize(context){ log = context.logger; }
> function getRelationships()
> { var r = new HashSet(); r.add(REL_FAILURE); r.add(REL_SUCCESS); return r; }
> function validate(context){ return null; }
> function getPropertyDescriptor(name) {
> if(name.equals(strPassword2)){
> return prop_password2;
> } else if(name.equals(strPassword3)){
> return prop_password3;
> } else{{
> return null;
> }
>
> }
> function onPropertyModified(descriptor, oldValue, newValue) { return null; }
> function getPropertyDescriptors(){
> var r = new LinkedList();
> r.add(prop_password2);
> r.add(prop_password3);
> return r;
> }
> function getIdentifier(){ return null; }
> function onTrigger(context, sessionFactory) {
> var session = sessionFactory.createSession();
> try{
> executeScript(session, context, log, REL_SUCCESS, REL_FAILURE);
> session.commit();
> }catch (t) {
> log.error("{} failed to process due to {}; rolling back session",
> Java.to([this, t], "java.lang.Object[]"));
> session.rollback(true);
> throw t;
> }
> }
> processor = this;</value>
> </property>}}
> <property>}}
> <name>Module Directory</name>}}
> </property>}}
> <property>}}
> <name>Password2</name>}}
> <value>xyz</value>}}
> </property>}}
> <property>}}
> <name>Password3</name>}}
> <value>enc
> {da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
> </value>}}
> </property>}}
> </processor>}}
>
> {code}
>
> to recover I had to uncomment the code and remove the property: Password3
> using a text editor on the flow.xml
>
> Many thanks to mburgess the cookbook author.
> I'm new to NiFi so sorry if I've simply made a programming error.
> It seems like I'm not the only one with the similar problem.
> I try to createa flow that login to provenance repository in a NiFi with LDAP
> using the InvokeHttp in a later step. To login I need to fetch a JWT-token
> and then call the provenanc repo REST apis using the token. When fetching the
> token I want to hide tha password. Maybe there is a better way. I also would
> like to store the token in a secure way and not store it as attribute. I use
> the DistributedMapCacheServer to cache the token until expiry but maybe
> there is a better way.
> [https://host:port/nifi-api/access/token] POST
> This endpoint only supports form based authentication but if it supported
> basic authentcation too it would be simpler becasue then the InvokeHttp
> processor could access it and the password would be hidden by sensitive
> property. It doesn't help to actually hide the token in the flow. Howdo you
> store sensitve values at attribute in the flow? Maybe it is wrong thinking to
> do so.
> It seems like many are struggling with oAuth2 token based authentication as
> well. client credentials grant and password credentials grant and so on.
> Havne't found any cookbook for that.
>
>
> I'm on Ubuntu and java 1.11 sap machine
> In Dell Boomi they have [https://ace.c9.io/] as script editor. It's almost a
> bug not to have a decent script editor for InvokeScriptedProcessor and the
> other script processors. The tiny little NiFi script window is painful to
> wotk with and closes when you press enter to create a new line.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
