Re: Problem with property descriptor in InvokeScriptedProcessor

[Elemir Stevko]

Hi Denes,

Yes, that was it. I missed the import you mentioned. Thanks a lot for your help 
and also for suggesting setting the logging level to debug!

Best regards,
Elemir

From: Denes Arvay 
Reply-To: "users@nifi.apache.org" 
Date: Thursday, 21 March 2019 at 6:22 pm
To: "users@nifi.apache.org" 
Subject: Re: Problem with property descriptor in InvokeScriptedProcessor

Hi Elemir,

Setting the log level to DEBUG outputs more details, it seems the cause of the 
exception is the following:
javax.script.ScriptException: NameError: global name 'StandardValidators' is 
not defined in 

Re: Using NiFi Expression Language outside of NiFi

[Matt Burgess]

Tim,

Sorry I lost track of this, I meant to respond earlier. If you can use
a tool/language that brings in dependencies easily, such as Groovy,
then using NiFi Expression Language to evaluate expressions is fairly
straightforward, here's a Groovy script I use to do just that (the
extra gravy is for CLI users):

@Grab(group='org.apache.nifi', module='nifi-expression-language',
version='1.9.1')
import org.apache.nifi.attribute.expression.language.*

def cli = new CliBuilder(usage:'groovy testEL.groovy [options] [expressions]',
  header:'Options:')
cli.help('print this message')
cli.D(args:2, valueSeparator:'=', argName:'attribute=value',
   'set value for given attribute')
def options = cli.parse(args)
if(!options.arguments()) {
  cli.usage()
  return 1
}

def attrMap = [:]
def currKey = null
options.Ds?.eachWithIndex {o,i ->
  if(i%2==0) {
currKey = o
  } else {
attrMap[currKey] = o
  }
}
options.arguments()?.each {
  def q = Query.compile(it)
  println q.evaluate(attrMap ?: null)
}

Regards,
Matt

On Thu, Mar 21, 2019 at 5:42 PM Tim Zimmerman  wrote:
>
> Sorry for the delay in responding.
> After looking at a few different options it looks like it will be pretty
> easy to accomplish what we need using  Java Unified Expression Language
>   .
>
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Sorry for the delay in responding.
After looking at a few different options it looks like it will be pretty
easy to accomplish what we need using  Java Unified Expression Language
  .





--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Nifi backup & recovery

[Vishal Jadhav (BLOOMBERG/ 731 LEX)]

Hello All,

I would like to have the ability to recreate the nifi system and wondering what 
 status I need to store/preserver. It's a single node system at this time.

I am backing up the following at this time.

Repos:
Disk back up of repositories directories - content, provenence & flowfile, 
keystore/trustsotres files and related keys and of course the conf directory.

Process flows:
Backed up via nifi-registry + git

Is there anything missing from the users/groups and their policies, if I have 
recreate the nifi instance?

thanks,
- Vishal

Re: sensitive variable values ?

[Shawn Weeks]

Are you not able to update the properties for the controller service. It looks 
like you use something like PUT /controller-services/{id} with some json kinda 
like this.

{ "revision": {…}, "id": "value", "uri": "value", "position": {…}, 
"permissions": {…}, "bulletins": [{…}], "disconnectedNodeAcknowledged": true, 
"parentGroupId": "value", "component": {…}, "operatePermissions": {…}, 
"status": {…} }

Filling in component with something like this and trying to set the properties.

{ "id": "value", "versionedComponentId": "value", "parentGroupId": "value", 
"position": {…}, "name": "value", "type": "value", "bundle": {…}, 
"controllerServiceApis": [{…}], "comments": "value", "state": "value", 
"persistsState": true, "restricted": true, "deprecated": true, 
"multipleVersionsAvailable": true, "properties": { "name": "value" }, 
"descriptors": { "name": {…} }, "customUiUrl": "value", "annotationData": 
"value", "referencingComponents": [{…}], "validationErrors": ["value"], 
"validationStatus": "value", "extensionMissing": true }

Check https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

What JSON have you tried so far?

Thanks
Shawn


From: l vic 
Reply-To: "users@nifi.apache.org" 
Date: Thursday, March 21, 2019 at 3:38 PM
To: "users@nifi.apache.org" 
Subject: Re: sensitive variable values ?

I don't see how to do it for service from NiFi REST api... Any suggestions?
Thank you

Re: sensitive variable values ?

[l vic]

I don't see how to do it for service from NiFi REST api... Any suggestions?
Thank you


>
>

RE: Automate the Enablement of a StandardHttpContextMap

[William Gosse]

Ok I got this working by configuring each Http component's 
StandardHttpContextMap  individual instead having them all ganged all onto one.

Each of the StandardHttpContextMap  is stills coped for the process group that 
all the components are in.

I'll admit I'm still a little shaky on controller services but it seem to work 
now.  

I'm able to enable all the controller services form the root group with 
.\conf\cli.properties
PS C:\Dev\aim\nifi-toolkit-1.9.1\bin> .\cli.bat nifi pg-enable-services -pgid 
911b83b1-0169-1000-17bb-823bb530cf8c -p ..\conf\cli.properties

Thanks for the help it is much approceiated.

-Original Message-
From: William Gosse  
Sent: Thursday, March 21, 2019 2:55 PM
To: users@nifi.apache.org
Subject: RE: Automate the Enablement of a StandardHttpContextMap

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.] 

[WARNING: Potential spoofed mail]
The sender of this email used an @kodakalaris.com email address, but the email 
did not come from a known Kodak Alaris email system. This email might not be 
legitimate. Delete this email unless it was sent from a known, trusted source.
To avoid this notice if this email was sent from a known trusted source please 
submit an IT_KAre request to have this sender added as a valid email sender.

Proofpoint rule module.access.rule.anti_spoofing_rule Message ID 2rceu6r68c-1 
2rceu6r68c

I'm having a problem with the command you showed below. When I run it for my 
stuff I get the following:
PS C:\Dev\aim\nifi-toolkit-1.9.1\bin> .\cli.bat nifi pg-enable-services -pgid 
aee943fb-48ac-351e-aef7-29db27d33b0f -p ..
\conf\cli.properties

ERROR: Error executing command 'pg-enable-services' : One or more services 
could not be enabled, run command with -verbose to obtain more details

The are for components referencing the StandardHttpContextMap, one 
HandleHttpRequest component and three  HandleHttpResponse components.

Any suggestion on how to handle this error?

-Original Message-
From: Bryan Bende 
Sent: Thursday, March 21, 2019 12:27 PM
To: users@nifi.apache.org
Subject: Re: Automate the Enablement of a StandardHttpContextMap

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.] 

When a template, or a versioned flow, is instantiated, all the processors are 
initially stopped and the services are disabled. This is done on purpose since 
starting and enabling components may cause interactions with external systems, 
and it may not be clear or desired that just by instantiating the template that 
all of these interactions are automatically going to take place. So it requires 
an extra step where the user (or your script) must explicitly enabled and start 
the components.

The NIFI CLI has a command "pg-enable-services" that can be used to enable all 
the services in a given process group. You could call this from your script 
like "./bin/cli.sh nifi pg-enable-service -pgid someid".

On Thu, Mar 21, 2019 at 11:29 AM William Gosse  
wrote:
>
> I’m trying to automate the deployment of a template that contains a 
> HandleHttpRequest processor.
>
>
>
> My issue is that this always comes up with an Invalid component marker after 
> a fresh deployment of the template.
>
> The cause of this marker is that the StandardHttpContextMap in the 
> HandleHttpRequest processor needs to be manually enabled in Controller 
> Services.
>
>
>
> Is there any way to avoid this manual enablement?
>
>
>
>

Re: sensitive variable values ?

[Bryan Bende]

Sensitive variables is something we'd like to support, but we
currently don't have that capability yet.

If you are using a script to create components, or to instantiate
templates, you should be able to set the property values of those
components using the REST API.

You would be making the same REST call that is made if you were in the
UI and were on the config window, entered the password, and hit the
Apply button.

On Thu, Mar 21, 2019 at 3:17 PM l vic  wrote:
>
> Is there a mechanism in Nifi to use sensitive variable values? For example: 
> would it be possible to store/set password value in 
> StandardRestrictedSSLContextService without exposing it? I am looking for the 
> way to set password in StandardRestrictedSSLContextService from the automated 
> deployment script
> Thank you,
> -V

Re: sensitive variable values ?

[Joe Witt]

Hello

The variables of a pg are not, at this time, for sensitive values.  You can
set the sens values programatically to ensure they are never shown.

We will likely add support for secrets (ie sensitive variables) but eta
there depends on progress in the community.

thanks

On Thu, Mar 21, 2019, 3:17 PM l vic  wrote:

> Is there a mechanism in Nifi to use sensitive variable values? For
> example: would it be possible to store/set password value
> in StandardRestrictedSSLContextService without exposing it? I am looking
> for the way to set password in StandardRestrictedSSLContextService from the
> automated deployment script
> Thank you,
> -V
>

sensitive variable values ?

[l vic]

Is there a mechanism in Nifi to use sensitive variable values? For example:
would it be possible to store/set password value
in StandardRestrictedSSLContextService without exposing it? I am looking
for the way to set password in StandardRestrictedSSLContextService from the
automated deployment script
Thank you,
-V

Re: Automate the Enablement of a StandardHttpContextMap

[Bryan Bende]

Also meant to say that it can only be enabled if it is valid, meaning
there are no validation messages on the component.

So it might be worth checking in the UI to see if there are any
validation issues on the service.

On Thu, Mar 21, 2019 at 3:02 PM Bryan Bende  wrote:
>
> If you run it with -verbose does it produce anything more helpful?
>
> On Thu, Mar 21, 2019 at 2:55 PM William Gosse
>  wrote:
> >
> > I'm having a problem with the command you showed below. When I run it for 
> > my stuff I get the following:
> > PS C:\Dev\aim\nifi-toolkit-1.9.1\bin> .\cli.bat nifi pg-enable-services 
> > -pgid aee943fb-48ac-351e-aef7-29db27d33b0f -p ..
> > \conf\cli.properties
> >
> > ERROR: Error executing command 'pg-enable-services' : One or more services 
> > could not be enabled, run command with -verbose to obtain more details
> >
> > The are for components referencing the StandardHttpContextMap, one 
> > HandleHttpRequest component and three  HandleHttpResponse components.
> >
> > Any suggestion on how to handle this error?
> >
> > -Original Message-
> > From: Bryan Bende 
> > Sent: Thursday, March 21, 2019 12:27 PM
> > To: users@nifi.apache.org
> > Subject: Re: Automate the Enablement of a StandardHttpContextMap
> >
> > [CAUTION: This email originated from outside of Kodak Alaris. Do not click 
> > links or open attachments unless you recognize the sender and know the 
> > content is safe.] 
> >
> > When a template, or a versioned flow, is instantiated, all the processors 
> > are initially stopped and the services are disabled. This is done on 
> > purpose since starting and enabling components may cause interactions with 
> > external systems, and it may not be clear or desired that just by 
> > instantiating the template that all of these interactions are automatically 
> > going to take place. So it requires an extra step where the user (or your 
> > script) must explicitly enabled and start the components.
> >
> > The NIFI CLI has a command "pg-enable-services" that can be used to enable 
> > all the services in a given process group. You could call this from your 
> > script like "./bin/cli.sh nifi pg-enable-service -pgid someid".
> >
> > On Thu, Mar 21, 2019 at 11:29 AM William Gosse 
> >  wrote:
> > >
> > > I’m trying to automate the deployment of a template that contains a 
> > > HandleHttpRequest processor.
> > >
> > >
> > >
> > > My issue is that this always comes up with an Invalid component marker 
> > > after a fresh deployment of the template.
> > >
> > > The cause of this marker is that the StandardHttpContextMap in the 
> > > HandleHttpRequest processor needs to be manually enabled in Controller 
> > > Services.
> > >
> > >
> > >
> > > Is there any way to avoid this manual enablement?
> > >
> > >
> > >
> > >

Re: Automate the Enablement of a StandardHttpContextMap

[Bryan Bende]

If you run it with -verbose does it produce anything more helpful?

On Thu, Mar 21, 2019 at 2:55 PM William Gosse
 wrote:
>
> I'm having a problem with the command you showed below. When I run it for my 
> stuff I get the following:
> PS C:\Dev\aim\nifi-toolkit-1.9.1\bin> .\cli.bat nifi pg-enable-services -pgid 
> aee943fb-48ac-351e-aef7-29db27d33b0f -p ..
> \conf\cli.properties
>
> ERROR: Error executing command 'pg-enable-services' : One or more services 
> could not be enabled, run command with -verbose to obtain more details
>
> The are for components referencing the StandardHttpContextMap, one 
> HandleHttpRequest component and three  HandleHttpResponse components.
>
> Any suggestion on how to handle this error?
>
> -Original Message-
> From: Bryan Bende 
> Sent: Thursday, March 21, 2019 12:27 PM
> To: users@nifi.apache.org
> Subject: Re: Automate the Enablement of a StandardHttpContextMap
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.] 
>
> When a template, or a versioned flow, is instantiated, all the processors are 
> initially stopped and the services are disabled. This is done on purpose 
> since starting and enabling components may cause interactions with external 
> systems, and it may not be clear or desired that just by instantiating the 
> template that all of these interactions are automatically going to take 
> place. So it requires an extra step where the user (or your script) must 
> explicitly enabled and start the components.
>
> The NIFI CLI has a command "pg-enable-services" that can be used to enable 
> all the services in a given process group. You could call this from your 
> script like "./bin/cli.sh nifi pg-enable-service -pgid someid".
>
> On Thu, Mar 21, 2019 at 11:29 AM William Gosse  
> wrote:
> >
> > I’m trying to automate the deployment of a template that contains a 
> > HandleHttpRequest processor.
> >
> >
> >
> > My issue is that this always comes up with an Invalid component marker 
> > after a fresh deployment of the template.
> >
> > The cause of this marker is that the StandardHttpContextMap in the 
> > HandleHttpRequest processor needs to be manually enabled in Controller 
> > Services.
> >
> >
> >
> > Is there any way to avoid this manual enablement?
> >
> >
> >
> >

RE: Automate the Enablement of a StandardHttpContextMap

[William Gosse]

I'm having a problem with the command you showed below. When I run it for my 
stuff I get the following:
PS C:\Dev\aim\nifi-toolkit-1.9.1\bin> .\cli.bat nifi pg-enable-services -pgid 
aee943fb-48ac-351e-aef7-29db27d33b0f -p ..
\conf\cli.properties

ERROR: Error executing command 'pg-enable-services' : One or more services 
could not be enabled, run command with -verbose to obtain more details

The are for components referencing the StandardHttpContextMap, one 
HandleHttpRequest component and three  HandleHttpResponse components.

Any suggestion on how to handle this error?

-Original Message-
From: Bryan Bende  
Sent: Thursday, March 21, 2019 12:27 PM
To: users@nifi.apache.org
Subject: Re: Automate the Enablement of a StandardHttpContextMap

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.] 

When a template, or a versioned flow, is instantiated, all the processors are 
initially stopped and the services are disabled. This is done on purpose since 
starting and enabling components may cause interactions with external systems, 
and it may not be clear or desired that just by instantiating the template that 
all of these interactions are automatically going to take place. So it requires 
an extra step where the user (or your script) must explicitly enabled and start 
the components.

The NIFI CLI has a command "pg-enable-services" that can be used to enable all 
the services in a given process group. You could call this from your script 
like "./bin/cli.sh nifi pg-enable-service -pgid someid".

On Thu, Mar 21, 2019 at 11:29 AM William Gosse  
wrote:
>
> I’m trying to automate the deployment of a template that contains a 
> HandleHttpRequest processor.
>
>
>
> My issue is that this always comes up with an Invalid component marker after 
> a fresh deployment of the template.
>
> The cause of this marker is that the StandardHttpContextMap in the 
> HandleHttpRequest processor needs to be manually enabled in Controller 
> Services.
>
>
>
> Is there any way to avoid this manual enablement?
>
>
>
>

Re: Automate the Enablement of a StandardHttpContextMap

[Bryan Bende]

When a template, or a versioned flow, is instantiated, all the
processors are initially stopped and the services are disabled. This
is done on purpose since starting and enabling components may cause
interactions with external systems, and it may not be clear or desired
that just by instantiating the template that all of these interactions
are automatically going to take place. So it requires an extra step
where the user (or your script) must explicitly enabled and start the
components.

The NIFI CLI has a command "pg-enable-services" that can be used to
enable all the services in a given process group. You could call this
from your script like "./bin/cli.sh nifi pg-enable-service -pgid
someid".

On Thu, Mar 21, 2019 at 11:29 AM William Gosse
 wrote:
>
> I’m trying to automate the deployment of a template that contains a 
> HandleHttpRequest processor.
>
>
>
> My issue is that this always comes up with an Invalid component marker after 
> a fresh deployment of the template.
>
> The cause of this marker is that the StandardHttpContextMap in the 
> HandleHttpRequest processor needs to be manually enabled in Controller 
> Services.
>
>
>
> Is there any way to avoid this manual enablement?
>
>
>
>

Automate the Enablement of a StandardHttpContextMap

[William Gosse]

I'm trying to automate the deployment of a template that contains a 
HandleHttpRequest processor.

My issue is that this always comes up with an Invalid component marker after a 
fresh deployment of the template.
The cause of this marker is that the StandardHttpContextMap in the 
HandleHttpRequest processor needs to be manually enabled in Controller Services.

Is there any way to avoid this manual enablement?

Re: Problem with property descriptor in InvokeScriptedProcessor

[Denes Arvay]

Hi Elemir,

Setting the log level to DEBUG outputs more details, it seems the cause of
the exception is the following:
javax.script.ScriptException: NameError: global name 'StandardValidators'
is not defined in 

Problem with property descriptor in InvokeScriptedProcessor

[Elemir Stevko]

Hello,

I would like to implement a stateless S3 lister as a python script using 
InvokeScriptedProcessor in NiFi 1.9.0. I have based my script on the 
CompressFlowFile processor:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/test/resources/jython/test_compress.py

I want to declare my property 'Bucket' to be required and non-empty, so I have 
changed the getPropertyDescriptors method to:

def getPropertyDescriptors(self) :
descriptor = 
PropertyDescriptor.Builder().name("Bucket").addValidator(StandardValidators.NON_EMPTY_VALIDATOR).required(True).build()

I am getting an error when storing the script code in the processor:

Unable to get property descriptors from Processor: 
java.lang.reflect.UndeclaredThrowableException.

If I define the descriptor using allowableValues, it works fine:

descriptor = 
PropertyDescriptor.Builder().name("Bucket").allowableValues("compress", 
"decompress").required(True).build()

Could you please let me know what is causing this problem and how I can fix it?

Best regards,
Elemir

Here is the complete test code of the script:

import sys
import traceback
from org.apache.nifi.processor import Processor
from org.apache.nifi.processor import Relationship
from org.apache.nifi.components import PropertyDescriptor
from org.apache.nifi.processor.io import StreamCallback

class PyStreamCallback(StreamCallback):
  def __init__(self, text):
self.text = text
pass

  def process(self, inputStream, outputStream):
outputStream.write(bytearray(self.text.encode('utf-8')))

class ListS3Files(Processor) :
__rel_success = 
Relationship.Builder().description("Success").name("success").build()

def __init__(self) :
pass

def initialize(self, context) :
pass

def getRelationships(self) :
return set([self.__rel_success])

def validate(self, context) :
pass

def getPropertyDescriptors(self) :
descriptor = 
PropertyDescriptor.Builder().name("Bucket").addValidator(StandardValidators.NON_EMPTY_VALIDATOR).required(True).build()
#descriptor = 
PropertyDescriptor.Builder().name("Bucket").allowableValues("val1", 
"val2").required(True).build()
return [descriptor]

def onPropertyModified(self, descriptor, newValue, oldValue) :
pass

def onTrigger(self, context, sessionFactory) :
session = sessionFactory.createSession()
try :
bucket = context.getProperty("Bucket").getValue()
if not bucket:
return

for file in ['file1', 'file2']:
flowfile = session.create()
if flowfile is None :
return

flowfile = session.write(flowfile, 
PyStreamCallback(bucket+'/'+file))
# transfer
session.transfer(flowfile, self.__rel_success)

session.commit()
except :
print sys.exc_info()[0]
print "Exception in TestReader:"
print '-' * 60
traceback.print_exc(file=sys.stdout)
print '-' * 60

session.rollback(true)
raise

processor = ListS3Files()