Re: InvokeScriptedProcessor

2016-10-12 Thread Matt Burgess 
summary, there are 2 buggy behaviors that have been huge gotchas (aka.
> hours wasted for nothing) for me with InvokeScriptedProcessor:
> 1)  After correcting syntax errors in my property definitions, I had
> to completely delete and recreate my InvokeScriptedProcessor instance
> instead of just updating the text in the processor – otherwise the error
> still reoccurs even after fixing the code
> 2)  After throwing an exception while using a DBCPService I had to
> disable/reenable the service – otherwise the Processor still crashes even
> after fixing the code
>
> Not sure if there’s even a way to prevent those from happening, but man is
> it misleading when you (correctly) fix your code, start things up again,
> and the error is still there!  Perhaps some missing cleanup?  Maybe it’s
> just the nature of the beast.  Anyway, sorry for the bother, just figured I
> post this solution for completeness.  Cheers!
>
>
> <http://homecarepulse.com/?utm_source=logoLink&utm_medium=Jason&utm_campaign=EmailSignature>
> *Jason Hamilton*
> Senior Software Developer
> Home Care Pulse, LLC
> www.homecarepulse.com
> <http://homecarepulse.com/?utm_source=websiteLink&utm_medium=Jason&utm_campaign=EmailSignature>
>
> 208.228.0895 (Direct) <2082280895>
> 877.307.8573 (Office) <8773078573>
> <http://www.linkedin.com/company/home-care-pulse>
> <http://facebook.com/homecarepulse> <http://twitter.com/homecarepulse>
> <http://plus.google.com/+Homecarepulse/posts>
> Satisfaction Management
> <http://www.homecarepulse.com/program-details?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | Benchmarking
> <http://benchmarking.homecarepulse.com/?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | BestofHomeCare.com
> <http://www.bestofhomecare.com/?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | Blog
> <http://www.homecarepulse.com/blog?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>
>
> *From:* Jason Hamilton
> *Sent:* Tuesday, October 11, 2016 5:10 PM
> *To:* 'dev@nifi.apache.org' 
> *Subject:* RE: InvokeScriptedProcessor
>
> Sorry, it looks like the mailing list hates attachments.  The script is:
>
> import org.apache.nifi.dbcp.DBCPService
> import groovy.sql.Sql
> import groovy.json.JsonBuilder
>
> class MySQLToJSON implements Processor {
>
> def REL_SUCCESS = new Relationship.Builder()
> .name("success")
> .description("The flowfile with the
> specified query results was successfully transferred.")
> .build();
>
> // TODO: figure out how to get exception messages to show
> in the bulletins or go out to a failure relationship
> def REL_FAILURE = new Relationship.Builder()
> .name("failure")
> .description("An error occured while
> running the specified query.")
> .build();
>
> def DBCP_SERVICE = new PropertyDescriptor.Builder()
> .name('Database Connection Pooling
> Service')
> .description("The Controller Service that
> is used to obtain connection to database.")
> .required(true)
> .identifiesControllerService(
> DBCPService.class)
> .build()
>
> def DB_QUERY = new PropertyDescriptor.Builder()
> .name('SQL Query')
> .description('SQL query to be executed.')
> .required(true)
> .expressionLanguageSupported(true)
> .addValidator(Validator.VALID)
> .build()
>
> def FILENAME = new PropertyDescriptor.Builder()
> .name('File Name')
> .description('Sets the filename attribute
> of the flowfile (do not include the extension).')
> .required(true)
> .expressionLanguageSupported(true)
> .addValidator(Validator.VALID)
> .build()
>
> def ComponentLog log
>
> @Override
> void initialize(ProcessorInitial

Re: InvokeScriptedProcessor

2016-10-11 Thread Andy LoPresto 
Jason,

Sorry to hear you had a frustrating experience with InvokeScriptedProcessor. 
While it is marked with the “Experimental” annotation for just this reason 
(it’s performed well with standard behaviors but we’re sure there are still 
some edge cases remaining), no one should accept that as an excuse for a less 
than satisfactory user experience.

If you could please submit these issues as a Jira ticket [1], we can work to 
identify the cause and address them in a coming release. While I appreciate 
that this may seem like duplicate work after you described the behavior in your 
email, capturing it in a Jira will help us gather specific information like 
hardware, OS, JRE, and NiFi version in a more permanent and trackable system 
and ensure a developer addresses it.

As you seem quite comfortable developing custom processor code, you may be able 
to reduce the development lifecycle cost by testing your scripts 
programmatically rather than running a full environment to perform the initial 
testing. Matt has a good post on the same site you referenced for achieving 
this [2].

Thank you for sharing the solution to your issues with the list. Please let us 
know if there is anything else we can do to make NiFi easier to use/more 
productive as a tool for your use cases.

[1] https://issues.apache.org/jira/secure/CreateIssue!default.jspa
[2] 
https://funnifi.blogspot.com/2016/06/testing-executescript-processor-scripts.html
 
<https://funnifi.blogspot.com/2016/06/testing-executescript-processor-scripts.html>


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 11, 2016, at 4:37 PM, Jason Hamilton  wrote:
> 
> Well, I solved it myself – the script was fine, but apparently after some 
> earlier failed attempts that threw exceptions, the DBCPService was hosed.
> 
> SOLUTION: I had to disable and then re-enable the DBCPService
> 
> In summary, there are 2 buggy behaviors that have been huge gotchas (aka. 
> hours wasted for nothing) for me with InvokeScriptedProcessor:
> 1)  After correcting syntax errors in my property definitions, I had to 
> completely delete and recreate my InvokeScriptedProcessor instance instead of 
> just updating the text in the processor – otherwise the error still reoccurs 
> even after fixing the code
> 2)  After throwing an exception while using a DBCPService I had to 
> disable/reenable the service – otherwise the Processor still crashes even 
> after fixing the code
> 
> Not sure if there’s even a way to prevent those from happening, but man is it 
> misleading when you (correctly) fix your code, start things up again, and the 
> error is still there!  Perhaps some missing cleanup?  Maybe it’s just the 
> nature of the beast.  Anyway, sorry for the bother, just figured I post this 
> solution for completeness.  Cheers!
> 
>  
> <http://homecarepulse.com/?utm_source=logoLink&utm_medium=Jason&utm_campaign=EmailSignature>
> Jason Hamilton
> Senior Software Developer
> Home Care Pulse, LLC
> www.homecarepulse.com 
> <http://homecarepulse.com/?utm_source=websiteLink&utm_medium=Jason&utm_campaign=EmailSignature>
> 208.228.0895 (Direct) 
> 877.307.8573 (Office) 
>  <http://www.linkedin.com/company/home-care-pulse> 
> <http://facebook.com/homecarepulse> <http://twitter.com/homecarepulse> 
> <http://plus.google.com/+Homecarepulse/posts>
> Satisfaction Management 
> <http://www.homecarepulse.com/program-details?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | Benchmarking 
> <http://benchmarking.homecarepulse.com/?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | BestofHomeCare.com 
> <http://www.bestofhomecare.com/?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
>  | Blog 
> <http://www.homecarepulse.com/blog?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
> 
> 
> From: Jason Hamilton
> Sent: Tuesday, October 11, 2016 5:10 PM
> To: 'dev@nifi.apache.org' 
> Subject: RE: InvokeScriptedProcessor
> 
> Sorry, it looks like the mailing list hates attachments.  The script is:
> 
> import org.apache.nifi.dbcp.DBCPService
> import groovy.sql.Sql
> import groovy.json.JsonBuilder
> 
> class MySQLToJSON implements Processor {
> 
> def REL_SUCCESS = new Relationship.Builder()
> .name("success")
> .description("The flowfile with the specified 
> query results was successfully transferred.")
> .build();
> 
> // TODO: figure out how to get exception messages to show in

RE: InvokeScriptedProcessor

2016-10-11 Thread Jason Hamilton 
Well, I solved it myself - the script was fine, but apparently after some 
earlier failed attempts that threw exceptions, the DBCPService was hosed.

SOLUTION: I had to disable and then re-enable the DBCPService

In summary, there are 2 buggy behaviors that have been huge gotchas (aka. hours 
wasted for nothing) for me with InvokeScriptedProcessor:

1)  After correcting syntax errors in my property definitions, I had to 
completely delete and recreate my InvokeScriptedProcessor instance instead of 
just updating the text in the processor - otherwise the error still reoccurs 
even after fixing the code

2)  After throwing an exception while using a DBCPService I had to 
disable/reenable the service - otherwise the Processor still crashes even after 
fixing the code

Not sure if there's even a way to prevent those from happening, but man is it 
misleading when you (correctly) fix your code, start things up again, and the 
error is still there!  Perhaps some missing cleanup?  Maybe it's just the 
nature of the beast.  Anyway, sorry for the bother, just figured I post this 
solution for completeness.  Cheers!

[photo]<http://homecarepulse.com/?utm_source=logoLink&utm_medium=Jason&utm_campaign=EmailSignature>

Jason Hamilton
Senior Software Developer
Home Care Pulse, LLC
www.homecarepulse.com<http://homecarepulse.com?utm_source=websiteLink&utm_medium=Jason&utm_campaign=EmailSignature>
208.228.0895 (Direct)
877.307.8573 (Office)


[Linnked 
In]<http://www.linkedin.com/company/home-care-pulse>[Facebook]<http://facebook.com/homecarepulse>[Twitter]<http://twitter.com/homecarepulse>[Google+]<http://plus.google.com/+Homecarepulse/posts>

Satisfaction 
Management<http://www.homecarepulse.com/program-details?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
 | 
Benchmarking<http://benchmarking.homecarepulse.com?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
 | 
BestofHomeCare.com<http://www.bestofhomecare.com?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>
 | 
Blog<http://www.homecarepulse.com/blog?utm_source=bottomLinks&utm_medium=Jason&utm_campaign=EmailSignature>



From: Jason Hamilton
Sent: Tuesday, October 11, 2016 5:10 PM
To: 'dev@nifi.apache.org' 
Subject: RE: InvokeScriptedProcessor

Sorry, it looks like the mailing list hates attachments.  The script is:

import org.apache.nifi.dbcp.DBCPService
import groovy.sql.Sql
import groovy.json.JsonBuilder

class MySQLToJSON implements Processor {

def REL_SUCCESS = new Relationship.Builder()
.name("success")
.description("The flowfile with the specified 
query results was successfully transferred.")
.build();

// TODO: figure out how to get exception messages to show in 
the bulletins or go out to a failure relationship
def REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("An error occured while running 
the specified query.")
.build();

def DBCP_SERVICE = new PropertyDescriptor.Builder()
.name('Database Connection Pooling Service')
.description("The Controller Service that is 
used to obtain connection to database.")
.required(true)
.identifiesControllerService(DBCPService.class)
.build()

def DB_QUERY = new PropertyDescriptor.Builder()
.name('SQL Query')
.description('SQL query to be executed.')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def FILENAME = new PropertyDescriptor.Builder()
.name('File Name')
.description('Sets the filename attribute of 
the flowfile (do not include the extension).')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def ComponentLog log

@Override
void initialize(ProcessorInitializationContext context) {
log = context.getLogger()
}

@Override

Set getRelationships() {
return [REL_SUCCESS, REL_FAILURE] as Set
}

@Override
void onTrigger(ProcessContext co

RE: InvokeScriptedProcessor

2016-10-11 Thread Jason Hamilton 
Sorry, it looks like the mailing list hates attachments.  The script is:

import org.apache.nifi.dbcp.DBCPService
import groovy.sql.Sql
import groovy.json.JsonBuilder

class MySQLToJSON implements Processor {

def REL_SUCCESS = new Relationship.Builder()
.name("success")
.description("The flowfile with the specified 
query results was successfully transferred.")
.build();

// TODO: figure out how to get exception messages to show in 
the bulletins or go out to a failure relationship
def REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("An error occured while running 
the specified query.")
.build();

def DBCP_SERVICE = new PropertyDescriptor.Builder()
.name('Database Connection Pooling Service')
.description("The Controller Service that is 
used to obtain connection to database.")
.required(true)
.identifiesControllerService(DBCPService.class)
.build()

def DB_QUERY = new PropertyDescriptor.Builder()
.name('SQL Query')
.description('SQL query to be executed.')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def FILENAME = new PropertyDescriptor.Builder()
.name('File Name')
.description('Sets the filename attribute of 
the flowfile (do not include the extension).')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def ComponentLog log

@Override
void initialize(ProcessorInitializationContext context) {
log = context.getLogger()
}

@Override

Set getRelationships() {
return [REL_SUCCESS, REL_FAILURE] as Set
}

@Override
void onTrigger(ProcessContext context, ProcessSessionFactory 
sessionFactory) throws ProcessException {
try {

def session = sessionFactory.createSession()
//def flowFile = session.get() // use existing flowfile if one is 
there
def flowFile = session.create() // create new flowfile if not
log.info("Flowfile created")

def filename = 
context.getProperty(FILENAME)?.evaluateAttributeExpressions()?.getValue()
log.info("Filename set")

// TODO: Evaluate expression 
langauge for getProperty calls
def dbcpService = 
context.getProperty(DBCP_SERVICE).asControllerService(DBCPService.class)
log.info("dbcpService set")
def conn = 
dbcpService.getConnection()
log.info("getConnection() 
called")
def sql = new Sql(conn)
log.info("Sql(conn) called set")
def dbResults = sql.rows(context.getProperty(DB_QUERY).value)
log.info("dbResults set")
def jsonResults = new 
JsonBuilder(dbResults).toPrettyString()
log.info("jsonResults set")
flowFile = 
session.write(flowFile, { outputStream ->

outputStream.write(jsonResults.getBytes('UTF-8'))

log.info("outputStream written")
} as 
OutputStreamCallback)

//flowFile = 
session.putAttribute(flowFile, "executesql.row.count", dbResults.size())
flowFile = 
session.putAttribute(flowFile, "mime.type", "application/json")
log.info("mime.type attribute 
set")
flowFile = 
session.putAttribute(flowFile, "filename", "test")
log.info("fi