date:20161011

Re: InvokeScriptedProcessor

2016-10-11 Thread Andy LoPresto

Jason,

Sorry to hear you had a frustrating experience with InvokeScriptedProcessor. 
While it is marked with the “Experimental” annotation for just this reason 
(it’s performed well with standard behaviors but we’re sure there are still 
some edge cases remaining), no one should accept that as an excuse for a less 
than satisfactory user experience.

If you could please submit these issues as a Jira ticket [1], we can work to 
identify the cause and address them in a coming release. While I appreciate 
that this may seem like duplicate work after you described the behavior in your 
email, capturing it in a Jira will help us gather specific information like 
hardware, OS, JRE, and NiFi version in a more permanent and trackable system 
and ensure a developer addresses it.

As you seem quite comfortable developing custom processor code, you may be able 
to reduce the development lifecycle cost by testing your scripts 
programmatically rather than running a full environment to perform the initial 
testing. Matt has a good post on the same site you referenced for achieving 
this [2].

Thank you for sharing the solution to your issues with the list. Please let us 
know if there is anything else we can do to make NiFi easier to use/more 
productive as a tool for your use cases.

[1] https://issues.apache.org/jira/secure/CreateIssue!default.jspa
[2] 
https://funnifi.blogspot.com/2016/06/testing-executescript-processor-scripts.html

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 11, 2016, at 4:37 PM, Jason Hamilton  wrote:
> 
> Well, I solved it myself – the script was fine, but apparently after some 
> earlier failed attempts that threw exceptions, the DBCPService was hosed.
> 
> SOLUTION: I had to disable and then re-enable the DBCPService
> 
> In summary, there are 2 buggy behaviors that have been huge gotchas (aka. 
> hours wasted for nothing) for me with InvokeScriptedProcessor:
> 1)  After correcting syntax errors in my property definitions, I had to 
> completely delete and recreate my InvokeScriptedProcessor instance instead of 
> just updating the text in the processor – otherwise the error still reoccurs 
> even after fixing the code
> 2)  After throwing an exception while using a DBCPService I had to 
> disable/reenable the service – otherwise the Processor still crashes even 
> after fixing the code
> 
> Not sure if there’s even a way to prevent those from happening, but man is it 
> misleading when you (correctly) fix your code, start things up again, and the 
> error is still there!  Perhaps some missing cleanup?  Maybe it’s just the 
> nature of the beast.  Anyway, sorry for the bother, just figured I post this 
> solution for completeness.  Cheers!
> 
>  
> 
> Jason Hamilton
> Senior Software Developer
> Home Care Pulse, LLC
> www.homecarepulse.com 
> 
> 208.228.0895 (Direct) 
> 877.307.8573 (Office) 
>   
>   
> 
> Satisfaction Management 
> 
>  | Benchmarking 
> 
>  | BestofHomeCare.com 
> 
>  | Blog 
> 
> 
> 
> From: Jason Hamilton
> Sent: Tuesday, October 11, 2016 5:10 PM
> To: 'dev@nifi.apache.org' 
> Subject: RE: InvokeScriptedProcessor
> 
> Sorry, it looks like the mailing list hates attachments.  The script is:
> 
> import org.apache.nifi.dbcp.DBCPService
> import groovy.sql.Sql
> import groovy.json.JsonBuilder
> 
> class MySQLToJSON implements Processor {
> 
> def REL_SUCCESS = new Relationship.Builder()
> .name("success")
> .description("The flowfile with the specified 
> query results was successfully transferred.")
> .build();
> 
> // TODO: figure out how to get exception messages to show in 
> the bulletins or go out to a failure relationship
> def REL_FAILURE = new Relationship.Builder()
> .name("failure")
> .description("An error occured while running 
> the specified query.")
> .build();
> 
> def

RE: InvokeScriptedProcessor

2016-10-11 Thread Jason Hamilton

Well, I solved it myself - the script was fine, but apparently after some 
earlier failed attempts that threw exceptions, the DBCPService was hosed.

SOLUTION: I had to disable and then re-enable the DBCPService

In summary, there are 2 buggy behaviors that have been huge gotchas (aka. hours 
wasted for nothing) for me with InvokeScriptedProcessor:

1)  After correcting syntax errors in my property definitions, I had to 
completely delete and recreate my InvokeScriptedProcessor instance instead of 
just updating the text in the processor - otherwise the error still reoccurs 
even after fixing the code

2)  After throwing an exception while using a DBCPService I had to 
disable/reenable the service - otherwise the Processor still crashes even after 
fixing the code

Not sure if there's even a way to prevent those from happening, but man is it 
misleading when you (correctly) fix your code, start things up again, and the 
error is still there!  Perhaps some missing cleanup?  Maybe it's just the 
nature of the beast.  Anyway, sorry for the bother, just figured I post this 
solution for completeness.  Cheers!

[photo]

Jason Hamilton
Senior Software Developer
Home Care Pulse, LLC
www.homecarepulse.com
208.228.0895 (Direct)
877.307.8573 (Office)


[Linnked 
In][Facebook][Twitter][Google+]

Satisfaction 
Management
 | 
Benchmarking
 | 
BestofHomeCare.com
 | 
Blog



From: Jason Hamilton
Sent: Tuesday, October 11, 2016 5:10 PM
To: 'dev@nifi.apache.org' 
Subject: RE: InvokeScriptedProcessor

Sorry, it looks like the mailing list hates attachments.  The script is:

import org.apache.nifi.dbcp.DBCPService
import groovy.sql.Sql
import groovy.json.JsonBuilder

class MySQLToJSON implements Processor {

def REL_SUCCESS = new Relationship.Builder()
.name("success")
.description("The flowfile with the specified 
query results was successfully transferred.")
.build();

// TODO: figure out how to get exception messages to show in 
the bulletins or go out to a failure relationship
def REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("An error occured while running 
the specified query.")
.build();

def DBCP_SERVICE = new PropertyDescriptor.Builder()
.name('Database Connection Pooling Service')
.description("The Controller Service that is 
used to obtain connection to database.")
.required(true)
.identifiesControllerService(DBCPService.class)
.build()

def DB_QUERY = new PropertyDescriptor.Builder()
.name('SQL Query')
.description('SQL query to be executed.')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def FILENAME = new PropertyDescriptor.Builder()
.name('File Name')
.description('Sets the filename attribute of 
the flowfile (do not include the extension).')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def ComponentLog log

@Override
void initialize(ProcessorInitializationContext context) {
log = context.getLogger()
}

@Override

Set getRelationships() {
return [REL_SUCCESS, REL_FAILURE] as Set
}

@Override
void onTrigger(ProcessContext context, ProcessSessionFactory 
sessionFactory) throws ProcessException {
try {

def session = sessionFactory.createSession()
//def flowFile = session.get() // use existing flowfile if one is 
there
def flowFile =

RE: InvokeScriptedProcessor

2016-10-11 Thread Jason Hamilton

Sorry, it looks like the mailing list hates attachments.  The script is:

import org.apache.nifi.dbcp.DBCPService
import groovy.sql.Sql
import groovy.json.JsonBuilder

class MySQLToJSON implements Processor {

def REL_SUCCESS = new Relationship.Builder()
.name("success")
.description("The flowfile with the specified 
query results was successfully transferred.")
.build();

// TODO: figure out how to get exception messages to show in 
the bulletins or go out to a failure relationship
def REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("An error occured while running 
the specified query.")
.build();

def DBCP_SERVICE = new PropertyDescriptor.Builder()
.name('Database Connection Pooling Service')
.description("The Controller Service that is 
used to obtain connection to database.")
.required(true)
.identifiesControllerService(DBCPService.class)
.build()

def DB_QUERY = new PropertyDescriptor.Builder()
.name('SQL Query')
.description('SQL query to be executed.')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def FILENAME = new PropertyDescriptor.Builder()
.name('File Name')
.description('Sets the filename attribute of 
the flowfile (do not include the extension).')
.required(true)
.expressionLanguageSupported(true)
.addValidator(Validator.VALID)
.build()

def ComponentLog log

@Override
void initialize(ProcessorInitializationContext context) {
log = context.getLogger()
}

@Override

Set getRelationships() {
return [REL_SUCCESS, REL_FAILURE] as Set
}

@Override
void onTrigger(ProcessContext context, ProcessSessionFactory 
sessionFactory) throws ProcessException {
try {

def session = sessionFactory.createSession()
//def flowFile = session.get() // use existing flowfile if one is 
there
def flowFile = session.create() // create new flowfile if not
log.info("Flowfile created")

def filename = 
context.getProperty(FILENAME)?.evaluateAttributeExpressions()?.getValue()
log.info("Filename set")

// TODO: Evaluate expression 
langauge for getProperty calls
def dbcpService = 
context.getProperty(DBCP_SERVICE).asControllerService(DBCPService.class)
log.info("dbcpService set")
def conn = 
dbcpService.getConnection()
log.info("getConnection() 
called")
def sql = new Sql(conn)
log.info("Sql(conn) called set")
def dbResults = sql.rows(context.getProperty(DB_QUERY).value)
log.info("dbResults set")
def jsonResults = new 
JsonBuilder(dbResults).toPrettyString()
log.info("jsonResults set")
flowFile = 
session.write(flowFile, { outputStream ->

outputStream.write(jsonResults.getBytes('UTF-8'))

log.info("outputStream written")
} as 
OutputStreamCallback)

//flowFile = 
session.putAttribute(flowFile, "executesql.row.count", dbResults.size())
flowFile = 
session.putAttribute(flowFile, "mime.type", "application/json")
log.info("mime.type attribute 
set")
flowFile = 
session.putAttribute(flowFile, "filename", "test")

InvokeScriptedProcessor

2016-10-11 Thread Jason Hamilton

Hello everyone,

I have been struggling all day trying to make a simple MySQL to JSON processor 
(avoids the nasty issues with type casting MySQL numerics to Avro just to go to 
JSON anyway) in Groovy (see attached file).  There's more interesting 
processors I need to make but this is my starting point.  I have carefully used 
the following resources:
http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html
http://funnifi.blogspot.com/2016_02_01_archive.html

I am using Nifi 1.0.0 download as a binary from the website on a clean install 
of CentOS 7 x64 with Oracle JDK 8.

I have verified the database pool works by running the same test query from an 
ExecuteSQL Processor and it works.  From what I can tell the connection is even 
established from the script, but for the life of me I can't figure why it is 
now getting the following useless error:

---
InvokeScriptedProcessor[id=b5e567cf-0157-1000-cea6-4dc6cbaed0e3] 
InvokeScriptedProcessor[id=b5e567cf-0157-1000-cea6-4dc6cbaed0e3] failed to 
process session due to java.lang.reflect.UndeclaredThrowableException: 
java.lang.reflect.UndeclaredThrowableException
---

I have debugging log.info lines on every call to see where it breaks down, and 
the results seem inconsistent - I've seen it log the  "filename attribute set" 
clear at the bottom, but no data! What silly thing am I missing here?

[photo]

Jason Hamilton
Senior Software Developer
Home Care Pulse, LLC
www.homecarepulse.com
208.228.0895 (Direct)
877.307.8573 (Office)


[Linnked 
In][Facebook][Twitter][Google+]

Satisfaction 
Management
 | 
Benchmarking
 | 
BestofHomeCare.com
 | 
Blog

Provenance repo corruption

2016-10-11 Thread Brandon DeVries

Devs,

I just opened a ticket to address an issue we've encountered with
Provenance repo corruption[1].  This would address (as is currently
partially being done) how to recover from a corrupt provenance repo.
However, the question is whether we can avoid this sort of corruption in
the first place.  The immediate thought that jumped to mind was wrapping
the writes to lucene with a write ahead log.  Obviously, this would
increase the overhead on something that is already fairly expensive.
However, in cases where provenance is *really* important, it might be worth
considering.  This could potentially be another flavor of
ProvenanceEventRepository, e.g.  WriteAheadPersistentProvenanceRepository
or FaultTolerantProvenanceRepository.  Does anyone have any thoughts /
opinions?

Brandon

[1] https://issues.apache.org/jira/browse/NIFI-2890

Re: [DISCUSS] Closing in on a 0.x release

2016-10-11 Thread Tony Kurc

Awesome. I propose we start building a release candidate off of
40618364e70a966f9c1e425674b53b22b1fb0fb0 soon.

I believe I was the sole volunteer to RM, and unless I hear otherwise, I
presume I will be doing so. I'd like to give the commit at least a good 24
hours for some people to bang on it before I start pulling together an RC.



On Tue, Oct 11, 2016 at 11:30 AM, Michael Moser  wrote:

> NIFI-2774 is now complete and merged to both master and 0.x branches.  +1
> on a release from the 0.x branch now.
>
> -- Mike
>
>
> On Mon, Oct 10, 2016 at 10:24 AM, Michael Moser 
> wrote:
>
> > I feel that Oleg was really close, and it would be nice for this to be in
> > 0.7.1 but it isn't necessary. I did functional testing on the current
> state
> > of the PR and I am +1 in that respect.
> >
> > -- Mike
> >
> > On Oct 10, 2016 9:40 AM, "Tony Kurc"  wrote:
> >
> >> So in reviewing the Jiras, it looks like the two tickets NIFI-2429,
> >> NIFI-2874 were merged in and NIFI-2774 is still under discussion. Oleg,
> >> Mike, are we feeling like we're close, or would this best fit in the
> next
> >> 0.x release?
> >>
> >> Tony
> >>
> >>
> >>
> >> On Fri, Oct 7, 2016 at 3:21 PM, Michael Moser 
> wrote:
> >>
> >> > Thanks Joe Witt, I reviewed that PR and got it into 0.x.
> >> >
> >> > Since we decided that our next 0.x release will be 0.7.1, I am going
> >> > through JIRA and for all Resolved tickets marked against 0.8.0 I am
> >> > changing their Fix Version to 0.7.1.  Open tickets I will not change.
> >> >
> >> > -- Mike
> >> >
> >> >
> >> > On Fri, Oct 7, 2016 at 9:56 AM, Joe Witt  wrote:
> >> >
> >> > > Team,
> >> > >
> >> > > Mark Payne just opened this one: https://issues.apache.org/
> >> > > jira/browse/NIFI-2874
> >> > >
> >> > > It should probably be in this release if able.
> >> > >
> >> > > Thanks
> >> > > Joe
> >> > >
> >> > > On Mon, Oct 3, 2016 at 10:48 AM, Michael Moser 
> >> > wrote:
> >> > > > I am reviewing the PR for NIFI-2774 ConsumeJMS and we need someone
> >> to
> >> > > > review the PR for NIFI-2429 PersistentProvenanceRepository.  Once
> >> > those
> >> > > are
> >> > > > complete I think we can start the process to cut 0.7.1.
> >> > > >
> >> > > > -- Mike
> >> > > >
> >> > > > On Mon, Oct 3, 2016 at 9:49 AM, Tony Kurc 
> wrote:
> >> > > >
> >> > > >> So, sounds like we have enough support to go ahead. How are we
> >> feeling
> >> > > >> about what our timeline should be on this?
> >> > > >>
> >> > > >>
> >> > > >> On Wed, Sep 28, 2016 at 11:14 PM, Joe Witt 
> >> > wrote:
> >> > > >>
> >> > > >> > +1 to an 0.7.1 with the bugs that have been addressed already.
> >> > > >> > Even bigger +1 to Tony volunteering as RM!
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > Joe
> >> > > >> >
> >> > > >> > On Tue, Sep 27, 2016 at 10:24 PM, Brandon DeVries  >
> >> > > wrote:
> >> > > >> > > I agree sooner rather than later for cutting 0.7.1. I think
> >> Mike's
> >> > > >> > question
> >> > > >> > > to some degree was whether or not some of those tickets were
> >> worth
> >> > > >> fixing
> >> > > >> > > in 0.x. For example, I'm not sure how much I care about:
> >> > > >> > >
> >> > > >> > > NIFI-2571 deprecate NiFiProperties.getInstance()
> >> > > >> > > NIFI-2163 nifi.sh follow the Linux service spec
> >> > > >> > >
> >> > > >> > > On the other, there are some I would like to see, even if its
> >> in
> >> > > 0.7.2
> >> > > >> or
> >> > > >> > > 0.8.0, e.g.:
> >> > > >> > >
> >> > > >> > > NIFI-2433 "Primary Node Only" processors
> >> > > >> > > NIFI-2562 PutHDFS data corruption
> >> > > >> > >
> >> > > >> > > But, there are a number of things that are currently
> committed
> >> (or
> >> > > have
> >> > > >> > > patch available) that I'd like to see available as soon as
> >> > > possible. So
> >> > > >> > > rather than wait for more "nice to haves", I'd rather address
> >> the
> >> > > >> > immediate
> >> > > >> > > needs... Immediately.
> >> > > >> > >
> >> > > >> > > Brandon
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > On Tue, Sep 27, 2016 at 10:15 PM Tony Kurc  >
> >> > > wrote:
> >> > > >> > >
> >> > > >> > >> I think I brought this up before, I sort of expected we may
> do
> >> > more
> >> > > >> 0.x
> >> > > >> > >> releases. I certainly think the more the bugs we can fix,
> the
> >> > > merrier,
> >> > > >> > and
> >> > > >> > >> it seems like your list is a good initial strawman for a bug
> >> fix
> >> > > >> > release of
> >> > > >> > >> we collectively would like to put one together.
> >> > > >> > >>
> >> > > >> > >> While the tickets with work to do on them would be great to
> >> have
> >> > > >> fixed,
> >> > > >> > I
> >> > > >> > >> personally would rather see a release with some fixes and a
> >> > couple
> >> > > >> known
> >> > > >> > >> issues than holding off for "perfection", especially as a
> lot
> >>

Re: Which are the underlying files for NiFi?

2016-10-11 Thread Jeff

Hello Russ!  I'm glad you brought this up on the list.  I've been thinking
about reworking/enhancing the NiFi testing framework to enable testing of
flows and flow templates from a programmatic perspective, without needing
to stand up an entire NiFi instance.  A lot of this is in the idea stage
right now, and could require a fair chunk of refactoring.  I've discussed
this a bit with other members of the community, and it has been pointed out
that we don't want to require deep knowledge of mocking frameworks, or the
NiFi framework itself, to allow extension developers to test the various
use cases that their extensions would be required to handle gracefully.
Improvements could be made to NiFi's testing framework to put extensions
through a number of predetermined use cases, such as a processor's
onTrigger invocation where the flowfile is null, or that processing can be
stopped within a reasonable amount of time when the processor is told to
stop or is disabled, in addition to the developer's own explicit test cases.

My goal would be to enable the testing of any use case, whether it be a
unit or integration test, and be able to allow the developer to mock
endpoints that would be external to NiFi itself (web services, databases,
etc) at the processor or flow/template level.  Developers should be able to
verify a flow/template's behavior throughout the flow/template as well.  I
have some ideas on how we can achieve this, and I'll be working on a
different way to write extensions to allow for easier mocking/injection of
collaborators/dependencies.

If you would like to collaborate on achieving these testing goals, I'd be
happy to discuss any ideas with you and the rest of the community.

Thanks,
Jeff

On Tue, Oct 11, 2016 at 12:43 PM Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Andy,
>
> First, many thanks for your answer which was very complete. It's exactly
> what I expected and I'm sure I had seen /flow.xml.gz/ in some thread
> before, but I could not remember it nor did I pay attention to that file
> while looking through the NiFi subdirectories (/*_repository/, /run/,
> /state/ and /work/) as I didn't think to look in /conf/.
>
> However, as I peruse testing [4] below, I don't find anything that
> readily explains (or even hints) to me how to replace (what's below
> serves only as the "essence" of) a JUnit test with all or part of a
> /flow.xml[.gz]/ or inject it:
>
> TestRunner  runner = TestRunners.newTestRunner( new MyProcessor() );
> InputStream data   = new FileInputStream( "sample.data" );
>
> runner.enqueue( data );
> runner.run( 1 );
> runner.assertQueueEmpty();
>
> List< MockFlowFile > results = runner.getFlowFilesForRelationship(
> MyProcessor.SUCCESS );
>
> MockFlowFile result = results.get( 0 );
> String   actual = new String( runner.getContentAsByteArray(
> result ) );
>
>
> I have long written fairly exhaustive unit tests for the features of my
> custom processors and am very pleased with the result. Only rarely have
> I done something that worked in a unit test, but did not in production
> (and I think it was something I was over-looking anyway).
>
> Being able to run a whole- or partial flow from the unit-testing
> framework would be immeasurably cheaper and easier to set up than
> configuring an instance of a server, NiFi, and kicking it off. I need to
> be able to set up test code that will run two or more processors in this
> framework with the appropriate inter-connections. (You were suggesting
> that this is possible, right?) And I'm very happy to consume something
> like /flow.xml/ in doing it.
>
> What I (lightly) reproach in the developer guide is that one needs to be
> pretty familiar already with writing custom processors before reading it
> in order to map its advice to actual code because of its gross lack of
> real code samples. Early on, I found NiFi Rules! samples and actual NiFi
> standard-processor code to be much more useful to me than this guide.
>
> Russ
>
>
> On 10/10/2016 08:14 PM, Andy LoPresto wrote:
> > Hi Russell,
> >
> > The “flow” (the processors on the canvas and the relationships between
> > them) is stored on disk as /conf/flow.xml.gz [1]. You can
> > also export selections from the flow as “templates” [2], along with a
> > name and description, which can then be saved as XML files and loaded
> > into a new instance of NiFi with (non-sensitive) configuration values
> > and properties maintained. You may also be interested in the ongoing
> > work to support the “Software Development Lifecycle (SDLC)” or
> > “Development to Production Pipeline (D2P)” efforts to allow users to
> > “develop” flows in testbed environments and “promote” them through
> > code control/repositories to QE/QA and then Production environments [3].
> >
> > If you are attempting to perform automated testing of flows or flow
> > segments, you can do this with the test runner [4]. I would encourage
> > you to store the flow

Re: Which are the underlying files for NiFi?

2016-10-11 Thread Russell Bateman


Andy,

First, many thanks for your answer which was very complete. It's exactly 
what I expected and I'm sure I had seen /flow.xml.gz/ in some thread 
before, but I could not remember it nor did I pay attention to that file 
while looking through the NiFi subdirectories (/*_repository/, /run/, 
/state/ and /work/) as I didn't think to look in /conf/.


However, as I peruse testing [4] below, I don't find anything that 
readily explains (or even hints) to me how to replace (what's below 
serves only as the "essence" of) a JUnit test with all or part of a 
/flow.xml[.gz]/ or inject it:


   TestRunner  runner = TestRunners.newTestRunner( new MyProcessor() );
   InputStream data   = new FileInputStream( "sample.data" );

   runner.enqueue( data );
   runner.run( 1 );
   runner.assertQueueEmpty();

   List< MockFlowFile > results = runner.getFlowFilesForRelationship(
   MyProcessor.SUCCESS );

   MockFlowFile result = results.get( 0 );
   String   actual = new String( runner.getContentAsByteArray(
   result ) );


I have long written fairly exhaustive unit tests for the features of my 
custom processors and am very pleased with the result. Only rarely have 
I done something that worked in a unit test, but did not in production 
(and I think it was something I was over-looking anyway).


Being able to run a whole- or partial flow from the unit-testing 
framework would be immeasurably cheaper and easier to set up than 
configuring an instance of a server, NiFi, and kicking it off. I need to 
be able to set up test code that will run two or more processors in this 
framework with the appropriate inter-connections. (You were suggesting 
that this is possible, right?) And I'm very happy to consume something 
like /flow.xml/ in doing it.


What I (lightly) reproach in the developer guide is that one needs to be 
pretty familiar already with writing custom processors before reading it 
in order to map its advice to actual code because of its gross lack of 
real code samples. Early on, I found NiFi Rules! samples and actual NiFi 
standard-processor code to be much more useful to me than this guide.


Russ


On 10/10/2016 08:14 PM, Andy LoPresto wrote:

Hi Russell,

The “flow” (the processors on the canvas and the relationships between 
them) is stored on disk as /conf/flow.xml.gz [1]. You can 
also export selections from the flow as “templates” [2], along with a 
name and description, which can then be saved as XML files and loaded 
into a new instance of NiFi with (non-sensitive) configuration values 
and properties maintained. You may also be interested in the ongoing 
work to support the “Software Development Lifecycle (SDLC)” or 
“Development to Production Pipeline (D2P)” efforts to allow users to 
“develop” flows in testbed environments and “promote” them through 
code control/repositories to QE/QA and then Production environments [3].


If you are attempting to perform automated testing of flows or flow 
segments, you can do this with the test runner [4]. I would encourage 
you to store the flow in a template or full flow file (flow.xml.gz, 
not a “flowfile”) and load that during the test rather than 
programmatically instantiating and configuring each component and 
relationship in the test code, but you are able to do both. Also, be 
aware that the TestRunner is just that, and approximates/mocks some 
elements of the NiFi environment, so you should understand what 
decisions are being made behind the scenes to ensure that what is 
covered by the test is exactly what you expect.


Hope this helps clarify some things. Good luck.

[1] 
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#terminology

[2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates
[3] 
https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
[4] 
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing


Andy LoPresto
alopre...@apache.org 
/alopresto.apa...@gmail.com /
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Oct 10, 2016, at 3:28 PM, Russell Bateman 
> wrote:


Thanks, Joe. I've digested the document you linked.

I think my real question is simpler. I don't think I'm interested in 
hacking NiFi's internal details. I assumed that once I "paint" my 
canvas with processors and relationships, it's harvestable in some 
form, in some /.xml/ file somewhere, and easily transported to, then 
dropped into, a new NiFi installation on some server or host for 
running it. I assumed my testing would be thus configured, then 
kicked off somehow.


Now, I've long used the mocking from the JUnit 
/org.apache.nifi.util.TestRunner/, but it never dawned on me (still 
hasn't, in fact) that I can erect tests using it to run data through 
a whole flow. Were you suggesting that this is the case?


Thanks,
Russ

Re: [DISCUSS] Closing in on a 0.x release

2016-10-11 Thread Michael Moser

NIFI-2774 is now complete and merged to both master and 0.x branches.  +1
on a release from the 0.x branch now.

-- Mike


On Mon, Oct 10, 2016 at 10:24 AM, Michael Moser  wrote:

> I feel that Oleg was really close, and it would be nice for this to be in
> 0.7.1 but it isn't necessary. I did functional testing on the current state
> of the PR and I am +1 in that respect.
>
> -- Mike
>
> On Oct 10, 2016 9:40 AM, "Tony Kurc"  wrote:
>
>> So in reviewing the Jiras, it looks like the two tickets NIFI-2429,
>> NIFI-2874 were merged in and NIFI-2774 is still under discussion. Oleg,
>> Mike, are we feeling like we're close, or would this best fit in the next
>> 0.x release?
>>
>> Tony
>>
>>
>>
>> On Fri, Oct 7, 2016 at 3:21 PM, Michael Moser  wrote:
>>
>> > Thanks Joe Witt, I reviewed that PR and got it into 0.x.
>> >
>> > Since we decided that our next 0.x release will be 0.7.1, I am going
>> > through JIRA and for all Resolved tickets marked against 0.8.0 I am
>> > changing their Fix Version to 0.7.1.  Open tickets I will not change.
>> >
>> > -- Mike
>> >
>> >
>> > On Fri, Oct 7, 2016 at 9:56 AM, Joe Witt  wrote:
>> >
>> > > Team,
>> > >
>> > > Mark Payne just opened this one: https://issues.apache.org/
>> > > jira/browse/NIFI-2874
>> > >
>> > > It should probably be in this release if able.
>> > >
>> > > Thanks
>> > > Joe
>> > >
>> > > On Mon, Oct 3, 2016 at 10:48 AM, Michael Moser 
>> > wrote:
>> > > > I am reviewing the PR for NIFI-2774 ConsumeJMS and we need someone
>> to
>> > > > review the PR for NIFI-2429 PersistentProvenanceRepository.  Once
>> > those
>> > > are
>> > > > complete I think we can start the process to cut 0.7.1.
>> > > >
>> > > > -- Mike
>> > > >
>> > > > On Mon, Oct 3, 2016 at 9:49 AM, Tony Kurc  wrote:
>> > > >
>> > > >> So, sounds like we have enough support to go ahead. How are we
>> feeling
>> > > >> about what our timeline should be on this?
>> > > >>
>> > > >>
>> > > >> On Wed, Sep 28, 2016 at 11:14 PM, Joe Witt 
>> > wrote:
>> > > >>
>> > > >> > +1 to an 0.7.1 with the bugs that have been addressed already.
>> > > >> > Even bigger +1 to Tony volunteering as RM!
>> > > >> >
>> > > >> > Thanks
>> > > >> > Joe
>> > > >> >
>> > > >> > On Tue, Sep 27, 2016 at 10:24 PM, Brandon DeVries 
>> > > wrote:
>> > > >> > > I agree sooner rather than later for cutting 0.7.1. I think
>> Mike's
>> > > >> > question
>> > > >> > > to some degree was whether or not some of those tickets were
>> worth
>> > > >> fixing
>> > > >> > > in 0.x. For example, I'm not sure how much I care about:
>> > > >> > >
>> > > >> > > NIFI-2571 deprecate NiFiProperties.getInstance()
>> > > >> > > NIFI-2163 nifi.sh follow the Linux service spec
>> > > >> > >
>> > > >> > > On the other, there are some I would like to see, even if its
>> in
>> > > 0.7.2
>> > > >> or
>> > > >> > > 0.8.0, e.g.:
>> > > >> > >
>> > > >> > > NIFI-2433 "Primary Node Only" processors
>> > > >> > > NIFI-2562 PutHDFS data corruption
>> > > >> > >
>> > > >> > > But, there are a number of things that are currently committed
>> (or
>> > > have
>> > > >> > > patch available) that I'd like to see available as soon as
>> > > possible. So
>> > > >> > > rather than wait for more "nice to haves", I'd rather address
>> the
>> > > >> > immediate
>> > > >> > > needs... Immediately.
>> > > >> > >
>> > > >> > > Brandon
>> > > >> > >
>> > > >> > >
>> > > >> > > On Tue, Sep 27, 2016 at 10:15 PM Tony Kurc 
>> > > wrote:
>> > > >> > >
>> > > >> > >> I think I brought this up before, I sort of expected we may do
>> > more
>> > > >> 0.x
>> > > >> > >> releases. I certainly think the more the bugs we can fix, the
>> > > merrier,
>> > > >> > and
>> > > >> > >> it seems like your list is a good initial strawman for a bug
>> fix
>> > > >> > release of
>> > > >> > >> we collectively would like to put one together.
>> > > >> > >>
>> > > >> > >> While the tickets with work to do on them would be great to
>> have
>> > > >> fixed,
>> > > >> > I
>> > > >> > >> personally would rather see a release with some fixes and a
>> > couple
>> > > >> known
>> > > >> > >> issues than holding off for "perfection", especially as a lot
>> of
>> > > our
>> > > >> > effort
>> > > >> > >> is on 1.x. Are you asking if effort would be wasted if patches
>> > were
>> > > >> > >> developed for the 0.x issues?
>> > > >> > >>
>> > > >> > >> Fwiw, I certainly could do the RM work if there is
>> > interest/demand
>> > > >> > signal
>> > > >> > >> for in another 0.x.
>> > > >> > >>
>> > > >> > >> On Sep 27, 2016 5:28 PM, "Michael Moser" 
>> > > wrote:
>> > > >> > >>
>> > > >> > >> > All,
>> > > >> > >> >
>> > > >> > >> > I would like to start the discussion of making the next
>> > official
>> > > >> > release
>> > > >> > >> of
>> > > >> > >> > the 0.x branch.  I propose that this release be numbered
>> 0.7.1
>> > >

Re: InvokeScriptedProcessor

RE: InvokeScriptedProcessor

RE: InvokeScriptedProcessor

InvokeScriptedProcessor

Provenance repo corruption

Re: [DISCUSS] Closing in on a 0.x release

Re: Which are the underlying files for NiFi?

Re: Which are the underlying files for NiFi?

Re: [DISCUSS] Closing in on a 0.x release

9 matches

Site Navigation

Mail list logo

Footer information