Re: QueryDatabaseTable Processor

[Guillaume Pool]

Thanks Matt, will do

Get Outlook for iOS




On Wed, Sep 14, 2016 at 7:34 PM +0200, "Matt Burgess" 
> wrote:

Guillaume, that is certainly something that could be added. Would you
mind filing a Jira for this at
https://issues.apache.org/jira/browse/NIFI ?

Thanks,
Matt

On Wed, Sep 14, 2016 at 12:52 AM, Guillaume Pool  wrote:
> Hi,
>
> On the Processor there are pre-processing options for handling Oracle dates.
> Is there a possibility of adding a with (nolock) option for mssql option?
>
> I have suppliers that use MSSQL tables for vehicle telematics data and
> reading from that table is problematic.
>
> Thanks
>
> Get Outlook for iOS
>

Re: Best Practice for backing up NiFi Flows

[Joe Witt]

Tijo

Most conf file related changes are intentionally application startup
based.  Which properties are you having to change that you'd like to
see honored without a restart?

Thanks
Joe

On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas  wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to  restart the cluster  when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically  when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing"  wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right?  I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them?  I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files.  But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi.  The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc.  Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi.  NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment.  As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems.  Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> Is there a best practice for keeping a backup of all the data flows we
>>> are developing in NiFi?
>>>
>>>
>>>
>>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>>> folder (also in our source control). Also, we keep a copy of all Config
>>> files in source control.
>>>
>>>
>>>
>>> · We are assuming that using flow.xml.gz and Config files, we
>>> will be able to restore the NiFi in case of any failure or if someone makes
>>> some mistake. Is this assumption correct? Is there a better way to deal with
>>> this?
>>>
>>> · When we move to production (or some other environment), will it
>>> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along
>>> with making some environment related changes? Or, should we use templates on
>>> Dev, and import on Prod?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Manish
>>>
>>>
>>
>>
>>
>
>

Re: Best Practice for backing up NiFi Flows

[Tijo Thomas]

Hi

Our nifi cluster need to be up continuously as our user is over different
time zones.
Most of the times we often need to  restart the cluster  when ever there is
any changes in conf.

Is there any way to load the conf automatically  when ever there is any
change.

Tijo

On 15-Sep-2016 4:36 am, "James Wing"  wrote:

Mark,

Loading templates from the file system on startup is a new feature in
1.0.0, right?  I've been using the same API deployment Dan describes in 0.x.

Thanks,

James

On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:

> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them?  I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files.  But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi.  The difference is the live data in your flow, in the
> provenance repository, in state variables, etc.  Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi.  NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment.  As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems.  Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>

Re: Best Practice for backing up NiFi Flows

[James Wing]

Mark,

Loading templates from the file system on startup is a new feature in
1.0.0, right?  I've been using the same API deployment Dan describes in 0.x.

Thanks,

James

On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:

> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them?  I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files.  But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi.  The difference is the live data in your flow, in the
> provenance repository, in state variables, etc.  Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi.  NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment.  As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems.  Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>

RE: Best Practice for backing up NiFi Flows

[Manish Gupta 8]

Thanks James. Using Templates makes more sense. So what we are going to do is:

· Take backup of Conf Folder in source control (scheduled).

· Create Template for the top level Processor Group(s) periodically 
(will try to automate it using NiFi REST API for Processor Group).

Regards,
Manish
Cell: +1 646-744-7606
Email: mgupt...@sapient.com

From: James Wing [mailto:jvw...@gmail.com]
Sent: Wednesday, September 14, 2016 3:02 PM
To: users@nifi.apache.org
Subject: Re: Best Practice for backing up NiFi Flows

Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
But I would carefully distinguish between using these backups to recreate an 
equivalent new NiFi, versus attempting to reset the state of your existing 
NiFi.  The difference is the live data in your flow, in the provenance 
repository, in state variables, etc.  Restoring a flow definition that no 
longer matches your content and provenance data may have unexpected results for 
you, and for systems connecting with NiFi.  NiFi does try hard to handle these 
changes smoothly, but it isn't a magic time machine.

Deploying flow.xml.gz can work, especially when deployed with conf files that 
reference IDs in the flow (like authorizations.xml), or the 
nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I 
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the 
flow logic
3.) Workflow that allows, encourages, forces the administrator to address 
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the 
command-and-control model, which is controversial and different from most other 
systems.  Templates fit within command-and-control, overwriting flow.xml.gz 
suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
Hello Everyone,

Is there a best practice for keeping a backup of all the data flows we are 
developing in NiFi?

Currently we take a copy of flow.xml.gz every hour and keep it in backup folder 
(also in our source control). Also, we keep a copy of all Config files in 
source control.


• We are assuming that using flow.xml.gz and Config files, we will be 
able to restore the NiFi in case of any failure or if someone makes some 
mistake. Is this assumption correct? Is there a better way to deal with this?

• When we move to production (or some other environment), will it be as 
simple as dropping flow.xml.gz in a new NiFi installation on NCM along with 
making some environment related changes? Or, should we use templates on Dev, 
and import on Prod?

Thanks,
Manish

RE: ExecuteSQL processor running in duplicate?

[Crystal Deschamps Fogdall (cdeschampsfo)]

Thanks for all of your help Lee!

From: Lee Laim [mailto:lee.l...@gmail.com]
Sent: Tuesday, September 13, 2016 13:08 PM
To: users@nifi.apache.org
Subject: Re: ExecuteSQL processor running in duplicate?

Hi Crystal,

While this may not be the root cause of your specific case, I suspect the 
sequence of:
->stopping the processor,  ->changing concurrent task setting, ->then starting 
the timer driven processor

generated the rogue flowfile in the queue.  Timer driven processors are 
triggered on start, regardless of when the previous trigger occurred.  I've 
observed similar events in my dataflows and have switched over to a Cron Driven 
scheduling to start most processes.

For troubleshooting and one-time back-fill operations on a cron-initialed 
process, I also include a timer-driven processor that can  initiate the flow.   
The timer-driven version is always-off, until it is required: I  start it, then 
immediately stop it.(The run schedule is set for 100 seconds to prevent 
multiple runs between the start-click and stop-click).  This provides 
predictable cron-based scheduling with convenient ad-hoc backfills that do not 
require re-configuring the cron.


For your second question on monitoring, I've had success with the Monitor 
Activity processor paired with a Put Email processor to indicate something is 
awry. Depending on the amount of detail you require and your definition of 
failure, you can branch data into a process to check integrity and notify you 
when something about the data is not correct.


Here are some references that show different approaches to error-capture that 
have been useful to me:

  *   
http://stackoverflow.com/questions/37430915/how-to-capture-bulletin-messages-in-apache-nifi
  *   
http://stackoverflow.com/questions/36869043/get-failure-reason-in-apache-nifi
  *   
https://kisstechdocs.wordpress.com/2015/01/15/creating-a-limited-failure-loop-in-nifi/


Thanks,
-Lee


On Mon, Sep 12, 2016 at 2:39 PM, Crystal Deschamps Fogdall (cdeschampsfo) 
> wrote:
I'm using NiFi for a very simple, yet non-standard way - as a POC scheduler to 
call an existing stored procedure using an ExecuteSQL processor. The processor 
is timer driven (every 4 hours), uses a basic JDBC connection, and executes a 
stored procedure call that contains one hard-coded variable. I have failure and 
success relationships defined only to collect flow files to troubleshoot this 
POC.

The processor had been running successfully for a day and a half. Based on the 
NiFi status history, I could see a single FlowFile out every 4 hours. My sp 
takes the last known copy of the target table, renames it, and builds a new 
table that has a load timestamp. The timestamps in both tables reflected what I 
saw in the history.

After monkeying with the concurrent tasks (originally at 1, I moved it to 4 
just to see what would happen), I suddenly saw two FlowFiles appear every 4 
hours in the history and it appeared, based on the timestamps in my tables, 
that the processor would run once, kick off again a few minutes later (the new 
table's timestamp would be a few minutes later than the last known copy 
timestamp) and they lay dormant again until 4 hours had passed.

Because I don't have data inputs / outputs, troubleshooting the issue is really 
difficult and I'm finding the out-of-the-box monitoring tools (e.g. NiFi 
summary tabs, status history, etc) lacking. I'm wondering if anyone has seen 
this kind of behavior and, if so, what was its cause? (Certainly, it can't be 
the concurrent tasks could it?!) Secondly, I'm wondering if anyone has created 
their own or have external resources they can point to for troubleshooting and 
monitoring of process groups. My background is in traditional ETL so I'm 
looking for scheduler-type application reporting like ActiveBatch.

Thanks, in advance, for your help,
Crystal

Re: Best Practice for backing up NiFi Flows

[Mark Payne]

Dan,

Yes, you should be able to pre-deploy templates. When NiFi starts up, it looks 
in the conf/templates
directory (by default - this directory can be changed in the nifi.properties 
file). It looks for any file that
has a suffix of ".template" or ".xml" so you need to be sure that you are 
naming the files properly. Also, if you
are starting a cluster, you need to ensure that all nodes in the cluster have 
the templates or they will
not show up.

Thanks
-Mark


> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
> 
> James, 
> 
> Related to this question, is there a way to pre-deploy templates to the 
> template directory and have nifi recognize them?  I recently tried this and 
> nifi would not recognize the template until after I manually uploaded it 
> through the UI. 
> 
> I did this with nifi 0.7.0. 
> 
> I'd like to be able to use tools like salt or puppet to pre-position 
> templates. 
> 
> Thanks,
> 
> Dan 
> M: 443-992-2848
> GV: 410-861-0206
> 
> On Sep 14, 2016, at 3:02 PM, James Wing  > wrote:
> 
>> Manish, you are absolutely right to back up your flow.xml.gz and conf files. 
>>  But I would carefully distinguish between using these backups to recreate 
>> an equivalent new NiFi, versus attempting to reset the state of your 
>> existing NiFi.  The difference is the live data in your flow, in the 
>> provenance repository, in state variables, etc.  Restoring a flow definition 
>> that no longer matches your content and provenance data may have unexpected 
>> results for you, and for systems connecting with NiFi.  NiFi does try hard 
>> to handle these changes smoothly, but it isn't a magic time machine.
>> 
>> Deploying flow.xml.gz can work, especially when deployed with conf files 
>> that reference IDs in the flow (like authorizations.xml), or the 
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
>> you still have the data migration problem.
>> 
>> Templates are the current recommended best practice for deployment.  As I 
>> understand it, templates provide:
>> 
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations from 
>> the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address 
>> migration from the existing flow to incorporate the new template
>> 
>> Personally, I think it centers on acceptance or rejection of the 
>> command-and-control model, which is controversial and different from most 
>> other systems.  Templates fit within command-and-control, overwriting 
>> flow.xml.gz suggests a different model.
>> 
>> I know there are many other opinions on this.  
>> 
>> Thanks,
>> 
>> James
>> 
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 > > wrote:
>> Hello Everyone,
>> 
>>  
>> 
>> Is there a best practice for keeping a backup of all the data flows we are 
>> developing in NiFi?
>> 
>>  
>> 
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup 
>> folder (also in our source control). Also, we keep a copy of all Config 
>> files in source control.
>> 
>>  
>> 
>> · We are assuming that using flow.xml.gz and Config files, we will 
>> be able to restore the NiFi in case of any failure or if someone makes some 
>> mistake. Is this assumption correct? Is there a better way to deal with this?
>> 
>> · When we move to production (or some other environment), will it be 
>> as simple as dropping flow.xml.gz in a new NiFi installation on NCM along 
>> with making some environment related changes? Or, should we use templates on 
>> Dev, and import on Prod?
>> 
>>  
>> 
>> Thanks,
>> 
>> Manish
>> 
>>  
>> 
>> 

Re: Best Practice for backing up NiFi Flows

[Oleg Zhurakousky]

Just to add to what James has already stated, templates also have been polished 
quite a bit to be more deterministic and source control friendly so they are an 
ideal artifact to be versioned and kept in source control repo.

Cheers
Oleg

On Sep 14, 2016, at 3:02 PM, James Wing 
> wrote:

Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
But I would carefully distinguish between using these backups to recreate an 
equivalent new NiFi, versus attempting to reset the state of your existing 
NiFi.  The difference is the live data in your flow, in the provenance 
repository, in state variables, etc.  Restoring a flow definition that no 
longer matches your content and provenance data may have unexpected results for 
you, and for systems connecting with NiFi.  NiFi does try hard to handle these 
changes smoothly, but it isn't a magic time machine.

Deploying flow.xml.gz can work, especially when deployed with conf files that 
reference IDs in the flow (like authorizations.xml), or the 
nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I 
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the 
flow logic
3.) Workflow that allows, encourages, forces the administrator to address 
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the 
command-and-control model, which is controversial and different from most other 
systems.  Templates fit within command-and-control, overwriting flow.xml.gz 
suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
Hello Everyone,

Is there a best practice for keeping a backup of all the data flows we are 
developing in NiFi?

Currently we take a copy of flow.xml.gz every hour and keep it in backup folder 
(also in our source control). Also, we keep a copy of all Config files in 
source control.


• We are assuming that using flow.xml.gz and Config files, we will be 
able to restore the NiFi in case of any failure or if someone makes some 
mistake. Is this assumption correct? Is there a better way to deal with this?

• When we move to production (or some other environment), will it be as 
simple as dropping flow.xml.gz in a new NiFi installation on NCM along with 
making some environment related changes? Or, should we use templates on Dev, 
and import on Prod?

Thanks,
Manish

Re: Best Practice for backing up NiFi Flows

[Dan Morris]

James, 

Related to this question, is there a way to pre-deploy templates to the 
template directory and have nifi recognize them?  I recently tried this and 
nifi would not recognize the template until after I manually uploaded it 
through the UI. 

I did this with nifi 0.7.0. 

I'd like to be able to use tools like salt or puppet to pre-position templates. 

Thanks,

Dan 
M: 443-992-2848
GV: 410-861-0206

> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
> 
> Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
> But I would carefully distinguish between using these backups to recreate an 
> equivalent new NiFi, versus attempting to reset the state of your existing 
> NiFi.  The difference is the live data in your flow, in the provenance 
> repository, in state variables, etc.  Restoring a flow definition that no 
> longer matches your content and provenance data may have unexpected results 
> for you, and for systems connecting with NiFi.  NiFi does try hard to handle 
> these changes smoothly, but it isn't a magic time machine.
> 
> Deploying flow.xml.gz can work, especially when deployed with conf files that 
> reference IDs in the flow (like authorizations.xml), or the 
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
> you still have the data migration problem.
> 
> Templates are the current recommended best practice for deployment.  As I 
> understand it, templates provide:
> 
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations from 
> the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address 
> migration from the existing flow to incorporate the new template
> 
> Personally, I think it centers on acceptance or rejection of the 
> command-and-control model, which is controversial and different from most 
> other systems.  Templates fit within command-and-control, overwriting 
> flow.xml.gz suggests a different model.
> 
> I know there are many other opinions on this.  
> 
> Thanks,
> 
> James
> 
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8  wrote:
>> Hello Everyone,
>> 
>>  
>> 
>> Is there a best practice for keeping a backup of all the data flows we are 
>> developing in NiFi?
>> 
>>  
>> 
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup 
>> folder (also in our source control). Also, we keep a copy of all Config 
>> files in source control.
>> 
>>  
>> 
>> · We are assuming that using flow.xml.gz and Config files, we will 
>> be able to restore the NiFi in case of any failure or if someone makes some 
>> mistake. Is this assumption correct? Is there a better way to deal with this?
>> 
>> · When we move to production (or some other environment), will it be 
>> as simple as dropping flow.xml.gz in a new NiFi installation on NCM along 
>> with making some environment related changes? Or, should we use templates on 
>> Dev, and  import on Prod?
>> 
>>  
>> 
>> Thanks,
>> 
>> Manish
>> 
>>  
>> 
> 

Re: Best Practice for backing up NiFi Flows

[James Wing]

Manish, you are absolutely right to back up your flow.xml.gz and conf
files.  But I would carefully distinguish between using these backups to
recreate an equivalent new NiFi, versus attempting to reset the state of
your existing NiFi.  The difference is the live data in your flow, in the
provenance repository, in state variables, etc.  Restoring a flow
definition that no longer matches your content and provenance data may have
unexpected results for you, and for systems connecting with NiFi.  NiFi
does try hard to handle these changes smoothly, but it isn't a magic time
machine.

Deploying flow.xml.gz can work, especially when deployed with conf files
that reference IDs in the flow (like authorizations.xml), or the
nifi.sensitive.props.key setting, etc.  But if you overwrite a running
flow, you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from
the flow logic
3.) Workflow that allows, encourages, forces the administrator to address
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the
command-and-control model, which is controversial and different from most
other systems.  Templates fit within command-and-control, overwriting
flow.xml.gz suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
wrote:

> Hello Everyone,
>
>
>
> Is there a best practice for keeping a backup of all the data flows we are
> developing in NiFi?
>
>
>
> Currently we take a copy of flow.xml.gz every hour and keep it in backup
> folder (also in our source control). Also, we keep a copy of all Config
> files in source control.
>
>
>
> · We are assuming that using flow.xml.gz and Config files, we
> will be able to restore the NiFi in case of any failure or if someone makes
> some mistake. Is this assumption correct? Is there a better way to deal
> with this?
>
> · When we move to production (or some other environment), will it
> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
> along with making some environment related changes? Or, should we use
> templates on Dev, and import on Prod?
>
>
>
> Thanks,
>
> Manish
>
>
>

RE: OnTrigger - FlowFile is Null

[Manish Gupta 8]

Apologies for the typo. I meant *Mark.


Regards,
Manish
Cell: +1 646-744-7606
Email: mgupt...@sapient.com

From: Manish Gupta 8
Sent: Wednesday, September 14, 2016 2:55 PM
To: users@nifi.apache.org
Subject: RE: OnTrigger - FlowFile is Null

Thanks Matt. This is very helpful. In my case also, it's the 3rd scenario. I 
remember, this error started showing up only when I had increased the 
concurrent tasks to greater than 1 sometime earlier today.

Regards,
Manish
Cell: +1 646-744-7606
Email: mgupt...@sapient.com

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 14, 2016 2:50 PM
To: users@nifi.apache.org
Subject: Re: OnTrigger - FlowFile is Null

Manish,

This happens for a few reasons:

* Processor has no incoming connections (is a source processor)
* Processor has @ScheduleWhenEmpty annotation
* Processor has more than 1 concurrent task

The main reason is the third one above. If you have multiple concurrent tasks, 
Thread 1 can determine that there is 1
FlowFile queued. Thread 2 then determines that there is 1 FlowFile queued. Both 
threads call
onTrigger(). Thread 1 gets the FlowFIle, and Thread 2 gets null.

Does this help?

Thanks
-Mark


On Sep 14, 2016, at 2:45 PM, Manish Gupta 8 
> wrote:

Hi,

In one of our custom processor, I forgot to mention the following check, and 
the error started showing up randomly.

if (flowFile == null) {
return;
}


I am just curious to know, in what situation would onTrigger will get called if 
there is no FlowFile?

Regards,
Manish

RE: OnTrigger - FlowFile is Null

[Manish Gupta 8]

Thanks Matt. This is very helpful. In my case also, it's the 3rd scenario. I 
remember, this error started showing up only when I had increased the 
concurrent tasks to greater than 1 sometime earlier today.

Regards,
Manish
Cell: +1 646-744-7606
Email: mgupt...@sapient.com

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 14, 2016 2:50 PM
To: users@nifi.apache.org
Subject: Re: OnTrigger - FlowFile is Null

Manish,

This happens for a few reasons:

* Processor has no incoming connections (is a source processor)
* Processor has @ScheduleWhenEmpty annotation
* Processor has more than 1 concurrent task

The main reason is the third one above. If you have multiple concurrent tasks, 
Thread 1 can determine that there is 1
FlowFile queued. Thread 2 then determines that there is 1 FlowFile queued. Both 
threads call
onTrigger(). Thread 1 gets the FlowFIle, and Thread 2 gets null.

Does this help?

Thanks
-Mark


On Sep 14, 2016, at 2:45 PM, Manish Gupta 8 
> wrote:

Hi,

In one of our custom processor, I forgot to mention the following check, and 
the error started showing up randomly.

if (flowFile == null) {
return;
}


I am just curious to know, in what situation would onTrigger will get called if 
there is no FlowFile?

Regards,
Manish

Re: OnTrigger - FlowFile is Null

[Mark Payne]

Manish,

This happens for a few reasons:

* Processor has no incoming connections (is a source processor)
* Processor has @ScheduleWhenEmpty annotation
* Processor has more than 1 concurrent task

The main reason is the third one above. If you have multiple concurrent tasks, 
Thread 1 can determine that there is 1
FlowFile queued. Thread 2 then determines that there is 1 FlowFile queued. Both 
threads call
onTrigger(). Thread 1 gets the FlowFIle, and Thread 2 gets null.

Does this help?

Thanks
-Mark


> On Sep 14, 2016, at 2:45 PM, Manish Gupta 8  wrote:
> 
> Hi,
>  
> In one of our custom processor, I forgot to mention the following check, and 
> the error started showing up randomly.
>  
> if (flowFile == null) {
> return;
> }
>  
> I am just curious to know, in what situation would onTrigger will get called 
> if there is no FlowFile?
>  
> Regards,
> Manish

OnTrigger - FlowFile is Null

[Manish Gupta 8]

Hi,

In one of our custom processor, I forgot to mention the following check, and 
the error started showing up randomly.

if (flowFile == null) {
return;
}


I am just curious to know, in what situation would onTrigger will get called if 
there is no FlowFile?

Regards,
Manish

Re: PermissionBasedStatusMergerSpec is failing

[Jeff]

Ok, sounds good!  Please let us know!

On Wed, Sep 14, 2016 at 12:04 AM Tijo Thomas  wrote:

> Sorry to reply  late. I was on vacation for last 4 days.
>
> I have not modified any files.
>
> I think there is some problem with my repo.  I will make a new repo and
> try again.  Still the problem  exist I will post it again in the group.
>
> Thank you very much for your support.
>
> Tijo
>
> On 10-Sep-2016 6:44 pm, "Jeff"  wrote:
>
>> Tijo,
>>
>> Have you modified ProcessorStatusSnapshotDTO.java or
>> PermissionBasedStatusMergerSpec.groovy?
>>
>> On Sat, Sep 10, 2016 at 7:48 AM Tijo Thomas 
>> wrote:
>>
>>> Hi Jeff
>>>
>>> I recently  rebase  from master.
>>> Then I cloned again and ran mvn  package
>>>
>>> Tijo
>>>
>>> On 09-Sep-2016 9:12 pm, "Jeff"  wrote:
>>>
 Tijo,

 I just ran this test on master and it's passing for me.  Can you
 provide some details about the branch you're on when running the tests?  I
 see that tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000,
 and that's why the JSON isn't matching.

 On Thu, Sep 8, 2016 at 4:58 PM Tijo Thomas 
 wrote:

> Hi
> Nifi test case is failing (PermissionBasedStatusMergerSpec) .
> This is written in Grovy .. not comfortable with Groovy .
>
> Running org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
> Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.922
> sec <<< FAILURE! - in
> org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
> Merge
> ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec)
> Time elapsed: 0.144 sec  <<< FAILURE!
> org.spockframework.runtime.SpockComparisonFailure: Condition not
> satisfied:
>
> returnedJson == expectedJson
> ||  |
> ||
> {"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}
> |false
> |1 difference (99% similarity)
> |
> {"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
> |
> {"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
> {"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}
>
> at
> org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec.Merge
> ProcessorStatusSnapshotDTO(PermissionBasedStatusMergerSpec.groovy:257)
>
> Merge
> ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec)
> Time elapsed: 0.01 sec  <<< FAILURE!
> org.spockframework.runtime.SpockComparisonFailure: Condition not
> satisfied:
>
> Tijo
>
>
>
>