Re: Best Practice for backing up NiFi Flows

[Tijo Thomas]

Hi Joe, 

Scenarios we have encountered is1. If a node goes down , we need to restart the 
cluster. A graceful degradation of service is not possible . I would be better 
if Nifi cluster can still run with existing nodes.
2. Zookeeper.properties : In our cluster we configured to scale  zookeeper 
nodes when ever the load in the zookeeper cluster increases.  We want to add 
this zookeeper ip to zookeeper.properties. 

3. it will be great if Nifi version upgrade can be done with out shutting down 
the cluster. I have noticed some thing like rolling upgrades  which is 
available in Hbase and some other  components also . 

Out of this 1st one is more crucial . We can implement auto scaling 
(specifically scale down) if Nifi node can run even if a node is removed.  

Thanks & RegardsTijo Thomas



On Thursday, 15 September 2016 8:16 AM, Joe Witt  wrote:
 

 Tijo

Most conf file related changes are intentionally application startup
based.  Which properties are you having to change that you'd like to
see honored without a restart?

Thanks
Joe

On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas  wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to  restart the cluster  when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically  when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing"  wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right?  I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them?  I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files.  But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi.  The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc.  Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi.  NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment.  As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems.  Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> Is there a best practice for keeping a backup of all the data flows we
>>> are developing in NiFi?
>>>
>>>
>>>
>>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>>> folder (also in our source control). Also, we keep a copy of all Config
>>> files in source control.
>>>
>>>
>>>
>>> ·        We are assuming that using flow.xml.gz and Config files, we
>>> will be able to restore the NiFi in case of any failure or if someone makes
>>> some mistake. Is this assumption c

Re: Best Practice for backing up NiFi Flows

[Joe Witt]

Tijo

Most conf file related changes are intentionally application startup
based.  Which properties are you having to change that you'd like to
see honored without a restart?

Thanks
Joe

On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas  wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to  restart the cluster  when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically  when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing"  wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right?  I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them?  I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files.  But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi.  The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc.  Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi.  NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment.  As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems.  Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> Is there a best practice for keeping a backup of all the data flows we
>>> are developing in NiFi?
>>>
>>>
>>>
>>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>>> folder (also in our source control). Also, we keep a copy of all Config
>>> files in source control.
>>>
>>>
>>>
>>> · We are assuming that using flow.xml.gz and Config files, we
>>> will be able to restore the NiFi in case of any failure or if someone makes
>>> some mistake. Is this assumption correct? Is there a better way to deal with
>>> this?
>>>
>>> · When we move to production (or some other environment), will it
>>> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along
>>> with making some environment related changes? Or, should we use templates on
>>> Dev, and import on Prod?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Manish
>>>
>>>
>>
>>
>>
>
>

Re: Best Practice for backing up NiFi Flows

[Tijo Thomas]

Hi

Our nifi cluster need to be up continuously as our user is over different
time zones.
Most of the times we often need to  restart the cluster  when ever there is
any changes in conf.

Is there any way to load the conf automatically  when ever there is any
change.

Tijo

On 15-Sep-2016 4:36 am, "James Wing"  wrote:

Mark,

Loading templates from the file system on startup is a new feature in
1.0.0, right?  I've been using the same API deployment Dan describes in 0.x.

Thanks,

James

On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:

> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them?  I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files.  But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi.  The difference is the live data in your flow, in the
> provenance repository, in state variables, etc.  Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi.  NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment.  As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems.  Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>

Re: Best Practice for backing up NiFi Flows

[James Wing]

Mark,

Loading templates from the file system on startup is a new feature in
1.0.0, right?  I've been using the same API deployment Dan describes in 0.x.

Thanks,

James

On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:

> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them?  I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files.  But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi.  The difference is the live data in your flow, in the
> provenance repository, in state variables, etc.  Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi.  NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment.  As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems.  Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>

RE: Best Practice for backing up NiFi Flows

[Manish Gupta 8]

Thanks James. Using Templates makes more sense. So what we are going to do is:

· Take backup of Conf Folder in source control (scheduled).

· Create Template for the top level Processor Group(s) periodically 
(will try to automate it using NiFi REST API for Processor Group).

Regards,
Manish
Cell: +1 646-744-7606
Email: mgupt...@sapient.com<mailto:mgupt...@sapient.com>

From: James Wing [mailto:jvw...@gmail.com]
Sent: Wednesday, September 14, 2016 3:02 PM
To: users@nifi.apache.org
Subject: Re: Best Practice for backing up NiFi Flows

Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
But I would carefully distinguish between using these backups to recreate an 
equivalent new NiFi, versus attempting to reset the state of your existing 
NiFi.  The difference is the live data in your flow, in the provenance 
repository, in state variables, etc.  Restoring a flow definition that no 
longer matches your content and provenance data may have unexpected results for 
you, and for systems connecting with NiFi.  NiFi does try hard to handle these 
changes smoothly, but it isn't a magic time machine.

Deploying flow.xml.gz can work, especially when deployed with conf files that 
reference IDs in the flow (like authorizations.xml), or the 
nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I 
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the 
flow logic
3.) Workflow that allows, encourages, forces the administrator to address 
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the 
command-and-control model, which is controversial and different from most other 
systems.  Templates fit within command-and-control, overwriting flow.xml.gz 
suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
mailto:mgupt...@sapient.com>> wrote:
Hello Everyone,

Is there a best practice for keeping a backup of all the data flows we are 
developing in NiFi?

Currently we take a copy of flow.xml.gz every hour and keep it in backup folder 
(also in our source control). Also, we keep a copy of all Config files in 
source control.


• We are assuming that using flow.xml.gz and Config files, we will be 
able to restore the NiFi in case of any failure or if someone makes some 
mistake. Is this assumption correct? Is there a better way to deal with this?

• When we move to production (or some other environment), will it be as 
simple as dropping flow.xml.gz in a new NiFi installation on NCM along with 
making some environment related changes? Or, should we use templates on Dev, 
and import on Prod?

Thanks,
Manish

Re: Best Practice for backing up NiFi Flows

[Mark Payne]

Dan,

Yes, you should be able to pre-deploy templates. When NiFi starts up, it looks 
in the conf/templates
directory (by default - this directory can be changed in the nifi.properties 
file). It looks for any file that
has a suffix of ".template" or ".xml" so you need to be sure that you are 
naming the files properly. Also, if you
are starting a cluster, you need to ensure that all nodes in the cluster have 
the templates or they will
not show up.

Thanks
-Mark


> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
> 
> James, 
> 
> Related to this question, is there a way to pre-deploy templates to the 
> template directory and have nifi recognize them?  I recently tried this and 
> nifi would not recognize the template until after I manually uploaded it 
> through the UI. 
> 
> I did this with nifi 0.7.0. 
> 
> I'd like to be able to use tools like salt or puppet to pre-position 
> templates. 
> 
> Thanks,
> 
> Dan 
> M: 443-992-2848
> GV: 410-861-0206
> 
> On Sep 14, 2016, at 3:02 PM, James Wing  > wrote:
> 
>> Manish, you are absolutely right to back up your flow.xml.gz and conf files. 
>>  But I would carefully distinguish between using these backups to recreate 
>> an equivalent new NiFi, versus attempting to reset the state of your 
>> existing NiFi.  The difference is the live data in your flow, in the 
>> provenance repository, in state variables, etc.  Restoring a flow definition 
>> that no longer matches your content and provenance data may have unexpected 
>> results for you, and for systems connecting with NiFi.  NiFi does try hard 
>> to handle these changes smoothly, but it isn't a magic time machine.
>> 
>> Deploying flow.xml.gz can work, especially when deployed with conf files 
>> that reference IDs in the flow (like authorizations.xml), or the 
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
>> you still have the data migration problem.
>> 
>> Templates are the current recommended best practice for deployment.  As I 
>> understand it, templates provide:
>> 
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations from 
>> the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address 
>> migration from the existing flow to incorporate the new template
>> 
>> Personally, I think it centers on acceptance or rejection of the 
>> command-and-control model, which is controversial and different from most 
>> other systems.  Templates fit within command-and-control, overwriting 
>> flow.xml.gz suggests a different model.
>> 
>> I know there are many other opinions on this.  
>> 
>> Thanks,
>> 
>> James
>> 
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 > > wrote:
>> Hello Everyone,
>> 
>>  
>> 
>> Is there a best practice for keeping a backup of all the data flows we are 
>> developing in NiFi?
>> 
>>  
>> 
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup 
>> folder (also in our source control). Also, we keep a copy of all Config 
>> files in source control.
>> 
>>  
>> 
>> · We are assuming that using flow.xml.gz and Config files, we will 
>> be able to restore the NiFi in case of any failure or if someone makes some 
>> mistake. Is this assumption correct? Is there a better way to deal with this?
>> 
>> · When we move to production (or some other environment), will it be 
>> as simple as dropping flow.xml.gz in a new NiFi installation on NCM along 
>> with making some environment related changes? Or, should we use templates on 
>> Dev, and import on Prod?
>> 
>>  
>> 
>> Thanks,
>> 
>> Manish
>> 
>>  
>> 
>> 

Re: Best Practice for backing up NiFi Flows

[Oleg Zhurakousky]

Just to add to what James has already stated, templates also have been polished 
quite a bit to be more deterministic and source control friendly so they are an 
ideal artifact to be versioned and kept in source control repo.

Cheers
Oleg

On Sep 14, 2016, at 3:02 PM, James Wing 
mailto:jvw...@gmail.com>> wrote:

Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
But I would carefully distinguish between using these backups to recreate an 
equivalent new NiFi, versus attempting to reset the state of your existing 
NiFi.  The difference is the live data in your flow, in the provenance 
repository, in state variables, etc.  Restoring a flow definition that no 
longer matches your content and provenance data may have unexpected results for 
you, and for systems connecting with NiFi.  NiFi does try hard to handle these 
changes smoothly, but it isn't a magic time machine.

Deploying flow.xml.gz can work, especially when deployed with conf files that 
reference IDs in the flow (like authorizations.xml), or the 
nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I 
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the 
flow logic
3.) Workflow that allows, encourages, forces the administrator to address 
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the 
command-and-control model, which is controversial and different from most other 
systems.  Templates fit within command-and-control, overwriting flow.xml.gz 
suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
mailto:mgupt...@sapient.com>> wrote:
Hello Everyone,

Is there a best practice for keeping a backup of all the data flows we are 
developing in NiFi?

Currently we take a copy of flow.xml.gz every hour and keep it in backup folder 
(also in our source control). Also, we keep a copy of all Config files in 
source control.


• We are assuming that using flow.xml.gz and Config files, we will be 
able to restore the NiFi in case of any failure or if someone makes some 
mistake. Is this assumption correct? Is there a better way to deal with this?

• When we move to production (or some other environment), will it be as 
simple as dropping flow.xml.gz in a new NiFi installation on NCM along with 
making some environment related changes? Or, should we use templates on Dev, 
and import on Prod?

Thanks,
Manish

Re: Best Practice for backing up NiFi Flows

[Dan Morris]

James, 

Related to this question, is there a way to pre-deploy templates to the 
template directory and have nifi recognize them?  I recently tried this and 
nifi would not recognize the template until after I manually uploaded it 
through the UI. 

I did this with nifi 0.7.0. 

I'd like to be able to use tools like salt or puppet to pre-position templates. 

Thanks,

Dan 
M: 443-992-2848
GV: 410-861-0206

> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
> 
> Manish, you are absolutely right to back up your flow.xml.gz and conf files.  
> But I would carefully distinguish between using these backups to recreate an 
> equivalent new NiFi, versus attempting to reset the state of your existing 
> NiFi.  The difference is the live data in your flow, in the provenance 
> repository, in state variables, etc.  Restoring a flow definition that no 
> longer matches your content and provenance data may have unexpected results 
> for you, and for systems connecting with NiFi.  NiFi does try hard to handle 
> these changes smoothly, but it isn't a magic time machine.
> 
> Deploying flow.xml.gz can work, especially when deployed with conf files that 
> reference IDs in the flow (like authorizations.xml), or the 
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow, 
> you still have the data migration problem.
> 
> Templates are the current recommended best practice for deployment.  As I 
> understand it, templates provide:
> 
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations from 
> the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address 
> migration from the existing flow to incorporate the new template
> 
> Personally, I think it centers on acceptance or rejection of the 
> command-and-control model, which is controversial and different from most 
> other systems.  Templates fit within command-and-control, overwriting 
> flow.xml.gz suggests a different model.
> 
> I know there are many other opinions on this.  
> 
> Thanks,
> 
> James
> 
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8  wrote:
>> Hello Everyone,
>> 
>>  
>> 
>> Is there a best practice for keeping a backup of all the data flows we are 
>> developing in NiFi?
>> 
>>  
>> 
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup 
>> folder (also in our source control). Also, we keep a copy of all Config 
>> files in source control.
>> 
>>  
>> 
>> · We are assuming that using flow.xml.gz and Config files, we will 
>> be able to restore the NiFi in case of any failure or if someone makes some 
>> mistake. Is this assumption correct? Is there a better way to deal with this?
>> 
>> · When we move to production (or some other environment), will it be 
>> as simple as dropping flow.xml.gz in a new NiFi installation on NCM along 
>> with making some environment related changes? Or, should we use templates on 
>> Dev, and  import on Prod?
>> 
>>  
>> 
>> Thanks,
>> 
>> Manish
>> 
>>  
>> 
> 

Re: Best Practice for backing up NiFi Flows

[James Wing]

Manish, you are absolutely right to back up your flow.xml.gz and conf
files.  But I would carefully distinguish between using these backups to
recreate an equivalent new NiFi, versus attempting to reset the state of
your existing NiFi.  The difference is the live data in your flow, in the
provenance repository, in state variables, etc.  Restoring a flow
definition that no longer matches your content and provenance data may have
unexpected results for you, and for systems connecting with NiFi.  NiFi
does try hard to handle these changes smoothly, but it isn't a magic time
machine.

Deploying flow.xml.gz can work, especially when deployed with conf files
that reference IDs in the flow (like authorizations.xml), or the
nifi.sensitive.props.key setting, etc.  But if you overwrite a running
flow, you still have the data migration problem.

Templates are the current recommended best practice for deployment.  As I
understand it, templates provide:

1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from
the flow logic
3.) Workflow that allows, encourages, forces the administrator to address
migration from the existing flow to incorporate the new template

Personally, I think it centers on acceptance or rejection of the
command-and-control model, which is controversial and different from most
other systems.  Templates fit within command-and-control, overwriting
flow.xml.gz suggests a different model.

I know there are many other opinions on this.

Thanks,

James

On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
wrote:

> Hello Everyone,
>
>
>
> Is there a best practice for keeping a backup of all the data flows we are
> developing in NiFi?
>
>
>
> Currently we take a copy of flow.xml.gz every hour and keep it in backup
> folder (also in our source control). Also, we keep a copy of all Config
> files in source control.
>
>
>
> · We are assuming that using flow.xml.gz and Config files, we
> will be able to restore the NiFi in case of any failure or if someone makes
> some mistake. Is this assumption correct? Is there a better way to deal
> with this?
>
> · When we move to production (or some other environment), will it
> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
> along with making some environment related changes? Or, should we use
> templates on Dev, and import on Prod?
>
>
>
> Thanks,
>
> Manish
>
>
>

Best Practice for backing up NiFi Flows

[Manish Gupta 8]

Hello Everyone,

Is there a best practice for keeping a backup of all the data flows we are 
developing in NiFi?

Currently we take a copy of flow.xml.gz every hour and keep it in backup folder 
(also in our source control). Also, we keep a copy of all Config files in 
source control.


* We are assuming that using flow.xml.gz and Config files, we will be 
able to restore the NiFi in case of any failure or if someone makes some 
mistake. Is this assumption correct? Is there a better way to deal with this?

* When we move to production (or some other environment), will it be as 
simple as dropping flow.xml.gz in a new NiFi installation on NCM along with 
making some environment related changes? Or, should we use templates on Dev, 
and import on Prod?

Thanks,
Manish