Tls toolkit the

[Tijo Thomas]

Hi 
Can any one guide on how to use tls toolkit.  In the code I found that there 
are 3  services.  Standalone,  server and  client.  I created a cluster with 
Standalone service by following  a blog.   but not sure where client and server 
 will be used.  
Any doc on this is highly appreciated 


Sent from Yahoo Mail on Android

enforce run only in promary node $ multiple primary node

[Tijo Thomas]

 Hi ,
1. While writing a processor is it possible to enforce to run only in primary 
node. I saw a Jira for this but appears to unresolved. 
[NIFI-543] Provide extensions a way to indicate that they can run only on 
primary node, if clustered - ASF JIRA  

  
|  
|   
|   
|   ||

   |

  |
|  
|   |  
[NIFI-543] Provide extensions a way to indicate that they can run only on p...
   |   |

  |

  |

 
2. Currently my Primary node is heavily loaded  as i have many  processor which 
will run only in Primary node.  Is it possible to define multiple primary nodes 
. or is it possible to configure processors not to run in primary node. 
Tijo 

Re: Configure Multiple NCM

[Tijo Thomas]

Thank you Andrew, 

Is there any Blog or Jira to understand how the new approch works. 

I read the following blog Apache NiFi 1.0.0 – Cluster setup

  
|  
|   
|   
|   ||

   |

  |
|  
|   |  
Apache NiFi 1.0.0 – Cluster setup
 As you may know a version 1.0.0-BETA of Apache NiFi has been released few days 
ago. The upcoming 1.0.0 release w...  |   |

  |

  |

 
Is there any Jira / some other blog which explain the design or how it works. 

Tijo 
 

On Sunday, 18 September 2016 9:39 PM, Andrew Grande  
wrote:
 

 Hi Tijo,Take a look at clustering docs for NiFi 1.0. Zero-master clustering 
changed a few things, any node can be elected to be a primary or coordinator 
now. The 0.x concept of NCM is gone.From the UI access standpoint, one can hit 
any node in a cluster to get the same experience.Andrew
On Sun, Sep 18, 2016, 11:56 AM Tijo Thomas  wrote:

Hi ,
Is there any way to configure multiple NCMs and can do load balance amount 
NCMs. 
Tijo 


   

Configure Multiple NCM

[Tijo Thomas]

Hi ,
Is there any way to configure multiple NCMs and can do load balance amount 
NCMs. 
Tijo 

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Hi , 

I tried replacing with  Locale.getDefault(). Still getting the same error. 

Sorry 

Tijo 
 

On Saturday, 17 September 2016 8:43 PM, Tijo Thomas  
wrote:
 

 Great..   I will  try this tomorrow and let u know.  
Tijo

Sent from Yahoo Mail on Android 
 
 On Fri, 16 Sep, 2016 at 11:31 pm, Jeff wrote:  I looked 
into the merger code and found a possible issue.  I haven't been able to 
reproduce the test failure you're getting by setting my locale to en_IN, but I 
have a sneaking suspicion that FormatUtils might be the culprit here when it's 
formatting the value for TasksDuration in the target DTO.  Can you please 
change line 86 in FormatUtils to use Locale.getDefault() instead of Locale.US 
and rerun the tests?
On Fri, Sep 16, 2016 at 1:12 PM Tijo Thomas  wrote:

Hi Jeff,Yes,  I took a fresh clone from git hub .  I cleaned my maven repo as 
well before building .  Attached StatusMerger.java and 
ProcessorStatusSnapshotDTO.java
All,It will be great if others also check if it is happening when u build. I am 
worried whether I am doing some thing really stupid. Thanks & RegardsTijo Thomas
On 16-Sep-2016 7:47 pm, "Jeff"  wrote:

Thank you for the information.  Did you try running the tests on a fresh clone 
of the github repo?
Could you please link me to or include the contents of StatusMerger.java and 
ProcessorStatusSnapshotDTO.java?
On Fri, Sep 16, 2016 at 4:17 AM Tijo Thomas  wrote:


Output when I ran through IDE, 


Condition not satisfied:

returnedJson == expectedJson
|    |  |
|    |  
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}
|    false
|    1 difference (99% similarity)
|    
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
|    
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}

Expected 
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}

Actual   
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden&qu

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Great..   I will  try this tomorrow and let u know.  
Tijo

Sent from Yahoo Mail on Android 
 
  On Fri, 16 Sep, 2016 at 11:31 pm, Jeff wrote:   I looked 
into the merger code and found a possible issue.  I haven't been able to 
reproduce the test failure you're getting by setting my locale to en_IN, but I 
have a sneaking suspicion that FormatUtils might be the culprit here when it's 
formatting the value for TasksDuration in the target DTO.  Can you please 
change line 86 in FormatUtils to use Locale.getDefault() instead of Locale.US 
and rerun the tests?
On Fri, Sep 16, 2016 at 1:12 PM Tijo Thomas  wrote:


Hi Jeff,

Yes,  I took a fresh clone from git hub .  I cleaned my maven repo as well 
before building . 

 Attached StatusMerger.java and ProcessorStatusSnapshotDTO.java


All,

It will be great if others also check if it is happening when u build. I am 
worried whether I am doing some thing really stupid. 

Thanks & Regards

Tijo Thomas

On 16-Sep-2016 7:47 pm, "Jeff"  wrote:

Thank you for the information.  Did you try running the tests on a fresh clone 
of the github repo?
Could you please link me to or include the contents of StatusMerger.java and 
ProcessorStatusSnapshotDTO.java?
On Fri, Sep 16, 2016 at 4:17 AM Tijo Thomas  wrote:


Output when I ran through IDE, 


Condition not satisfied:

returnedJson == expectedJson
|    |  |
|    |  
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}
|    false
|    1 difference (99% similarity)
|    
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
|    
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}

Expected 
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0 
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0 
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}

Actual   
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
 bytes","written":"0 bytes","flowF

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Hi Jeff,

Yes,  I took a fresh clone from git hub .  I cleaned my maven repo as well
before building .

 Attached StatusMerger.java and ProcessorStatusSnapshotDTO.java

All,

It will be great if others also check if it is happening when u build. I am
worried whether I am doing some thing really stupid.

Thanks & Regards

Tijo Thomas

On 16-Sep-2016 7:47 pm, "Jeff"  wrote:

> Thank you for the information.  Did you try running the tests on a fresh
> clone of the github repo?
>
> Could you please link me to or include the contents of StatusMerger.java
> and ProcessorStatusSnapshotDTO.java?
>
> On Fri, Sep 16, 2016 at 4:17 AM Tijo Thomas  wrote:
>
>>
>> Output when I ran through IDE,
>>
>>
>> Condition not satisfied:
>>
>> returnedJson == expectedJson
>> ||  |
>> ||  {"id":"hidden","groupId":"hidd
>> en","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","
>> tasksDuration":"00:00:00.000","activeThreadCount":0}
>> |false
>> |1 difference (99% similarity)
>> |{"id":"hidden","groupId":"hidden","name":"hidden","type":"
>> hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","
>> tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
>> |{"id":"hidden","groupId":"hidden","name":"hidden","type":"
>> hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","
>> tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
>> {"id":"hidden","groupId":"hidden","name":"hidden","type":"
>> hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","
>> tasksDuration":"00:30:00.000","activeThreadCount":0}
>>
>> Expected :{"id":"hidden","groupId":"hidden","name":"hidden","type":"
>> hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","
>> tasksDuration":"00:00:00.000","activeThreadCount":0}
>>
>> Actual   :{"id":"hidden","groupId":"hidden","name":"hidden","type":"
>> hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tas

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

milarity)
|
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
|
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}

Expected
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}

Actual
:{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}


mvn --version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T22:11:47+05:30)
Maven home: /home/tijo/apache-maven-3.3.9
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /home/tijo/jdk1.8.0_91/jre
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-95-generic", arch: "amd64", family:
"unix"


Tijo



On Thu, Sep 15, 2016 at 8:07 PM, Jeff  wrote:

> Hello,
>
> Could you try building a fresh clone of the NIFI repo?  What is your dev
> environment?  What versions of java and maven are you running?
>
> Does PermissionBasedStatusMergerSpec work when you run it directly via
> maven and/or an IDE?
>
> On Wed, Sep 14, 2016 at 2:37 PM Tijo Thomas  wrote:
>
>> Hi Jeff,
>>
>> I tried replacing the mvn repo and  did  mvn clean package.
>>
>> I am still getting this error.
>>
>> I am sorry I am not getting any clue why it is failing.
>>
>> Can you check again.
>>
>> Tijo
>>
>> On Wed, Sep 14, 2016 at 9:27 PM, Jeff  wrote:
>>
>>> Ok, sounds good!  Please let us know!
>>>
>>> On Wed, Sep 14, 2016 at 12:04 AM Tijo Thomas 
>>> wrote:
>>>
>>>> Sorry to reply  late. I was on vacation for last 4 days.
>>>>
>>>> I have not modified any files.
>>>>
>>>> I think there is some problem with my repo.  I will make a new repo and
>>>> try again.  Still the problem  exist I will post it again in the group.
>>>>
>>>> Thank you very much for your support.
>>>>
>>>> Tijo
>>>>
>>>> On 10-Sep-2016 6:44 pm, "Jeff"  wrote:
>>>>
>>>>> Tijo,
>>>>>
>>>>> Have you modified ProcessorStatusSnapshotDTO.java or
>>>>> PermissionBasedStatusMergerSpec.groovy?
>>>>>
>>>>> On Sat, Sep 10, 2016 at 7:48 AM Tijo Thomas 
>>>>> wrote:
>>>>>
>>>>>> Hi Jeff
>>>>>>
>>>>>> I recently  rebase  from master.
>>>>>> Then I cloned again and ran mvn  package
>>>>>>
>>>>>> Tijo
>>>>>>
>>>>>> On 09-Sep-2016 9:12 pm, "Jeff"  wrote:
>>>>>>
>>>>>>> Tijo,
>>>>>>>
>>>>>>> I just ran this test on master and it's passing for me.  Can you
>>>>>>> provide some details about the branch you're on when running the tests? 
>>>>>>>  I
>>>>>>> see that tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000,
>>>>>>> and that's why the JSON isn't matching.
>>>>>>>
>>>>>>> On Thu, Sep 8, 2016 at 4:58 PM Tijo Thomas 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>> Nifi test case is failing (PermissionBasedStatusMergerSpec) .
>>>>>>>> This is written in Grovy .. not comfortable with Groovy .
>>>>>>>>
>>>>>>>> Running org.apache.nifi.cluster.manager.
>>>>>>>> PermissionBasedStatusMergerSpec
>>>>>>>> Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>>>>>>>> 0.922 sec <<< FAILURE! - in org.apache.nifi.cluster.manager.
>>>>>>>> PermissionBasedStatusMergerSpec
>>>>>>>> Merge ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.
>>>>>>>> manager.PermissionBasedStatusMergerSpec)  Time elapsed: 0.144 sec
>>>>>>>> <<< FAILURE!
>>>>>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>>>>>> satisfied:
>>>>>>>>
>>>>>>>> returnedJson == expectedJson
>>>>>>>> ||  |
>>>>>>>> ||  {"id":"hidden","groupId":"
>>>>>>>> hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0
>>>>>>>> (0 bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>>>> "0","tasksDuration":"00:00:00.000","activeThreadCount":0}
>>>>>>>> |false
>>>>>>>> |1 difference (99% similarity)
>>>>>>>> |{"id":"hidden","groupId":"
>>>>>>>> hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0
>>>>>>>> (0 bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>>>> "0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
>>>>>>>> |{"id":"hidden","groupId":"
>>>>>>>> hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0
>>>>>>>> (0 bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>>>> "0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
>>>>>>>> {"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>>>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0
>>>>>>>> (0 bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>>>> "0","tasksDuration":"00:30:00.000","activeThreadCount":0}
>>>>>>>>
>>>>>>>> at org.apache.nifi.cluster.manager.
>>>>>>>> PermissionBasedStatusMergerSpec.Merge ProcessorStatusSnapshotDTO(
>>>>>>>> PermissionBasedStatusMergerSpec.groovy:257)
>>>>>>>>
>>>>>>>> Merge ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.
>>>>>>>> manager.PermissionBasedStatusMergerSpec)  Time elapsed: 0.01 sec
>>>>>>>> <<< FAILURE!
>>>>>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>>>>>> satisfied:
>>>>>>>>
>>>>>>>> Tijo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>

Re: Best Practice for backing up NiFi Flows

[Tijo Thomas]

Hi Joe, 

Scenarios we have encountered is1. If a node goes down , we need to restart the 
cluster. A graceful degradation of service is not possible . I would be better 
if Nifi cluster can still run with existing nodes.
2. Zookeeper.properties : In our cluster we configured to scale  zookeeper 
nodes when ever the load in the zookeeper cluster increases.  We want to add 
this zookeeper ip to zookeeper.properties. 

3. it will be great if Nifi version upgrade can be done with out shutting down 
the cluster. I have noticed some thing like rolling upgrades  which is 
available in Hbase and some other  components also . 

Out of this 1st one is more crucial . We can implement auto scaling 
(specifically scale down) if Nifi node can run even if a node is removed.  

Thanks & RegardsTijo Thomas



On Thursday, 15 September 2016 8:16 AM, Joe Witt  wrote:
 

 Tijo

Most conf file related changes are intentionally application startup
based.  Which properties are you having to change that you'd like to
see honored without a restart?

Thanks
Joe

On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas  wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to  restart the cluster  when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically  when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing"  wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right?  I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them?  I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files.  But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi.  The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc.  Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi.  NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc.  But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment.  As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems.  Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016

Re: Best Practice for backing up NiFi Flows

[Tijo Thomas]

Hi

Our nifi cluster need to be up continuously as our user is over different
time zones.
Most of the times we often need to  restart the cluster  when ever there is
any changes in conf.

Is there any way to load the conf automatically  when ever there is any
change.

Tijo

On 15-Sep-2016 4:36 am, "James Wing"  wrote:

Mark,

Loading templates from the file system on startup is a new feature in
1.0.0, right?  I've been using the same API deployment Dan describes in 0.x.

Thanks,

James

On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne  wrote:

> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris  wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them?  I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing  wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files.  But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi.  The difference is the live data in your flow, in the
> provenance repository, in state variables, etc.  Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi.  NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc.  But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment.  As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems.  Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Hi Jeff,

I tried replacing the mvn repo and  did  mvn clean package.

I am still getting this error.

I am sorry I am not getting any clue why it is failing.

Can you check again.

Tijo

On Wed, Sep 14, 2016 at 9:27 PM, Jeff  wrote:

> Ok, sounds good!  Please let us know!
>
> On Wed, Sep 14, 2016 at 12:04 AM Tijo Thomas 
> wrote:
>
>> Sorry to reply  late. I was on vacation for last 4 days.
>>
>> I have not modified any files.
>>
>> I think there is some problem with my repo.  I will make a new repo and
>> try again.  Still the problem  exist I will post it again in the group.
>>
>> Thank you very much for your support.
>>
>> Tijo
>>
>> On 10-Sep-2016 6:44 pm, "Jeff"  wrote:
>>
>>> Tijo,
>>>
>>> Have you modified ProcessorStatusSnapshotDTO.java or
>>> PermissionBasedStatusMergerSpec.groovy?
>>>
>>> On Sat, Sep 10, 2016 at 7:48 AM Tijo Thomas 
>>> wrote:
>>>
>>>> Hi Jeff
>>>>
>>>> I recently  rebase  from master.
>>>> Then I cloned again and ran mvn  package
>>>>
>>>> Tijo
>>>>
>>>> On 09-Sep-2016 9:12 pm, "Jeff"  wrote:
>>>>
>>>>> Tijo,
>>>>>
>>>>> I just ran this test on master and it's passing for me.  Can you
>>>>> provide some details about the branch you're on when running the tests?  I
>>>>> see that tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000,
>>>>> and that's why the JSON isn't matching.
>>>>>
>>>>> On Thu, Sep 8, 2016 at 4:58 PM Tijo Thomas 
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>> Nifi test case is failing (PermissionBasedStatusMergerSpec) .
>>>>>> This is written in Grovy .. not comfortable with Groovy .
>>>>>>
>>>>>> Running org.apache.nifi.cluster.manager.
>>>>>> PermissionBasedStatusMergerSpec
>>>>>> Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
>>>>>> 0.922 sec <<< FAILURE! - in org.apache.nifi.cluster.manager.
>>>>>> PermissionBasedStatusMergerSpec
>>>>>> Merge ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.manager.
>>>>>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.144 sec  <<<
>>>>>> FAILURE!
>>>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>>>> satisfied:
>>>>>>
>>>>>> returnedJson == expectedJson
>>>>>> ||  |
>>>>>> ||  {"id":"hidden","groupId":"
>>>>>> hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>> "0","tasksDuration":"00:00:00.000","activeThreadCount":0}
>>>>>> |false
>>>>>> |1 difference (99% similarity)
>>>>>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>> "0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
>>>>>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>> "0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
>>>>>> {"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>>>> "0","tasksDuration":"00:30:00.000","activeThreadCount":0}
>>>>>>
>>>>>> at 
>>>>>> org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec.Merge
>>>>>> ProcessorStatusSnapshotDTO(PermissionBasedStatusMergerSpe
>>>>>> c.groovy:257)
>>>>>>
>>>>>> Merge ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.manager.
>>>>>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.01 sec  <<<
>>>>>> FAILURE!
>>>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>>>> satisfied:
>>>>>>
>>>>>> Tijo
>>>>>>
>>>>>>
>>>>>>
>>>>>>

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Sorry to reply  late. I was on vacation for last 4 days.

I have not modified any files.

I think there is some problem with my repo.  I will make a new repo and try
again.  Still the problem  exist I will post it again in the group.

Thank you very much for your support.

Tijo

On 10-Sep-2016 6:44 pm, "Jeff"  wrote:

> Tijo,
>
> Have you modified ProcessorStatusSnapshotDTO.java or
> PermissionBasedStatusMergerSpec.groovy?
>
> On Sat, Sep 10, 2016 at 7:48 AM Tijo Thomas  wrote:
>
>> Hi Jeff
>>
>> I recently  rebase  from master.
>> Then I cloned again and ran mvn  package
>>
>> Tijo
>>
>> On 09-Sep-2016 9:12 pm, "Jeff"  wrote:
>>
>>> Tijo,
>>>
>>> I just ran this test on master and it's passing for me.  Can you provide
>>> some details about the branch you're on when running the tests?  I see that
>>> tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000, and that's
>>> why the JSON isn't matching.
>>>
>>> On Thu, Sep 8, 2016 at 4:58 PM Tijo Thomas 
>>> wrote:
>>>
>>>> Hi
>>>> Nifi test case is failing (PermissionBasedStatusMergerSpec) .
>>>> This is written in Grovy .. not comfortable with Groovy .
>>>>
>>>> Running org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
>>>> Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.922
>>>> sec <<< FAILURE! - in org.apache.nifi.cluster.manager.
>>>> PermissionBasedStatusMergerSpec
>>>> Merge ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.manager.
>>>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.144 sec  <<< FAILURE!
>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>> satisfied:
>>>>
>>>> returnedJson == expectedJson
>>>> ||  |
>>>> ||  {"id":"hidden","groupId":"
>>>> hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
>>>> bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>> "0","tasksDuration":"00:00:00.000","activeThreadCount":0}
>>>> |false
>>>> |1 difference (99% similarity)
>>>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>>>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>> "0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
>>>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>>>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>> "0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
>>>> {"id":"hidden","groupId":"hidden","name":"hidden","type"
>>>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>>>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>>>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>>>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>>>> "0","tasksDuration":"00:30:00.000","activeThreadCount":0}
>>>>
>>>> at 
>>>> org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec.Merge
>>>> ProcessorStatusSnapshotDTO(PermissionBasedStatusMergerSpec.groovy:257)
>>>>
>>>> Merge ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.manager.
>>>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.01 sec  <<< FAILURE!
>>>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>>>> satisfied:
>>>>
>>>> Tijo
>>>>
>>>>
>>>>
>>>>

Re: PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Hi Jeff

I recently  rebase  from master.
Then I cloned again and ran mvn  package

Tijo

On 09-Sep-2016 9:12 pm, "Jeff"  wrote:

> Tijo,
>
> I just ran this test on master and it's passing for me.  Can you provide
> some details about the branch you're on when running the tests?  I see that
> tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000, and that's
> why the JSON isn't matching.
>
> On Thu, Sep 8, 2016 at 4:58 PM Tijo Thomas  wrote:
>
>> Hi
>> Nifi test case is failing (PermissionBasedStatusMergerSpec) .
>> This is written in Grovy .. not comfortable with Groovy .
>>
>> Running org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
>> Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.922
>> sec <<< FAILURE! - in org.apache.nifi.cluster.manager.
>> PermissionBasedStatusMergerSpec
>> Merge ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.manager.
>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.144 sec  <<< FAILURE!
>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>> satisfied:
>>
>> returnedJson == expectedJson
>> ||  |
>> ||  {"id":"hidden","groupId":"hidden","name":"hidden","type"
>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>> "0","tasksDuration":"00:00:00.000","activeThreadCount":0}
>> |false
>> |1 difference (99% similarity)
>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>> "0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
>> |{"id":"hidden","groupId":"hidden","name":"hidden","type"
>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>> "0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
>> {"id":"hidden","groupId":"hidden","name":"hidden","type"
>> :"hidden","bytesRead":0,"bytesWritten":0,"read":"0 bytes","written":"0
>> bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
>> bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
>> bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":
>> "0","tasksDuration":"00:30:00.000","activeThreadCount":0}
>>
>> at org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec.Merge
>> ProcessorStatusSnapshotDTO(PermissionBasedStatusMergerSpec.groovy:257)
>>
>> Merge ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.manager.
>> PermissionBasedStatusMergerSpec)  Time elapsed: 0.01 sec  <<< FAILURE!
>> org.spockframework.runtime.SpockComparisonFailure: Condition not
>> satisfied:
>>
>> Tijo
>>
>>
>>
>>

PermissionBasedStatusMergerSpec is failing

[Tijo Thomas]

Hi
Nifi test case is failing (PermissionBasedStatusMergerSpec) .
This is written in Grovy .. not comfortable with Groovy .

Running org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.922 sec
<<< FAILURE! - in
org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec
Merge
ProcessorStatusSnapshotDTO[0](org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec)
Time elapsed: 0.144 sec  <<< FAILURE!
org.spockframework.runtime.SpockComparisonFailure: Condition not satisfied:

returnedJson == expectedJson
||  |
||
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:00:00.000","activeThreadCount":0}
|false
|1 difference (99% similarity)
|
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(3)0:00.000","activeThreadCount":0}
|
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:(0)0:00.000","activeThreadCount":0}
{"id":"hidden","groupId":"hidden","name":"hidden","type":"hidden","bytesRead":0,"bytesWritten":0,"read":"0
bytes","written":"0 bytes","flowFilesIn":0,"bytesIn":0,"input":"0 (0
bytes)","flowFilesOut":0,"bytesOut":0,"output":"0 (0
bytes)","taskCount":0,"tasksDurationNanos":0,"tasks":"0","tasksDuration":"00:30:00.000","activeThreadCount":0}

at
org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec.Merge
ProcessorStatusSnapshotDTO(PermissionBasedStatusMergerSpec.groovy:257)

Merge
ProcessorStatusSnapshotDTO[1](org.apache.nifi.cluster.manager.PermissionBasedStatusMergerSpec)
Time elapsed: 0.01 sec  <<< FAILURE!
org.spockframework.runtime.SpockComparisonFailure: Condition not satisfied:

Tijo

Re: Appending files in Hadoop with PutHDFS ...

[Tijo Thomas]

Hi ,

I found this Jira open . https://issues.apache.org/jira/browse/NIFI-1322

In some of the commercial distributions of hadoop version(Huawei Hadoop
distribution)  small file features are supported so that  the Nifi can
write lot of small files and later merge into a bingger one.  I think a
good  approach is to  put in kafka/hbase and move it to HDFS based on size.

http://blog.cloudera.com/blog/2009/07/file-appends-in-hdfs/   suggest not
to append .  In that  case  do we need to keep the Jira open ?
I was also an impression that Nifi is going to implement append some how
in future.

Tijo


On Thu, Sep 8, 2016 at 1:28 AM, Jeff  wrote:

> Hello,
>
> The JIRA to which you are referring (probably https://issues.
> apache.org/jira/browse/NIFI-958) was closed and marked as a duplicate of
> https://issues.apache.org/jira/browse/NIFI-1321 and http
> s://issues.apache.org/jira/browse/NIFI-1322, so that work can be tracked
> separately for PutHDFS and PutFile.  There's some discussions on those
> JIRAs, but to my knowledge (and JIRA-searching skills) there is no
> capability to append files on HDFS.
>
> On Wed, Sep 7, 2016 at 12:24 PM Kulkarni, Suyog 
> wrote:
>
>> Hi,
>>
>> I just wanted to find out if PutHDFS now supports appending files in HDFS
>> or not. I noticed there was a Jira with status "Resolved" for this, but I
>> wanted to know which version has this feature or if there is any patch
>> available for this. Also would like to know if anyone has tried it
>> successfully or not. We are currently running version 0.6.
>>
>> Thanks,
>> Suyog Kulkarni
>> suyog_kulka...@csx.com
>>
>>
>>
>>
>> This email transmission and any accompanying attachments may contain CSX
>> privileged and confidential information intended only for the use of the
>> intended addressee. Any dissemination, distribution, copying or action
>> taken in reliance on the contents of this email by anyone other than the
>> intended recipient is strictly prohibited. If you have received this email
>> in error please immediately delete it and notify sender at the above CSX
>> email address. Sender and CSX accept no liability for any damage caused
>> directly or indirectly by receipt of this email.
>>
>

Re: What next with NiFi

[Tijo Thomas]

Hi Joe,

I am not sure whether I sounds stupid.  I was also researching on the
option 2.
I felt nifi lack the capability  to process the big data jobs where data
need to be processed  with locality.

Initially  I was thinking  about making the modification  to run nifi on
yarn.  But this approach  also will not solve many of the problems like
splitting  input for parallel  execution etc.  Of course we can write
processor to do that.  But I wish the frame work should take a call on
that,  not processor.

I got lost when I thought  about running  nifi on other framework  like
flink.  But  I feel beam is a better  option .

I also feel that nifi as a beam runner is not a good idea.  May be I am not
aware what we will gain out of this.

Earlier  I used aws pipeline and  Azure data factory,  I felt the concept
of data source node is a good and easy to use.  Do we have a plan to
introduce  the data source concept.

Tijo

On 06-Sep-2016 7:38 pm, "Joe Witt"  wrote:

> Gunjan
>
> No plans at this point.  What sort of beam related integration are you
> envisioning?  There are a couple scenarios that come to mind:
> 1) NiFi as a beam runner.  Probably not a great fit as by design we'd
> not be attempting to address and important cross section of the types
> of processing you could define in a beam API.  Storm/Flink/Spark
> already have those or are under way and we'd just integrate there.
>
> 2) NiFi being able to submit processing flows to Beam.  No idea what
> this would really mean yet.
>
> Thanks
> Joe
>
> On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave 
> wrote:
> > Thank Joe,
> > w.r.t to NiFi externals, like we currently have NiFi spark and storm
> > externals, any plans for NiFi beam external? Reason is I saw your name as
> > one of the contributors there in incubation proposal.
> > I was studying beam in more depth hence the question.
> >
> >
> > On Tue, Sep 6, 2016, 6:42 PM Joe Witt  wrote:
> >>
> >> Gunjan
> >>
> >> The best indicator of areas of focus I think are listed here
> >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> >>
> >> In a roadmap discussion thread back in January of this year the items
> >> mentioned specifically as trailing the 1.0 release were extension and
> >> variable registries.
> >>
> >> Which items will get addressed in which order is perhaps less precise
> >> and more a function of where contributions occur.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave  >
> >> wrote:
> >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the
> >> > next
> >> > big ticket plans?
> >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor
> enhacements,
> >> > but
> >> > what are the larger plans from the existing roadmap?
> >> >
>