Extract the min and max date of the input file

2017-05-30 Thread suman....@cuddle.ai
Hi ALL, I want to extract the min and max date of the input csv file. I have a date column in that file. Also please let me know how to add some custom attribute depending on the date column value. Any help would be appreciated. -- View this message in context: http://apache-nifi-developer-li

Re: unstable cluster

2017-05-30 Thread Mark Bean
Updated to external ZooKeeper last Friday. Over the weekend, there are no reports of SUSPENDED or RECONNECTED. Are there plans to upgrade the embedded ZooKeeper to the latest version, 3.4.10? Thanks, Mark On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > looked at a secured cluster and the s

Re: unstable cluster

2017-05-30 Thread Jeff
Mark, I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to log4j issues) once it's out and stable, There are issues with the way that ZK refers to log4j classes in the code that cause issues for NiFi and our Toolkit.. However there has been some back and forth [2] (in 3.4.0, which

Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura

2017-05-30 Thread Jeff
Contrats, Koji! On Sun, May 28, 2017 at 12:33 PM Matt Burgess wrote: > Congratulations and welcome aboard Koji! Looking forward to even more > great contributions and blogs/docs! > > On Fri, May 26, 2017 at 10:09 PM, Tony Kurc wrote: > > On behalf of the Apache NiFi PMC, I am pleased to announc

Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-05-30 Thread Matt Gilman
I do not believe this issue has been addressed yet. There is an open JIRA [1]. Matt [1] https://issues.apache.org/jira/browse/NIFI-3719 On Sun, May 28, 2017 at 10:16 PM, Joe Witt wrote: > Peter > > Probably best to go ahead and file a JIRA. In it you can reliably > post the attachments. There

Re: unstable cluster

2017-05-30 Thread Mark Bean
Jeff, The Nodes are disconnecting from the Cluster due to the problem reported in [1]. ZK fixed this in 3.4.10. This was the reason for inquiring about upgrading the embedded ZK to 3.4.10. While I understand there are additional reasons (log4j) to wait for a later ZK release so they can be include

Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura

2017-05-30 Thread Scott Aslan
Congrats Koji! On Tue, May 30, 2017 at 8:44 AM, Jeff wrote: > Contrats, Koji! > > On Sun, May 28, 2017 at 12:33 PM Matt Burgess > wrote: > > > Congratulations and welcome aboard Koji! Looking forward to even more > > great contributions and blogs/docs! > > > > On Fri, May 26, 2017 at 10:09 PM,

Re: unstable cluster

2017-05-30 Thread Joe Skora
Jeff, Does that mean NiFi 1.x will be unstable when using embedded ZooKeeper until the ZK version is upgrade? By "components outside of NiFi" do you mean the NiFi toolkit and other parts of the NiFi release? Joe On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > Mark, > > I did report a JIRA [1]

Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura

2017-05-30 Thread Bryan Rosander
Congrats Koji! On Tue, May 30, 2017 at 10:38 AM, Scott Aslan wrote: > Congrats Koji! > > On Tue, May 30, 2017 at 8:44 AM, Jeff wrote: > > > Contrats, Koji! > > > > On Sun, May 28, 2017 at 12:33 PM Matt Burgess > > wrote: > > > > > Congratulations and welcome aboard Koji! Looking forward to eve

MiNiFi C++ Unit Test Example

2017-05-30 Thread Andrew Christianson
Hi All, I am currently working on final preparations for the merge request on MINIFI-244. I see that a unit test framework has been added to the code base. Looking through the extant unit tests, there is a significant amount of boilerplate and it is unclear to me what would be an idiomatic unit

Re: MiNiFi C++ Unit Test Example

2017-05-30 Thread Marc
Andrew, That is something we would love to see I think. We could certainly facilitate testing and is something that has been discussed. I'm unsure if a ticket exists, though. Feel free to make a ticket to begin building some of the test framework for functional components. I'll be happy to help

Re: unstable cluster

2017-05-30 Thread Jeff
Mark, we can certainly take smaller steps rather than waiting for 3.5.2/3.6.0 to come out. I was just bringing that JIRA up as another scenario that entices us to upgrade. Joe, I'm referring to NiFi, the toolkit, and things non-NiFi that provide a ZK server to which NiFi or the ZK Migration Toolk

Re: unstable cluster

2017-05-30 Thread Joe Skora
Jeff, If I understand the issue correctly, this means NiFi 1.x has always been broken for clustering with an embedded ZooKeeper. That has never communicated until now, we clearly build for and explain how to use an embedded ZooKeeper in documentation. Any external non-NiFi elements that are cons

Re: unstable cluster

2017-05-30 Thread Jeff
Joe, My own direct and indirect experiences with NiFi 1.x clustering have been good for both embedded and external zookeeper but we have certainly seen some emails on mailing-list about it. Those have been for high load case where the embedded approach would be susceptible to timing issues and res

Re: unstable cluster

2017-05-30 Thread Juan Sequeiros
Hello all, I'll like to chime in on this interesting discussion thread. I'll like to add that my system(s) too have seen unstable ZK interaction with both embedded and eventually external ZK ( granted external has been better ) interaction. We have resolved them with NIFI restarts. And it's to th

Example Processors that....

2017-05-30 Thread Otto Fowler
Does anyone know of a processor that uses a file or files loaded into the system as part of it’s configuration? I’m looking for best practices. These files do not need to be in the repo or transferred. Thanks in advance.

Re: Example Processors that....

2017-05-30 Thread Bryan Bende
Hi Otto, Do you mean something specific by "files loaded into the system" or are you just referring to a processor having a property that points to an external file? There are a few examples of the latter in the code base, you can look for usage of FILE_EXISTS_VALIDATOR as an easy way to find the

Re: Example Processors that....

2017-05-30 Thread Otto Fowler
Thanks, I’ll take a look! On May 30, 2017 at 16:12:22, Bryan Bende (bbe...@gmail.com) wrote: Hi Otto, Do you mean something specific by "files loaded into the system" or are you just referring to a processor having a property that points to an external file? There are a few examples of

Re: Example Processors that....

2017-05-30 Thread Otto Fowler
Bryan, This will be a great start for what I’m looking for.  Thank you! On May 30, 2017 at 17:23:47, Otto Fowler (ottobackwa...@gmail.com) wrote: Thanks, I’ll take a look! On May 30, 2017 at 16:12:22, Bryan Bende (bbe...@gmail.com) wrote: Hi Otto, Do you mean something specific by "files lo

RE: [EXT] Re: NiFi Processors show 30 Second Execution time, 0 executions

2017-05-30 Thread Peter Wicks (pwicks)
Thanks Matt. Interestingly enough I'm at +09:00, and not a half hour interval, but hopefully this will fix it regardless. -Original Message- From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: Tuesday, May 30, 2017 9:05 PM To: dev@nifi.apache.org Subject: [EXT] Re: NiFi Processors

Re: unstable cluster

2017-05-30 Thread Joe Witt
Just scanning through the items currently on master that would show up in the 1.3.0 release we see numerous cluster related bug fixes. More consistent port alignment across cluster https://issues.apache.org/jira/browse/NIFI-3981 Ensure controller service lifecycle handled better with different