Extract the min and max date of the input file
Hi ALL, I want to extract the min and max date of the input csv file. I have a date column in that file. Also please let me know how to add some custom attribute depending on the date column value. Any help would be appreciated. -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Extract-the-min-and-max-date-of-the-input-file-tp16012.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: unstable cluster
Updated to external ZooKeeper last Friday. Over the weekend, there are no reports of SUSPENDED or RECONNECTED. Are there plans to upgrade the embedded ZooKeeper to the latest version, 3.4.10? Thanks, Mark On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > looked at a secured cluster and the send times are routinely at 100ms > similar to yours. I think what i was flagging as potentially > interesting is not interesting at all. > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt wrote: > > Ok. Well as a point of comparison i'm looking at heartbeat logs from > > another cluster and the times are consistently 1-3 millis for the > > send. Yours above show 100+ms typical with one north of 900ms. Not > > sure how relevant that is but something i noticed. > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean > wrote: > >> ping shows acceptably fast response time between servers, approximately > >> 0.100-0.150 ms > >> > >> > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt wrote: > >> > >>> have you evaluated latency across the machines in your cluster? I ask > >>> because 122ms is pretty long and 917ms is very long. Are these nodes > >>> across a WAN link? > >>> > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean > wrote: > >>> > Update: now all 5 nodes, regardless of ZK server, are indicating > >>> SUSPENDED > >>> > -> RECONNECTED. > >>> > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean > >>> wrote: > >>> > > >>> >> I reduced the number of embedded ZooKeeper servers on the 5-Node > NiFi > >>> >> Cluster from 5 to 3. This has improved the situation. I do not see > any > >>> of > >>> >> the three Nodes which are also ZK servers > disconnecting/reconnecting to > >>> the > >>> >> cluster as before. However, the two Nodes which are not running ZK > >>> continue > >>> >> to disconnect and reconnect. The following is taken from one of the > >>> non-ZK > >>> >> Nodes. It's curious that some messages are issued twice from the > same > >>> >> thread, but reference a different object > >>> >> > >>> >> nifi-app.log > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state. > >>> ConnectionStateManager > >>> >> State change: SUSPENDED > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1] o.a.n.c.c. > >>> ClusterProtocolHeaertbeater > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to FQDN:PORT at > >>> >> 2017-05-25 13:39:45,627; send took 122 millis > >>> >> 2017-05-25 13:39:50,862 INFO [Clustering Tasks Thread-1] o.a.n.c.c. > >>> ClusterProtocolHeaertbeater > >>> >> Heartbeat create at 2017-05-25 13:39:50,732 and sent to FQDN:PORT at > >>> >> 2017-05-25 13:39:50,862; send took 122 millis > >>> >> 2017-05-25 13:39:56,089 INFO [Clustering Tasks Thread-1] o.a.n.c.c. > >>> ClusterProtocolHeaertbeater > >>> >> Heartbeat create at 2017-05-25 13:39:55,966 and sent to FQDN:PORT at > >>> >> 2017-05-25 13:39:56,089; send took 129 millis > >>> >> 2017-05-25 13:40:01,629 INFO [Curator-ConnectionStateManager-0] > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller. > >>> >> leader.election.CuratorLeaderElectionManager$ > ElectionListener@68f8b6a2 > >>> >> Connection State changed to SUSPENDED > >>> >> 2017-05-25 13:40:01,629 INFO [Curator-ConnectionStateManager-0] > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller. > >>> >> leader.election.CuratorLeaderElectionManager$ > ElectionListener@663f55cd > >>> >> Connection State changed to SUSPENDED > >>> >> 2017-05-25 13:40:02,412 INFO [main-EventThread] o.a.c.f.state. > >>> ConnectinoStateManager > >>> >> State change: RECONNECTED > >>> >> 2017-05-25 13:40:02,413 INFO [Curator-ConnectionStateManager-0] > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller. > >>> >> leader.election.CuratorLeaderElectionManager$ > ElectionListener@68f8b6a2 > >>> >> Connection State changed to RECONNECTED > >>> >> 2017-05-25 13:40:02,413 INFO [Curator-ConnectionStateManager-0] > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > org.apache.nifi.controller. > >>> >> leader.election.CuratorLeaderElectionManager$ > ElectionListener@663f55cd > >>> >> Connection State changed to RECONNECTED > >>> >> 2017-05-25 13:40:02,550 INFO [Clustering Tasks Thread-1] o.a.n.c.c. > >>> ClusterProtocolHeaertbeater > >>> >> Heartbeat create at 2017-05-25 13:40:01,632 and sent to FQDN:PORT at > >>> >> 2017-05-25 13:40:02,550; send took 917 millis > >>> >> 2017-05-25 13:40:07,787 INFO [Clustering Tasks Thread-1] o.a.n.c.c. > >>> ClusterProtocolHeaertbeater > >>> >> Heartbeat create at 2017-05-25 13:40:07,657 and sent to FQDN:PORT at > >>> >> 2017-05-25 13:40:07,787; send took 129 millis > >>> >> > >>> >> I will work on setting up an external ZK next, but would still like > some > >>> >> insight to what is being observed with the embedded ZK. > >>> >> > >>> >> Thanks, > >>> >> Mark > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> On Wed, May 24, 2017 at 3:57 PM, Mark Bean > >>> wrote: > >>> >> > >>> >>> Yes, we are u
Re: unstable cluster
Mark, I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to log4j issues) once it's out and stable, There are issues with the way that ZK refers to log4j classes in the code that cause issues for NiFi and our Toolkit.. However there has been some back and forth [2] (in 3.4.0, which doesn't fix the issue, but moves towards fixing it), [3], and [4] on the changes being implemented in versions 3.5.2 and 3.6.0. Also, it looks like ZK 3.6.0 is headed toward using log4j 2 [5]. There are many components outside of NiFi that are still using ZK 3.4.6, so it may be a while before we can move to 3.4.10. I don't currently know anything about the forward compatibility of 3.4.6. Are there improvements/fixes in 3.4.10 which you need? [1] https://issues.apache.org/jira/browse/NIFI-3067 [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 - Jeff On Tue, May 30, 2017 at 8:15 AM Mark Bean wrote: > Updated to external ZooKeeper last Friday. Over the weekend, there are no > reports of SUSPENDED or RECONNECTED. > > Are there plans to upgrade the embedded ZooKeeper to the latest version, > 3.4.10? > > Thanks, > Mark > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > > > looked at a secured cluster and the send times are routinely at 100ms > > similar to yours. I think what i was flagging as potentially > > interesting is not interesting at all. > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt wrote: > > > Ok. Well as a point of comparison i'm looking at heartbeat logs from > > > another cluster and the times are consistently 1-3 millis for the > > > send. Yours above show 100+ms typical with one north of 900ms. Not > > > sure how relevant that is but something i noticed. > > > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean > > wrote: > > >> ping shows acceptably fast response time between servers, > approximately > > >> 0.100-0.150 ms > > >> > > >> > > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt > wrote: > > >> > > >>> have you evaluated latency across the machines in your cluster? I > ask > > >>> because 122ms is pretty long and 917ms is very long. Are these nodes > > >>> across a WAN link? > > >>> > > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean > > wrote: > > >>> > Update: now all 5 nodes, regardless of ZK server, are indicating > > >>> SUSPENDED > > >>> > -> RECONNECTED. > > >>> > > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean > > > >>> wrote: > > >>> > > > >>> >> I reduced the number of embedded ZooKeeper servers on the 5-Node > > NiFi > > >>> >> Cluster from 5 to 3. This has improved the situation. I do not see > > any > > >>> of > > >>> >> the three Nodes which are also ZK servers > > disconnecting/reconnecting to > > >>> the > > >>> >> cluster as before. However, the two Nodes which are not running ZK > > >>> continue > > >>> >> to disconnect and reconnect. The following is taken from one of > the > > >>> non-ZK > > >>> >> Nodes. It's curious that some messages are issued twice from the > > same > > >>> >> thread, but reference a different object > > >>> >> > > >>> >> nifi-app.log > > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state. > > >>> ConnectionStateManager > > >>> >> State change: SUSPENDED > > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1] > o.a.n.c.c. > > >>> ClusterProtocolHeaertbeater > > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to FQDN:PORT > at > > >>> >> 2017-05-25 13:39:45,627; send took 122 millis > > >>> >> 2017-05-25 13:39:50,862 INFO [Clustering Tasks Thread-1] > o.a.n.c.c. > > >>> ClusterProtocolHeaertbeater > > >>> >> Heartbeat create at 2017-05-25 13:39:50,732 and sent to FQDN:PORT > at > > >>> >> 2017-05-25 13:39:50,862; send took 122 millis > > >>> >> 2017-05-25 13:39:56,089 INFO [Clustering Tasks Thread-1] > o.a.n.c.c. > > >>> ClusterProtocolHeaertbeater > > >>> >> Heartbeat create at 2017-05-25 13:39:55,966 and sent to FQDN:PORT > at > > >>> >> 2017-05-25 13:39:56,089; send took 129 millis > > >>> >> 2017-05-25 13:40:01,629 INFO [Curator-ConnectionStateManager-0] > > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > > org.apache.nifi.controller. > > >>> >> leader.election.CuratorLeaderElectionManager$ > > ElectionListener@68f8b6a2 > > >>> >> Connection State changed to SUSPENDED > > >>> >> 2017-05-25 13:40:01,629 INFO [Curator-ConnectionStateManager-0] > > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > > org.apache.nifi.controller. > > >>> >> leader.election.CuratorLeaderElectionManager$ > > ElectionListener@663f55cd > > >>> >> Connection State changed to SUSPENDED > > >>> >> 2017-05-25 13:40:02,412 INFO [main-EventThread] o.a.c.f.state. > > >>> ConnectinoStateManager > > >>> >> State change: RECONNECTED > > >>> >> 2017-05-25 13:40:02,413 INFO [Curator-ConnectionStateManager-0] > > >>> >> o.a.n.c.l.e.Curator
Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura
Contrats, Koji! On Sun, May 28, 2017 at 12:33 PM Matt Burgess wrote: > Congratulations and welcome aboard Koji! Looking forward to even more > great contributions and blogs/docs! > > On Fri, May 26, 2017 at 10:09 PM, Tony Kurc wrote: > > On behalf of the Apache NiFi PMC, I am pleased to announce that Koji > > Kawamura has accepted the PMC's invitation to join the Apache NiFi PMC. > We > > greatly appreciate all of Koji's hard work and generous contributions to > > the project. We look forward to continued involvement in the project. > > > > Koji has been contributing code NiFi since 2015, taking point on > important > > parts of the project like improving site-to-site. Beyond code, Koji is > > active in building the community through blogging on technical topics, > > generating interest on social media (@apachenifi #NiFi) like twitter, and > > giving talks at conferences. Koji has been keeping the project strong by > > being active in reviewing new contributions and verifying releases. > > > > Please join us in congratulating and welcoming Koji to the Apache NiFi > PMC. > > > > Congratulations and welcome, Koji! > > > > -- Tony >
Re: NiFi Processors show 30 Second Execution time, 0 executions
I do not believe this issue has been addressed yet. There is an open JIRA [1]. Matt [1] https://issues.apache.org/jira/browse/NIFI-3719 On Sun, May 28, 2017 at 10:16 PM, Joe Witt wrote: > Peter > > Probably best to go ahead and file a JIRA. In it you can reliably > post the attachments. There was a potentially related timezone > handling issue as I recall in this past release so perhaps there is > some relationship. > > Thanks > Joe > > On Sun, May 28, 2017 at 10:04 PM, Peter Wicks (pwicks) > wrote: > > I wanted to re-open this discussion, it's been a while and I'm still > seeing > > the issue even with the latest version. I'm still seeing this issue > running > > a stock NiFi v1.2.0. By stock I mean no custom NAR’s, etc… just original > > vanilla code, in this case with no configuration, so running unsecured, > > empty canvas (except for my test case). > > > > > > > > I’ve expanded my test scenarios. > > > > > > > > Scenario 1 is Windows 7, code built using mvn, using Oracle Java. > > > > Java Version: > > > > java version "1.8.0_91" > > > > Java(TM) SE Runtime Environment (build 1.8.0_91-b15) > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b15, mixed mode) > > > > > > > > Scenario 2 is RHEL 7.3, the NiFi build is v1.2.0 downloaded from the NiFi > > website. Running OpenJDK. > > > > > > > > openjdk version "1.8.0_102" > > > > OpenJDK Runtime Environment (build 1.8.0_102-b14) > > > > OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode) > > > > > > > > I’ve attempted to attach a screenshot (my attachments seem to not make it > > very often on this list). In it I show the onscreen Tasks/Time for two > > processors: one shows 1 / 00:30:04.292 and the other 0 / 00:30:00.000. > > > > > > > > Thanks! > > > > Peter > > > > > > > > -Original Message- > > From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > > Sent: Friday, March 31, 2017 10:54 PM > > To: dev@nifi.apache.org > > Subject: Re: NiFi Processors show 30 Second Execution time, 0 executions > > > > > > > > What version of Java are you running on ? Major_minor? > > > > > > > > On Fri, Mar 31, 2017 at 10:49 AM, Peter Wicks (pwicks) < > pwi...@micron.com> > > > > wrote: > > > > > > > >> I misread my own screenshot, it says 30 minutes, not seconds. Also, I > > > >> did a restart of NiFi and opened it up in a fresh instance of Chrome; > > > >> no change. I kicked off a GenerateFlowFile processor and the > > > >> milliseconds are going up, but the 30 minutes is remaining the same... > > > >> > > > >> > > > >> > > > >> -Original Message- > > > >> From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > >> Sent: Friday, March 31, 2017 10:41 PM > > > >> To: dev@nifi.apache.org > > > >> Subject: Re: NiFi Processors show 30 Second Execution time, 0 > > > >> executions > > > >> > > > >> Doing a clean build of 091359b450a7d0fb6bb04e2238c9171728cd2720, I > > > >> will have to see if I have a windows 7 VM anywhere, I know Witt was > > > >> using Win10 and didnt see it... I find it odd that ALL your processors > > > >> have a 30 second number not just the UpdateAttribute. Anything else > > > >> about your environment you can share that may be unique? > > > >> > > > >> On Fri, Mar 31, 2017 at 10:15 AM, Peter Wicks (pwicks) > > > >> > > > >> wrote: > > > >> > > > >> > 091359b450a7d0fb6bb04e2238c9171728cd2720, so just one commit behind > > > >> > master. > > > >> > I am testing on Windows 7. > > > >> > > > > >> > Lee, yield isn't a bad idea, but UpdateAttribute in my screenshot > > > >> > has never run; not even once. I don't think it's had the opportunity > > > >> > to > > > >> yield. > > > >> > > > > >> > -Original Message- > > > >> > From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > >> > Sent: Friday, March 31, 2017 9:37 PM > > > >> > To: dev@nifi.apache.org > > > >> > Subject: Re: NiFi Processors show 30 Second Execution time, 0 > > > >> > executions > > > >> > > > > >> > I just built the latest and am unable to see the issue as well. I > > > >> > also played with yield duration with no luck. > > > >> > > > > >> > Can you provide us what build your on so I can check that one out > > > >> exactly? > > > >> > I did my last trunk test as of - > > > >> > 6a64b3cd9cca70e6a27b9034eba520ae0c0cb6ca > > > >> > > > > >> > git rev-parse HEAD > > > >> > > > > >> > > > > >> > On Fri, Mar 31, 2017 at 9:21 AM, Lee Laim wrote: > > > >> > > > > >> > > Is it potentially related to yield duration? > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Mar 31, 2017, at 6:58 AM, Peter Wicks (pwicks) > > > >> > > > > > > >> > > wrote: > > > >> > > > > > > >> > > > channel. > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Joseph > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Joseph > > > >> > > > > > > > > > > > > > > > > -- > > > > Joseph >
Re: unstable cluster
Jeff, The Nodes are disconnecting from the Cluster due to the problem reported in [1]. ZK fixed this in 3.4.10. This was the reason for inquiring about upgrading the embedded ZK to 3.4.10. While I understand there are additional reasons (log4j) to wait for a later ZK release so they can be included as well. But, can we take two smaller steps (especially since ZK 3.5.2 or 3.6.0 is a somewhat unknown timeframe) rather than one big step? [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044 On Tue, May 30, 2017 at 8:42 AM, Jeff wrote: > Mark, > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to log4j > issues) once it's out and stable, There are issues with the way that ZK > refers to log4j classes in the code that cause issues for NiFi and our > Toolkit.. However there has been some back and forth [2] (in 3.4.0, which > doesn't fix the issue, but moves towards fixing it), [3], and [4] on the > changes being implemented in versions 3.5.2 and 3.6.0. Also, it looks like > ZK 3.6.0 is headed toward using log4j 2 [5]. > > There are many components outside of NiFi that are still using ZK 3.4.6, so > it may be a while before we can move to 3.4.10. I don't currently know > anything about the forward compatibility of 3.4.6. Are there > improvements/fixes in 3.4.10 which you need? > > [1] https://issues.apache.org/jira/browse/NIFI-3067 > [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 > [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 > [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 > [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 > > - Jeff > > On Tue, May 30, 2017 at 8:15 AM Mark Bean wrote: > > > Updated to external ZooKeeper last Friday. Over the weekend, there are no > > reports of SUSPENDED or RECONNECTED. > > > > Are there plans to upgrade the embedded ZooKeeper to the latest version, > > 3.4.10? > > > > Thanks, > > Mark > > > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > > > > > looked at a secured cluster and the send times are routinely at 100ms > > > similar to yours. I think what i was flagging as potentially > > > interesting is not interesting at all. > > > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt wrote: > > > > Ok. Well as a point of comparison i'm looking at heartbeat logs from > > > > another cluster and the times are consistently 1-3 millis for the > > > > send. Yours above show 100+ms typical with one north of 900ms. Not > > > > sure how relevant that is but something i noticed. > > > > > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean > > > wrote: > > > >> ping shows acceptably fast response time between servers, > > approximately > > > >> 0.100-0.150 ms > > > >> > > > >> > > > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt > > wrote: > > > >> > > > >>> have you evaluated latency across the machines in your cluster? I > > ask > > > >>> because 122ms is pretty long and 917ms is very long. Are these > nodes > > > >>> across a WAN link? > > > >>> > > > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean > > > > wrote: > > > >>> > Update: now all 5 nodes, regardless of ZK server, are indicating > > > >>> SUSPENDED > > > >>> > -> RECONNECTED. > > > >>> > > > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean < > mark.o.b...@gmail.com > > > > > > >>> wrote: > > > >>> > > > > >>> >> I reduced the number of embedded ZooKeeper servers on the 5-Node > > > NiFi > > > >>> >> Cluster from 5 to 3. This has improved the situation. I do not > see > > > any > > > >>> of > > > >>> >> the three Nodes which are also ZK servers > > > disconnecting/reconnecting to > > > >>> the > > > >>> >> cluster as before. However, the two Nodes which are not running > ZK > > > >>> continue > > > >>> >> to disconnect and reconnect. The following is taken from one of > > the > > > >>> non-ZK > > > >>> >> Nodes. It's curious that some messages are issued twice from the > > > same > > > >>> >> thread, but reference a different object > > > >>> >> > > > >>> >> nifi-app.log > > > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state. > > > >>> ConnectionStateManager > > > >>> >> State change: SUSPENDED > > > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:45,627; send took 122 millis > > > >>> >> 2017-05-25 13:39:50,862 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:50,732 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:50,862; send took 122 millis > > > >>> >> 2017-05-25 13:39:56,089 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:55,966 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:56,089; send took 129 millis > > > >>> >> 2017-05-25 13:40:01,629
Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura
Congrats Koji!
On Tue, May 30, 2017 at 8:44 AM, Jeff wrote:
> Contrats, Koji!
>
> On Sun, May 28, 2017 at 12:33 PM Matt Burgess
> wrote:
>
> > Congratulations and welcome aboard Koji! Looking forward to even more
> > great contributions and blogs/docs!
> >
> > On Fri, May 26, 2017 at 10:09 PM, Tony Kurc wrote:
> > > On behalf of the Apache NiFi PMC, I am pleased to announce that Koji
> > > Kawamura has accepted the PMC's invitation to join the Apache NiFi PMC.
> > We
> > > greatly appreciate all of Koji's hard work and generous contributions
> to
> > > the project. We look forward to continued involvement in the project.
> > >
> > > Koji has been contributing code NiFi since 2015, taking point on
> > important
> > > parts of the project like improving site-to-site. Beyond code, Koji is
> > > active in building the community through blogging on technical topics,
> > > generating interest on social media (@apachenifi #NiFi) like twitter,
> and
> > > giving talks at conferences. Koji has been keeping the project strong
> by
> > > being active in reviewing new contributions and verifying releases.
> > >
> > > Please join us in congratulating and welcoming Koji to the Apache NiFi
> > PMC.
> > >
> > > Congratulations and welcome, Koji!
> > >
> > > -- Tony
> >
>
--
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"email":
"scottyas...@gmail.com ","linkedin":
"http://www.linkedin.com/in/scottyaslan
"}});*
Re: unstable cluster
Jeff, Does that mean NiFi 1.x will be unstable when using embedded ZooKeeper until the ZK version is upgrade? By "components outside of NiFi" do you mean the NiFi toolkit and other parts of the NiFi release? Joe On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > Mark, > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to log4j > issues) once it's out and stable, There are issues with the way that ZK > refers to log4j classes in the code that cause issues for NiFi and our > Toolkit.. However there has been some back and forth [2] (in 3.4.0, which > doesn't fix the issue, but moves towards fixing it), [3], and [4] on the > changes being implemented in versions 3.5.2 and 3.6.0. Also, it looks like > ZK 3.6.0 is headed toward using log4j 2 [5]. > > There are many components outside of NiFi that are still using ZK 3.4.6, so > it may be a while before we can move to 3.4.10. I don't currently know > anything about the forward compatibility of 3.4.6. Are there > improvements/fixes in 3.4.10 which you need? > > [1] https://issues.apache.org/jira/browse/NIFI-3067 > [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 > [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 > [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 > [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 > > - Jeff > > On Tue, May 30, 2017 at 8:15 AM Mark Bean wrote: > > > Updated to external ZooKeeper last Friday. Over the weekend, there are no > > reports of SUSPENDED or RECONNECTED. > > > > Are there plans to upgrade the embedded ZooKeeper to the latest version, > > 3.4.10? > > > > Thanks, > > Mark > > > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > > > > > looked at a secured cluster and the send times are routinely at 100ms > > > similar to yours. I think what i was flagging as potentially > > > interesting is not interesting at all. > > > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt wrote: > > > > Ok. Well as a point of comparison i'm looking at heartbeat logs from > > > > another cluster and the times are consistently 1-3 millis for the > > > > send. Yours above show 100+ms typical with one north of 900ms. Not > > > > sure how relevant that is but something i noticed. > > > > > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean > > > wrote: > > > >> ping shows acceptably fast response time between servers, > > approximately > > > >> 0.100-0.150 ms > > > >> > > > >> > > > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt > > wrote: > > > >> > > > >>> have you evaluated latency across the machines in your cluster? I > > ask > > > >>> because 122ms is pretty long and 917ms is very long. Are these > nodes > > > >>> across a WAN link? > > > >>> > > > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean > > > > wrote: > > > >>> > Update: now all 5 nodes, regardless of ZK server, are indicating > > > >>> SUSPENDED > > > >>> > -> RECONNECTED. > > > >>> > > > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean < > mark.o.b...@gmail.com > > > > > > >>> wrote: > > > >>> > > > > >>> >> I reduced the number of embedded ZooKeeper servers on the 5-Node > > > NiFi > > > >>> >> Cluster from 5 to 3. This has improved the situation. I do not > see > > > any > > > >>> of > > > >>> >> the three Nodes which are also ZK servers > > > disconnecting/reconnecting to > > > >>> the > > > >>> >> cluster as before. However, the two Nodes which are not running > ZK > > > >>> continue > > > >>> >> to disconnect and reconnect. The following is taken from one of > > the > > > >>> non-ZK > > > >>> >> Nodes. It's curious that some messages are issued twice from the > > > same > > > >>> >> thread, but reference a different object > > > >>> >> > > > >>> >> nifi-app.log > > > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state. > > > >>> ConnectionStateManager > > > >>> >> State change: SUSPENDED > > > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:45,627; send took 122 millis > > > >>> >> 2017-05-25 13:39:50,862 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:50,732 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:50,862; send took 122 millis > > > >>> >> 2017-05-25 13:39:56,089 INFO [Clustering Tasks Thread-1] > > o.a.n.c.c. > > > >>> ClusterProtocolHeaertbeater > > > >>> >> Heartbeat create at 2017-05-25 13:39:55,966 and sent to > FQDN:PORT > > at > > > >>> >> 2017-05-25 13:39:56,089; send took 129 millis > > > >>> >> 2017-05-25 13:40:01,629 INFO [Curator-ConnectionStateManager-0] > > > >>> >> o.a.n.c.l.e.CuratorLeaderElectionManager > > > org.apache.nifi.controller. > > > >>> >> leader.election.CuratorLeaderElectionManager$ > > > ElectionListener@68f8b6a2 > > > >>> >> Connection State changed to SUSPENDED > > > >>> >>
Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura
Congrats Koji!
On Tue, May 30, 2017 at 10:38 AM, Scott Aslan wrote:
> Congrats Koji!
>
> On Tue, May 30, 2017 at 8:44 AM, Jeff wrote:
>
> > Contrats, Koji!
> >
> > On Sun, May 28, 2017 at 12:33 PM Matt Burgess
> > wrote:
> >
> > > Congratulations and welcome aboard Koji! Looking forward to even more
> > > great contributions and blogs/docs!
> > >
> > > On Fri, May 26, 2017 at 10:09 PM, Tony Kurc wrote:
> > > > On behalf of the Apache NiFi PMC, I am pleased to announce that Koji
> > > > Kawamura has accepted the PMC's invitation to join the Apache NiFi
> PMC.
> > > We
> > > > greatly appreciate all of Koji's hard work and generous contributions
> > to
> > > > the project. We look forward to continued involvement in the project.
> > > >
> > > > Koji has been contributing code NiFi since 2015, taking point on
> > > important
> > > > parts of the project like improving site-to-site. Beyond code, Koji
> is
> > > > active in building the community through blogging on technical
> topics,
> > > > generating interest on social media (@apachenifi #NiFi) like twitter,
> > and
> > > > giving talks at conferences. Koji has been keeping the project strong
> > by
> > > > being active in reviewing new contributions and verifying releases.
> > > >
> > > > Please join us in congratulating and welcoming Koji to the Apache
> NiFi
> > > PMC.
> > > >
> > > > Congratulations and welcome, Koji!
> > > >
> > > > -- Tony
> > >
> >
>
>
>
> --
> *Scott Aslan = new WebDeveloper(*
> *{"location": {"city": "Saint Cloud","state": "FL",
> "zip": "34771"},"contact": {"email":
> "scottyas...@gmail.com ","linkedin":
> "http://www.linkedin.com/in/scottyaslan
> "}});*
>
MiNiFi C++ Unit Test Example
Hi All, I am currently working on final preparations for the merge request on MINIFI-244. I see that a unit test framework has been added to the code base. Looking through the extant unit tests, there is a significant amount of boilerplate and it is unclear to me what would be an idiomatic unit test for MiNiFi C++. Is there/could someone produce an example minimal unit test source code for a MiNiFi processor? Alternatively, are there plans/efforts to extract out some of this boilerplate? Regards, Andy
Re: MiNiFi C++ Unit Test Example
Andrew, That is something we would love to see I think. We could certainly facilitate testing and is something that has been discussed. I'm unsure if a ticket exists, though. Feel free to make a ticket to begin building some of the test framework for functional components. I'll be happy to help when I have some cycles. On May 30, 2017 11:14 AM, "Andrew Christianson" < andrew.christian...@nextcentury.com> wrote: > Hi All, > > I am currently working on final preparations for the merge request on > MINIFI-244. I see that a unit test framework has been added to the code > base. Looking through the extant unit tests, there is a significant amount > of boilerplate and it is unclear to me what would be an idiomatic unit test > for MiNiFi C++. > > Is there/could someone produce an example minimal unit test source code > for a MiNiFi processor? Alternatively, are there plans/efforts to extract > out some of this boilerplate? > > Regards, > > Andy
Re: unstable cluster
Mark, we can certainly take smaller steps rather than waiting for 3.5.2/3.6.0 to come out. I was just bringing that JIRA up as another scenario that entices us to upgrade. Joe, I'm referring to NiFi, the toolkit, and things non-NiFi that provide a ZK server to which NiFi or the ZK Migration Toolkit are clients. I'm not saying we can't or shouldn't upgrade, but we do need to test to make sure that no issues are introduced by NiFi shipping with ZK 3.4.10. Being that it's a bugfix version change, it's probably fine. - Jeff On Tue, May 30, 2017 at 10:46 AM Joe Skora wrote: > Jeff, > > Does that mean NiFi 1.x will be unstable when using embedded ZooKeeper > until the ZK version is upgrade? > > By "components outside of NiFi" do you mean the NiFi toolkit and other > parts of the NiFi release? > > Joe > > On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > > > Mark, > > > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to > log4j > > issues) once it's out and stable, There are issues with the way that ZK > > refers to log4j classes in the code that cause issues for NiFi and our > > Toolkit.. However there has been some back and forth [2] (in 3.4.0, > which > > doesn't fix the issue, but moves towards fixing it), [3], and [4] on the > > changes being implemented in versions 3.5.2 and 3.6.0. Also, it looks > like > > ZK 3.6.0 is headed toward using log4j 2 [5]. > > > > There are many components outside of NiFi that are still using ZK 3.4.6, > so > > it may be a while before we can move to 3.4.10. I don't currently know > > anything about the forward compatibility of 3.4.6. Are there > > improvements/fixes in 3.4.10 which you need? > > > > [1] https://issues.apache.org/jira/browse/NIFI-3067 > > [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 > > [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 > > [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 > > [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 > > > > - Jeff > > > > On Tue, May 30, 2017 at 8:15 AM Mark Bean wrote: > > > > > Updated to external ZooKeeper last Friday. Over the weekend, there are > no > > > reports of SUSPENDED or RECONNECTED. > > > > > > Are there plans to upgrade the embedded ZooKeeper to the latest > version, > > > 3.4.10? > > > > > > Thanks, > > > Mark > > > > > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt wrote: > > > > > > > looked at a secured cluster and the send times are routinely at 100ms > > > > similar to yours. I think what i was flagging as potentially > > > > interesting is not interesting at all. > > > > > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt > wrote: > > > > > Ok. Well as a point of comparison i'm looking at heartbeat logs > from > > > > > another cluster and the times are consistently 1-3 millis for the > > > > > send. Yours above show 100+ms typical with one north of 900ms. > Not > > > > > sure how relevant that is but something i noticed. > > > > > > > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean > > > > > wrote: > > > > >> ping shows acceptably fast response time between servers, > > > approximately > > > > >> 0.100-0.150 ms > > > > >> > > > > >> > > > > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt > > > wrote: > > > > >> > > > > >>> have you evaluated latency across the machines in your cluster? > I > > > ask > > > > >>> because 122ms is pretty long and 917ms is very long. Are these > > nodes > > > > >>> across a WAN link? > > > > >>> > > > > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean < > mark.o.b...@gmail.com > > > > > > > wrote: > > > > >>> > Update: now all 5 nodes, regardless of ZK server, are > indicating > > > > >>> SUSPENDED > > > > >>> > -> RECONNECTED. > > > > >>> > > > > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean < > > mark.o.b...@gmail.com > > > > > > > > >>> wrote: > > > > >>> > > > > > >>> >> I reduced the number of embedded ZooKeeper servers on the > 5-Node > > > > NiFi > > > > >>> >> Cluster from 5 to 3. This has improved the situation. I do not > > see > > > > any > > > > >>> of > > > > >>> >> the three Nodes which are also ZK servers > > > > disconnecting/reconnecting to > > > > >>> the > > > > >>> >> cluster as before. However, the two Nodes which are not > running > > ZK > > > > >>> continue > > > > >>> >> to disconnect and reconnect. The following is taken from one > of > > > the > > > > >>> non-ZK > > > > >>> >> Nodes. It's curious that some messages are issued twice from > the > > > > same > > > > >>> >> thread, but reference a different object > > > > >>> >> > > > > >>> >> nifi-app.log > > > > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state. > > > > >>> ConnectionStateManager > > > > >>> >> State change: SUSPENDED > > > > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1] > > > o.a.n.c.c. > > > > >>> ClusterProtocolHeaertbeater > > > > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to > > FQDN:PORT > > > at > > > > >>> >> 2017-05-25 13:39:45,627; send took 12
Re: unstable cluster
Jeff, If I understand the issue correctly, this means NiFi 1.x has always been broken for clustering with an embedded ZooKeeper. That has never communicated until now, we clearly build for and explain how to use an embedded ZooKeeper in documentation. Any external non-NiFi elements that are considered in design and dependency decisions need to be clearly understood by the entire community. What things non-NiFi are you thinking of that drive ZooKeeper dependencies? Joe On Tue, May 30, 2017 at 9:11 AM, Jeff wrote: > Mark, we can certainly take smaller steps rather than waiting for > 3.5.2/3.6.0 to come out. I was just bringing that JIRA up as another > scenario that entices us to upgrade. > > Joe, I'm referring to NiFi, the toolkit, and things non-NiFi that provide a > ZK server to which NiFi or the ZK Migration Toolkit are clients. I'm not > saying we can't or shouldn't upgrade, but we do need to test to make sure > that no issues are introduced by NiFi shipping with ZK 3.4.10. Being that > it's a bugfix version change, it's probably fine. > > - Jeff > > On Tue, May 30, 2017 at 10:46 AM Joe Skora wrote: > > > Jeff, > > > > Does that mean NiFi 1.x will be unstable when using embedded ZooKeeper > > until the ZK version is upgrade? > > > > By "components outside of NiFi" do you mean the NiFi toolkit and other > > parts of the NiFi release? > > > > Joe > > > > On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > > > > > Mark, > > > > > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to > > log4j > > > issues) once it's out and stable, There are issues with the way that ZK > > > refers to log4j classes in the code that cause issues for NiFi and our > > > Toolkit.. However there has been some back and forth [2] (in 3.4.0, > > which > > > doesn't fix the issue, but moves towards fixing it), [3], and [4] on > the > > > changes being implemented in versions 3.5.2 and 3.6.0. Also, it looks > > like > > > ZK 3.6.0 is headed toward using log4j 2 [5]. > > > > > > There are many components outside of NiFi that are still using ZK > 3.4.6, > > so > > > it may be a while before we can move to 3.4.10. I don't currently know > > > anything about the forward compatibility of 3.4.6. Are there > > > improvements/fixes in 3.4.10 which you need? > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-3067 > > > [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 > > > [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 > > > [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 > > > [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 > > > > > > - Jeff > > > > > > On Tue, May 30, 2017 at 8:15 AM Mark Bean > wrote: > > > > > > > Updated to external ZooKeeper last Friday. Over the weekend, there > are > > no > > > > reports of SUSPENDED or RECONNECTED. > > > > > > > > Are there plans to upgrade the embedded ZooKeeper to the latest > > version, > > > > 3.4.10? > > > > > > > > Thanks, > > > > Mark > > > > > > > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt > wrote: > > > > > > > > > looked at a secured cluster and the send times are routinely at > 100ms > > > > > similar to yours. I think what i was flagging as potentially > > > > > interesting is not interesting at all. > > > > > > > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt > > wrote: > > > > > > Ok. Well as a point of comparison i'm looking at heartbeat logs > > from > > > > > > another cluster and the times are consistently 1-3 millis for the > > > > > > send. Yours above show 100+ms typical with one north of 900ms. > > Not > > > > > > sure how relevant that is but something i noticed. > > > > > > > > > > > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean < > mark.o.b...@gmail.com > > > > > > > > wrote: > > > > > >> ping shows acceptably fast response time between servers, > > > > approximately > > > > > >> 0.100-0.150 ms > > > > > >> > > > > > >> > > > > > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt > > > > wrote: > > > > > >> > > > > > >>> have you evaluated latency across the machines in your cluster? > > I > > > > ask > > > > > >>> because 122ms is pretty long and 917ms is very long. Are these > > > nodes > > > > > >>> across a WAN link? > > > > > >>> > > > > > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean < > > mark.o.b...@gmail.com > > > > > > > > > wrote: > > > > > >>> > Update: now all 5 nodes, regardless of ZK server, are > > indicating > > > > > >>> SUSPENDED > > > > > >>> > -> RECONNECTED. > > > > > >>> > > > > > > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean < > > > mark.o.b...@gmail.com > > > > > > > > > > >>> wrote: > > > > > >>> > > > > > > >>> >> I reduced the number of embedded ZooKeeper servers on the > > 5-Node > > > > > NiFi > > > > > >>> >> Cluster from 5 to 3. This has improved the situation. I do > not > > > see > > > > > any > > > > > >>> of > > > > > >>> >> the three Nodes which are also ZK servers > > > > > disconnecting/reconnecting to > > > > > >>> the > > > > > >>> >> cluster as bef
Re: unstable cluster
Joe, My own direct and indirect experiences with NiFi 1.x clustering have been good for both embedded and external zookeeper but we have certainly seen some emails on mailing-list about it. Those have been for high load case where the embedded approach would be susceptible to timing issues and resolved by using an external system. Mark Bean's report is interesting though since it happens under no real load at all. I suspect ZOOKEEPER-2044 will help that though there are several comments [1] (and others on that JIRA) that describe the issue as minor/false reporting/cosmetic/an improvement. Updating to ZooKeeper 3.4.10 suggests that this rare issue can be resolved in NiFi, but we'll have to do our due diligence to make sure that no new issues are raised with the upgrade for NiFi or its ability to interface with external systems. We'll have to do testing with other dependencies that use ZooKeeper 3.4.6 to ensure that forward capability. [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2044?focusedCommentId=15024616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15024616 Thanks, Jeff On Tue, May 30, 2017 at 1:15 PM Joe Skora wrote: > Jeff, > > If I understand the issue correctly, this means NiFi 1.x has always been > broken for clustering with an embedded ZooKeeper. That has never > communicated until now, we clearly build for and explain how to use an > embedded ZooKeeper in documentation. > > Any external non-NiFi elements that are considered in design and dependency > decisions need to be clearly understood by the entire community. What > things non-NiFi are you thinking of that drive ZooKeeper dependencies? > > Joe > > On Tue, May 30, 2017 at 9:11 AM, Jeff wrote: > > > Mark, we can certainly take smaller steps rather than waiting for > > 3.5.2/3.6.0 to come out. I was just bringing that JIRA up as another > > scenario that entices us to upgrade. > > > > Joe, I'm referring to NiFi, the toolkit, and things non-NiFi that > provide a > > ZK server to which NiFi or the ZK Migration Toolkit are clients. I'm not > > saying we can't or shouldn't upgrade, but we do need to test to make sure > > that no issues are introduced by NiFi shipping with ZK 3.4.10. Being > that > > it's a bugfix version change, it's probably fine. > > > > - Jeff > > > > On Tue, May 30, 2017 at 10:46 AM Joe Skora wrote: > > > > > Jeff, > > > > > > Does that mean NiFi 1.x will be unstable when using embedded ZooKeeper > > > until the ZK version is upgrade? > > > > > > By "components outside of NiFi" do you mean the NiFi toolkit and other > > > parts of the NiFi release? > > > > > > Joe > > > > > > On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > > > > > > > Mark, > > > > > > > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to > > > log4j > > > > issues) once it's out and stable, There are issues with the way that > ZK > > > > refers to log4j classes in the code that cause issues for NiFi and > our > > > > Toolkit.. However there has been some back and forth [2] (in 3.4.0, > > > which > > > > doesn't fix the issue, but moves towards fixing it), [3], and [4] on > > the > > > > changes being implemented in versions 3.5.2 and 3.6.0. Also, it > looks > > > like > > > > ZK 3.6.0 is headed toward using log4j 2 [5]. > > > > > > > > There are many components outside of NiFi that are still using ZK > > 3.4.6, > > > so > > > > it may be a while before we can move to 3.4.10. I don't currently > know > > > > anything about the forward compatibility of 3.4.6. Are there > > > > improvements/fixes in 3.4.10 which you need? > > > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-3067 > > > > [2] https://issues.apache.org/jira/browse/ZOOKEEPER-850 > > > > [3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371 > > > > [4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393 > > > > [5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342 > > > > > > > > - Jeff > > > > > > > > On Tue, May 30, 2017 at 8:15 AM Mark Bean > > wrote: > > > > > > > > > Updated to external ZooKeeper last Friday. Over the weekend, there > > are > > > no > > > > > reports of SUSPENDED or RECONNECTED. > > > > > > > > > > Are there plans to upgrade the embedded ZooKeeper to the latest > > > version, > > > > > 3.4.10? > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > On Thu, May 25, 2017 at 11:56 AM, Joe Witt > > wrote: > > > > > > > > > > > looked at a secured cluster and the send times are routinely at > > 100ms > > > > > > similar to yours. I think what i was flagging as potentially > > > > > > interesting is not interesting at all. > > > > > > > > > > > > On Thu, May 25, 2017 at 11:34 AM, Joe Witt > > > wrote: > > > > > > > Ok. Well as a point of comparison i'm looking at heartbeat > logs > > > from > > > > > > > another cluster and the times are consistently 1-3 millis for > the > > > > > > > send. Yours above show 100+ms typical with one north of 900ms. > > > Not > > > > > > > sure how r
Re: unstable cluster
Hello all, I'll like to chime in on this interesting discussion thread. I'll like to add that my system(s) too have seen unstable ZK interaction with both embedded and eventually external ZK ( granted external has been better ) interaction. We have resolved them with NIFI restarts. And it's to the point that we are hesitant to roll up to NIFI 1.X mainly because of this ( we have DEV NIFI 1.X ) I also would like to add that we are greatly anticipating ZK release 3.5.X for its TLS implementation, and as such have not voiced our experience with NIFI / ZOOKEEPER assuming that once ZOOKEEPER 3.5.X is out of ALPHA that it would be added in to NIFI NAR framework fairly fast and fix the oddities. I would say though that we have been hoping for a newer client on NIFI ZK side since the current one suggests its based off 3.4.6 ZOOKEEPER which was released on *MAR 2014*. # jar tc nifi-framework-nar-1.1.1.nar | grep zoo META-INF/bundled-dependencies/zookeper-3.4.6.jar And now I wonder how long it would take for NIFI to code release a client based off 3.5.X once it goes official given hesitation on forward capability. On Tue, May 30, 2017 at 2:52 PM Jeff wrote: > Joe, > > My own direct and indirect experiences with NiFi 1.x clustering have been > good for both embedded and external zookeeper but we have certainly seen > some emails on mailing-list about it. Those have been for high load case > where the embedded approach would be susceptible to timing issues and > resolved by using an external system. Mark Bean's report is interesting > though since it happens under no real load at all. > > I suspect ZOOKEEPER-2044 will help that though there are several comments > [1] (and others on that JIRA) that describe the issue as minor/false > reporting/cosmetic/an improvement. Updating to ZooKeeper 3.4.10 suggests > that this rare issue can be resolved in NiFi, but we'll have to do our due > diligence to make sure that no new issues are raised with the upgrade for > NiFi or its ability to interface with external systems. We'll have to do > testing with other dependencies that use ZooKeeper 3.4.6 to ensure that > forward capability. > > [1] > > https://issues.apache.org/jira/browse/ZOOKEEPER-2044?focusedCommentId=15024616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15024616 > > Thanks, > Jeff > > On Tue, May 30, 2017 at 1:15 PM Joe Skora wrote: > > > Jeff, > > > > If I understand the issue correctly, this means NiFi 1.x has always been > > broken for clustering with an embedded ZooKeeper. That has never > > communicated until now, we clearly build for and explain how to use an > > embedded ZooKeeper in documentation. > > > > Any external non-NiFi elements that are considered in design and > dependency > > decisions need to be clearly understood by the entire community. What > > things non-NiFi are you thinking of that drive ZooKeeper dependencies? > > > > Joe > > > > On Tue, May 30, 2017 at 9:11 AM, Jeff wrote: > > > > > Mark, we can certainly take smaller steps rather than waiting for > > > 3.5.2/3.6.0 to come out. I was just bringing that JIRA up as another > > > scenario that entices us to upgrade. > > > > > > Joe, I'm referring to NiFi, the toolkit, and things non-NiFi that > > provide a > > > ZK server to which NiFi or the ZK Migration Toolkit are clients. I'm > not > > > saying we can't or shouldn't upgrade, but we do need to test to make > sure > > > that no issues are introduced by NiFi shipping with ZK 3.4.10. Being > > that > > > it's a bugfix version change, it's probably fine. > > > > > > - Jeff > > > > > > On Tue, May 30, 2017 at 10:46 AM Joe Skora wrote: > > > > > > > Jeff, > > > > > > > > Does that mean NiFi 1.x will be unstable when using embedded > ZooKeeper > > > > until the ZK version is upgrade? > > > > > > > > By "components outside of NiFi" do you mean the NiFi toolkit and > other > > > > parts of the NiFi release? > > > > > > > > Joe > > > > > > > > On Tue, May 30, 2017 at 5:42 AM, Jeff wrote: > > > > > > > > > Mark, > > > > > > > > > > I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due > to > > > > log4j > > > > > issues) once it's out and stable, There are issues with the way > that > > ZK > > > > > refers to log4j classes in the code that cause issues for NiFi and > > our > > > > > Toolkit.. However there has been some back and forth [2] (in > 3.4.0, > > > > which > > > > > doesn't fix the issue, but moves towards fixing it), [3], and [4] > on > > > the > > > > > changes being implemented in versions 3.5.2 and 3.6.0. Also, it > > looks > > > > like > > > > > ZK 3.6.0 is headed toward using log4j 2 [5]. > > > > > > > > > > There are many components outside of NiFi that are still using ZK > > > 3.4.6, > > > > so > > > > > it may be a while before we can move to 3.4.10. I don't currently > > know > > > > > anything about the forward compatibility of 3.4.6. Are there > > > > > improvements/fixes in 3.4.10 which you need? > > > > > > > >
Example Processors that....
Does anyone know of a processor that uses a file or files loaded into the system as part of it’s configuration? I’m looking for best practices. These files do not need to be in the repo or transferred. Thanks in advance.
Re: Example Processors that....
Hi Otto, Do you mean something specific by "files loaded into the system" or are you just referring to a processor having a property that points to an external file? There are a few examples of the latter in the code base, you can look for usage of FILE_EXISTS_VALIDATOR as an easy way to find them: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateXml.java#L63-L68 Let us know if that is not what you meant. Thanks, Bryan On Tue, May 30, 2017 at 3:59 PM, Otto Fowler wrote: > Does anyone know of a processor that uses a file or files loaded into the > system as part of it’s configuration? I’m looking for best practices. > These files do not need to be in the repo or transferred. > > Thanks in advance.
Re: Example Processors that....
Thanks, I’ll take a look! On May 30, 2017 at 16:12:22, Bryan Bende (bbe...@gmail.com) wrote: Hi Otto, Do you mean something specific by "files loaded into the system" or are you just referring to a processor having a property that points to an external file? There are a few examples of the latter in the code base, you can look for usage of FILE_EXISTS_VALIDATOR as an easy way to find them: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateXml.java#L63-L68 Let us know if that is not what you meant. Thanks, Bryan On Tue, May 30, 2017 at 3:59 PM, Otto Fowler wrote: > Does anyone know of a processor that uses a file or files loaded into the > system as part of it’s configuration? I’m looking for best practices. > These files do not need to be in the repo or transferred. > > Thanks in advance.
Re: Example Processors that....
Bryan, This will be a great start for what I’m looking for. Thank you! On May 30, 2017 at 17:23:47, Otto Fowler (ottobackwa...@gmail.com) wrote: Thanks, I’ll take a look! On May 30, 2017 at 16:12:22, Bryan Bende (bbe...@gmail.com) wrote: Hi Otto, Do you mean something specific by "files loaded into the system" or are you just referring to a processor having a property that points to an external file? There are a few examples of the latter in the code base, you can look for usage of FILE_EXISTS_VALIDATOR as an easy way to find them: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateXml.java#L63-L68 Let us know if that is not what you meant. Thanks, Bryan On Tue, May 30, 2017 at 3:59 PM, Otto Fowler wrote: > Does anyone know of a processor that uses a file or files loaded into the > system as part of it’s configuration? I’m looking for best practices. > These files do not need to be in the repo or transferred. > > Thanks in advance.
RE: [EXT] Re: NiFi Processors show 30 Second Execution time, 0 executions
Thanks Matt. Interestingly enough I'm at +09:00, and not a half hour interval, but hopefully this will fix it regardless. -Original Message- From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: Tuesday, May 30, 2017 9:05 PM To: dev@nifi.apache.org Subject: [EXT] Re: NiFi Processors show 30 Second Execution time, 0 executions I do not believe this issue has been addressed yet. There is an open JIRA [1]. Matt [1] https://issues.apache.org/jira/browse/NIFI-3719 On Sun, May 28, 2017 at 10:16 PM, Joe Witt wrote: > Peter > > Probably best to go ahead and file a JIRA. In it you can reliably > post the attachments. There was a potentially related timezone > handling issue as I recall in this past release so perhaps there is > some relationship. > > Thanks > Joe > > On Sun, May 28, 2017 at 10:04 PM, Peter Wicks (pwicks) > wrote: > > I wanted to re-open this discussion, it's been a while and I'm still > seeing > > the issue even with the latest version. I'm still seeing this issue > running > > a stock NiFi v1.2.0. By stock I mean no custom NAR’s, etc… just > > original vanilla code, in this case with no configuration, so > > running unsecured, empty canvas (except for my test case). > > > > > > > > I’ve expanded my test scenarios. > > > > > > > > Scenario 1 is Windows 7, code built using mvn, using Oracle Java. > > > > Java Version: > > > > java version "1.8.0_91" > > > > Java(TM) SE Runtime Environment (build 1.8.0_91-b15) > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.91-b15, mixed mode) > > > > > > > > Scenario 2 is RHEL 7.3, the NiFi build is v1.2.0 downloaded from the > > NiFi website. Running OpenJDK. > > > > > > > > openjdk version "1.8.0_102" > > > > OpenJDK Runtime Environment (build 1.8.0_102-b14) > > > > OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode) > > > > > > > > I’ve attempted to attach a screenshot (my attachments seem to not > > make it very often on this list). In it I show the onscreen > > Tasks/Time for two > > processors: one shows 1 / 00:30:04.292 and the other 0 / 00:30:00.000. > > > > > > > > Thanks! > > > > Peter > > > > > > > > -Original Message- > > From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > > Sent: Friday, March 31, 2017 10:54 PM > > To: dev@nifi.apache.org > > Subject: Re: NiFi Processors show 30 Second Execution time, 0 > > executions > > > > > > > > What version of Java are you running on ? Major_minor? > > > > > > > > On Fri, Mar 31, 2017 at 10:49 AM, Peter Wicks (pwicks) < > pwi...@micron.com> > > > > wrote: > > > > > > > >> I misread my own screenshot, it says 30 minutes, not seconds. Also, > >> I > > > >> did a restart of NiFi and opened it up in a fresh instance of > >> Chrome; > > > >> no change. I kicked off a GenerateFlowFile processor and the > > > >> milliseconds are going up, but the 30 minutes is remaining the same... > > > >> > > > >> > > > >> > > > >> -Original Message- > > > >> From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > >> Sent: Friday, March 31, 2017 10:41 PM > > > >> To: dev@nifi.apache.org > > > >> Subject: Re: NiFi Processors show 30 Second Execution time, 0 > > > >> executions > > > >> > > > >> Doing a clean build of 091359b450a7d0fb6bb04e2238c9171728cd2720, I > > > >> will have to see if I have a windows 7 VM anywhere, I know Witt was > > > >> using Win10 and didnt see it... I find it odd that ALL your > >> processors > > > >> have a 30 second number not just the UpdateAttribute. Anything else > > > >> about your environment you can share that may be unique? > > > >> > > > >> On Fri, Mar 31, 2017 at 10:15 AM, Peter Wicks (pwicks) > > > >> > > > >> wrote: > > > >> > > > >> > 091359b450a7d0fb6bb04e2238c9171728cd2720, so just one commit > >> > behind > > > >> > master. > > > >> > I am testing on Windows 7. > > > >> > > > > >> > Lee, yield isn't a bad idea, but UpdateAttribute in my screenshot > > > >> > has never run; not even once. I don't think it's had the > >> > opportunity > > > >> > to > > > >> yield. > > > >> > > > > >> > -Original Message- > > > >> > From: Joseph Niemiec [mailto:josephx...@gmail.com] > > > >> > Sent: Friday, March 31, 2017 9:37 PM > > > >> > To: dev@nifi.apache.org > > > >> > Subject: Re: NiFi Processors show 30 Second Execution time, 0 > > > >> > executions > > > >> > > > > >> > I just built the latest and am unable to see the issue as well. I > > > >> > also played with yield duration with no luck. > > > >> > > > > >> > Can you provide us what build your on so I can check that one out > > > >> exactly? > > > >> > I did my last trunk test as of - > > > >> > 6a64b3cd9cca70e6a27b9034eba520ae0c0cb6ca > > > >> > > > > >> > git rev-parse HEAD > > > >> > > > > >> > > > > >> > On Fri, Mar 31, 2017 at 9:21 AM, Lee Laim wrote: > > > >> > > > > >> > > Is it potentially related to yield duration? > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Mar 31, 2017, at 6:58 AM, Peter Wicks (pwicks) > > > >> > > > > > > >> > > wrote: > > > >> >
Re: unstable cluster
Just scanning through the items currently on master that would show up in the 1.3.0 release we see numerous cluster related bug fixes. More consistent port alignment across cluster https://issues.apache.org/jira/browse/NIFI-3981 Ensure controller service lifecycle handled better with different timing/dependencies https://issues.apache.org/jira/browse/NIFI-3972 Insufficient heartbeat handling causing improper clustering behavior https://issues.apache.org/jira/browse/NIFI-3933 Improve timing of component startup relative to other lifecycle items when clustered https://issues.apache.org/jira/browse/NIFI-3923 Inconsistent scheduled state in some cluster settings https://issues.apache.org/jira/browse/NIFI-3900 Improved fingerprinted/non-fingerprinted settings enforcement and handling in clusters https://issues.apache.org/jira/browse/NIFI-1963 These are nifi specific cluster behavior things. For nifi and zookeeper interaction specifically most of the focus this far has been about NiFi itself as the above JIRAs show and also of course the cases where a given system that is so resource contended will simply not have a nice embedded ZK/nifi experience. MarkB, your testing above suggests you were using a nifi 1.x which means a zookeeper 3.4.6 client against a Zookeeper 3.4.10 server cluster and behavior was much better. Could you possibly run the same cluster evaluation against the latest master but with an embedded zookeeper 3.4.10 version in nifi (which means both server and client are on latest zk 3.4.10 release)? This would be helpful data. Assuming that goes well the only other concern that jumps to mind is if us using a zookeeper 3.4.10 client presents problems for us talking to older server versions (still 3.4 though so probably ok, i'd hope). In general we should be safe thanks to classloader isolation but we've seen some pretty magical JVM/system classloader level changes happen for Kerberized environments. Thanks Joe On Tue, May 30, 2017 at 3:21 PM, Juan Sequeiros wrote: > Hello all, > > I'll like to chime in on this interesting discussion thread. > > I'll like to add that my system(s) too have seen unstable ZK interaction > with both embedded and eventually external ZK ( granted external has been > better ) interaction. > We have resolved them with NIFI restarts. And it's to the point that we are > hesitant to roll up to NIFI 1.X mainly because of this ( we have DEV NIFI > 1.X ) > > I also would like to add that we are greatly anticipating ZK release 3.5.X > for its TLS implementation, and as such have not voiced our experience with > NIFI / ZOOKEEPER assuming that once ZOOKEEPER 3.5.X is out of ALPHA that it > would be added in to NIFI NAR framework fairly fast and fix the oddities. > > I would say though that we have been hoping for a newer client on NIFI ZK > side since the current one suggests its based off 3.4.6 ZOOKEEPER which was > released on *MAR 2014*. > > # jar tc nifi-framework-nar-1.1.1.nar | grep zoo > META-INF/bundled-dependencies/zookeper-3.4.6.jar > > And now I wonder how long it would take for NIFI to code release a client > based off 3.5.X once it goes official given hesitation on forward > capability. > > > On Tue, May 30, 2017 at 2:52 PM Jeff wrote: > >> Joe, >> >> My own direct and indirect experiences with NiFi 1.x clustering have been >> good for both embedded and external zookeeper but we have certainly seen >> some emails on mailing-list about it. Those have been for high load case >> where the embedded approach would be susceptible to timing issues and >> resolved by using an external system. Mark Bean's report is interesting >> though since it happens under no real load at all. >> >> I suspect ZOOKEEPER-2044 will help that though there are several comments >> [1] (and others on that JIRA) that describe the issue as minor/false >> reporting/cosmetic/an improvement. Updating to ZooKeeper 3.4.10 suggests >> that this rare issue can be resolved in NiFi, but we'll have to do our due >> diligence to make sure that no new issues are raised with the upgrade for >> NiFi or its ability to interface with external systems. We'll have to do >> testing with other dependencies that use ZooKeeper 3.4.6 to ensure that >> forward capability. >> >> [1] >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-2044?focusedCommentId=15024616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15024616 >> >> Thanks, >> Jeff >> >> On Tue, May 30, 2017 at 1:15 PM Joe Skora wrote: >> >> > Jeff, >> > >> > If I understand the issue correctly, this means NiFi 1.x has always been >> > broken for clustering with an embedded ZooKeeper. That has never >> > communicated until now, we clearly build for and explain how to use an >> > embedded ZooKeeper in documentation. >> > >> > Any external non-NiFi elements that are considered in design and >> dependency >> > decisions need to be clearly understood by the entire community. What >> > things non-NiFi are you
