Yes, we are using the embedded ZK. We will try instantiating and external ZK and see if that resolves the problem.
The load on the system is extremely small. Currently (as Nodes are disconnecting/reconnecting) all input ports to the flow are turned off. The only data in the flow is from a single GenerateFlow generating 5B every 30 secs. Also, it is a 5-node cluster with embedded ZK on each node. First, I will try reducing ZK to only 3 nodes. Then, I will try a 3-node external ZK. Thanks, Mark On Wed, May 24, 2017 at 11:49 AM, Joe Witt <joe.w...@gmail.com> wrote: > Are you using the embedded Zookeeper? If yes we recommend using an > external zookeeper. > > What type of load are the systems under when this occurs (cpu, > network, memory, disk io)? Under high load the default timeouts for > clustering are too aggressive. You can relax these for higher load > clusters and should see good behavior. Even if the system overall is > not under all that high of load if you're seeing garbage collection > pauses that are lengthy and/or frequent it can cause the same high > load effect as far as the JVM is concerned. > > Thanks > Joe > > On Wed, May 24, 2017 at 9:11 AM, Mark Bean <mark.o.b...@gmail.com> wrote: > > We have a cluster which is showing signs of instability. The Primary Node > > and Coordinator are reassigned to different nodes every several minutes. > I > > believe this is due to lack of heartbeat or other coordination. The > > following error occurs periodically in the nifi-app.log > > > > ERROR [CommitProcessor:1] o.apache.zookeeper.server.NIOServerCnxn > > Unexpected Exception: > > java.nio.channels.CancelledKeyException: null > > at sun.nio.ch.SelectionKeyImpl.ensureValid(SectionKeyImpl. > java:73) > > at sun.nio.ch.SelectionKeyImpl.interestOps(SelctionKeyImpl. > java:77) > > at > > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer( > NIOServerCnxn.java:151) > > at > > org.apache.zookeeper.server.NIOServerCnXn.sendResopnse( > NIOServerCnxn.java:1081) > > at > > org.apache.zookeeper.server.FinalRequestProcessor.processRequest( > FinalRequestProcessor.java:404) > > at > > org.apache.zookeeper.server.quorum.CommitProcessor.run( > CommitProcessor.java:74) > > > > Apache NiFi 1.2.0 > > > > Thoughts? >
