Morning, We started running 4.2.1 w/ Hazelcast (hz.cluster.tcpip.enabled=true) on Linux VMs (RedHat variant) a couple of weeks ago with three nodes on the same subnet. Things seemed fine initially, but a couple of days ago started getting cluster errors starting with heartbeat timeout, several (dis)connects, attempted repartitions, and ending with the cluster frozen.
Has anyone experience this? E.g. > 2016-06-01 21:01:25,330 WARN > [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [-------.50]:5701 > [dev] [3.6] Removing Member [------.55]:5701 because it has not sent any > heartbeats for 5000 ms. Last heartbeat time was Wed Jun 01 21:01:20 PDT 2016 > 2016-06-01 21:01:25,330 INFO [com.hazelcast.cluster.ClusterService] - > [------.50]:5701 [dev] [3.6] Old master Address[------.55]:5701 left the > cluster, assigning new master Member [128.120.39.50]:5701 this ... > 2016-06-01 21:01:29,167 WARN > [com.hazelcast.partition.InternalPartitionService] - [------.50]:5701 [dev] > [3.6] This is the master node and received a PartitionRuntimeState from > Address[------.55]:5701. Ignoring incoming state! ... > 2016-06-01 21:05:16,046 INFO > [com.hazelcast.cluster.impl.operations.JoinCheckOperation] - [------.50]:5701 > [dev] [3.6] Ignoring join check from Address[------.55]:5701, because cluster > is in FROZEN state ... Interestingly enough, if we shut down one of the nodes (leaving two), the issue does not recur--at least in the time we've been monitoring. The only recourse seems to be a full cluster restart. Thanks for any advice! Tom. -- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscr...@apereo.org. To post to this group, send email to cas-user@apereo.org. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/B8F20E5F-0BC3-44AE-B53F-BCFD1B181E3D%40ucdavis.edu. For more options, visit https://groups.google.com/a/apereo.org/d/optout.
