+ [email protected] On Fri, Jun 17, 2016 at 8:49 AM, Anthony Baker <[email protected]> wrote:
> I would start by turning on GC logging and see if you have any long > collections. A stop-the-world collection that takes longer than the member > timeout could cause the behavior you are observing. > > You may want to consider using an off-heap region. Also check out the > other eviction algorithms (entry count, region size) to see if those would > be appropriate. > > Anthony > > > On Jun 17, 2016, at 8:33 AM, Avinash Dongre <[email protected]> > wrote: > > > > Thanks Anthony > > > > *>>>> At the CRITICAL threshold further writes are blocked. * > > > > Does client get any kind of warning/error or does put or putAll fails. > > My client is just putting KVs and not aware the server State. and I am > > getting > > > > com.gemstone.gemfire.ForcedDisconnectException: Member isn't responding > to > > heartbeat requests, and after this server is attempting to reconnect. > > > > Thanks > > Avinash > > > > > > > > On Fri, Jun 17, 2016 at 7:50 PM, Anthony Baker <[email protected]> > wrote: > > > >> You need enough heap to retain all your keys in memory. Values begin to > >> be evicted at the EVICTION threshold. At the CRITICAL threshold further > >> writes are blocked. Your GC settings should be tuned to match these > >> thresholds (search the mailing list archives and/or wiki for some > advice on > >> GC tuning). > >> > >> Anthony > >> > >>> On Jun 17, 2016, at 6:42 AM, Avinash Dongre <[email protected]> > >> wrote: > >>> > >>> Thanks Anthony, > >>> > >>> I know that my heap is not sufficient for the data I want to put. > >>> but My Region is configured as PARTITION_OVERFLOW, so I was hoping that > >>> once Geode reaches to critical heap , it will start overflowing > >>> the data. > >>> > >>> Is this assumption correct ? > >>> > >>> thanks > >>> Avinash > >>> > >>> > >>> On Thu, Jun 16, 2016 at 8:21 PM, Anthony Baker <[email protected]> > >> wrote: > >>> > >>>> Hi Avinash, > >>>> > >>>> The question to answer is “Why was a member removed from the cluster?” > >>>> Some things to investigate: > >>>> > >>>> - Insufficient heap for the data volume > >>>> - Excessive GC causing the member to be unresponsive > >>>> - OutOfMemory errors in the log > >>>> - Overloaded CPU causing delayed heartbeats responses > >>>> > >>>> HTH, > >>>> Anthony > >>>> > >>>>> On Jun 16, 2016, at 6:48 AM, Avinash Dongre < > [email protected]> > >>>> wrote: > >>>>> > >>>>> Hello All, > >>>>> > >>>>> I am getting following exception when I try to load my system with > >> large > >>>>> amount of data. > >>>>> > >>>>> My Setup Details: > >>>>> 1 locator, 3 cacheservers with 8g and all the regions are disk > >>>> persistence > >>>>> enabled. ( All this is running on single AWS cluster node ) > >>>>> > >>>>> Please give me some clues what I am missing here ? > >>>>> > >>>>> > >>>>> [severe 2016/06/16 12:51:15.552 UTC S1 <Notification Handler> > tid=0x40] > >>>>> Uncaught exception in thread Thread[Notification > >>>>> Handler,10,ResourceListenerInvokerThreadGroup] > >>>>> > >> com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: > >>>>> DistributedSystem is shutting down, caused by > >>>>> com.gemstone.gemfire.ForcedDisconnectException: Member isn't > responding > >>>> to > >>>>> heartbeat requests > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1719) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1897) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.DistributionChannel.send(DistributionChannel.java:87) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3427) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3468) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1828) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.ResourceAdvisor$ResourceProfileMessage.send(ResourceAdvisor.java:185) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.ResourceAdvisor.updateRemoteProfile(ResourceAdvisor.java:448) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.processLocalEvent(HeapMemoryMonitor.java:677) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:485) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:448) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor$2.run(HeapMemoryMonitor.java:718) > >>>>> at > >>>>> > >>>> > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > >>>>> at > >>>>> > >>>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > >>>>> at java.lang.Thread.run(Thread.java:745) > >>>>> Caused by: com.gemstone.gemfire.ForcedDisconnectException: Member > isn't > >>>>> responding to heartbeat requests > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2551) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:885) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:578) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1540) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1061) > >>>>> at org.jgroups.JChannel.invokeCallback(JChannel.java:816) > >>>>> at org.jgroups.JChannel.up(JChannel.java:741) > >>>>> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) > >>>>> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) > >>>>> at org.jgroups.protocols.FlowControl.up(FlowControl.java:392) > >>>>> at > >>>> org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064) > >>>>> at > >>>>> org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779) > >>>>> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:69) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:74) > >>>>> at org.jgroups.protocols.TP.passMessageUp(TP.java:1567) > >>>>> at org.jgroups.protocols.TP > >>>> $SingleMessageHandler.run(TP.java:1783) > >>>>> at > >> org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) > >>>>> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1695) > >>>>> at org.jgroups.protocols.TP.receive(TP.java:1620) > >>>>> at > >>>>> > >>>> > >> > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:158) > >>>>> at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) > >>>>> ... 1 more > >>>> > >>>> > >> > >> > >
