Hi Azeez, If two members in a cluster cannot communicate with each other for few mins (lets say 5 mins), can that end up with the members going to an OOM situation.
Recently we experience an OOM situation a api-gateway cluster (not under a heavy load). When checking the logs, we found the following log being printed many times around the time of incidents. TID: [0] [AM] [2015-10-23 21:46:51,424] ERROR {org.wso2.carbon.caching.impl.CacheCleanupTask} - Error occurred while running CacheCleanupTask {org.wso2.carbon.caching.impl.CacheCleanupTask} com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! Invocation{ serviceName='hz:impl:mapService', op=SetOperation{$cache.$domain[clouddemo]registryCacheManager#$_timestamp_$REG_PATH_CACHE}, partitionId=30, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.0.11.12]:4000, backupsExpected=0, backupsCompleted=0} No response has been received! backups-expected:0 backups-completed: 0 at com.hazelcast.spi.impl.operationservice.impl.Invocation.newOperationTimeoutException(Invocation.java:491) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:277) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435) at com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487) at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171) at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161) at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149) at org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939) at org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) at ------ End remote and begin local stack-trace ------.(Unknown Source) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435) at com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487) at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171) at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161) at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149) at org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939) at org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- *Amila Maharachchi* Senior Technical Lead WSO2, Inc.; http://wso2.com Blog: http://maharachchi.blogspot.com Mobile: +94719371446
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev