Hi Azeez,
If two members in a cluster cannot communicate with each other for few mins
(lets say 5 mins), can that end up with the members going to an OOM
situation.
Recently we experience an OOM situation a api-gateway cluster (not under a
heavy load). When checking the logs, we found the following log being
printed many times around the time of incidents.
TID: [0] [AM] [2015-10-23 21:46:51,424] ERROR
{org.wso2.carbon.caching.impl.CacheCleanupTask} - Error occurred while
running CacheCleanupTask {org.wso2.carbon.caching.impl.CacheCleanupTask}
com.hazelcast.core.OperationTimeoutException: No response for 120000 ms.
Aborting invocation! Invocation{ serviceName='hz:impl:mapService',
op=SetOperation{$cache.$domain[clouddemo]registryCacheManager#$_timestamp_$REG_PATH_CACHE},
partitionId=30, replicaIndex=0, tryCount=250, tryPauseMillis=500,
invokeCount=1, callTimeout=60000, target=Address[10.0.11.12]:4000,
backupsExpected=0, backupsCompleted=0} No response has been received!
backups-expected:0 backups-completed: 0
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.newOperationTimeoutException(Invocation.java:491)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:277)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435)
at
com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487)
at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171)
at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161)
at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149)
at
org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939)
at
org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at ------ End remote and begin local stack-trace ------.(Unknown
Source)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435)
at
com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487)
at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171)
at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161)
at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149)
at
org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939)
at
org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--
*Amila Maharachchi*
Senior Technical Lead
WSO2, Inc.; http://wso2.com
Blog: http://maharachchi.blogspot.com
Mobile: +94719371446
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
