Hi Azeez,

If two members in a cluster cannot communicate with each other for few mins
(lets say 5 mins), can that end up with the members going to an OOM
situation.

Recently we experience an OOM situation a api-gateway cluster (not under a
heavy load). When checking the logs, we found the following log being
printed many times around the time of incidents.

TID: [0] [AM] [2015-10-23 21:46:51,424] ERROR
{org.wso2.carbon.caching.impl.CacheCleanupTask} -  Error occurred while
running CacheCleanupTask {org.wso2.carbon.caching.impl.CacheCleanupTask}

com.hazelcast.core.OperationTimeoutException: No response for 120000 ms.
Aborting invocation! Invocation{ serviceName='hz:impl:mapService',
op=SetOperation{$cache.$domain[clouddemo]registryCacheManager#$_timestamp_$REG_PATH_CACHE},
partitionId=30, replicaIndex=0, tryCount=250, tryPauseMillis=500,
invokeCount=1, callTimeout=60000, target=Address[10.0.11.12]:4000,
backupsExpected=0, backupsCompleted=0} No response has been received!
backups-expected:0 backups-completed: 0

        at
com.hazelcast.spi.impl.operationservice.impl.Invocation.newOperationTimeoutException(Invocation.java:491)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:277)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)

        at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435)

        at
com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487)

        at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171)

        at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161)

        at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149)

        at
org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939)

        at
org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

        at ------ End remote and begin local stack-trace ------.(Unknown
Source)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)

        at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)

        at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:435)

        at
com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:487)

        at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:171)

        at
com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:161)

        at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.put(HazelcastDistributedMapProvider.java:149)

        at
org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:939)

        at
org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

-- 
*Amila Maharachchi*
Senior Technical Lead
WSO2, Inc.; http://wso2.com

Blog: http://maharachchi.blogspot.com
Mobile: +94719371446
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Reply via email to