Ok, better to check the heap dumps then.
Regards,
Manoj
*Manoj Kumara*
WSO2 Inc. *| **lean. enterprise. middleware.*
*Mobile:* +94 713 448188
On Wed, Nov 4, 2015 at 11:49 AM, Milinda Perera <milin...@wso2.com> wrote:
> Hi Manoj,
>
> Yes it contains hazelcast_3.5.2.wso2v1
>
> Thanks,
> Milinda
>
> On Wed, Nov 4, 2015 at 5:13 PM, Manoj Kumara <ma...@wso2.com> wrote:
>
>> Can you check the pack contain the Hazelcast 3.5.2 upgrade patch.
>>
>> Regards,
>> Manoj
>>
>> *Manoj Kumara*
>> WSO2 Inc. *| **lean. enterprise. middleware.*
>> *Mobile:* +94 713 448188
>>
>> On Wed, Nov 4, 2015 at 11:30 AM, Milinda Perera <milin...@wso2.com>
>> wrote:
>>
>>> Hi,
>>>
>>> When we perform load test for BRS 2.2.0 snapshot in 2 node cluster setup
>>> (for [1]) with simple rule, we get following error and one node goes OOM
>>> (Below log is from Master node. In another test worker node goes OOM and
>>> ERROR {org.wso2.carbon.caching.impl.CacheCleanupTask} - Error occurred
>>> while running CacheCleanupTask is shown in the Master).
>>>
>>> FYI : Currently we have upgraded BRS to carbon kernel 4.4.2
>>>
>>> [2015-11-04 13:02:17,583] ERROR
>>> {org.wso2.carbon.caching.impl.CacheCleanupTask} - Error occurred while
>>> running CacheCleanupTask
>>> com.hazelcast.core.OperationTimeoutException: No response for 120000 ms.
>>> Aborting invocation! Invocation{ serviceName='hz:impl:mapService',
>>> op=ClearOperation{}, partitionId=0, replicaIndex=0, tryCount=250,
>>> tryPauseMillis=500, invokeCount=1, callTimeout=60000,
>>> target=Address[10.100.5.92]:4001, backupsExpected=0, backupsCompleted=0} No
>>> response has been received! backups-expected:0 backups-completed: 0
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.Invocation.newOperationTimeoutException(Invocation.java:491)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:277)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.retryFailedPartitions(InvokeOnPartitions.java:131)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invoke(InvokeOnPartitions.java:67)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:326)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxySupport.clearInternal(MapProxySupport.java:914)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxyImpl.clearInternal(MapProxyImpl.java:71)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxyImpl.clear(MapProxyImpl.java:532)
>>> at
>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.clear(HazelcastDistributedMapProvider.java:172)
>>> at org.wso2.carbon.caching.impl.CacheImpl.stop(CacheImpl.java:734)
>>> at
>>> org.wso2.carbon.caching.impl.CarbonCacheManager.removeCache(CarbonCacheManager.java:168)
>>> at org.wso2.carbon.caching.impl.CacheImpl.expire(CacheImpl.java:769)
>>> at
>>> org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:931)
>>> at
>>> org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> at ------ End remote and begin local stack-trace ------.(Unknown
>>> Source)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.retryFailedPartitions(InvokeOnPartitions.java:131)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invoke(InvokeOnPartitions.java:67)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:326)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxySupport.clearInternal(MapProxySupport.java:914)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxyImpl.clearInternal(MapProxyImpl.java:71)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxyImpl.clear(MapProxyImpl.java:532)
>>> at
>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.clear(HazelcastDistributedMapProvider.java:172)
>>> at org.wso2.carbon.caching.impl.CacheImpl.stop(CacheImpl.java:734)
>>> at
>>> org.wso2.carbon.caching.impl.CarbonCacheManager.removeCache(CarbonCacheManager.java:168)
>>> at org.wso2.carbon.caching.impl.CacheImpl.expire(CacheImpl.java:769)
>>> at
>>> org.wso2.carbon.caching.impl.CacheImpl.runCacheExpiry(CacheImpl.java:931)
>>> at
>>> org.wso2.carbon.caching.impl.CacheCleanupTask.run(CacheCleanupTask.java:61)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to
>>> /home/milinda/WSO2/BPS/BRS/cluster/ClusterScript/WSO2BRS_CLUSTER/MASTER/wso2brs-2.2.0-SNAPSHOT/repository/logs/heap-dump.hprof
>>> ...
>>> Heap dump file created [1072448408 bytes in 37.326 secs]
>>> java.lang.OutOfMemoryError: Java heap space
>>> [2015-11-04 13:09:02,283] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.partition.impl.SyncReplicaVersion.createCheckReplicaVersion(SyncReplicaVersion.java:103)
>>> [2015-11-04 13:09:35,256] ERROR
>>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - Error
>>> while running deployment scheduler..
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.partition.impl.SyncReplicaVersion.invokeCheckReplicaVersion(SyncReplicaVersion.java:79)
>>> at
>>> com.hazelcast.partition.impl.SyncReplicaVersion.run(SyncReplicaVersion.java:70)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:137)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processOperation(OperationThread.java:154)
>>> [2015-11-04 13:10:50,321] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:110)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>>> Exception in thread "RMI RenewClean-[127.0.1.1:11111]" [2015-11-04
>>> 13:13:30,851] ERROR {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> Exception in thread "PoolCleaner[5658920:1446620775007]" [2015-11-04
>>> 13:14:02,061] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> [2015-11-04 13:14:27,858] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.toByteArray(ByteArrayObjectDataOutput.java:345)
>>> [2015-11-04 13:14:55,185] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
>>> [2015-11-04 13:16:04,942] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> [2015-11-04 13:16:04,942] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:233)
>>> at
>>> com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:207)
>>> [2015-11-04 13:16:10,601] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:194)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.send(OperationServiceImpl.java:357)
>>> at
>>> com.hazelcast.partition.impl.SyncReplicaVersion.invokeCheckReplicaVersion(SyncReplicaVersion.java:89)
>>> at
>>> com.hazelcast.partition.impl.SyncReplicaVersion.run(SyncReplicaVersion.java:70)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:137)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processOperation(OperationThread.java:154)
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:110)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)[2015-11-04
>>> 13:18:02,568] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>>> [2015-11-04 13:21:19,509] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.partition.PartitionRuntimeState.setPartitions(PartitionRuntimeState.java:73)
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.partition.PartitionRuntimeState.<init>(PartitionRuntimeState.java:61)
>>> [2015-11-04 13:21:07,693] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> [2015-11-04 13:20:11,152] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> [2015-11-04 13:18:07,393] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> [2015-11-04 13:18:04,470] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.partition.impl.InternalPartitionServiceImpl.createPartitionState(InternalPartitionServiceImpl.java:486)[2015-11-04
>>> 13:30:11,421] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> [2015-11-04 13:30:04,959] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> Exception in thread
>>> "hz.wso2.carbon.domain.instance.SlowOperationDetectorThread" at
>>> com.hazelcast.partition.impl.InternalPartitionServiceImpl.publishPartitionRuntimeState(InternalPartitionServiceImpl.java:510)
>>> at
>>> com.hazelcast.partition.impl.InternalPartitionServiceImpl.access$800(InternalPartitionServiceImpl.java:109)[2015-11-04
>>> 13:29:01,781] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.partition.impl.InternalPartitionServiceImpl$SendClusterStateTask.run(InternalPartitionServiceImpl.java:1592)
>>> at
>>> com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)[2015-11-04 13:28:43,809]
>>> ERROR {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
>>> at
>>> com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.ThreadDeath[2015-11-04 13:28:10,538] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> Exception in thread "[Timer] - Main Queue Handler" SEVERE:Memory usage
>>> is low, parachute is non existent, your system may start failing.
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at java.lang.Thread.stop(Thread.java:836)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.handleStuckThread(CarbonStuckThreadDetectionValve.java:125)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.backgroundProcess(CarbonStuckThreadDetectionValve.java:178)
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1368)
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> at
>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> java.lang.OutOfMemoryError: Java heap space
>>> SEVERE:Memory usage is low, parachute is non existent, your system may
>>> start failing.
>>> [2015-11-04 13:28:10,538] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$SocketProcessor} -
>>> [2015-11-04 13:28:00,993] ERROR {unknown} -
>>> [2015-11-04 13:27:58,984] ERROR {com.hazelcast.nio.tcp.WriteHandler} -
>>> [10.100.5.92]:4000 [wso2.carbon.domain] [3.5.2] Fatal Error at WriteHandler
>>> for endPoint: Address[10.100.5.92]:4001
>>> [2015-11-04 13:27:42,818] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
>>> [2015-11-04 13:27:39,982] ERROR {unknown} -
>>> [2015-11-04 13:23:54,357] ERROR
>>> {org.apache.tomcat.util.net.NioEndpoint$Acceptor} -
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> Exception in thread "http-nio-9763-exec-211" java.lang.OutOfMemoryError:
>>> Java heap space
>>>
>>>
>>> Any Idea which cause this?
>>> Does hazelcast caused for the OOM?
>>>
>>> [1] https://wso2.org/jira/browse/BRS-100
>>>
>>> Thanks,
>>> Milinda
>>>
>>> --
>>> Milinda Perera
>>> Software Engineer;
>>> WSO2 Inc. http://wso2.com ,
>>> Mobile: (+94) 714 115 032
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>
>
> --
> Milinda Perera
> Software Engineer;
> WSO2 Inc. http://wso2.com ,
> Mobile: (+94) 714 115 032
>
>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
