Lets analyse the heap dump and see what caused the issue... also check the metrics recorded (via matric library) when server went OOM.
On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage <pum...@wso2.com> wrote: > Hi, > > I came across out of memory exception while doing long running test on mb > cluster with oracle database as message store. Following was the scenario i > tested. > > 1. Start two clustered mb nodes with serializers (nodeA(coordinator), > nodeB). > 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" > messages to nodeA with average 1500 TPS. > 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" > messages to nodeB with average 1000 TPS. > 4. After around 14 hours, om exception has thrown from nodeA and all > operations has halted. > > After analysing heap dump i found which thread has thrown the error, and > it's as follows. > > "pool-25-thread-1" prio=5 tid=174 RUNNABLE > at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48) > at java.lang.Long.toString(Long.java:267) > at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) > Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 > Local Variable: com.google.gson.JsonArray#1 > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) > at com.google.gson.internal.Streams.parse(Streams.java:44) > at com.google.gson.JsonParser.parse(JsonParser.java:84) > at com.google.gson.JsonParser.parse(JsonParser.java:59) > Local Variable: com.google.gson.stream.JsonReader#1 > at com.google.gson.JsonParser.parse(JsonParser.java:45) > at > org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) > at > org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) > at > com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) > Local Variable: > com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 > at > com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) > at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) > at com.hazelcast.map.MapService.toObject(MapService.java:872) > at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) > at > org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) > Local Variable: java.util.TreeSet#1 > at > org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) > Local Variable: java.lang.String#2380 > Local Variable: java.lang.String#2379 > at > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) > Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 > at > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) > Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 > Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) > Local Variable: org.apache.thrift.transport.TSocket#2 > Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 > Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 > Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > Local Variable: > org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 > Local Variable: java.util.concurrent.ThreadPoolExecutor#26 > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 > at java.lang.Thread.run(Thread.java:745) > > Have anyone faced similar issue ? What is the best method to handle this > out of memory issue? > > > Thanks, > -- > Pumudu Ruhunage > Associate Software Engineer | WSO2 Inc > M: +94 779 664493 | http://wso2.com > > _______________________________________________ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
