[Dev] [MB] Out of memory error in long running mb cluster
Hi, I came across out of memory exception while doing long running test on mb cluster with oracle database as message store. Following was the scenario i tested. 1. Start two clustered mb nodes with serializers (nodeA(coordinator), nodeB). 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" messages to nodeA with average 1500 TPS. 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" messages to nodeB with average 1000 TPS. 4. After around 14 hours, om exception has thrown from nodeA and all operations has halted. After analysing heap dump i found which thread has thrown the error, and it's as follows. "pool-25-thread-1" prio=5 tid=174 RUNNABLE at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48) at java.lang.Long.toString(Long.java:267) at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 Local Variable: com.google.gson.JsonArray#1 at com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) at com.google.gson.internal.Streams.parse(Streams.java:44) at com.google.gson.JsonParser.parse(JsonParser.java:84) at com.google.gson.JsonParser.parse(JsonParser.java:59) Local Variable: com.google.gson.stream.JsonReader#1 at com.google.gson.JsonParser.parse(JsonParser.java:45) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) at org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) Local Variable: com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) at com.hazelcast.map.MapService.toObject(MapService.java:872) at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) at org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) Local Variable: java.util.TreeSet#1 at org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) Local Variable: java.lang.String#2380 Local Variable: java.lang.String#2379 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 at org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) Local Variable: org.apache.thrift.transport.TSocket#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 Local Variable: org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Local Variable: org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 Local Variable: java.util.concurrent.ThreadPoolExecutor#26 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 at java.lang.Thread.run(Thread.java:745) Have anyone faced similar issue ? What is the best method to handle this out of memory issue? Thanks, -- Pumudu Ruhunage Associate Software Engineer | WSO2 Inc M: +94 779 664493 | http://wso2.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] Out of memory error in long running mb cluster
Lets analyse the heap dump and see what caused the issue... also check the metrics recorded (via matric library) when server went OOM. On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage wrote: > Hi, > > I came across out of memory exception while doing long running test on mb > cluster with oracle database as message store. Following was the scenario i > tested. > > 1. Start two clustered mb nodes with serializers (nodeA(coordinator), > nodeB). > 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" > messages to nodeA with average 1500 TPS. > 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" > messages to nodeB with average 1000 TPS. > 4. After around 14 hours, om exception has thrown from nodeA and all > operations has halted. > > After analysing heap dump i found which thread has thrown the error, and > it's as follows. > > "pool-25-thread-1" prio=5 tid=174 RUNNABLE > at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48) > at java.lang.Long.toString(Long.java:267) > at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) >Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 >Local Variable: com.google.gson.JsonArray#1 > at > com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) > at com.google.gson.internal.Streams.parse(Streams.java:44) > at com.google.gson.JsonParser.parse(JsonParser.java:84) > at com.google.gson.JsonParser.parse(JsonParser.java:59) >Local Variable: com.google.gson.stream.JsonReader#1 > at com.google.gson.JsonParser.parse(JsonParser.java:45) > at > org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) > at > org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) > at > com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >Local Variable: > com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 > at > com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) > at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) > at com.hazelcast.map.MapService.toObject(MapService.java:872) > at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) > at > org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) >Local Variable: java.util.TreeSet#1 > at > org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) >Local Variable: java.lang.String#2380 >Local Variable: java.lang.String#2379 > at > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) >Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 > at > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) >Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 >Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) >Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >Local Variable: org.apache.thrift.transport.TSocket#2 >Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 >Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 >Local Variable: > org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >Local Variable: > org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 >Local Variable: java.util.concurrent.ThreadPoolExecutor#26 > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 > at java.lang.Thread.run(Thread.java:745) > > Have anyone faced similar issue ? What is the best method to handle this > out of memory issue? > > > Thanks, > -- > Pumudu Ruhunage > Associate Software Engineer | WSO2 Inc > M: +94 779 664493 | http://wso2.com > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 ___ Dev m
Re: [Dev] [MB] Out of memory error in long running mb cluster
Hi, Maybe the error is not really in this tread, but has been occurred after accumulating data from a memory leak. Better analyse the memory allocations. Thanks, Akalanka. On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe wrote: > Lets analyse the heap dump and see what caused the issue... > also check the metrics recorded (via matric library) when server went OOM. > > On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage wrote: > >> Hi, >> >> I came across out of memory exception while doing long running test on mb >> cluster with oracle database as message store. Following was the scenario i >> tested. >> >> 1. Start two clustered mb nodes with serializers (nodeA(coordinator), >> nodeB). >> 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" >> messages to nodeA with average 1500 TPS. >> 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" >> messages to nodeB with average 1000 TPS. >> 4. After around 14 hours, om exception has thrown from nodeA and all >> operations has halted. >> >> After analysing heap dump i found which thread has thrown the error, and >> it's as follows. >> >> "pool-25-thread-1" prio=5 tid=174 RUNNABLE >> at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48) >> at java.lang.Long.toString(Long.java:267) >> at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) >>Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 >>Local Variable: com.google.gson.JsonArray#1 >> at >> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) >> at com.google.gson.internal.Streams.parse(Streams.java:44) >> at com.google.gson.JsonParser.parse(JsonParser.java:84) >> at com.google.gson.JsonParser.parse(JsonParser.java:59) >>Local Variable: com.google.gson.stream.JsonReader#1 >> at com.google.gson.JsonParser.parse(JsonParser.java:45) >> at >> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) >> at >> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) >> at >> com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >>Local Variable: >> com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 >> at >> com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) >> at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) >> at com.hazelcast.map.MapService.toObject(MapService.java:872) >> at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) >> at >> org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) >>Local Variable: java.util.TreeSet#1 >> at >> org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) >>Local Variable: java.lang.String#2380 >>Local Variable: java.lang.String#2379 >> at >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) >>Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 >> at >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) >>Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 >>Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 >> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) >>Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 >> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >> at >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >>Local Variable: org.apache.thrift.transport.TSocket#2 >>Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 >>Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 >>Local Variable: >> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>Local Variable: >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 >>Local Variable: java.util.concurrent.ThreadPoolExecutor#26 >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41 >> at java.lang.Thread.run(Thread.java:745) >> >> Have anyone faced similar issue ? What is the best method to handle this >> out of memory issue? >> >> >> Thanks, >> -- >> Pumudu Ruhunage >> Associate Software Engineer | WSO2 Inc >> M: +94 779 664493 |
Re: [Dev] [MB] Out of memory error in long running mb cluster
Hi, We found the cause for this error. Will fix this and run the tests again. Thank you, Sajini On Wed, Mar 18, 2015 at 5:36 PM, Akalanka Pagoda Arachchi < darsha...@wso2.com> wrote: > Hi, > > Maybe the error is not really in this tread, but has been occurred after > accumulating data from a memory leak. Better analyse the memory allocations. > > Thanks, > Akalanka. > > On Wed, Mar 18, 2015 at 4:58 PM, Ramith Jayasinghe > wrote: > >> Lets analyse the heap dump and see what caused the issue... >> also check the metrics recorded (via matric library) when server went OOM. >> >> On Wed, Mar 18, 2015 at 4:55 PM, Pumudu Ruhunage wrote: >> >>> Hi, >>> >>> I came across out of memory exception while doing long running test on >>> mb cluster with oracle database as message store. Following was the >>> scenario i tested. >>> >>> 1. Start two clustered mb nodes with serializers (nodeA(coordinator), >>> nodeB). >>> 2. Add queue subscriber "MyQueue1", to nodeB and publish "MyQueue1" >>> messages to nodeA with average 1500 TPS. >>> 3. Add queue subscriber "MyQueue2" to nodeA and publish "MyQueue2" >>> messages to nodeB with average 1000 TPS. >>> 4. After around 14 hours, om exception has thrown from nodeA and all >>> operations has halted. >>> >>> After analysing heap dump i found which thread has thrown the error, and >>> it's as follows. >>> >>> "pool-25-thread-1" prio=5 tid=174 RUNNABLE >>> at java.lang.OutOfMemoryError.(OutOfMemoryError.java:48) >>> at java.lang.Long.toString(Long.java:267) >>> at com.google.gson.stream.JsonReader.nextString(JsonReader.java:816) >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:648) >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:659) >>>Local Variable: com.google.gson.internal.bind.TypeAdapters$25#1 >>>Local Variable: com.google.gson.JsonArray#1 >>> at >>> com.google.gson.internal.bind.TypeAdapters$25.read(TypeAdapters.java:642) >>> at com.google.gson.internal.Streams.parse(Streams.java:44) >>> at com.google.gson.JsonParser.parse(JsonParser.java:84) >>> at com.google.gson.JsonParser.parse(JsonParser.java:59) >>>Local Variable: com.google.gson.stream.JsonReader#1 >>> at com.google.gson.JsonParser.parse(JsonParser.java:45) >>> at >>> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:53) >>> at >>> org.wso2.andes.server.cluster.coordination.hazelcast.custom.serializer.TreeSetLongWrapperSerializer.read(TreeSetLongWrapperSerializer.java:33) >>> at >>> com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59) >>>Local Variable: >>> com.hazelcast.nio.serialization.ByteArrayObjectDataInput#1 >>> at >>> com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221) >>> at >>> com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:156) >>> at com.hazelcast.map.MapService.toObject(MapService.java:872) >>> at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53) >>> at >>> org.wso2.andes.kernel.slot.SlotManagerClusterMode.updateMessageID(SlotManagerClusterMode.java:347) >>>Local Variable: java.util.TreeSet#1 >>> at >>> org.wso2.andes.thrift.SlotManagementServiceImpl.updateMessageId(SlotManagementServiceImpl.java:56) >>>Local Variable: java.lang.String#2380 >>>Local Variable: java.lang.String#2379 >>> at >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:440) >>>Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_result#2 >>> at >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId.getResult(SlotManagementService.java:429) >>>Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$updateMessageId_args#1 >>>Local Variable: org.wso2.andes.thrift.SlotManagementServiceImpl#1 >>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) >>>Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor$updateMessageId#1 >>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) >>> at >>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) >>>Local Variable: org.apache.thrift.transport.TSocket#2 >>>Local Variable: org.apache.thrift.protocol.TBinaryProtocol#2 >>>Local Variable: org.apache.thrift.protocol.TBinaryProtocol#3 >>>Local Variable: >>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Processor#1 >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>Local Variable: >>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess#1 >>>Local Variable: java.util.concurrent.ThreadPoolExecutor#26 >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>Local Variable: java.ut
