Hi All, Observed following error on AS 520 manager Node when restarting the AS worker Nodes. Issue seams Intermittent. Have we observed this issue before ? Any solutions/explanations?
*Cluster Details:* niginx fronted 1 Manager/ 2 worker node AS 520 (with Service Pack1-hazelcast_3.0.1.wso2v1.jar) cluster, each node runs on separate machines. Local registry mounted to H2 Config and Governance registry mounted on a Mysql Database Manager Node: 192.168.48.90 Wrk Node 1: 192.168.48.91 Wrk Node 2: 192.168.48.92 *Steps followed:* 1. Cluster was running nearly 2 weeks continuously 2. AS cluster used to host back-end services required for ESB 3. AS Services invoked thorough ESB continously as a long running/load test 4. Due to a OOM issue on worker Node2 , restarted the cluster *Issue:* TID: [0] [AS] [2015-03-17 02:06:34,207] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member left [a3ee426a-153e-4ebc-95d5-95292370ec23]: /192.168.48.92:4000 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} TID: [0] [AS] [2015-03-17 04:49:47,567] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [a5d7cd86-fc94-409b-9720-e68210e6906c]: /192.168.48.92:4000 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} TID: [0] [AS] [2015-03-17 04:50:18,641] ERROR {com.hazelcast.spi.impl.PartitionIteratingOperation} - [192.168.48.90]:4000 [wso2.as.domain] [unknown] java.lang.NullPointerException {com.hazelcast.spi.impl.PartitionIteratingOperation} com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.NullPointerException at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:298) at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:254) at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:262) at com.hazelcast.spi.impl.PartitionIteratingOperation$PartitionResponse.writeData(PartitionIteratingOperation.java:130) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36) at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:36) at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:252) at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:262) at com.hazelcast.spi.impl.NormalResponse.writeData(NormalResponse.java:79) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36) at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:48) at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:176) at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:157) at com.hazelcast.spi.impl.NodeEngineImpl.toData(NodeEngineImpl.java:151) at com.hazelcast.spi.impl.BasicOperationService.send(BasicOperationService.java:639) at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:120) at com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:385) at com.hazelcast.spi.impl.BasicOperationService.processPacket(BasicOperationService.java:309) at com.hazelcast.spi.impl.BasicOperationService.access$400(BasicOperationService.java:102) at com.hazelcast.spi.impl.BasicOperationService$BasicOperationProcessorImpl.process(BasicOperationService.java:756) at com.hazelcast.spi.impl.BasicOperationScheduler$ProcessTask.run(BasicOperationScheduler.java:186) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) Caused by: java.lang.NullPointerException at com.hazelcast.map.MapEntrySet.writeData(MapEntrySet.java:60) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:129) at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:36) at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:36) at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:252) ... 25 more TID: [0] [AS] [2015-03-17 04:52:41,699] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [83db16fc-65be-41de-9750-36fb5dd6f6d7]: /192.168.48.91:4000 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -- Asanka Vithanage Senior Software Engineer -QA Mobile: +94 0716286708 Email: asan...@wso2.com WSO2 Inc. www.wso2.com
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev