Hi everyone, When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has reported some Hazlecast issues, specifically [1] and [2]. Some of the errors as follows:-
TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - Unable to test existence of resource {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000, partitionId: 5, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246) at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127) at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76) at ------ End remote and begin local stack-trace ------.(Unknown Source) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456) at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275) at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82) at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141) at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203) ==================================================================== TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - Unable to test existence of resource {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} com.hazelcast.spi.exception.CallerNotMemberException: Not Member! caller:Address[172.18.1.228]:4000, partitionId: 30, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336) at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) at com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) ==================================================================== ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} - [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response: JoinRequest{packetVersion=4, buildNumber=20150826, address=Address[172.18.1.229]:4000, uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 {com.hazelcast.cluster.impl.operations.JoinCheckOperation} com.hazelcast.core.HazelcastException: Cannot send response: JoinRequest{packetVersion=4, buildNumber=20150826, address=Address[172.18.1.229]:4000, uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131) at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204) ==================================================================== Have any one of you guys seen issues similar to the above in EC2? .. and what are the possibilities this could arise, the most obvious thing would be networking issues that can arise in EC2, which maybe related to a higher latency that's there or something. Any of you guys know how to maybe tune HZ in this situations to have a higher timeout values or anything like that, or else, doing this in EC2 is altogether not recommended? .. [1] https://wso2.org/jira/browse/DAS-301 [2] https://wso2.org/jira/browse/DAS-302 Cheers, Anjana. -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev