Hi, So I'd a chat with Azeez, and the first thing he mentioned was, we should be using the AWS mode in EC2 it seems, the docs can be found for this here [1]. And his comments are, these kind of issues can be seen sometimes there, but if it recovers eventually it would be okay, but if it continues to give same kind of errors always, that would be a problem.
[1] https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode Cheers, Anjana. On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando <anj...@wso2.com> wrote: > Hi everyone, > > When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has > reported some Hazlecast issues, specifically [1] and [2]. Some of the > errors as follows:- > > TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR > {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - > Unable to test existence of resource > {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} > com.hazelcast.spi.exception.WrongTargetException: WrongTarget! > this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000, > partitionId: 5, replicaIndex: 0, operation: > com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246) > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127) > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76) > at ------ End remote and begin local stack-trace ------.(Unknown > Source) > at > com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) > at > com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) > at > com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) > at > com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) > at > com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456) > at > com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275) > at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82) > at > org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141) > at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203) > > ==================================================================== > > > TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR > {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - > Unable to test existence of resource > {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} > com.hazelcast.spi.exception.CallerNotMemberException: Not Member! > caller:Address[172.18.1.228]:4000, partitionId: 30, operation: > com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336) > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) > at > com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) > > ==================================================================== > > ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} - > [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response: > JoinRequest{packetVersion=4, buildNumber=20150826, > address=Address[172.18.1.229]:4000, > uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, > memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 > {com.hazelcast.cluster.impl.operations.JoinCheckOperation} > com.hazelcast.core.HazelcastException: Cannot send response: > JoinRequest{packetVersion=4, buildNumber=20150826, > address=Address[172.18.1.229]:4000, > uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, > memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 > at > com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131) > at > com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204) > > ==================================================================== > > Have any one of you guys seen issues similar to the above in EC2? .. and > what are the possibilities this could arise, the most obvious thing would > be networking issues that can arise in EC2, which maybe related to a higher > latency that's there or something. Any of you guys know how to maybe tune > HZ in this situations to have a higher timeout values or anything like > that, or else, doing this in EC2 is altogether not recommended? .. > > [1] https://wso2.org/jira/browse/DAS-301 > [2] https://wso2.org/jira/browse/DAS-302 > > Cheers, > Anjana. > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
