Hi everyone,
When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
reported some Hazlecast issues, specifically [1] and [2]. Some of the
errors as follows:-
TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
Unable to test existence of resource
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
partitionId: 5, replicaIndex: 0, operation:
com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
at ------ End remote and begin local stack-trace ------.(Unknown Source)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
at
com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)
====================================================================
TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
Unable to test existence of resource
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
====================================================================
ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} -
[172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response:
JoinRequest{packetVersion=4, buildNumber=20150826,
address=Address[172.18.1.229]:4000,
uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
{com.hazelcast.cluster.impl.operations.JoinCheckOperation}
com.hazelcast.core.HazelcastException: Cannot send response:
JoinRequest{packetVersion=4, buildNumber=20150826,
address=Address[172.18.1.229]:4000,
uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
at
com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204)
====================================================================
Have any one of you guys seen issues similar to the above in EC2? .. and
what are the possibilities this could arise, the most obvious thing would
be networking issues that can arise in EC2, which maybe related to a higher
latency that's there or something. Any of you guys know how to maybe tune
HZ in this situations to have a higher timeout values or anything like
that, or else, doing this in EC2 is altogether not recommended? ..
[1] https://wso2.org/jira/browse/DAS-301
[2] https://wso2.org/jira/browse/DAS-302
Cheers,
Anjana.
--
*Anjana Fernando*
Senior Technical Lead
WSO2 Inc. | http://wso2.com
lean . enterprise . middleware
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
