We have not tested AWS clustering with old kernel. We moved to AWS clustering with new kernel. I thought it was due to new kernel version as I have heard AWS was working properly with products released with older kernel. Here I have started each node one after the other. And once a member was stopped also any message was not printed.
On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne <im...@wso2.com> wrote: > Member joined message may not come with AWS membership scheme if all the > members were started at the same time. However the member left message > should appear if a member is stopped afterwards. > > > On Monday, October 26, 2015, Maninda Edirisooriya <mani...@wso2.com> > wrote: > >> I have setup the AWS clustering according to [1] but it seems the >> "AWSBasedMembershipScheme >> Member joined" message is not displayed once a new member is joined and >> no message when it left. This seems to be an issue with new kernel version. >> >> <clustering >> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" >> enable="true"> >> >> >> <parameter name="membershipScheme">aws</parameter> >> >> >> <parameter name="localMemberPort">5701</parameter> >> >> >> <parameter name="accessKey">xxxxxxx</parameter> >> <parameter name="secretKey">zzzzzzz</parameter> >> <parameter name="securityGroup">sg-16f4b571</parameter> >> <parameter name="region">us-east-1</parameter> >> <parameter name="tagKey">a_tag_key</parameter> >> <parameter name="tagValue">a_tag_value</parameter> >> >> >> [1] >> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >> >> >> Thanks. >> >> >> *Maninda Edirisooriya* >> Senior Software Engineer >> >> *WSO2, Inc.*lean.enterprise.middleware. >> >> *Blog* : http://maninda.blogspot.com/ >> *E-mail* : mani...@wso2.com >> *Skype* : @manindae >> *Twitter* : @maninda >> >> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez <az...@wso2.com> wrote: >> >>> In EC2 with AWS mode, you should use ports 5701 to 5800 >>> >>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera <nira...@wso2.com> >>> wrote: >>> >>>> Hi Thusitha, >>>> >>>> 4000 is the local host port specified in the axis2.xml >>>> >>>> quote axis2.xml >>>> " <!-- The host name or IP address of this member --> >>>> >>>> <parameter name="localMemberHost">10.100.5.109</parameter> >>>> >>>> <!-- >>>> The bind adress of this member. The difference between >>>> localMemberHost & localMemberBindAddress >>>> is that localMemberHost is the one that is advertised by >>>> this member, while localMemberBindAddress >>>> is the address to which this member is bound to. >>>> --> >>>> <!-- >>>> <parameter >>>> name="localMemberBindAddress">10.100.5.109</parameter> >>>> --> >>>> >>>> <!-- >>>> The TCP port used by this member. This is the port through >>>> which other nodes will >>>> contact this member >>>> --> >>>> <parameter name="localMemberPort">4000</parameter>" >>>> >>>> in DAS we are using wka as the membership scheme. >>>> >>>> cheers >>>> >>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne < >>>> thusit...@wso2.com> wrote: >>>> >>>>> Hi Anjana, >>>>> >>>>> I noticed following error line on the error stack that you posted. >>>>> >>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>> this:Address[172.18.1.227]:4000 >>>>> >>>>> >>>>> Here it tries to connect using the port 4000. But AFAIK we usually use >>>>> ports between 5701 to 5800. Did you set the port to 4000 in any instance? >>>>> >>>>> Thanks >>>>> Thusitha >>>>> >>>>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando <anj...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> So I'd a chat with Azeez, and the first thing he mentioned was, we >>>>>> should be using the AWS mode in EC2 it seems, the docs can be found for >>>>>> this here [1]. And his comments are, these kind of issues can be seen >>>>>> sometimes there, but if it recovers eventually it would be okay, but if >>>>>> it >>>>>> continues to give same kind of errors always, that would be a problem. >>>>>> >>>>>> [1] >>>>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA >>>>>>> has reported some Hazlecast issues, specifically [1] and [2]. Some of >>>>>>> the >>>>>>> errors as follows:- >>>>>>> >>>>>>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR >>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>> Unable to test existence of resource >>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000, >>>>>>> partitionId: 5, replicaIndex: 0, operation: >>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>> hz:impl:mapService >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76) >>>>>>> at ------ End remote and begin local stack-trace ------.(Unknown >>>>>>> Source) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) >>>>>>> at >>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456) >>>>>>> at >>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275) >>>>>>> at >>>>>>> com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82) >>>>>>> at >>>>>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141) >>>>>>> at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203) >>>>>>> >>>>>>> ==================================================================== >>>>>>> >>>>>>> >>>>>>> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR >>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>> Unable to test existence of resource >>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>> com.hazelcast.spi.exception.CallerNotMemberException: Not Member! >>>>>>> caller:Address[172.18.1.228]:4000, partitionId: 30, operation: >>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>> hz:impl:mapService >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>> >>>>>>> ==================================================================== >>>>>>> >>>>>>> ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} - >>>>>>> [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response: >>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>> address=Address[172.18.1.229]:4000, >>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>> {com.hazelcast.cluster.impl.operations.JoinCheckOperation} >>>>>>> com.hazelcast.core.HazelcastException: Cannot send response: >>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>> address=Address[172.18.1.229]:4000, >>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>> at >>>>>>> com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131) >>>>>>> at >>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204) >>>>>>> >>>>>>> ==================================================================== >>>>>>> >>>>>>> Have any one of you guys seen issues similar to the above in EC2? .. >>>>>>> and what are the possibilities this could arise, the most obvious thing >>>>>>> would be networking issues that can arise in EC2, which maybe related >>>>>>> to a >>>>>>> higher latency that's there or something. Any of you guys know how to >>>>>>> maybe >>>>>>> tune HZ in this situations to have a higher timeout values or anything >>>>>>> like >>>>>>> that, or else, doing this in EC2 is altogether not recommended? .. >>>>>>> >>>>>>> [1] https://wso2.org/jira/browse/DAS-301 >>>>>>> [2] https://wso2.org/jira/browse/DAS-302 >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@wso2.org >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thusitha Dayaratne >>>>> Software Engineer >>>>> WSO2 Inc. - lean . enterprise . middleware | wso2.com >>>>> >>>>> Mobile +94712756809 >>>>> Blog alokayasoya.blogspot.com >>>>> About http://about.me/thusithathilina >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Niranda Perera* >>>> Software Engineer, WSO2 Inc. >>>> Mobile: +94-71-554-8430 >>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>> https://pythagoreanscript.wordpress.com/ >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Afkham Azeez* >>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> Member; Apache Software Foundation; http://www.apache.org/ >>> * <http://www.apache.org/>* >>> *email: **az...@wso2.com* >>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>> *http://blog.afkham.org* <http://blog.afkham.org> >>> *twitter: **http://twitter.com/afkham_azeez* >>> <http://twitter.com/afkham_azeez> >>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>> <http://lk.linkedin.com/in/afkhamazeez>* >>> >>> *Lean . Enterprise . Middleware* >>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> > > -- > *Imesh Gunaratne* > Senior Technical Lead > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: http://imesh.gunaratne.org > Lean . Enterprise . Middleware > > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
