And also I have tested the cluster with WKA membership scheme with old kernel previously and it worked fine. But with new kernel version we got some issues and that is why we had to move to AWS clustering mode.
On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya <mani...@wso2.com> wrote: > We have not tested AWS clustering with old kernel. We moved to AWS > clustering with new kernel. I thought it was due to new kernel version as I > have heard AWS was working properly with products released with older > kernel. > Here I have started each node one after the other. And once a member was > stopped also any message was not printed. > > > > On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne <im...@wso2.com> wrote: > >> Member joined message may not come with AWS membership scheme if all the >> members were started at the same time. However the member left message >> should appear if a member is stopped afterwards. >> >> >> On Monday, October 26, 2015, Maninda Edirisooriya <mani...@wso2.com> >> wrote: >> >>> I have setup the AWS clustering according to [1] but it seems the >>> "AWSBasedMembershipScheme >>> Member joined" message is not displayed once a new member is joined and >>> no message when it left. This seems to be an issue with new kernel version. >>> >>> <clustering >>> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" >>> enable="true"> >>> >>> >>> <parameter name="membershipScheme">aws</parameter> >>> >>> >>> <parameter name="localMemberPort">5701</parameter> >>> >>> >>> <parameter name="accessKey">xxxxxxx</parameter> >>> <parameter name="secretKey">zzzzzzz</parameter> >>> <parameter name="securityGroup">sg-16f4b571</parameter> >>> <parameter name="region">us-east-1</parameter> >>> <parameter name="tagKey">a_tag_key</parameter> >>> <parameter name="tagValue">a_tag_value</parameter> >>> >>> >>> [1] >>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >>> >>> >>> Thanks. >>> >>> >>> *Maninda Edirisooriya* >>> Senior Software Engineer >>> >>> *WSO2, Inc.*lean.enterprise.middleware. >>> >>> *Blog* : http://maninda.blogspot.com/ >>> *E-mail* : mani...@wso2.com >>> *Skype* : @manindae >>> *Twitter* : @maninda >>> >>> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez <az...@wso2.com> wrote: >>> >>>> In EC2 with AWS mode, you should use ports 5701 to 5800 >>>> >>>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera <nira...@wso2.com> >>>> wrote: >>>> >>>>> Hi Thusitha, >>>>> >>>>> 4000 is the local host port specified in the axis2.xml >>>>> >>>>> quote axis2.xml >>>>> " <!-- The host name or IP address of this member --> >>>>> >>>>> <parameter name="localMemberHost">10.100.5.109</parameter> >>>>> >>>>> <!-- >>>>> The bind adress of this member. The difference between >>>>> localMemberHost & localMemberBindAddress >>>>> is that localMemberHost is the one that is advertised by >>>>> this member, while localMemberBindAddress >>>>> is the address to which this member is bound to. >>>>> --> >>>>> <!-- >>>>> <parameter >>>>> name="localMemberBindAddress">10.100.5.109</parameter> >>>>> --> >>>>> >>>>> <!-- >>>>> The TCP port used by this member. This is the port through >>>>> which other nodes will >>>>> contact this member >>>>> --> >>>>> <parameter name="localMemberPort">4000</parameter>" >>>>> >>>>> in DAS we are using wka as the membership scheme. >>>>> >>>>> cheers >>>>> >>>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne < >>>>> thusit...@wso2.com> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> I noticed following error line on the error stack that you posted. >>>>>> >>>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>>> this:Address[172.18.1.227]:4000 >>>>>> >>>>>> >>>>>> Here it tries to connect using the port 4000. But AFAIK we usually >>>>>> use ports between 5701 to 5800. Did you set the port to 4000 in any >>>>>> instance? >>>>>> >>>>>> Thanks >>>>>> Thusitha >>>>>> >>>>>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> So I'd a chat with Azeez, and the first thing he mentioned was, we >>>>>>> should be using the AWS mode in EC2 it seems, the docs can be found for >>>>>>> this here [1]. And his comments are, these kind of issues can be seen >>>>>>> sometimes there, but if it recovers eventually it would be okay, but if >>>>>>> it >>>>>>> continues to give same kind of errors always, that would be a problem. >>>>>>> >>>>>>> [1] >>>>>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando <anj...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA >>>>>>>> has reported some Hazlecast issues, specifically [1] and [2]. Some of >>>>>>>> the >>>>>>>> errors as follows:- >>>>>>>> >>>>>>>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>>> Unable to test existence of resource >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>>>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000, >>>>>>>> partitionId: 5, replicaIndex: 0, operation: >>>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>>> hz:impl:mapService >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76) >>>>>>>> at ------ End remote and begin local stack-trace >>>>>>>> ------.(Unknown Source) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141) >>>>>>>> at >>>>>>>> org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> >>>>>>>> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>>> Unable to test existence of resource >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>>> com.hazelcast.spi.exception.CallerNotMemberException: Not Member! >>>>>>>> caller:Address[172.18.1.228]:4000, partitionId: 30, operation: >>>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>>> hz:impl:mapService >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} >>>>>>>> - [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send >>>>>>>> response: >>>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>>> address=Address[172.18.1.229]:4000, >>>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>>> {com.hazelcast.cluster.impl.operations.JoinCheckOperation} >>>>>>>> com.hazelcast.core.HazelcastException: Cannot send response: >>>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>>> address=Address[172.18.1.229]:4000, >>>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> Have any one of you guys seen issues similar to the above in EC2? >>>>>>>> .. and what are the possibilities this could arise, the most obvious >>>>>>>> thing >>>>>>>> would be networking issues that can arise in EC2, which maybe related >>>>>>>> to a >>>>>>>> higher latency that's there or something. Any of you guys know how to >>>>>>>> maybe >>>>>>>> tune HZ in this situations to have a higher timeout values or anything >>>>>>>> like >>>>>>>> that, or else, doing this in EC2 is altogether not recommended? .. >>>>>>>> >>>>>>>> [1] https://wso2.org/jira/browse/DAS-301 >>>>>>>> [2] https://wso2.org/jira/browse/DAS-302 >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@wso2.org >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thusitha Dayaratne >>>>>> Software Engineer >>>>>> WSO2 Inc. - lean . enterprise . middleware | wso2.com >>>>>> >>>>>> Mobile +94712756809 >>>>>> Blog alokayasoya.blogspot.com >>>>>> About http://about.me/thusithathilina >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@wso2.org >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Niranda Perera* >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94-71-554-8430 >>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>> https://pythagoreanscript.wordpress.com/ >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Afkham Azeez* >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>> Member; Apache Software Foundation; http://www.apache.org/ >>>> * <http://www.apache.org/>* >>>> *email: **az...@wso2.com* >>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>> *twitter: **http://twitter.com/afkham_azeez* >>>> <http://twitter.com/afkham_azeez> >>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>> >>>> *Lean . Enterprise . Middleware* >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >> >> -- >> *Imesh Gunaratne* >> Senior Technical Lead >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: http://imesh.gunaratne.org >> Lean . Enterprise . Middleware >> >> >> >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
