When you say periodically, how often? In the initial description, it says newly active node's port gets open once it finish processing the queue. So events that send to the SP during that period going to lost??
On Fri, Jul 13, 2018 at 3:49 PM Damith Wickramasinghe <dami...@wso2.com> wrote: > Hi all, > > Please find my comments inline. These are from highlevel perspective. > > @Anoukh > > I'm assuming that "queue1" and "queue2" are actually per Source? >> > > Actually both would be just one. Reason being since we need to maintain > the order of the events. > > So at the moment the active node goes down, how does the passive node >> start receiving events? Will we be handling this or should the cluster be >> fronted by a separate load balancer? >> > > This should be handle from the client side by some kind of load balancing. > So if one node goes down, it will retry till a node can be accessible. > > > @Tishan > > But then we will need to shutdown thrift/binary receiving servers of >> passive node for the failover to happen. >> > > Yes the passive node will only have management ports open. All other ports > etc will be not accessible. > > > @Harshan > > According to this architecture, the passive node will only fetch event >> states and filter events only once it is active. This model might exhaust >> the queue 2 very soon depending on the rate of events. >> > > Sorry I missed that information. No. Periodically active node will persist > the state and last event timestamp. And passive node too will get the state > periodically and clear unrelated events from the queue. > > @Tishan/@Harshan > > Queue exhaustion could happen to both queues if the event rate is really > high. The mitigation action for some extend is, use a high performance > protocol to send events from active node to passive and clearing the > corresponding events from passive node periodically. So it will not > accumulate forever overtime . Yes we could use some broker queue concept. > We will discuss internally and update the thread. > > Regards, > Damith > > On Fri, Jul 13, 2018 at 3:30 AM, Tishan Dahanayakage <tis...@wso2.com> > wrote: > >> @Anoukh, >> We can configure a LB in front to handle failover. Actually that is the >> correct solution rather than publishing to both. But I think we need to >> figure out how to handle incoming databridge traffic as it cannot be load >> balanced. Yes we can utilize client side load balancing and directly >> publish to SP nodes. But then we will need to shutdown thrift/binary >> receiving servers of passive node for the failover to happen. >> >> @Damith, >> I have below questions/suggestions. >> 1. What is the expected behavior when queue is full in active node? >> First of all I suppose this is a in-memory queue. This is something we >> should think clearly and implement as even now we are facing quite a number >> of issues where server is stalled because queues are getting filled. When >> queue in active node is full >> - We can't let receiving threads to get block on the queue as it will >> cause server to freeze. >> - At the same time We can't also drop the event if exactly once >> processing or zero event drop is enforced. >> - Also we can't keep adding stuff to queue as it will go OOM. >> So IMO we need a way to persist the queue. In other words we need a >> proper broker queue to solve this issue. If user is fine with event loss >> then We can live with in-memory queue. >> >> 2. We need a automated purging task to clean queue2 to after every state >> persistance done by active node. >> >> /Tishan >> >> On Thu, Jul 12, 2018 at 10:41 PM, Anoukh Jayawardena <ano...@wso2.com> >> wrote: >> >>> Hi Damith, >>> >>> Just a few clarifications on the new architecture. >>> I'm assuming that "queue1" and "queue2" are actually per Source? Which >>> means if we have 3 sources, we'll have 3 queues in active node and 3 queues >>> in passive node? Does that mean there will be 3 threads working in async? >>> Also, the main difference with the current HA architecture is that both >>> nodes will not receive all the events. So at the moment the active node >>> goes down, how does the passive node start receiving events? Will we be >>> handling this or should the cluster be fronted by a separate load balancer? >>> >>> Thanks, >>> Anoukh >>> >>> On Thu, Jul 12, 2018 at 8:34 PM Damith Wickramasinghe <dami...@wso2.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We are in the process of refactor/improve the existing HA architecture >>>> due to various concerns found. >>>> >>>> Below is the high level design came up with. We will provide more >>>> in-depth details as the implementation carries on. >>>> >>>> >>>> >>>> >>>> As per above at a given point of time there will only be a one active >>>> node. Passive node will not consume any events. Electing active node will >>>> work as per current architecture via cluster coordination. >>>> A thread will work in the active node to put the data into a >>>> queue(queue1) and same thread will then publish the events to >>>> outside.Reason for having a queue here because to send events >>>> asynchronously to passive node. Here we are going to send events to passive >>>> node via TCP. (This we need to decide) Active node will persist the state >>>> periodically to the database. >>>> >>>> When active node goes down via cluster coordination passive node will >>>> become active. When active it will get the state from database(to sync the >>>> state) with latest event timestamp , and filter events from queue 2 (in >>>> order to stop processing already processed events) and send them out. Then >>>> open the ports in the newly active node and start receiving events from >>>> sources. In this architecture also at least one processing will be done. >>>> >>>> Nisala please add anything I missed. >>>> >>>> Regards, >>>> Damith >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Senior Software Engineer >>>> WSO2 Inc.; http://wso2.com >>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>> lean.enterprise.middleware >>>> >>>> mobile: *+94728671315* >>>> >>>> >>> >>> -- >>> *Anoukh Jayawardena* >>> *Software Engineer* >>> >>> *WSO2 Lanka (Private) Limited: http://wso2.com >>> <http://wso2.com/>lean.enterprise.middle-ware* >>> >>> >>> *phone: (+94) 77 99 28932* >>> <https://wso2.com/signature> >>> >> >> >> >> -- >> Tishan Dahanayakage >> Associate Technical Lead >> WSO2, Inc. >> Mobile:+94 716481328 >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, re-transmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> > > > > -- > Senior Software Engineer > WSO2 Inc.; http://wso2.com > <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> > lean.enterprise.middleware > > mobile: *+94728671315* > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > -- W.G. Gihan Anuruddha Associate Technical Lead | WSO2, Inc.
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
