Re: [Architecture] MB4 active-passive clustering implementation

[Asanka Abeyweera]

Hi Shammi,

On Fri, Sep 29, 2017 at 4:18 AM, Shammi Jayasinghe  wrote:

>
>
> On Thu, Aug 17, 2017 at 4:15 AM, Asanka Abeyweera 
> wrote:
>
>> Hi all,
>>
>> I am working on implementing active-passive clustering on top of the new
>> MB4 architecture. Current message delivery implementation in MB4 is not
>> capable to work in a multi-node environment. If we run more than one node
>> concurrently, messages can be duplicated since all nodes start delivering
>> messages available for a queue in the message store. We have to make the
>> MB4 nodes cluster aware to rectify this behavior.
>> High-level design
>>
>> We are planning to use the available RDBMS leader election mechanism to
>> decide on the active node. All the non-leader nodes will be passive nodes
>> and will reject all incoming requests. The clients should be configured
>> with failover URLs so that they can failover to the active node from
>> passive nodes.
>>
>
> Hi Asanka,
>
> Sorry for the late reply.  AFAIU, with this new design, although there are
> multiple message broker nodes in the deployment, Only one node will be
> working as the active node and others will be idle until they get selected
> as the ACTIVE node.
>
> In that kind of behavior, Are we deviating from the concept of distrubited
> message brokering?
>

> When there is a high load (Thousands of subscriptions or consumers/
> Thousands of publishers or producers), All this node is going to be
> applying on one node which is going to use high CPU power and memory. In
> that case, we have to define the numbers which can be handled with specific
> memory and CPU power of a box. If there is a higher load, it will increase
> the delay of the message delivery also. How are we going to address these
> in this new design?
>

We are bit deviating from the distributed queueing concept. The plan is to
bind a single queue to a single node and make all subscribers and
publishers to a queue connect to one broker. The reason for the decision
is, in practice a given queue tends to have only a small number of
subscribers (~10). Therefore we can avoid doing distributed coordination
per message which ultimately result in more simplified and performant
architecture. As the first iteration we will be focusing on releasing an
active-passive clustering implementation. But the long term plan will be
the one I described previously.


> Thanks
> shammi
>
>>
>> Tasks to do when the active node becomes passive
>>
>>1.
>>
>>Enable rejecting all requests to broker node except the CLOSE-OK
>>2.
>>
>>Close existing AMQ connections (both publishing and consuming)
>>
>>
>> Tasks to do when the passive node becomes active
>>
>>1.
>>
>>Start accepting all requests to broker node
>>
>>
>> The assumption here is the active components like the Inbound disruptor,
>> outbound disruptor, delivery task managers will stop working (consuming
>> resources) when the incoming requests are stopped and subscriptions are
>> removed in passive nodes.
>>
>>
>>
>>-
>>
>>Filtering message events will be handled by “InboundEventGatekeeper”.
>>-
>>
>>DeliveryHandler will also be modified to stop delivering messages
>>when a node becomes passive.
>>-
>>
>>All the connections will be closed using the server connection
>>registry
>>
>>
>> Suggestions and feedback are appreciated.
>>
>> --
>> Asanka Abeyweera
>> Senior Software Engineer
>> WSO2 Inc.
>>
>> Phone: +94 712228648 <+94%2071%20222%208648>
>> Blog: a5anka.github.io
>>
>> 
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Best Regards,
>
> *  Shammi Jayasinghe*
>
>
> *Senior Technical Lead*
> *WSO2, Inc.*
> *+1-812-391-7730 <(812)%20391-7730>*
> *+1-812-327-3505 <(812)%20327-3505>*
>
> *http://shammijayasinghe.blogspot.com
> *
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648
Blog: a5anka.github.io


___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] MB4 active-passive clustering implementation

[Shammi Jayasinghe]

On Thu, Aug 17, 2017 at 4:15 AM, Asanka Abeyweera  wrote:

> Hi all,
>
> I am working on implementing active-passive clustering on top of the new
> MB4 architecture. Current message delivery implementation in MB4 is not
> capable to work in a multi-node environment. If we run more than one node
> concurrently, messages can be duplicated since all nodes start delivering
> messages available for a queue in the message store. We have to make the
> MB4 nodes cluster aware to rectify this behavior.
> High-level design
>
> We are planning to use the available RDBMS leader election mechanism to
> decide on the active node. All the non-leader nodes will be passive nodes
> and will reject all incoming requests. The clients should be configured
> with failover URLs so that they can failover to the active node from
> passive nodes.
>

Hi Asanka,

Sorry for the late reply.  AFAIU, with this new design, although there are
multiple message broker nodes in the deployment, Only one node will be
working as the active node and others will be idle until they get selected
as the ACTIVE node.

In that kind of behavior, Are we deviating from the concept of distrubited
message brokering?

When there is a high load (Thousands of subscriptions or consumers/
Thousands of publishers or producers), All this node is going to be
applying on one node which is going to use high CPU power and memory. In
that case, we have to define the numbers which can be handled with specific
memory and CPU power of a box. If there is a higher load, it will increase
the delay of the message delivery also. How are we going to address these
in this new design?

Thanks
shammi

>
> Tasks to do when the active node becomes passive
>
>1.
>
>Enable rejecting all requests to broker node except the CLOSE-OK
>2.
>
>Close existing AMQ connections (both publishing and consuming)
>
>
> Tasks to do when the passive node becomes active
>
>1.
>
>Start accepting all requests to broker node
>
>
> The assumption here is the active components like the Inbound disruptor,
> outbound disruptor, delivery task managers will stop working (consuming
> resources) when the incoming requests are stopped and subscriptions are
> removed in passive nodes.
>
>
>
>-
>
>Filtering message events will be handled by “InboundEventGatekeeper”.
>-
>
>DeliveryHandler will also be modified to stop delivering messages when
>a node becomes passive.
>-
>
>All the connections will be closed using the server connection registry
>
>
> Suggestions and feedback are appreciated.
>
> --
> Asanka Abeyweera
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 712228648 <+94%2071%20222%208648>
> Blog: a5anka.github.io
>
> 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Best Regards,

*  Shammi Jayasinghe*


*Senior Technical Lead*
*WSO2, Inc.*
*+1-812-391-7730*
*+1-812-327-3505*

*http://shammijayasinghe.blogspot.com
*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] MB4 active-passive clustering implementation

[Asanka Abeyweera]

Hi all,

We have decided to change the default deployment configurations with the HA
active passive implementation[1], We now have two options for
"deployment/mode" [2][3]. They are,

   - standalone - This is the simplest mode a broker can be started. The
   node will assume that message store is not shared with another node.
   Therefore it will not try to coordinate with other nodes (possibly
   non-existent) to provide HA. This is the *default* configuration used.
   - clustered - Broker node will run in HA mode (active/passive).

WDYT?

[1] https://github.com/wso2/andes/pull/921
[2] https://github.com/wso2/andes/pull/923
[3] https://github.com/wso2/carbon-business-messaging/pull/585

On Fri, Aug 18, 2017 at 3:32 PM, Asanka Abeyweera  wrote:

> Hi Abimaran,
>
> On Fri, Aug 18, 2017 at 2:44 PM, Abimaran Kugathasan 
> wrote:
>
>> Hi Asanka,
>>
>> We are embedding MB components into APIM and for minimum HA in API
>> Manager, there will be two active nodes. So will MB components/features
>> decide which node should be active and passive based on start up order
>> initially?
>>
>
> Yes, the MB component will decide on the active node.
>
>
>>
>> Also, are you expecting a client to send messages to both nodes (multi
>> cast) and passive MB nodes will reject them? Because in API Manager
>> perspective both node as active, so there could be a chance to configure a
>> gateway with active MB node as failover node.
>>
>
> No, the clients don't have to publish messages to both node. You can
> configure the broker URL with failover options. If the client connects to
> the passive node first it will get rejected and try to connect to the
> second node in the broker list. After connecting to the active node clients
> can resume publishing messages. Therefore it will not be an issue even if
> you connect to the pssive node first.
>
> WDYT?
>
>
>> On Thu, Aug 17, 2017 at 4:45 PM, Asanka Abeyweera 
>> wrote:
>>
>>> Hi all,
>>>
>>> I am working on implementing active-passive clustering on top of the new
>>> MB4 architecture. Current message delivery implementation in MB4 is not
>>> capable to work in a multi-node environment. If we run more than one node
>>> concurrently, messages can be duplicated since all nodes start delivering
>>> messages available for a queue in the message store. We have to make the
>>> MB4 nodes cluster aware to rectify this behavior.
>>> High-level design
>>>
>>> We are planning to use the available RDBMS leader election mechanism to
>>> decide on the active node. All the non-leader nodes will be passive nodes
>>> and will reject all incoming requests. The clients should be configured
>>> with failover URLs so that they can failover to the active node from
>>> passive nodes.
>>>
>>> Tasks to do when the active node becomes passive
>>>
>>>1.
>>>
>>>Enable rejecting all requests to broker node except the CLOSE-OK
>>>2.
>>>
>>>Close existing AMQ connections (both publishing and consuming)
>>>
>>>
>>> Tasks to do when the passive node becomes active
>>>
>>>1.
>>>
>>>Start accepting all requests to broker node
>>>
>>>
>>> The assumption here is the active components like the Inbound disruptor,
>>> outbound disruptor, delivery task managers will stop working (consuming
>>> resources) when the incoming requests are stopped and subscriptions are
>>> removed in passive nodes.
>>>
>>>
>>>
>>>-
>>>
>>>Filtering message events will be handled by “InboundEventGatekeeper”.
>>>-
>>>
>>>DeliveryHandler will also be modified to stop delivering messages
>>>when a node becomes passive.
>>>-
>>>
>>>All the connections will be closed using the server connection
>>>registry
>>>
>>>
>>> Suggestions and feedback are appreciated.
>>>
>>> --
>>> Asanka Abeyweera
>>> Senior Software Engineer
>>> WSO2 Inc.
>>>
>>> Phone: +94 712228648 <+94%2071%20222%208648>
>>> Blog: a5anka.github.io
>>>
>>> 
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Thanks
>> Abimaran Kugathasan
>> Senior Software Engineer - API Technologies
>>
>> Email : abima...@wso2.com
>> Mobile : +94 773922820 <+94%2077%20392%202820>
>>
>> 
>> 
>>   
>> 
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Asanka Abeyweera
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 712228648 <+94%2071%20222%208648>
> Blog: a5anka.github.io
>
> 
>



-- 
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648
Blog: a5anka.github.io


___

Re: [Architecture] MB4 active-passive clustering implementation

[Asanka Abeyweera]

Hi Abimaran,

On Fri, Aug 18, 2017 at 2:44 PM, Abimaran Kugathasan 
wrote:

> Hi Asanka,
>
> We are embedding MB components into APIM and for minimum HA in API
> Manager, there will be two active nodes. So will MB components/features
> decide which node should be active and passive based on start up order
> initially?
>

Yes, the MB component will decide on the active node.


>
> Also, are you expecting a client to send messages to both nodes (multi
> cast) and passive MB nodes will reject them? Because in API Manager
> perspective both node as active, so there could be a chance to configure a
> gateway with active MB node as failover node.
>

No, the clients don't have to publish messages to both node. You can
configure the broker URL with failover options. If the client connects to
the passive node first it will get rejected and try to connect to the
second node in the broker list. After connecting to the active node clients
can resume publishing messages. Therefore it will not be an issue even if
you connect to the pssive node first.

WDYT?


> On Thu, Aug 17, 2017 at 4:45 PM, Asanka Abeyweera 
> wrote:
>
>> Hi all,
>>
>> I am working on implementing active-passive clustering on top of the new
>> MB4 architecture. Current message delivery implementation in MB4 is not
>> capable to work in a multi-node environment. If we run more than one node
>> concurrently, messages can be duplicated since all nodes start delivering
>> messages available for a queue in the message store. We have to make the
>> MB4 nodes cluster aware to rectify this behavior.
>> High-level design
>>
>> We are planning to use the available RDBMS leader election mechanism to
>> decide on the active node. All the non-leader nodes will be passive nodes
>> and will reject all incoming requests. The clients should be configured
>> with failover URLs so that they can failover to the active node from
>> passive nodes.
>>
>> Tasks to do when the active node becomes passive
>>
>>1.
>>
>>Enable rejecting all requests to broker node except the CLOSE-OK
>>2.
>>
>>Close existing AMQ connections (both publishing and consuming)
>>
>>
>> Tasks to do when the passive node becomes active
>>
>>1.
>>
>>Start accepting all requests to broker node
>>
>>
>> The assumption here is the active components like the Inbound disruptor,
>> outbound disruptor, delivery task managers will stop working (consuming
>> resources) when the incoming requests are stopped and subscriptions are
>> removed in passive nodes.
>>
>>
>>
>>-
>>
>>Filtering message events will be handled by “InboundEventGatekeeper”.
>>-
>>
>>DeliveryHandler will also be modified to stop delivering messages
>>when a node becomes passive.
>>-
>>
>>All the connections will be closed using the server connection
>>registry
>>
>>
>> Suggestions and feedback are appreciated.
>>
>> --
>> Asanka Abeyweera
>> Senior Software Engineer
>> WSO2 Inc.
>>
>> Phone: +94 712228648 <+94%2071%20222%208648>
>> Blog: a5anka.github.io
>>
>> 
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Thanks
> Abimaran Kugathasan
> Senior Software Engineer - API Technologies
>
> Email : abima...@wso2.com
> Mobile : +94 773922820 <+94%2077%20392%202820>
>
> 
> 
>   
> 
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: +94 712228648
Blog: a5anka.github.io


___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] MB4 active-passive clustering implementation

[Abimaran Kugathasan]

Hi Asanka,

We are embedding MB components into APIM and for minimum HA in API Manager,
there will be two active nodes. So will MB components/features decide which
node should be active and passive based on start up order initially?

Also, are you expecting a client to send messages to both nodes (multi
cast) and passive MB nodes will reject them? Because in API Manager
perspective both node as active, so there could be a chance to configure a
gateway with active MB node as failover node.

On Thu, Aug 17, 2017 at 4:45 PM, Asanka Abeyweera  wrote:

> Hi all,
>
> I am working on implementing active-passive clustering on top of the new
> MB4 architecture. Current message delivery implementation in MB4 is not
> capable to work in a multi-node environment. If we run more than one node
> concurrently, messages can be duplicated since all nodes start delivering
> messages available for a queue in the message store. We have to make the
> MB4 nodes cluster aware to rectify this behavior.
> High-level design
>
> We are planning to use the available RDBMS leader election mechanism to
> decide on the active node. All the non-leader nodes will be passive nodes
> and will reject all incoming requests. The clients should be configured
> with failover URLs so that they can failover to the active node from
> passive nodes.
>
> Tasks to do when the active node becomes passive
>
>1.
>
>Enable rejecting all requests to broker node except the CLOSE-OK
>2.
>
>Close existing AMQ connections (both publishing and consuming)
>
>
> Tasks to do when the passive node becomes active
>
>1.
>
>Start accepting all requests to broker node
>
>
> The assumption here is the active components like the Inbound disruptor,
> outbound disruptor, delivery task managers will stop working (consuming
> resources) when the incoming requests are stopped and subscriptions are
> removed in passive nodes.
>
>
>
>-
>
>Filtering message events will be handled by “InboundEventGatekeeper”.
>-
>
>DeliveryHandler will also be modified to stop delivering messages when
>a node becomes passive.
>-
>
>All the connections will be closed using the server connection registry
>
>
> Suggestions and feedback are appreciated.
>
> --
> Asanka Abeyweera
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 712228648 <+94%2071%20222%208648>
> Blog: a5anka.github.io
>
> 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Thanks
Abimaran Kugathasan
Senior Software Engineer - API Technologies

Email : abima...@wso2.com
Mobile : +94 773922820


  
  
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture