Create a <treeConnector> for ActiveMQ to provide better support for
hierarchical broker topologies
--------------------------------------------------------------------------------------------------
Key: AMQ-2875
URL: https://issues.apache.org/activemq/browse/AMQ-2875
Project: ActiveMQ
Issue Type: New Feature
Reporter: Adrian Trenaman
The ActiveMQ network connector is excellent at facilitating self-organizing
networks of ActiveMQ brokers with dynamically discovered routing, and can be
used in situations where network availability is sporadic and not guaranteed.
Network connectors can be used along with master-slave pairs to create a
distributed 'messaging fabric'. The network connector can also accommodate the
creation of hierarchical networks, however, I believe there is a good argument
for treating tree topologies as a special case in their own right. In this
note, I'm going to describe why trees should be treated as a special case, and
describe what a treeConnector might look like.
Hierarchical (or 'tree') networks are a special case of the abstract notion of
a network; they have numerous applications in the real world in
cross-geography, wide-area deployments. Consider a retail organization with
1,000's of stores, who wish to send and receive information to and from
head-quarters (HQ) and the stores: you can envisage a hierarchy of ActiveMQ
brokers: one for HQ, one for each of the regions (e.g. Ireland, UK, France,
Germany), and then a broker in each store/outlet in each of the regions. When
HQ wants to send a message to a store, it should be able to write a message to
the store's queue and have it dynamically routed to the store via the regional
broker. Likewise, if a store wants to send a message to HQ, it should be able
to write to a local queue in the store and have the message dynamically routed
to HQ.
Right now in ActiveMQ, there are a number of issues with using a network
connector to achieve a tree topology:
* Spillage of consumer advisories. When a consumer connects to a store, an
advisory message is generated and sent to all brokers within the networkTTL
range. These advisories don't just travel up the tree to the top (HQ): they can
also spill back down the tree to peers who are simply not interested. We have
seen this in a network with 1,000 brokers connected to a regional broker, and a
networkTTL of two: the advisory gets sent to the regional broker, HQ, *and* the
999 peers of the broker.
* Resource wastage (as a consequence of spillage). When a consumer advisory
from a store broker (say, broker 'A') reaches another broker within the
networkTTL (say, broker 'B'), then 'B' creates a subscription for the
destination (to do forwarding) *and* can result in the allocation of a thread
for this destination. So, it's possible that Broker B will allocate a thread
for the destination 'BrokerA.Incoming', despite the fact that it will never
receive or send to this queue. If there are only a small number of peers at
this level then this is not a problem; however, if there are many peers (as in
the case of 1000 stores per region) then this will be noticeable.
* Sensitivity to networkTTL. In order to minimize spillage and resource
wastage, you need to se the networkTTL on each broker to the distance between
the broker and the root - in this example, 2. However, a later reconfiguration
of the network could result in this setting being too low (which means messages
won't go the distance) or too high (which means you get inadvertent spillage)
These issues can be addressed if we create a new tree connector for ActiveMQ.
1) Brokers would identify themselves as 'leaf', 'branch' or 'root' nodes. There
should only ever be one root. If a broker has not been identified as either
'leaf' or 'branch' then it should consider itself to be the root.
2) Tree connectors would be configured using a 'many to one' approach. Leaf
brokers would configure a tree connector to their branch. Branch brokers would
configure a tree connector to the root. The root shouldn't need to configure
anything, except of course a transport listener.
So, a branch broker for the 'UK' region might have the following (to connect
to headquarters):
<treeConnector nodeType="branch"
uri=""tcp://headquarters.myorg.com:61616"/>
And, a leaf broker at a store or outlet in the UK might have the following (to
connect to the 'UK' region'):
<treeConnector nodeType="leaf" uri=""tcp://uk.myorg.com:61616"/>
3) If a consumer connects to a leaf node, then the consumer advisory travels up
the tree, all the way to root: the concept of networkTTL is ignored. If this
advisory travels through a 'branch' then the branch delegates the advisory
upwards. Branches do not pass advisory messages from children to their other
child nodes.
4) If a consumer connects to a branch node, then the consumer advisory travels
up the tree. Additionally, the branch broker will send the advisory downwards
to its children.
5) If a consumer connects to the root, then the consumer advisory is filtered
downwards to all children.
In this way, we reduce spillage of advisory messages, and ensure that trees can
self organize without too much intervention or worrying about the network
time-to-live.
Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
