Multiple Java Brokers - Use CasesPage edited by Martin RitchiePurposeThis page is intended to outline the known use cases for running multiple Java Brokers, addressing logged issues and limitations of the current implementation (as of V0.5). It is not about clustering proper. Use CasesHigh Volume Transient BrokerDescription This use case relates to applications with a high residual message load i.e. where message data on the broker remains in memory for some time or consumption lags production such that a backlog is constantly present in the broker queues. This paradigm is reasonably common, partly because publication threads are generally handling only the simple publish call where we often see consumption threads handling writes to RDBMS or other time expensive processing. Thus a rate gap opens up, and creates a data tailback. In this scenario, particularly for deployments on a 32bit VM, the broker can exhaust a 3GB heap or start to perform poorly as it approaches max heap. Result Broker side OoM or performance degradation requiring bounce. Messages in flight not processed, client applications experience connection loss. Possible Solution A - Load Balancing Module For our end users, we could potentially reduce the hassle factor in running 2 brokers by providing a solution comprised of load balancing module which would reside alongside the broker i.e. on server side. This module would intercept published messages and share them between multiple brokers (scaling according to app parameters). Consumers would require multiple connections, but publication would be unaffected and the burden of load balancing could be shifted from the user application to Qpid. Possible Solution B - No Message Order, 2 Brokers In this scenario, it would be possible to use 2 brokers and message order would not matter. Publishing clients would use 2 connections and publish alternately to each broker, providing a simplistic load balancing solution. Consuming clients would then consume from the 2 brokers, using the same topic name etc. The consumer could choose to consume in parallel, thus potentially speeding up processing time or by taking messages singly from the two sources alternately. Possible Solution C - With Message Order, Paired Flow Again, using 2 brokers but this time working with the assumption that the application data flows across the broker can be separated by source/destination. An easy solution for this is to simply divide the required traffic by source or destination and put an amount on each broker. This may necessitate multiple consuming connections (to each of the brokers) on the client side where there are multiple sources feeding the same client. Alternatively, for some applications, the clients can be segmented in pairs of publisher-consumer by flow. Possible Solution D - Redirect to Passive Broker An alternative approach might be to monitor heap use on the primary broker and kick off a second broker once the first is under heavy load. Client connections (publishing and consuming) would required to be redirected to the secondary broker until the first broker recovers. This is a kind of active-passive pair approach, indeed the secondary broker could be up all the time and simply redirected to as required. Rob mentioned that AMQP 1-0 has the concept of redirect, so it may be something we could look at to inform the solution. There are some questions around a redirect solution:
Considerations
Change Notification Preferences
View Online
|
View Change
|
Add Comment
|
