Interesting.
Kevin Kessler wrote:
We are trying to build a system that could be thought of as a sort of log
aggregator, with a small handful of listeners, and a much larger number of
publishers (~150). The messages are going to be 2-5K in size, and we may
burst up to 3000 messages per second. My main concern is that no Subscriber
or JMS issue affects the publishers. That means no flow-control, no-broker
problem causing a publisher to hang, etc. If something happens I just want
to lose messages, and the publishers continue their real jobs unimpeded.
Publish to a topic with no durable subscriber and use non-persistent
delivery and you should be fine I think. Though you might run into the
slow consumer problem, you can use the eviction strategy of ActiveMQ to
handle it.
http://www.activemq.org/site/slow-consumer-handling.html
Most of the publishers will be JBoss, although some are WebLogic. I’ve
played around with the embedded broker in JBoss, using log4j’s JMS appender
to publisher messages, and I was very happy with the performance. I think I
don’t want to use a traditional hub and spoke sort of configuration, where
the JMS clients connect to a centralized cluster of brokers, because I don’t
want to make a remote network call in-line with the publishing system’s
actual function call just for logging, and I don’t need durable
subscriptions or delivery guarantees that the hub and spoke design lends
itself to. I figured a more peer-to-peer approach of embedded brokers would
match the requirements better. My initial idea was to use unreliable
multicast to transmit the messages between the brokers, but, reading your
web site, it doesn’t look like that is a design that you favor for a
production environment. I’m concerned that making a TCP connection to every
other broker in the environment is going to be a big problem, especially
since the vast majority of the brokers are publishers, and have no interest
in what the other brokers are doing.
I would suggest using an embedded broker for each application that is
doing the logging, this decouples your publisher from any external
broker. Then creating a separate embedded broker with your consumer that
would network with the publishers broker to receive messages. There
could be a better way, but thats my two cents for now. :)
Do you think ActiveMQ is a match for this application, and what transport
protocol would you suggest?
I think ActiveMQ can handle it. VM transport with a TCP transport for
the publisher and consumer I think.
Thanks,
Kevin Kessler
