James, Yes, it sounds like the JEDI thing and the partitioning approach is what I need. And yes, I'm talking queues for the most part. For the partitioning, is that something that AMQ, or are you talking about me having a layer in front of AMQ that would do this. In this case, instead of having one big network of brokers, I would have several smaller networks of brokers?
Marc James.Strachan wrote: > > On 05/12/2007, Marc Zampetti <[EMAIL PROTECTED]> wrote: >> James, >> >> You are right about the HA comment I originally made. I was referring to >> fact that I'm not looking for persistent messages. But I am concerned >> about >> what happens when a broker fails and being able to recover from that >> quickly, even if that means losing messages. > > Master/Slave is all about replicating messages to avoid message loss - > it comes at a heavy performance cost. If you can't afford that cost, > you don't need master/slave. i.e. the normal failover of the JMS > client will do fine. > > >> I understand the point about the need for more information. Here is what >> I >> say so far. >> >> There will be around 30 - 50 processes producing messages, each one >> producing the same type of messages. Think it of a farm of processes >> doing >> the same basic work, spread out to handle processing load, scalability, >> fault tolerance, etc. These processes know nothing about what happens to >> these messages once they send them. >> >> On the consuming side, there are multiple farms of processes, each farm >> doing a particular type of processing on some subset of the messages. >> There >> is nothing in the message, other than a unique id, that can be used to >> determine which messages should be sent to each farm. Each of these farms >> would have anywhere from 2 - 30 processes, depending upon the >> functionality >> and load requirements. > > So we're talking queues right? And by the sounds of it you could > partition messages across multiple brokers for scalability. (Sounds > like a good use case for the JEDI transport that still needs > completing BTW to stripe message flows across different brokers to > avoid the downsides of the store and forward network of brokers). > > >> For the AMQ topologies, I was thinking of a Virtual Topic that all of the >> publishers send messages to, with queues behind that go to some sort of >> routing layer that in turns sends the message to the queue that the >> service >> the relevant consumer farm. > > Yeah. I guess it depends on how complex the routing is. It might be > easier to just have an input queue, then some consumer process (which > could be hosted inside the broker if need be) to do the routing; > whether its Camel or some custom JMS code or whatever. > > -- > James > ------- > http://macstrac.blogspot.com/ > > Open Source Integration > http://open.iona.com > > -- View this message in context: http://www.nabble.com/Questions-on-Network-of-Brokers-and-high-message-rates-tf4941283s2354.html#a14174653 Sent from the ActiveMQ - User mailing list archive at Nabble.com.
