Olivier - look at qpid dispatch router that¹s how I'd handle it. Jack
Jack Gibson Chief Architect Core Payments Platforms @ PayPal On 5/20/15, 5:31 AM, "Olivier Mallassi" <olivier.malla...@gmail.com> wrote: >Thank you rob. I will use non persitent messages for this. > >I like your idea of load-balancing at "message layer" instead of >"connection layer"! I need to figure out how to implement this, but this >is >clearly a more elegant solution (knowing we do not have thousands of >processes and connections) > >PS : for the record, we will also "functionally partitioned" : event of >type A goes to broker A (eg. a dedicated JVM), event of type B goes to >broker B > >On Wed, May 20, 2015 at 12:48 PM, Rob Godfrey <rob.j.godf...@gmail.com> >wrote: > >> On 20 May 2015 at 12:00, Olivier Mallassi <olivier.malla...@gmail.com> >> wrote: >> > Hi all >> > >> > my apologies in advance if I ask dummy questions and for this (maybe >>too) >> > long email but *i need some help in regards with "qpid clustering"*. >> > I have read some of the already asked questions on the ML and would >>like >> to >> > confirm (or not) some points. >> > >> > *Overview*: >> > I wanna use the Java Broker (mainly because I need to support several >>OS >> - >> > linux, solaris, windows....) but I am wondering our to scale out (or >>how >> to >> > cluster) qpid (target throughput will be around 150k events per sec, >> > message size variable but less than 1kB, connections number is not >>known >> > but dozens. >> > I saw some numbers on the ML (like 70k events per sec for a single >>broker >> > etc...) so it appears I will have to distribute my load on 2 or 3 >> brokers. >> > >> >> Are you using persistent or non persistent messages? The throughput >> the broker can achieve will obviously be very much tied to the type of >> messages and the sort of hardware you are using. >> >> > for now, I am not talking about HA. >> > >> > *1/ *I can cluster qipd using LVS or corosync. not ideal in my case >> because >> > it relies on OS specific solution >> > >> > *2/* I noticed that the messaging api uses the first IP in the >>specified >> > broker list (amqp://guest:guest@ >> > /default?failover='roundrobin?cyclecount='2''&brokerlist=") >> > >> > So I can use a custom strategy (round robin, zookeeper based), coded >>in >> > java, to shuffle the brokers list and ensure to randomly distribute >>the >> > connections (when they are established). it works for fail over, can >>be >> > tricky if you want to be "elastic" but it should work. >> > >> > what are your thought on that approach? >> >> Depending on exactly how your application works, that sounds >> reasonable... you'd obviously need to ensure that every destination >> produced to had consumers on... >> >> > >> > *3/* I am yet facing an issue because in some cases, I need to use >>node >> > affinity (based on JMSXGroupID...). so given a value of JMSXGroupID, I >> need >> > to ensure the same consumer jvm will get it. >> > so I need to ensure that for a given JMSXgroupID, publisher & consumer >> use >> > the same broker (which begin to be tricky, even using hashcode, in >>case >> of >> > failover etc...) >> >> >> At this point it might seem like it would make more sense for every >> producer/consumer to have connections to every broker... and to do the >> round-robining at a higher lever than establishing the connection. >> The client can then make the choice on a per-message basis as to which >> broker to send to - I have seen this pattern used elsewhere. >> >> > Does federation (Dispatch Router) helps in that case (knowing it will >>add >> > network hops, and so latency)? Is there any getting started/tutorial >>that >> > can help better understanding the fixedAdress and linkRoutePattern? >> > >> > 4/ what about HA? It appears I will finish with a topology like >> > | | A.Master --> A.Replica >> > Client | "hash" | B.Master --> B.Replica >> > | | C.Master --> C.Replica >> > >> >> Unfortunately I'm not very well acquainted with Dispatch - so I'll let >> others step in here. >> >> -- Rob >> >> > >> > Hope I was clear enough. many thanks for your help/feedbacks. >> > Regards >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
