Dear all, this is my first post to this list so please bear with my rather lengthy thoughts.
I'm working in a financial markets environment and am currently looking into alternatives for near-realtime trade data message delivery (contracts or deals, not market data ticks). Traditionally we are using Tibco TIB/Rendezvous (and I've extensive experience with it) which has its merits of being mature and rock-solid, but it is truely expensive. Much the same holds true for IBM's WebsphereMQ which is also used in our house already. We need guaranteed or certified delivery for our data, ideally with a quality of service of exactly-once-delivery, At-least-once may also be possible if duplicate delivery is a rare cornercase. To the best of my knowledge even TIB/Rv doesn't provide exactly-once-delivery though the docs don't really explicitly state this, at least in its standard CM (certified messaging) notion. But I've never personally encountered duplicated RV message delivery in a decade. Now, the obvious candidates for replacing the big beasts would be one of the AMQP bunch but of course all of those require a central broker/server. As I really like TIB/Rv's distributed nature I'm now evaluating the use of zeromq (as in: toying around with the idea). Alternatives that I can see: 1. Stay with TIB/Rv. + distributed architecture + proven battle-tested solution + reliable (in a sense of a company not about to vanish soon) commercial support by its creators - immense license and support fees. 2. Use one of the AMQP products (which?): + open source + open standard, so theoretically interchangeable implementations (see below for standards issues) + commercial products available (e.g. Red Hat MRG with QPid under the hood, or VMWare Pivotal RabbitMQ) + commercial support available - central broker/server as a single point of failure (needs its own HA/clustering/failover, dedicated monitoring, dedicated administration) - standardization process seems a mess and might actually have rendered the original AMQP initiative goals useless; though it looks like 1.0 now has rather widespread adoption I somehow have a lurking uncomfortable feeling 3. Roll your own solution, possibly based on zeromq + open source + open standard + full control over the architecture, can be fully distributed + flexibility, a lot of usage patterns + commercial support available (?) - development effort, maybe massive (?) Some additional alternatives might be: Moving this onto WebsphereMQ (also expensive), using Apache Kafka, maybe the Spread library (seems to have hard message size limits). I'm thinking about the possibilities of creating some kind of lightweight distributed pub-sub message bus with zeromq using multicast. Actually much like TIB/Rv conceptually so I wonder a bit why nobody seems to have done this yet with zeromq ;-). At least I'm not aware of it so I hope that's not a totally inappropriate way to use zeromq...? Here goes: 1. (E)PGM or NORM for multicast? zeromq-NORM is not yet in the latest release version of zeromq. 2. I understand it is not possible to use several publishers on one host using the same multicast group and port with (E)PGM due to NACKS only getting back to one publisher. (referring e.g. to this thread: http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm But I'm confused by these posts in this thread: http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm#2012080893pabdv3gkhg18wrqtbnp81gz8 http://grokbase.com/t/zeromq/zeromq-dev/127t25hcex/zmq-mcast-loop-with-pgm#20120809x9w6d4tw53aft99tgb1sf7mwy4 Is it impossible to have one publisher per host using the same multicast group + port? Why? Because if it actually is possible than it shouldn't be too hard to create s.th. much like TIB/Rv's daemon approach, i.e. a bus comprised of dedicated daemons on every host where the actually messaging clients connect to. The daemons handle all of the actual multicast communication. Does NORM also have this restriction/caveat? 3. Guaranteed or certified delivery (QoS) would have to be layered as an applicaton protocol on top of this. Again, this could borrow from TIB/Rv concepts, i.e. publishers storing sent messages persistently until they have been acknowledged by all known/registered receivers. Has anybody done s.th. like this already? I haven't been able to find something like it, apart from some general discussion on "reliability". 4. I understand zeromq does not impose limits on the message size. Is this correct? I'd be grateful for any insight. Best regards Holger Landesbank Baden-Wuerttemberg Anstalt des oeffentlichen Rechts Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz HRA 12704 Amtsgericht Stuttgart _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
