> In any case, one question of mine has remained unanswered so far: > How does one realize horizontal scaling? So far everyone seems to assume a > single machine/node. > > Regardless of the socket type used, I will run into some limit at some point. > Then I need to add nodes to the cluster. How is the load balanced between > them?
Maybe not … it sounds like you’re sending a small number of small messages infrequently, but to a VERY large number of recipients. With a point-to-point protocol like TCP you’re correct — you’ll hit some limit on what a single machine can handle, so will need to scale horizontally, which can be quite tricky. Which is the beauty of multicast — you’re leveraging the network itself to do the fan-out. So multicast may enable you to avoid scalability problems, but only if multicast works in your use case. If you need to go point-to-point, then you will likely need to do some kind of “sharding”, and like anything else simpler solutions are, well, simpler. For instance, when you exceed what a single machine can handle (quite a lot depending on the machine), then you go to two machines, then four, etc. At each step you need some kind of maintenance window to re-balance traffic across servers, and some criteria to decide which recipients belong in which “shard”. You can also make the sharding dynamic, but that is much more work — the advantage in your case is that it sounds like the traffic is not “transactional”, so you don’t care if a message gets delivered more than once, which makes the dynamic approach more feasible (but still tricky). _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
