>> If you have theoretical background in this area, please do check >> UPSTREAM/DOWNSTEAM sockets and let me know if you believe something is >> broken or missing there. > > The first is a very flexible, even autonomous load > distribution/balancing scheme(s) that take into consideration various > aspects of distributed pipeline processing like the distribution of > components (local, lan, wan, etc) as well as the various individual > machine characteristics like cpu, memory, etc. Currently I believe, > only a round-robin scheme is available.
I'm developing something similar for the network analytics company I work for. Unfortunately, you can't always rely on the messaging layer to do the partitioning for you — in a couple of our use cases, we're doing some key-based aggregation, and so we have to ensure that all records with the same key get sent to the same downstream partition. In theory, you could have an UP/DOWNSTREAM socket handle this for you, but you'd have to pass in a hash of the key for every message. Maybe at some point that becomes another message option? In the meantime, I just have all partitioning — both round-robin and key-based — happen at the application layer in separate components. That means that every 0mq socket is single-source, single-sink, which means (I think, correct me if I'm wrong) that I can get away with using PAIR sockets instead UP/STREAM sockets. –doug _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
