On Sep 13, 2010, at 1:00 PM, Stefan Majer wrote:
> Hi,
>
> I´m implementing a application which is basically a kind of log
> collection framework. It is basically divided into two parts:
> - message producers, many of them are installed on a node and
> produces log messages, send via zmq to 2 message consumers.
> - message consumers, as the name says, consumes the log messages from
> all producers and persist these messages in a NoSQL Datastore.
>
> The Idea is to be high available through 2 consumers and the use of
> the NoSQL datastore, but dont get duplicate log messages persisted.
>
> I started using a PUB Socket on the producer side, and a SUB Socket on
> the consumer side.
> I have high availability but both consumers got all messages. So i end
> up having every message persisted twice.
>
> After that i used a PUSH Socket on the producer side and still a SUB
> Socket on the consumer side.
> Now the messages produced from one producer get evenly distributed
> between the two consumers.
> But when i stop one consumer the other one did not get any messages
> anymore. If the stopped consumer is started again it get all messages
> sent during the down period.
> This is not bad, but i want to have the left produces to get all the
> messages send all the time.
You can't mix PUSH and SUB sockets. Those sockets are used for two different
patterns.
PUSH pairs with PULL.
PUB pairs with SUB.
The text in your last paragraph doesn't make a lot of sense ("i want to have
the left produces to get all the messages send all the time").
What I *think* you are asking for is how to have each consumer continue to
receive messages even when one consumer fails. I think you would do this by
setting a HWM (high water mark) on your PUSH socket to 1 so that no more than 1
message is ever queued at any given time. If a consumer fails (the PULL socket
goes down) then only 1 message is "lost"; then the PUSH socket should start to
load balance new messages to the remaining consumers.
cr
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
