On Mon, Jun 2, 2014 at 5:59 AM, Jeremy Richemont <jrichem...@gmail.com> wrote:
> Hi Tim. I agree, a message can only be deemed to have been 'sent' if an > acknowledgement for it is received. As soon as no ack is received the > server must start backing up messages - but only for that client. It's > perfectly possible to have one client up-to-date, one that's catching up > from 200 messages ago and another catching up from 1000 messages ago. The > server will need to know about each client - either because it's been > configured to expect a certain set of clients or through some discovery > mechanism whereby the client connects and tells the server about itself. > You're describing a fully synchronous topology where every message gets acked to a central broker by every client. You can of course write a 0mq broker to to this but you might as well save the time and effort and use AMQP or a simple database behind an HTTP server if those are you requirements. -Michel > > On 30 May 2014 20:03, Tim Crowder <crowd...@yahoo-inc.com> wrote: > >> Hi Jeremy- >> >> You might take a look at existing systems for how they handle it. >> Apache Kafka, for instance, always maintains a backlog of messages, >> via fast-append to logfiles. Clients keep track of the "offset" of the >> last message they processed, and actively pull new messages from >> the publisher. >> >> Even if you know when the connection dropped, it's hard to know >> which message was completely processed (vs delivered) last. >> So, without client pulls, you need explicit client acknowledgement of >> the last processed message. >> This means that you have to keep saving/buffering messages until all >> clients acknowledge them, then you can discard messages up to that point. >> >> Cheers! >> .timrc >> >> ------------------------------ >> *From:* zeromq-dev-boun...@lists.zeromq.org [ >> zeromq-dev-boun...@lists.zeromq.org] on behalf of Jeremy Richemont [ >> jrichem...@gmail.com] >> *Sent:* Friday, May 30, 2014 7:21 AM >> *To:* ZeroMQ development list >> *Subject:* Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list >> >> Thanks, Charles. I did, in fact, find that pattern. The problem is it >> does not match what I am trying to do. That pattern for when you have state >> + deltas. What I have is a continuous message stream which, once started to >> client x must be preserved even if client x dies for a bit (not forever of >> course, I put an SLA of 1 million messages/client) and then reconnects, >> every message it missed is replayed, in order, then the live stream >> resumes. >> >> It needs to handle n clients, any of which may drop and reconnect so >> each one will need an independent message cache. PUB/SUB will not do for >> this because I may need to send messages 10 - 100 to client x on reconnect >> but 50 - 200 to client y. >> >> Asking for state is a good idea - ask for missing updates in my case - >> but the question remains; how does the server know the client is no longer >> available and it must therefore start backing up messages from a PUB >> socket? The client can't tell it over OOB because it died already. >> >> If I could just query PUB and get a list of clients plus a notification >> when one drops that'd solve the problem I think. But how to do that? >> >> Jeremy >> >> On 30 May 2014 15:00, Charles Remes <li...@chuckremes.com> wrote: >> >>> Take a look at the Clone pattern in the zguide. >>> >>> http://zguide.zeromq.org/page:all#Reliable-Pub-Sub-Clone-Pattern >>> >>> This might be what you need. >>> >>> cr >>> >>> On May 29, 2014, at 11:20 AM, Jeremy Richemont <jrichem...@gmail.com> >>> wrote: >>> >>> > >>> > Hi. I am struggling to work out how to use zmq to implement the >>> architecture I need. I have a classic publish/subscribe situation except >>> that once client x has subscribed to a topic I need the topic data to be >>> sent to it to be cached if the client dies and resent on reconnect. The >>> data order is important and I can't miss messages should the client be >>> offline for a while. >>> > >>> > The PUB/SUB pattern doesn't seem to know about individual clients and >>> will just stop sending to client x if it dies. Plus I can't find out this >>> has happened and cache the messages, or know when it reconnects. >>> > >>> > To try to get around this I used the REQ/REP pattern so the clients >>> can announce themselves and have some persistence but this is not ideal for >>> a couple of reasons: >>> > >>> > 1) The clients must constantly ask "got any data for me?" which >>> offends my sensibilities >>> > >>> > 2) What happens if there's no data to send to client x but there is to >>> client y? Without zmq I'd have had a thread per client and simply block the >>> one with no data but I can't block client x without also blocking client y >>> in a single thread. >>> > >>> > Am I trying to shove a round peg in a square hole, here? Is there some >>> way I can get feedback from PUB saying 'failed to send to client x'? so I >>> can cache the messages instead? Or is there some other pattern I should be >>> using? >>> > >>> > Otherwise it's back to low level tcp for me... >>> > >>> > Many thanks; >>> > >>> > Jeremy >>> > >>> > _______________________________________________ >>> > zeromq-dev mailing list >>> > zeromq-dev@lists.zeromq.org >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
