I am also dealing with a file-sending case and want to avoid sending at read speed for the same reasons. I've decided to use the credits-based flow control approach Pieter suggested. For pub/sub, you really only need one subscriber sending credits, but the nice thing about the approach is that it can be used for either fastest subscriber or slowest subscriber flows depending on how you organize sending the credits.
On Saturday, September 22, 2012 10:23:16 PM Edwin Amsler wrote: > Heh. Not quite. > > In my particular case, I'm creating data at about 500MB/s (a file read > in this case that's limited by in-memory kernel file cache), and I'm > capping the outbound bandwidth at around 40Mb (~4MB/s) for OpenPGM. Not > setting a high water mark, I quickly eat all available memory and my > system swaps heavily (and indefinitely depending on the file I'm > sending). Setting it to 128 messages (~1MB/message which is big, I'll > admit), means that the first 128 megabytes go out just fine, then chunks > here and there make it out when the queues begin to empty. > > I also have some control data here and there in band, but I need to hope > I have room in the send queue so they don't just get silently thrown out. > > I have a mechanism out of band over TCP that re-requests pieces once the > transfer is done, but I'm never actually sure when it's done sending so > I just wait 1 minute before re-requesting. > > If I had some indicator of whether or not the message goes missing, I > could re-transmit or throttle back the 500MB/s to what the network is > actually able to provide. > > On 22/09/2012 10:04 PM, Bennie Kloosteman wrote: > > ADSL upload ??? > > > > On Sun, Sep 23, 2012 at 10:48 AM, Edwin Amsler > > <edwinams...@thinkboxsoftware.com > > > > <mailto:edwinams...@thinkboxsoftware.com>> wrote: > > It's unlikely that an application can produce more data per second > > than > > the network hardware is able to handle? > > > > On 22/09/2012 12:57 AM, Pieter Hintjens wrote: > > > On Sat, Sep 22, 2012 at 12:57 AM, Edwin Amsler > > > <edwinams...@thinkboxsoftware.com > > > > <mailto:edwinams...@thinkboxsoftware.com>> wrote: > > >> It was mentioned that under the hood, the PUB-SUB system had > > > > individual > > > > >> outgoing queues, each with their own water mark counters. What > > > > happens > > > > >> to a message when all queues are full? > > > > > > This is such an unlikely case... almost contrived. The real > > > > issue with > > > > > high-speed pub/sub is a small number of clients disconnecting or > > > getting swamped by other work, and their queues building up, and > > > causing memory exhaustion. > > > > > > The best strategy to keep data flowing but also ensure > > > > reliability is > > > > > then some kind of out-of-band recovery for clients that need it. > > > There's some ideas in the Clone pattern in the Guide (request > > > > snapshot > > > > > at startup, then apply changes as they arrive to the snapshot). > > > > > > -Pieter > > > _______________________________________________ > > > zeromq-dev mailing list > > > zeromq-dev@lists.zeromq.org <mailto:zeromq-dev@lists.zeromq.org> > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org <mailto:zeromq-dev@lists.zeromq.org> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
