Re: [zeromq-dev] Thinking out loud ...

2011-07-01 Thread Fabien NiƱoles
2011/6/30 Pieter Hintjens p...@imatix.com: Artur, an interesting paper. If you have running code you might want to refer to that in the spec. -Pieter Agree on this. It should probably be part of the guide with some introduction explaining how it compares to message routing protocols

Re: [zeromq-dev] Thinking out loud ...

2011-06-30 Thread Artur Brugeman
On Tue, Jun 28, 2011 at 12:06 PM, Martin Sustrik sust...@250bpm.com wrote: On 06/28/2011 05:57 AM, Artur Brugeman wrote: It's based on XREP sockets now. I've given quite a lot thinking to it, and can share my experience. Please, do so. Martin As the letter grew, I've decided to post

Re: [zeromq-dev] Thinking out loud ...

2011-06-30 Thread Andrew Hume
i don't understand what this sort of specification is meant to accomplish. it seems to be a description of a processing flow for a rudimentary job scheduling system. but its not complete enough for anyone to use nor implement. its hard to discern what is 0mq related, and what is job scheduler

Re: [zeromq-dev] Thinking out loud ...

2011-06-30 Thread Artur Brugeman
Hi, Andrew. i don't understand what this sort of specification is meant to accomplish. I said I didn't find a better place to post this on zmq website, so posted it at rfc.zeromq.org. It is simply a too big for an email message. It is meant to accomplish availability for people to read it. it

Re: [zeromq-dev] Thinking out loud ...

2011-06-30 Thread Pieter Hintjens
On Thu, Jun 30, 2011 at 4:27 AM, Andrew Hume and...@research.att.com wrote: and is the style to talk about such things on the list or on the web page? List is better, we didn't enable comments on the spec pages because such comments tend to become outdated very rapidly. Artur, an interesting

Re: [zeromq-dev] Thinking out loud ...

2011-06-28 Thread Martin Sustrik
On 06/28/2011 05:57 AM, Artur Brugeman wrote: It's based on XREP sockets now. I've given quite a lot thinking to it, and can share my experience. Please, do so. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org

Re: [zeromq-dev] Thinking out loud ...

2011-06-28 Thread Henry Baragar
Andrew, On June 24, 2011 08:41:13 PM Andrew Hume wrote: henry, i'm not sure i followed all your examples, but i detected a general meme which others have complained of in the past. i admit to being sensitised to this issue because i have worked in this area for over a decade now.

Re: [zeromq-dev] Thinking out loud ...

2011-06-28 Thread Andrew Hume
just to expand a little on what pieter said, take heartbeats. many people, when they first confront those, they think the problem is easy. at some level it is, but often people realise that a networking heartbeat does not imply application health, and so they end up doing an application level

Re: [zeromq-dev] Thinking out loud ...

2011-06-28 Thread Martin Sustrik
On 06/28/2011 04:50 PM, Henry Baragar wrote: Does queue management fall into the realm of messaging or job scheduling? If it falls into the realm of messaging, then shouldn't the queue be end-to-end? If it falls into the realm of job scheduling, then should 0mq have any queues at all? I

Re: [zeromq-dev] Thinking out loud ...

2011-06-28 Thread Gregg Irwin
MS I would say the usage of term queue originates from the olden days MS when 0MQ was still thought to be some kind of equivalent to enterprise MS queueing products. The queues are more like network buffers today. Buff Sockets? SoS (Sockets on Steroids)? SSS (Simply Smarter Sockets)? :) -- Gregg

Re: [zeromq-dev] Thinking out loud ...

2011-06-27 Thread Artur Brugeman
Naive approach would be to allow for explicit acks so that there could be at most one request sent to a specific peer at any single moment. That would at least allow load-balancer to distinguish busy peers from idle peers and base the load-balancing on that. However, that doesn't seem to

Re: [zeromq-dev] Thinking out loud ...

2011-06-26 Thread Andrew Hume
it is not synchronous versus asynchronous. it is somehow deeper than that, but confused because in one small part of the space, they overlap. i will take a stab out some text, post it here for comments, then pieter can think about if, or how, he might add it to the guide. andrew On Jun

Re: [zeromq-dev] Thinking out loud ...

2011-06-26 Thread Pieter Hintjens
On Sun, Jun 26, 2011 at 1:06 AM, Andrew Hume and...@research.att.com wrote: i will take a stab out some text, post it here for comments, then pieter can think about if, or how, he might add it to the guide. I'm more than happy to write up any useful patterns in the Guide. Given enough time,

Re: [zeromq-dev] Thinking out loud ...

2011-06-26 Thread Pieter Hintjens
On Fri, Jun 24, 2011 at 11:11 AM, Henry Baragar henry.bara...@instantiated.ca wrote: the slow joiner syndrome... It has to be handled on a pattern-by-pattern basis. My belief today is that if you want synchronization over push-pull, you must switch to a router-based pattern, in the same way as

Re: [zeromq-dev] Thinking out loud ...

2011-06-26 Thread Martin Sustrik
On 06/26/2011 04:26 PM, Pieter Hintjens wrote: It has to be handled on a pattern-by-pattern basis. My belief today is that if you want synchronization over push-pull, you must switch to a router-based pattern, in the same way as we use routers to synchronize request-reply workers. My feeling

Re: [zeromq-dev] Thinking out loud ...

2011-06-25 Thread Martin Sustrik
On 06/25/2011 02:41 AM, Andrew Hume wrote: i think this meme is quite common; nearly all instances of complaints about 0mq's scheduling and buffering, slow joiners and the like, are examples of this -- we want 0mq to trivially do job scheduling for us as well as all the other stuff. That's

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Martin Sustrik
Hi Henry, I have followed the 0MQ mailing list for about a year, experimented with 0MQ and contributed to the 0MQ adaptor for plack. I like many of the features of 0MQ, including asynchronous I/O, multi-language support, fan-out/fan-in connections and end-point connection syntax. But there

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Amr Ali
Yeah I think Henry gone all out on this one as well. These issues mainly are the problem of the application developer not 0MQ (or that's how I see it), basically what 0MQ offers is the infrastructure required to do lots of abstracted communication tasks in the form and shape of patterns. I've

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Martin Sustrik
On 06/24/2011 07:57 PM, Amr Ali wrote: Yeah I think Henry gone all out on this one as well. These issues mainly are the problem of the application developer not 0MQ (or that's how I see it), basically what 0MQ offers is the infrastructure required to do lots of abstracted communication

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Martin Sustrik
On 06/24/2011 09:20 PM, Henry Baragar wrote: Yes, I have a couple of workload distibution applications with highly asymmertric workloads (see my message from from 2011-04-04 Re: LRU broker queue in intuitive way on 3.0 http://lists.zeromq.org/pipermail/zeromq-dev/2011-April/010352.html). I

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Henry Baragar
On June 24, 2011 03:45:20 PM Martin Sustrik wrote: On 06/24/2011 09:20 PM, Henry Baragar wrote: Yes, I have a couple of workload distibution applications with highly asymmertric workloads (see my message from from 2011-04-04 Re: LRU broker queue in intuitive way on 3.0

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Andrew Hume
because the desired granularity (single job) is so small compared to all teh buffering involved, this kind of thing is, i think, best handled by REQ/REP initiated from the worker. i have handled similiar things before in this way, and even had the central job dispenser solve for optimal job

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Martin Sustrik
Hi Henry, OK. AFAIU what would solve your problem is to strictly limit the number of messages in flight to a particular peer, ie. honour HWM exactly without having excess messages in TCP buffers, right? Right. This have been in a pipeline for a long time but there were no

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Henry Baragar
Interesting idea... hooking up a client/server model backwards implements the pull architecure I need for my work distribution applications. It means that the ventilator has to do its own queuing, but I think I can handle that. I am going to give it a try. Thanks! Henry On June 24, 2011

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Henry Baragar
On June 24, 2011 03:59:41 PM Martin Sustrik wrote: Hi Henry, OK. AFAIU what would solve your problem is to strictly limit the number of messages in flight to a particular peer, ie. honour HWM exactly without having excess messages in TCP buffers, right? Right.

Re: [zeromq-dev] Thinking out loud ...

2011-06-24 Thread Andrew Hume
henry, i'm not sure i followed all your examples, but i detected a general meme which others have complained of in the past. i admit to being sensitised to this issue because i have worked in this area for over a decade now. the meme is that of conflating message routing with