I'm not sure I get it but I'll try to answer.
What I'm describing in the post is how logic can be expressed with
generators rather than callbacks. I'm assuming that the low level calls are
callback-style. So, there is no reference to any specific I/O library
and/or to back pressure.
The back pressure problem is a problem that I'm handling in streamline's
streams module. And I'm handling it with a simple pair of async calls:
stream.read(cb) and stream.write(cb, buffer) that are small wrappers around
node streams.
The read call pauses and resumes the underlying stream based on some
configurable high/low mark buffering limits (you can set them to 0 but then
the stream will pause every time it needs to buffer a chunk).
The write call deals with the drain event under the hood. If the lower
level call write call returns true, the callback is called immediately
(streamline trampolines so there is no risk of stack overflow in callback
mode). If it returns false, the callback is triggered by the drain event.
Pump loops can be written as:
while (data = input.read(_))
output.write(_, data);
In callback mode, streamline transforms this loop into something like:
(function loop() {
input.read(function(err, data) {
if (err) return cb(err);
if (data)
output.write(function(err) {
if (err) return cb(err);
loop():
}, data);
else cb();
});
})();
In generators mode it transforms it into:
while (data = yield input.read(_))
yield output.write(_, data)
The code looks very different but it execute just like the callback code
above. The input.read and output.write calls will use "invoke" and
callbacks to interact with the underlying node streams APIs. The run loop
that I've given in my post will exit at spot (c) every time a callback is
pending and the callback will reactivate it by calling resume; so, even
though the code does not look async and callback driven it is actually
completely async and callback driven.
With this pump loop, backpressure happens naturally when the two calls are
combined together. If the output is slower than the input when pumping from
an input stream to an output stream, the write call will start to wait for
drain events. This will naturally stop the pump loop. The input stream will
continue to receive data events but it will just buffer because read won't
be called by the pump loop any more. When buffering goes over the high
mark, the input stream will be paused. Then, at some point, the output
stream will receive a drain event. It will call its callback, which will
resume the pump loop. Read will be called and will get data that has been
buffered. The input stream will be resumed when buffering goes below the
low mark, etc., etc. If the drain event comes before the input reaches the
high water mark, the loop will be resumed and the input stream won't be
paused, which is what we want.
So, even though the pump loop is written as a simple while (data =
input.read(_)) output.write(_, data), it does handle the back pressure.
Bruno
On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote:
>
> How do you handle back pressure?
>
> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote:
>
> Yes, I fixed it. Thanks.
>
> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote:
>>
>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote:
>> >
>> http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/
>>
>>
>> shouldn't that be print(num) not print(n)
>>
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote:
>
> How do you handle back pressure?
>
> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote:
>
> Yes, I fixed it. Thanks.
>
> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote:
>>
>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote:
>> >
>> http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/
>>
>>
>> shouldn't that be print(num) not print(n)
>>
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote:
>
> How do you handle back pressure?
>
> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote:
>
> Yes, I fixed it. Thanks.
>
> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote:
>>
>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote:
>> >
>> http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/
>>
>>
>> shouldn't that be print(num) not print(n)
>>
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
On Saturday, May 19, 2012 9:40:03 PM UTC+2, Mikeal Rogers wrote:
>
> How do you handle back pressure?
>
> On May 19, 2012, at May 19, 20129:51 AM, Bruno Jouhier wrote:
>
> Yes, I fixed it. Thanks.
>
> On Saturday, May 19, 2012 3:15:33 PM UTC+2, Matthew Hazlett wrote:
>>
>> On 5/19/2012 6:20 AM, Bruno Jouhier wrote:
>> >
>> http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/
>>
>>
>> shouldn't that be print(num) not print(n)
>>
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
