Ok, that makes sense.
So the readable event is more of an advisory event. The docs should
probably say something about how you could possibly miss the event entirely
if you're doing some other IO before you try and read from the stream.
For posterity's sake, I adjusted my previous example:
var http = require('http');
http.get('http://www.google.com', function (response) {
console.log('got response with status ' + response.statusCode);
setTimeout(function () {
bufferStream(response, function (err, buffer) {
console.log(buffer.toString());
});
}, 1000);
});
function bufferStream(stream, callback) {
var chunks = [];
var chunk = stream.read();
if (chunk) {
chunks.push(chunk);
}
stream.on('readable', function () {
chunks.push(stream.read());
});
stream.on('error', function (error) {
callback(error);
});
stream.on('end', function () {
callback(null, Buffer.concat(chunks));
});
}
You can use the bufferStream function to catch all data on the stream, no
matter how far in the future you are.
--
Michael Jackson
@mjackson
On Mon, Mar 25, 2013 at 1:49 PM, Dean Landolt <d...@deanlandolt.com> wrote:
> You can always call `stream.read`, at any time. This is how data is *
> pulled* off the stream (instead of it being pushed to you, whether you're
> ready or not). Because of this you won't lose any data. With new streams
> there's no real notion of a paused state -- it's always paused. Once you
> grok that it may not seem so counter-intuitive.
>
> The `readable` event is like a corollary to `drain` -- there to tell you
> that it's worth bothering with a call to read. You don't *have* to listen
> for it -- a (needlessly inefficient) stream reader could just as easily
> poll stream.read for new data periodically.
>
>
> On Mon, Mar 25, 2013 at 4:42 PM, Michael Jackson <mjijack...@gmail.com>wrote:
>
>> readable is emitted after you've actually started reading.
>>>
>>
>> That's not what it says in the
>> docs<http://nodejs.org/api/stream.html#stream_event_readable>
>> .
>>
>> ###
>> Event: 'readable'
>> When there is data ready to be consumed, this event will fire.
>> When this event emits, call the read() method to consume the data.
>> ###
>>
>> Calling stream.read *before* you get the "readable" event is totally
>> counterintuitive.
>>
>> --
>> Michael Jackson
>> @mjackson
>>
>> In your example, you dont ever `response.read()`, so no readable event
>>> is ever emitted.
>>>
>>> As you said, streams start in paused state and ready to be read.
>>>
>>> On 03/25/13 22:28, Michael Jackson wrote:
>>> > Is it correct to assume that a Readable won't emit the "readable" event
>>> > until you're registered for it?
>>> >
>>> > Reading through the streams2 docs, I was under the impression that all
>>> > streams start out paused and don't start emitting data until you add
>>> > either a "data" (for old streams) or a "readable" listener. For new
>>> > streams, this should mean that they don't emit "readable" until at
>>> least
>>> > one listener is registered. Otherwise we still need to do some
>>> buffering
>>> > in order to capture all the data.
>>> >
>>> > For example, this code misses the readable event on node 0.10:
>>> >
>>> > var http = require('http');
>>> >
>>> > http.get('http://www.google.com', function (response) {
>>> > console.log('got response with status ' + response.statusCode);
>>> >
>>> > setTimeout(function () {
>>> > response.on('readable', function () {
>>> > console.log('readable');
>>> > });
>>> >
>>> > response.on('end', function () {
>>> > console.log('end');
>>> > });
>>> > }, 5);
>>> > });
>>> >
>>> > Here's my shell session:
>>> >
>>> > $ node -v
>>> > v0.10.0
>>> > $ node http-test.js
>>> > got response with status 200
>>> > $
>>> >
>>> > Is this the correct behavior?
>>> >
>>> > --
>>> > Michael Jackson
>>> > @mjackson
>>> >
>>> >
>>> > On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <i...@izs.me
>>> > <mailto:i...@izs.me>> wrote:
>>> >
>>> > re old-mode
>>> >
>>> > Yes, that's fine. If you just want to get all the data asap, use
>>> > on('data', handler). It'll work great, and it's still very fast.
>>> > pause()/resume(), the whole bit. (The difference is that it won't
>>> > emit data until you're listening, and pause() will *actually*
>>> pause.)
>>> >
>>> >
>>> > Re read(cb)
>>> >
>>> > It's problematic for reasons that I've discussed all of the places
>>> > where it's been brought up. That horse is dead, let's stop beating
>>> > it. (There were a few other proposals as well, btw. Reducibles
>>> and
>>> > some other monadic approaches come to mind.)
>>> >
>>> >
>>> > Re pipe() vs looping around read() vs custom Writable vs on('data')
>>> >
>>> > Whatever works for your case is fine. It's flexible on purpose,
>>> and
>>> > allows more types of consumption than streams1, and creating custom
>>> > writables is easier than it was in streams1.
>>> >
>>> > If you find something that the API can't do for you, or find
>>> yourself
>>> > doing a lot of backflips or overriding a lot of methods to get your
>>> > stuff working, then let's chat about it in a github issue. You
>>> might
>>> > be missing something, or you might have found a genuine
>>> shortcoming in
>>> > the API.
>>> >
>>> >
>>> >
>>> > On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson
>>> > <ziggy.jonsson....@gmail.com <mailto:ziggy.jonsson....@gmail.com>>
>>> > wrote:
>>> > > Thanks for all the answers. I almost forgot to look back at this
>>> > thread as
>>> > > the custom writeStreams have exceeded the high expectation I had
>>> > already for
>>> > > Streams 2.
>>> > > For me, the reference manual was a little confusing, as there are
>>> > complete
>>> > > examples on using the read method, no mention of "reading"
>>> through a
>>> > > writeStream endpoint.
>>> > >
>>> > > Marco, I agree that that read has more detailed control of
>>> minimum
>>> > incoming
>>> > > content. However I wonder if it would be more efficient to
>>> default
>>> > > pipe.chunkSize to a "lowWatermark" of the receiver (if defined).
>>> > This
>>> > > lowWatermark could be adjusted dynamically and the callback in
>>> the
>>> > writable
>>> > > should keep sequence of events under control?
>>> > >
>>> > > Anyway, thanks Node team, I'm very impressed!
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco Rogers
>>> wrote:
>>> > >>
>>> > >> @Nathan's response is right. Creating a writable stream is
>>> > preferable in
>>> > >> most cases. But I wanted to add a little context to that. If
>>> > you're dealing
>>> > >> with a base readable stream, it's just pushing chunks of data at
>>> > you off the
>>> > >> wire. Your first task is to collect those chunks into meaningful
>>> > data. So
>>> > >> IMO the reason creating a writable stream is preferable is
>>> because it
>>> > >> prompts you not just read off the stream, but to create
>>> semantics
>>> > around
>>> > >> what the new stream is supposed to be. The api reflects this
>>> > opinion and
>>> > >> that's why creating writable streams feels like the more natural
>>> > way, and
>>> > >> the ugliness of dealing with read() is wrapped up in the pipe()
>>> > method. It
>>> > >> was kind of designed that way.
>>> > >>
>>> > >> But the read() api was also designed for a use case. It's meant
>>> > to handle
>>> > >> low/high water marks effectively, as well as enable more
>>> > optimized special
>>> > >> parsing by reading off specific lengths of chunks. These were
>>> > things that
>>> > >> people kept needing, but the old api didn't support well. If
>>> you were
>>> > >> writing a library for a special parser, you might write a custom
>>> > Writable
>>> > >> stream and inside it you would be using the read(n) api to
>>> > control *how* you
>>> > >> read data off the socket. I hope that makes sense.
>>> > >>
>>> > >> :Marco
>>> > >>
>>> > >> On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir Jonsson
>>> wrote:
>>> > >>>
>>> > >>> The new streams have excellent support for high/low watermarks
>>> and
>>> > >>> auto-pausing/resuming, but the documentation confuses me a
>>> little...
>>> > >>> particularly the read method.
>>> > >>>
>>> > >>> When I read the new docs for the first time I was under the
>>> > impression
>>> > >>> that the optimal way to become a user of a stream is to write
>>> > loops around
>>> > >>> the read functio. However in practice I find myself simply
>>> > writing custom
>>> > >>> writeStreams and use the callback to control upstream pressure
>>> > (in addition
>>> > >>> to source Watermarks if needed). Here is an example where I
>>> > move the
>>> > >>> output to a queue that executes a custom function in parallel
>>> (i.e.
>>> > >>> uploading to a database)
>>> https://gist.github.com/ZJONSSON/5189249
>>> > >>>
>>> > >>> Are there any benefits to using the read method directly on a
>>> > stream vs.
>>> > >>> piping to a custom Writable stream?
>>> > >
>>> > > --
>>> > > --
>>> > > Job Board: http://jobs.nodejs.org/
>>> > > Posting guidelines:
>>> > >
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> > > You received this message because you are subscribed to the
>>> Google
>>> > > Groups "nodejs" group.
>>> > > To post to this group, send email to nodejs@googlegroups.com
>>> > <mailto:nodejs@googlegroups.com>
>>> > > To unsubscribe from this group, send email to
>>> > > nodejs+unsubscr...@googlegroups.com
>>> > <mailto:nodejs%2bunsubscr...@googlegroups.com>
>>> > > For more options, visit this group at
>>> > > http://groups.google.com/group/nodejs?hl=en?hl=en
>>> > >
>>> > > ---
>>> > > You received this message because you are subscribed to the
>>> Google
>>> > Groups
>>> > > "nodejs" group.
>>> > > To unsubscribe from this group and stop receiving emails from it,
>>> > send an
>>> > > email to nodejs+unsubscr...@googlegroups.com
>>> > <mailto:nodejs%2bunsubscr...@googlegroups.com>.
>>> > > For more options, visit https://groups.google.com/groups/opt_out
>>> .
>>> > >
>>> > >
>>> >
>>> > --
>>> > --
>>> > Job Board: http://jobs.nodejs.org/
>>> > Posting guidelines:
>>> >
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> > You received this message because you are subscribed to the Google
>>> > Groups "nodejs" group.
>>> > To post to this group, send email to nodejs@googlegroups.com
>>> > <mailto:nodejs@googlegroups.com>
>>> > To unsubscribe from this group, send email to
>>> > nodejs+unsubscr...@googlegroups.com
>>> > <mailto:nodejs%2bunsubscr...@googlegroups.com>
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>>> >
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> > Groups "nodejs" group.
>>> > To unsubscribe from this group and stop receiving emails from it,
>>> > send an email to nodejs+unsubscr...@googlegroups.com
>>> > <mailto:nodejs%2bunsubscr...@googlegroups.com>.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Job Board: http://jobs.nodejs.org/
>>> > Posting guidelines:
>>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> > You received this message because you are subscribed to the Google
>>> > Groups "nodejs" group.
>>> > To post to this group, send email to nodejs@googlegroups.com
>>> > To unsubscribe from this group, send email to
>>> > nodejs+unsubscr...@googlegroups.com
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>>> >
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> > Groups "nodejs" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an email to nodejs+unsubscr...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>> >
>>> >
>>>
>>>
>> --
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nodejs@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "nodejs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to nodejs+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
