I'm more and more convinced that need to go back and read all the available info about streams2. Answering these detail semantics questions is pretty important.
:Marco On Mon, Mar 25, 2013 at 2:29 PM, Michael Jackson <mjijack...@gmail.com>wrote: > If stream.read returns null that could mean one of two things: > > 1) the stream doesn't currently have any data, but still might have some > in the future > 2) the stream is already ended > > AFAICT, the only way you can know which state you're in is by checking the > stream.readable property which is marked as being a "legacy" property in > the > code<https://github.com/joyent/node/blob/master/lib/_stream_readable.js#L98-L99>, > so it seems like it's not a good idea to rely on that property either. > > -- > Michael Jackson > @mjackson > > > On Mon, Mar 25, 2013 at 1:50 PM, Dan Milon <danmi...@gmail.com> wrote: > >> You're right, my bad. >> >> But still, data stay in the buffer until someone tries to `.read()`. >> So, if you're being passed a stream that you dont know whether the >> first `readable` event has fired, you can try to actually read from >> it. If it returns null, then you wait for `readable`. >> >> On 03/25/13 22:42, Michael Jackson wrote: >> > readable is emitted after you've actually started reading. >> > >> > >> > That's not what it says in the docs >> > <http://nodejs.org/api/stream.html#stream_event_readable>. >> > >> > ### Event: 'readable' When there is data ready to be consumed, >> > this event will fire. When this event emits, call the read() method >> > to consume the data. ### >> > >> > Calling stream.read *before* you get the "readable" event is >> > totally counterintuitive. >> > >> > -- Michael Jackson @mjackson >> > >> > In your example, you dont ever `response.read()`, so no readable >> > event is ever emitted. >> > >> > As you said, streams start in paused state and ready to be read. >> > >> > On 03/25/13 22:28, Michael Jackson wrote: >> >> Is it correct to assume that a Readable won't emit the >> >> "readable" >> > event >> >> until you're registered for it? >> >> >> >> Reading through the streams2 docs, I was under the impression >> >> that all streams start out paused and don't start emitting data >> >> until you add either a "data" (for old streams) or a "readable" >> >> listener. For new streams, this should mean that they don't emit >> >> "readable" until at >> > least >> >> one listener is registered. Otherwise we still need to do some >> > buffering >> >> in order to capture all the data. >> >> >> >> For example, this code misses the readable event on node 0.10: >> >> >> >> var http = require('http'); >> >> >> >> http.get('http://www.google.com', function (response) { >> >> console.log('got response with status ' + response.statusCode); >> >> >> >> setTimeout(function () { response.on('readable', function () { >> >> console.log('readable'); }); >> >> >> >> response.on('end', function () { console.log('end'); }); }, 5); >> >> }); >> >> >> >> Here's my shell session: >> >> >> >> $ node -v v0.10.0 $ node http-test.js got response with status >> >> 200 $ >> >> >> >> Is this the correct behavior? >> >> >> >> -- Michael Jackson @mjackson >> >> >> >> >> >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <i...@izs.me >> > <mailto:i...@izs.me> >> >> <mailto:i...@izs.me <mailto:i...@izs.me>>> wrote: >> >> >> >> re old-mode >> >> >> >> Yes, that's fine. If you just want to get all the data asap, use >> >> on('data', handler). It'll work great, and it's still very fast. >> >> pause()/resume(), the whole bit. (The difference is that it >> >> won't emit data until you're listening, and pause() will >> >> *actually* >> > pause.) >> >> >> >> >> >> Re read(cb) >> >> >> >> It's problematic for reasons that I've discussed all of the >> >> places where it's been brought up. That horse is dead, let's >> >> stop >> > beating >> >> it. (There were a few other proposals as well, btw. >> > Reducibles and >> >> some other monadic approaches come to mind.) >> >> >> >> >> >> Re pipe() vs looping around read() vs custom Writable vs >> > on('data') >> >> >> >> Whatever works for your case is fine. It's flexible on >> > purpose, and >> >> allows more types of consumption than streams1, and creating >> > custom >> >> writables is easier than it was in streams1. >> >> >> >> If you find something that the API can't do for you, or find >> > yourself >> >> doing a lot of backflips or overriding a lot of methods to get >> > your >> >> stuff working, then let's chat about it in a github issue. >> > You might >> >> be missing something, or you might have found a genuine >> > shortcoming in >> >> the API. >> >> >> >> >> >> >> >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson >> >> <ziggy.jonsson....@gmail.com >> > <mailto:ziggy.jonsson....@gmail.com> >> > <mailto:ziggy.jonsson....@gmail.com >> > <mailto:ziggy.jonsson....@gmail.com>>> >> >> wrote: >> >>> Thanks for all the answers. I almost forgot to look back at >> >>> this >> >> thread as >> >>> the custom writeStreams have exceeded the high expectation I >> >>> had >> >> already for >> >>> Streams 2. For me, the reference manual was a little >> >>> confusing, as >> > there are >> >> complete >> >>> examples on using the read method, no mention of "reading" >> > through a >> >>> writeStream endpoint. >> >>> >> >>> Marco, I agree that that read has more detailed control of >> > minimum >> >> incoming >> >>> content. However I wonder if it would be more efficient to >> > default >> >>> pipe.chunkSize to a "lowWatermark" of the receiver (if >> >>> defined). >> >> This >> >>> lowWatermark could be adjusted dynamically and the callback >> > in the >> >> writable >> >>> should keep sequence of events under control? >> >>> >> >>> Anyway, thanks Node team, I'm very impressed! >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco Rogers >> > wrote: >> >>>> >> >>>> @Nathan's response is right. Creating a writable stream is >> >> preferable in >> >>>> most cases. But I wanted to add a little context to that. If >> >> you're dealing >> >>>> with a base readable stream, it's just pushing chunks of >> > data at >> >> you off the >> >>>> wire. Your first task is to collect those chunks into >> > meaningful >> >> data. So >> >>>> IMO the reason creating a writable stream is preferable is >> > because it >> >>>> prompts you not just read off the stream, but to create >> > semantics >> >> around >> >>>> what the new stream is supposed to be. The api reflects this >> >> opinion and >> >>>> that's why creating writable streams feels like the more >> > natural >> >> way, and >> >>>> the ugliness of dealing with read() is wrapped up in the >> >>>> pipe() >> >> method. It >> >>>> was kind of designed that way. >> >>>> >> >>>> But the read() api was also designed for a use case. It's >> >>>> meant >> >> to handle >> >>>> low/high water marks effectively, as well as enable more >> >> optimized special >> >>>> parsing by reading off specific lengths of chunks. These >> >>>> were >> >> things that >> >>>> people kept needing, but the old api didn't support well. >> > If you were >> >>>> writing a library for a special parser, you might write a >> > custom >> >> Writable >> >>>> stream and inside it you would be using the read(n) api to >> >> control *how* you >> >>>> read data off the socket. I hope that makes sense. >> >>>> >> >>>> :Marco >> >>>> >> >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir >> > Jonsson wrote: >> >>>>> >> >>>>> The new streams have excellent support for high/low >> > watermarks and >> >>>>> auto-pausing/resuming, but the documentation confuses me a >> > little... >> >>>>> particularly the read method. >> >>>>> >> >>>>> When I read the new docs for the first time I was under >> >>>>> the >> >> impression >> >>>>> that the optimal way to become a user of a stream is to >> >>>>> write >> >> loops around >> >>>>> the read functio. However in practice I find myself >> >>>>> simply >> >> writing custom >> >>>>> writeStreams and use the callback to control upstream >> >>>>> pressure >> >> (in addition >> >>>>> to source Watermarks if needed). Here is an example >> >>>>> where I >> >> move the >> >>>>> output to a queue that executes a custom function in >> > parallel (i.e. >> >>>>> uploading to a database) >> > https://gist.github.com/ZJONSSON/5189249 >> >>>>> >> >>>>> Are there any benefits to using the read method directly >> >>>>> on a >> >> stream vs. >> >>>>> piping to a custom Writable stream? >> >>> >> >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >>> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> > >> >> You received this message because you are subscribed to the >> > Google >> >>> Groups "nodejs" group. To post to this group, send email to >> >>> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> <mailto:nodejs@googlegroups.com >> >> <mailto:nodejs@googlegroups.com>> >> >>> To unsubscribe from this group, send email to >> >>> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >>> >> >>> --- You received this message because you are subscribed to >> >>> the >> > Google >> >> Groups >> >>> "nodejs" group. To unsubscribe from this group and stop >> >>> receiving emails >> > from it, >> >> send an >> >>> email to nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >> >>> For more options, visit >> > https://groups.google.com/groups/opt_out. >> >>> >> >>> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> > >> You received this message because you are subscribed to the >> > Google >> >> Groups "nodejs" group. To post to this group, send email to >> >> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> <mailto:nodejs@googlegroups.com <mailto:nodejs@googlegroups.com>> >> >> To unsubscribe from this group, send email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed to the >> >> Google Groups "nodejs" group. To unsubscribe from this group and >> >> stop receiving emails from it, send an email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >> >> For more options, visit >> >> https://groups.google.com/groups/opt_out. >> >> >> >> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> >> >> >> >> > You received this message because you are subscribed to the Google >> >> Groups "nodejs" group. To post to this group, send email to >> >> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> To unsubscribe from this group, send email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed to the >> >> Google Groups "nodejs" group. To unsubscribe from this group and >> >> stop receiving emails from it, send an email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com>. >> >> For more options, visit >> >> https://groups.google.com/groups/opt_out. >> >> >> >> >> > >> > >> >> > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to a topic in the > Google Groups "nodejs" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/nodejs/8VGu32aczR0/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- Marco Rogers marco.rog...@gmail.com | https://twitter.com/polotek Life is ten percent what happens to you and ninety percent how you respond to it. - Lou Holtz -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
