This thread is pretty huge. At this point, would people say there is more confusion about streams2 than old streams? I know that some of this is a little hard to get our heads around but i always got the feeling that only about 10 people really understood all of old streams.
-Mikeal On Mar 25, 2013, at 2:40PM, Marco Rogers <marco.rog...@gmail.com> wrote: > I'm more and more convinced that need to go back and read all the available > info about streams2. Answering these detail semantics questions is pretty > important. > > :Marco > > On Mon, Mar 25, 2013 at 2:29 PM, Michael Jackson <mjijack...@gmail.com> wrote: > If stream.read returns null that could mean one of two things: > > 1) the stream doesn't currently have any data, but still might have some in > the future > 2) the stream is already ended > > AFAICT, the only way you can know which state you're in is by checking the > stream.readable property which is marked as being a "legacy" property in the > code, so it seems like it's not a good idea to rely on that property either. > > -- > Michael Jackson > @mjackson > > > On Mon, Mar 25, 2013 at 1:50 PM, Dan Milon <danmi...@gmail.com> wrote: > You're right, my bad. > > But still, data stay in the buffer until someone tries to `.read()`. > So, if you're being passed a stream that you dont know whether the > first `readable` event has fired, you can try to actually read from > it. If it returns null, then you wait for `readable`. > > On 03/25/13 22:42, Michael Jackson wrote: > > readable is emitted after you've actually started reading. > > > > > > That's not what it says in the docs > > <http://nodejs.org/api/stream.html#stream_event_readable>. > > > > ### Event: 'readable' When there is data ready to be consumed, > > this event will fire. When this event emits, call the read() method > > to consume the data. ### > > > > Calling stream.read *before* you get the "readable" event is > > totally counterintuitive. > > > > -- Michael Jackson @mjackson > > > > In your example, you dont ever `response.read()`, so no readable > > event is ever emitted. > > > > As you said, streams start in paused state and ready to be read. > > > > On 03/25/13 22:28, Michael Jackson wrote: > >> Is it correct to assume that a Readable won't emit the > >> "readable" > > event > >> until you're registered for it? > >> > >> Reading through the streams2 docs, I was under the impression > >> that all streams start out paused and don't start emitting data > >> until you add either a "data" (for old streams) or a "readable" > >> listener. For new streams, this should mean that they don't emit > >> "readable" until at > > least > >> one listener is registered. Otherwise we still need to do some > > buffering > >> in order to capture all the data. > >> > >> For example, this code misses the readable event on node 0.10: > >> > >> var http = require('http'); > >> > >> http.get('http://www.google.com', function (response) { > >> console.log('got response with status ' + response.statusCode); > >> > >> setTimeout(function () { response.on('readable', function () { > >> console.log('readable'); }); > >> > >> response.on('end', function () { console.log('end'); }); }, 5); > >> }); > >> > >> Here's my shell session: > >> > >> $ node -v v0.10.0 $ node http-test.js got response with status > >> 200 $ > >> > >> Is this the correct behavior? > >> > >> -- Michael Jackson @mjackson > >> > >> > >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <i...@izs.me > > <mailto:i...@izs.me> > >> <mailto:i...@izs.me <mailto:i...@izs.me>>> wrote: > >> > >> re old-mode > >> > >> Yes, that's fine. If you just want to get all the data asap, use > >> on('data', handler). It'll work great, and it's still very fast. > >> pause()/resume(), the whole bit. (The difference is that it > >> won't emit data until you're listening, and pause() will > >> *actually* > > pause.) > >> > >> > >> Re read(cb) > >> > >> It's problematic for reasons that I've discussed all of the > >> places where it's been brought up. That horse is dead, let's > >> stop > > beating > >> it. (There were a few other proposals as well, btw. > > Reducibles and > >> some other monadic approaches come to mind.) > >> > >> > >> Re pipe() vs looping around read() vs custom Writable vs > > on('data') > >> > >> Whatever works for your case is fine. It's flexible on > > purpose, and > >> allows more types of consumption than streams1, and creating > > custom > >> writables is easier than it was in streams1. > >> > >> If you find something that the API can't do for you, or find > > yourself > >> doing a lot of backflips or overriding a lot of methods to get > > your > >> stuff working, then let's chat about it in a github issue. > > You might > >> be missing something, or you might have found a genuine > > shortcoming in > >> the API. > >> > >> > >> > >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson > >> <ziggy.jonsson....@gmail.com > > <mailto:ziggy.jonsson....@gmail.com> > > <mailto:ziggy.jonsson....@gmail.com > > <mailto:ziggy.jonsson....@gmail.com>>> > >> wrote: > >>> Thanks for all the answers. I almost forgot to look back at > >>> this > >> thread as > >>> the custom writeStreams have exceeded the high expectation I > >>> had > >> already for > >>> Streams 2. For me, the reference manual was a little > >>> confusing, as > > there are > >> complete > >>> examples on using the read method, no mention of "reading" > > through a > >>> writeStream endpoint. > >>> > >>> Marco, I agree that that read has more detailed control of > > minimum > >> incoming > >>> content. However I wonder if it would be more efficient to > > default > >>> pipe.chunkSize to a "lowWatermark" of the receiver (if > >>> defined). > >> This > >>> lowWatermark could be adjusted dynamically and the callback > > in the > >> writable > >>> should keep sequence of events under control? > >>> > >>> Anyway, thanks Node team, I'm very impressed! > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco Rogers > > wrote: > >>>> > >>>> @Nathan's response is right. Creating a writable stream is > >> preferable in > >>>> most cases. But I wanted to add a little context to that. If > >> you're dealing > >>>> with a base readable stream, it's just pushing chunks of > > data at > >> you off the > >>>> wire. Your first task is to collect those chunks into > > meaningful > >> data. So > >>>> IMO the reason creating a writable stream is preferable is > > because it > >>>> prompts you not just read off the stream, but to create > > semantics > >> around > >>>> what the new stream is supposed to be. The api reflects this > >> opinion and > >>>> that's why creating writable streams feels like the more > > natural > >> way, and > >>>> the ugliness of dealing with read() is wrapped up in the > >>>> pipe() > >> method. It > >>>> was kind of designed that way. > >>>> > >>>> But the read() api was also designed for a use case. It's > >>>> meant > >> to handle > >>>> low/high water marks effectively, as well as enable more > >> optimized special > >>>> parsing by reading off specific lengths of chunks. These > >>>> were > >> things that > >>>> people kept needing, but the old api didn't support well. > > If you were > >>>> writing a library for a special parser, you might write a > > custom > >> Writable > >>>> stream and inside it you would be using the read(n) api to > >> control *how* you > >>>> read data off the socket. I hope that makes sense. > >>>> > >>>> :Marco > >>>> > >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir > > Jonsson wrote: > >>>>> > >>>>> The new streams have excellent support for high/low > > watermarks and > >>>>> auto-pausing/resuming, but the documentation confuses me a > > little... > >>>>> particularly the read method. > >>>>> > >>>>> When I read the new docs for the first time I was under > >>>>> the > >> impression > >>>>> that the optimal way to become a user of a stream is to > >>>>> write > >> loops around > >>>>> the read functio. However in practice I find myself > >>>>> simply > >> writing custom > >>>>> writeStreams and use the callback to control upstream > >>>>> pressure > >> (in addition > >>>>> to source Watermarks if needed). Here is an example > >>>>> where I > >> move the > >>>>> output to a queue that executes a custom function in > > parallel (i.e. > >>>>> uploading to a database) > > https://gist.github.com/ZJONSSON/5189249 > >>>>> > >>>>> Are there any benefits to using the read method directly > >>>>> on a > >> stream vs. > >>>>> piping to a custom Writable stream? > >>> > >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >>> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > > > >> You received this message because you are subscribed to the > > Google > >>> Groups "nodejs" group. To post to this group, send email to > >>> nodejs@googlegroups.com > > <mailto:nodejs@googlegroups.com> > >> <mailto:nodejs@googlegroups.com > >> <mailto:nodejs@googlegroups.com>> > >>> To unsubscribe from this group, send email to > >>> nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com> > >> <mailto:nodejs%2bunsubscr...@googlegroups.com > > <mailto:nodejs%252bunsubscr...@googlegroups.com>> > >>> For more options, visit this group at > >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>> > >>> --- You received this message because you are subscribed to > >>> the > > Google > >> Groups > >>> "nodejs" group. To unsubscribe from this group and stop > >>> receiving emails > > from it, > >> send an > >>> email to nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com> > >> <mailto:nodejs%2bunsubscr...@googlegroups.com > > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. > >>> For more options, visit > > https://groups.google.com/groups/opt_out. > >>> > >>> > >> > >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > > > You received this message because you are subscribed to the > > Google > >> Groups "nodejs" group. To post to this group, send email to > >> nodejs@googlegroups.com > > <mailto:nodejs@googlegroups.com> > >> <mailto:nodejs@googlegroups.com <mailto:nodejs@googlegroups.com>> > >> To unsubscribe from this group, send email to > >> nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com> > >> <mailto:nodejs%2bunsubscr...@googlegroups.com > > <mailto:nodejs%252bunsubscr...@googlegroups.com>> > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> --- You received this message because you are subscribed to the > >> Google Groups "nodejs" group. To unsubscribe from this group and > >> stop receiving emails from it, send an email to > >> nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com> > >> <mailto:nodejs%2bunsubscr...@googlegroups.com > > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. > >> For more options, visit > >> https://groups.google.com/groups/opt_out. > >> > >> > >> > >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: > >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > >> > >> > > You received this message because you are subscribed to the Google > >> Groups "nodejs" group. To post to this group, send email to > >> nodejs@googlegroups.com > > <mailto:nodejs@googlegroups.com> > >> To unsubscribe from this group, send email to > >> nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com> > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> --- You received this message because you are subscribed to the > >> Google Groups "nodejs" group. To unsubscribe from this group and > >> stop receiving emails from it, send an email to > >> nodejs+unsubscr...@googlegroups.com > > <mailto:nodejs%2bunsubscr...@googlegroups.com>. > >> For more options, visit > >> https://groups.google.com/groups/opt_out. > >> > >> > > > > > > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to a topic in the Google > Groups "nodejs" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/nodejs/8VGu32aczR0/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > Marco Rogers > marco.rog...@gmail.com | https://twitter.com/polotek > > Life is ten percent what happens to you and ninety percent how you respond to > it. > - Lou Holtz > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
