Yeah, I think the old streams still had lots of confusion around them and the adoption wasn't great. If we can't beat that with streams2 then we are doing a lot of work for not much gain. I think there are common themes that barrier to understanding with both approaches. I've been trying to think about what those are in between the billion other things I've got going.
:Marco On Mon, Mar 25, 2013 at 2:41 PM, Mikeal Rogers <mikeal.rog...@gmail.com>wrote: > This thread is pretty huge. > > At this point, would people say there is more confusion about streams2 > than old streams? I know that some of this is a little hard to get our > heads around but i always got the feeling that only about 10 people really > understood all of old streams. > > -Mikeal > > On Mar 25, 2013, at 2:40PM, Marco Rogers <marco.rog...@gmail.com> wrote: > > I'm more and more convinced that need to go back and read all the > available info about streams2. Answering these detail semantics questions > is pretty important. > > :Marco > > On Mon, Mar 25, 2013 at 2:29 PM, Michael Jackson <mjijack...@gmail.com>wrote: > >> If stream.read returns null that could mean one of two things: >> >> 1) the stream doesn't currently have any data, but still might have >> some in the future >> 2) the stream is already ended >> >> AFAICT, the only way you can know which state you're in is by checking >> the stream.readable property which is marked as being a "legacy" property in >> the >> code<https://github.com/joyent/node/blob/master/lib/_stream_readable.js#L98-L99>, >> so it seems like it's not a good idea to rely on that property either. >> >> -- >> Michael Jackson >> @mjackson >> >> >> On Mon, Mar 25, 2013 at 1:50 PM, Dan Milon <danmi...@gmail.com> wrote: >> >>> You're right, my bad. >>> >>> But still, data stay in the buffer until someone tries to `.read()`. >>> So, if you're being passed a stream that you dont know whether the >>> first `readable` event has fired, you can try to actually read from >>> it. If it returns null, then you wait for `readable`. >>> >>> On 03/25/13 22:42, Michael Jackson wrote: >>> > readable is emitted after you've actually started reading. >>> > >>> > >>> > That's not what it says in the docs >>> > <http://nodejs.org/api/stream.html#stream_event_readable>. >>> > >>> > ### Event: 'readable' When there is data ready to be consumed, >>> > this event will fire. When this event emits, call the read() method >>> > to consume the data. ### >>> > >>> > Calling stream.read *before* you get the "readable" event is >>> > totally counterintuitive. >>> > >>> > -- Michael Jackson @mjackson >>> > >>> > In your example, you dont ever `response.read()`, so no readable >>> > event is ever emitted. >>> > >>> > As you said, streams start in paused state and ready to be read. >>> > >>> > On 03/25/13 22:28, Michael Jackson wrote: >>> >> Is it correct to assume that a Readable won't emit the >>> >> "readable" >>> > event >>> >> until you're registered for it? >>> >> >>> >> Reading through the streams2 docs, I was under the impression >>> >> that all streams start out paused and don't start emitting data >>> >> until you add either a "data" (for old streams) or a "readable" >>> >> listener. For new streams, this should mean that they don't emit >>> >> "readable" until at >>> > least >>> >> one listener is registered. Otherwise we still need to do some >>> > buffering >>> >> in order to capture all the data. >>> >> >>> >> For example, this code misses the readable event on node 0.10: >>> >> >>> >> var http = require('http'); >>> >> >>> >> http.get('http://www.google.com', function (response) { >>> >> console.log('got response with status ' + response.statusCode); >>> >> >>> >> setTimeout(function () { response.on('readable', function () { >>> >> console.log('readable'); }); >>> >> >>> >> response.on('end', function () { console.log('end'); }); }, 5); >>> >> }); >>> >> >>> >> Here's my shell session: >>> >> >>> >> $ node -v v0.10.0 $ node http-test.js got response with status >>> >> 200 $ >>> >> >>> >> Is this the correct behavior? >>> >> >>> >> -- Michael Jackson @mjackson >>> >> >>> >> >>> >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <i...@izs.me >>> > <mailto:i...@izs.me> >>> >> <mailto:i...@izs.me <mailto:i...@izs.me>>> wrote: >>> >> >>> >> re old-mode >>> >> >>> >> Yes, that's fine. If you just want to get all the data asap, use >>> >> on('data', handler). It'll work great, and it's still very fast. >>> >> pause()/resume(), the whole bit. (The difference is that it >>> >> won't emit data until you're listening, and pause() will >>> >> *actually* >>> > pause.) >>> >> >>> >> >>> >> Re read(cb) >>> >> >>> >> It's problematic for reasons that I've discussed all of the >>> >> places where it's been brought up. That horse is dead, let's >>> >> stop >>> > beating >>> >> it. (There were a few other proposals as well, btw. >>> > Reducibles and >>> >> some other monadic approaches come to mind.) >>> >> >>> >> >>> >> Re pipe() vs looping around read() vs custom Writable vs >>> > on('data') >>> >> >>> >> Whatever works for your case is fine. It's flexible on >>> > purpose, and >>> >> allows more types of consumption than streams1, and creating >>> > custom >>> >> writables is easier than it was in streams1. >>> >> >>> >> If you find something that the API can't do for you, or find >>> > yourself >>> >> doing a lot of backflips or overriding a lot of methods to get >>> > your >>> >> stuff working, then let's chat about it in a github issue. >>> > You might >>> >> be missing something, or you might have found a genuine >>> > shortcoming in >>> >> the API. >>> >> >>> >> >>> >> >>> >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson >>> >> <ziggy.jonsson....@gmail.com >>> > <mailto:ziggy.jonsson....@gmail.com> >>> > <mailto:ziggy.jonsson....@gmail.com >>> > <mailto:ziggy.jonsson....@gmail.com>>> >>> >> wrote: >>> >>> Thanks for all the answers. I almost forgot to look back at >>> >>> this >>> >> thread as >>> >>> the custom writeStreams have exceeded the high expectation I >>> >>> had >>> >> already for >>> >>> Streams 2. For me, the reference manual was a little >>> >>> confusing, as >>> > there are >>> >> complete >>> >>> examples on using the read method, no mention of "reading" >>> > through a >>> >>> writeStream endpoint. >>> >>> >>> >>> Marco, I agree that that read has more detailed control of >>> > minimum >>> >> incoming >>> >>> content. However I wonder if it would be more efficient to >>> > default >>> >>> pipe.chunkSize to a "lowWatermark" of the receiver (if >>> >>> defined). >>> >> This >>> >>> lowWatermark could be adjusted dynamically and the callback >>> > in the >>> >> writable >>> >>> should keep sequence of events under control? >>> >>> >>> >>> Anyway, thanks Node team, I'm very impressed! >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco Rogers >>> > wrote: >>> >>>> >>> >>>> @Nathan's response is right. Creating a writable stream is >>> >> preferable in >>> >>>> most cases. But I wanted to add a little context to that. If >>> >> you're dealing >>> >>>> with a base readable stream, it's just pushing chunks of >>> > data at >>> >> you off the >>> >>>> wire. Your first task is to collect those chunks into >>> > meaningful >>> >> data. So >>> >>>> IMO the reason creating a writable stream is preferable is >>> > because it >>> >>>> prompts you not just read off the stream, but to create >>> > semantics >>> >> around >>> >>>> what the new stream is supposed to be. The api reflects this >>> >> opinion and >>> >>>> that's why creating writable streams feels like the more >>> > natural >>> >> way, and >>> >>>> the ugliness of dealing with read() is wrapped up in the >>> >>>> pipe() >>> >> method. It >>> >>>> was kind of designed that way. >>> >>>> >>> >>>> But the read() api was also designed for a use case. It's >>> >>>> meant >>> >> to handle >>> >>>> low/high water marks effectively, as well as enable more >>> >> optimized special >>> >>>> parsing by reading off specific lengths of chunks. These >>> >>>> were >>> >> things that >>> >>>> people kept needing, but the old api didn't support well. >>> > If you were >>> >>>> writing a library for a special parser, you might write a >>> > custom >>> >> Writable >>> >>>> stream and inside it you would be using the read(n) api to >>> >> control *how* you >>> >>>> read data off the socket. I hope that makes sense. >>> >>>> >>> >>>> :Marco >>> >>>> >>> >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir >>> > Jonsson wrote: >>> >>>>> >>> >>>>> The new streams have excellent support for high/low >>> > watermarks and >>> >>>>> auto-pausing/resuming, but the documentation confuses me a >>> > little... >>> >>>>> particularly the read method. >>> >>>>> >>> >>>>> When I read the new docs for the first time I was under >>> >>>>> the >>> >> impression >>> >>>>> that the optimal way to become a user of a stream is to >>> >>>>> write >>> >> loops around >>> >>>>> the read functio. However in practice I find myself >>> >>>>> simply >>> >> writing custom >>> >>>>> writeStreams and use the callback to control upstream >>> >>>>> pressure >>> >> (in addition >>> >>>>> to source Watermarks if needed). Here is an example >>> >>>>> where I >>> >> move the >>> >>>>> output to a queue that executes a custom function in >>> > parallel (i.e. >>> >>>>> uploading to a database) >>> > https://gist.github.com/ZJONSSON/5189249 >>> >>>>> >>> >>>>> Are there any benefits to using the read method directly >>> >>>>> on a >>> >> stream vs. >>> >>>>> piping to a custom Writable stream? >>> >>> >>> >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >>> >>> >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> > >>> > >>> > >>> >> You received this message because you are subscribed to the >>> > Google >>> >>> Groups "nodejs" group. To post to this group, send email to >>> >>> nodejs@googlegroups.com >>> > <mailto:nodejs@googlegroups.com> >>> >> <mailto:nodejs@googlegroups.com >>> >> <mailto:nodejs@googlegroups.com>> >>> >>> To unsubscribe from this group, send email to >>> >>> nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >>> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >>> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >>> >>> For more options, visit this group at >>> >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >>> >>> >>> --- You received this message because you are subscribed to >>> >>> the >>> > Google >>> >> Groups >>> >>> "nodejs" group. To unsubscribe from this group and stop >>> >>> receiving emails >>> > from it, >>> >> send an >>> >>> email to nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >>> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >>> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >>> >>> For more options, visit >>> > https://groups.google.com/groups/opt_out. >>> >>> >>> >>> >>> >> >>> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >>> >> >>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> > >>> > >>> > >>> You received this message because you are subscribed to the >>> > Google >>> >> Groups "nodejs" group. To post to this group, send email to >>> >> nodejs@googlegroups.com >>> > <mailto:nodejs@googlegroups.com> >>> >> <mailto:nodejs@googlegroups.com <mailto:nodejs@googlegroups.com>> >>> >> To unsubscribe from this group, send email to >>> >> nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >>> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >>> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >>> >> For more options, visit this group at >>> >> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >>> >> --- You received this message because you are subscribed to the >>> >> Google Groups "nodejs" group. To unsubscribe from this group and >>> >> stop receiving emails from it, send an email to >>> >> nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >>> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >>> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >>> >> For more options, visit >>> >> https://groups.google.com/groups/opt_out. >>> >> >>> >> >>> >> >>> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >>> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> > >>> >> >>> >> >>> > You received this message because you are subscribed to the Google >>> >> Groups "nodejs" group. To post to this group, send email to >>> >> nodejs@googlegroups.com >>> > <mailto:nodejs@googlegroups.com> >>> >> To unsubscribe from this group, send email to >>> >> nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >>> >> For more options, visit this group at >>> >> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >>> >> --- You received this message because you are subscribed to the >>> >> Google Groups "nodejs" group. To unsubscribe from this group and >>> >> stop receiving emails from it, send an email to >>> >> nodejs+unsubscr...@googlegroups.com >>> > <mailto:nodejs%2bunsubscr...@googlegroups.com>. >>> >> For more options, visit >>> >> https://groups.google.com/groups/opt_out. >>> >> >>> >> >>> > >>> > >>> >>> >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "nodejs" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/nodejs/8VGu32aczR0/unsubscribe?hl=en. >> To unsubscribe from this group and all its topics, send an email to >> nodejs+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > > -- > Marco Rogers > marco.rog...@gmail.com | https://twitter.com/polotek > > Life is ten percent what happens to you and ninety percent how you respond > to it. > - Lou Holtz > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to a topic in the > Google Groups "nodejs" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/nodejs/8VGu32aczR0/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- Marco Rogers marco.rog...@gmail.com | https://twitter.com/polotek Life is ten percent what happens to you and ninety percent how you respond to it. - Lou Holtz -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
