On Mar 25, 2013, at 4:40PM, Michael Jackson <mjijack...@gmail.com> wrote:
> I don't think there's necessarily any *more* confusion over streams2 than > there ever was over old streams, just a different set of questions to answer. > > Part of the reason I was so excited about streams2 is that I thought it meant > I could finally deprecate BufferedStream in favor of 0.10's Readable. I wrote > BufferedStream back in the 0.4 days when streams would spew data regardless > of whether or not you had registered for "data" events or called pause(). The > basic idea behind the library was that you could pass whatever you wanted to > the constructor (a stream, a string, a Buffer, or what have you) and that you > would get back a predictable interface for handling streaming data without > losing any events. > > It was incredibly difficult to get all the semantics of streams right, but I > think we came pretty close. But streams2 opens a whole new can of worms. Now > we're not just keeping compatibility with the old behavior (well, pausing > seems to work a lot better now) Is this true? In request I'm just abusing the backwards compatibility mode by calling .resume() on stuff a lot. In a future release i'll pull in readable-stream and inherit from it and using the old stream wrappers so that all of request's code is using the new API. It seems like we get to pick one API or the other. I can make all of my code use the new API or I can abuse the reverse compatibility, either way I'm only using one API. > but introducing an entirely new method of consuming streams on top of that. > I've found myself diving into the _stream_readable.js code a lot more than I > would have liked over the past few weeks just to figure out how things work. > It's an incredibly complex piece of code. Hats off to the core team - I know > how tricky it can get. But honestly, when you start cracking open node core > code just to figure out how things work you start relying on implementation > details that may not even be specified fully. So it's kind of a precarious > place to be. > > Looking back, the data/pause/resume/end/error API isn't too bad. The main > problem with it was that a lot of streams didn't really care if you paused > them. Fixing that piece and leaving the rest (buffering, duplex, transform, > etc.) up to npm modules would have been a great baby step without introducing > an entirely new API. > > Related: the most intuitive streams implementation I've ever used is (gasp!) > PHP's. You just call fopen with a URL (file://, http://, zlib://, etc.) and > boom, you've got a handle to a stream of data that every other function in > the stdlib knows how to use. You can specify custom wrappers that know how to > open different URL schemes, stack as many filters as you like on top of them. > You never need to worry about data events, or what state different streams > are in. You can seek to different positions in the stream, read just a > certain number of bytes, whatever. It's just super simple and super powerful. > > -- > Michael Jackson > @mjackson > > > On Mon, Mar 25, 2013 at 2:41 PM, Mikeal Rogers <mikeal.rog...@gmail.com> > wrote: > This thread is pretty huge. > > At this point, would people say there is more confusion about streams2 than > old streams? I know that some of this is a little hard to get our heads > around but i always got the feeling that only about 10 people really > understood all of old streams. > > -Mikeal > > On Mar 25, 2013, at 2:40PM, Marco Rogers <marco.rog...@gmail.com> wrote: > >> I'm more and more convinced that need to go back and read all the available >> info about streams2. Answering these detail semantics questions is pretty >> important. >> >> :Marco >> >> On Mon, Mar 25, 2013 at 2:29 PM, Michael Jackson <mjijack...@gmail.com> >> wrote: >> If stream.read returns null that could mean one of two things: >> >> 1) the stream doesn't currently have any data, but still might have some >> in the future >> 2) the stream is already ended >> >> AFAICT, the only way you can know which state you're in is by checking the >> stream.readable property which is marked as being a "legacy" property in the >> code, so it seems like it's not a good idea to rely on that property either. >> >> -- >> Michael Jackson >> @mjackson >> >> >> On Mon, Mar 25, 2013 at 1:50 PM, Dan Milon <danmi...@gmail.com> wrote: >> You're right, my bad. >> >> But still, data stay in the buffer until someone tries to `.read()`. >> So, if you're being passed a stream that you dont know whether the >> first `readable` event has fired, you can try to actually read from >> it. If it returns null, then you wait for `readable`. >> >> On 03/25/13 22:42, Michael Jackson wrote: >> > readable is emitted after you've actually started reading. >> > >> > >> > That's not what it says in the docs >> > <http://nodejs.org/api/stream.html#stream_event_readable>. >> > >> > ### Event: 'readable' When there is data ready to be consumed, >> > this event will fire. When this event emits, call the read() method >> > to consume the data. ### >> > >> > Calling stream.read *before* you get the "readable" event is >> > totally counterintuitive. >> > >> > -- Michael Jackson @mjackson >> > >> > In your example, you dont ever `response.read()`, so no readable >> > event is ever emitted. >> > >> > As you said, streams start in paused state and ready to be read. >> > >> > On 03/25/13 22:28, Michael Jackson wrote: >> >> Is it correct to assume that a Readable won't emit the >> >> "readable" >> > event >> >> until you're registered for it? >> >> >> >> Reading through the streams2 docs, I was under the impression >> >> that all streams start out paused and don't start emitting data >> >> until you add either a "data" (for old streams) or a "readable" >> >> listener. For new streams, this should mean that they don't emit >> >> "readable" until at >> > least >> >> one listener is registered. Otherwise we still need to do some >> > buffering >> >> in order to capture all the data. >> >> >> >> For example, this code misses the readable event on node 0.10: >> >> >> >> var http = require('http'); >> >> >> >> http.get('http://www.google.com', function (response) { >> >> console.log('got response with status ' + response.statusCode); >> >> >> >> setTimeout(function () { response.on('readable', function () { >> >> console.log('readable'); }); >> >> >> >> response.on('end', function () { console.log('end'); }); }, 5); >> >> }); >> >> >> >> Here's my shell session: >> >> >> >> $ node -v v0.10.0 $ node http-test.js got response with status >> >> 200 $ >> >> >> >> Is this the correct behavior? >> >> >> >> -- Michael Jackson @mjackson >> >> >> >> >> >> On Thu, Mar 21, 2013 at 4:27 PM, Isaac Schlueter <i...@izs.me >> > <mailto:i...@izs.me> >> >> <mailto:i...@izs.me <mailto:i...@izs.me>>> wrote: >> >> >> >> re old-mode >> >> >> >> Yes, that's fine. If you just want to get all the data asap, use >> >> on('data', handler). It'll work great, and it's still very fast. >> >> pause()/resume(), the whole bit. (The difference is that it >> >> won't emit data until you're listening, and pause() will >> >> *actually* >> > pause.) >> >> >> >> >> >> Re read(cb) >> >> >> >> It's problematic for reasons that I've discussed all of the >> >> places where it's been brought up. That horse is dead, let's >> >> stop >> > beating >> >> it. (There were a few other proposals as well, btw. >> > Reducibles and >> >> some other monadic approaches come to mind.) >> >> >> >> >> >> Re pipe() vs looping around read() vs custom Writable vs >> > on('data') >> >> >> >> Whatever works for your case is fine. It's flexible on >> > purpose, and >> >> allows more types of consumption than streams1, and creating >> > custom >> >> writables is easier than it was in streams1. >> >> >> >> If you find something that the API can't do for you, or find >> > yourself >> >> doing a lot of backflips or overriding a lot of methods to get >> > your >> >> stuff working, then let's chat about it in a github issue. >> > You might >> >> be missing something, or you might have found a genuine >> > shortcoming in >> >> the API. >> >> >> >> >> >> >> >> On Thu, Mar 21, 2013 at 2:01 PM, Sigurgeir Jonsson >> >> <ziggy.jonsson....@gmail.com >> > <mailto:ziggy.jonsson....@gmail.com> >> > <mailto:ziggy.jonsson....@gmail.com >> > <mailto:ziggy.jonsson....@gmail.com>>> >> >> wrote: >> >>> Thanks for all the answers. I almost forgot to look back at >> >>> this >> >> thread as >> >>> the custom writeStreams have exceeded the high expectation I >> >>> had >> >> already for >> >>> Streams 2. For me, the reference manual was a little >> >>> confusing, as >> > there are >> >> complete >> >>> examples on using the read method, no mention of "reading" >> > through a >> >>> writeStream endpoint. >> >>> >> >>> Marco, I agree that that read has more detailed control of >> > minimum >> >> incoming >> >>> content. However I wonder if it would be more efficient to >> > default >> >>> pipe.chunkSize to a "lowWatermark" of the receiver (if >> >>> defined). >> >> This >> >>> lowWatermark could be adjusted dynamically and the callback >> > in the >> >> writable >> >>> should keep sequence of events under control? >> >>> >> >>> Anyway, thanks Node team, I'm very impressed! >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Wednesday, March 20, 2013 4:45:32 AM UTC-4, Marco Rogers >> > wrote: >> >>>> >> >>>> @Nathan's response is right. Creating a writable stream is >> >> preferable in >> >>>> most cases. But I wanted to add a little context to that. If >> >> you're dealing >> >>>> with a base readable stream, it's just pushing chunks of >> > data at >> >> you off the >> >>>> wire. Your first task is to collect those chunks into >> > meaningful >> >> data. So >> >>>> IMO the reason creating a writable stream is preferable is >> > because it >> >>>> prompts you not just read off the stream, but to create >> > semantics >> >> around >> >>>> what the new stream is supposed to be. The api reflects this >> >> opinion and >> >>>> that's why creating writable streams feels like the more >> > natural >> >> way, and >> >>>> the ugliness of dealing with read() is wrapped up in the >> >>>> pipe() >> >> method. It >> >>>> was kind of designed that way. >> >>>> >> >>>> But the read() api was also designed for a use case. It's >> >>>> meant >> >> to handle >> >>>> low/high water marks effectively, as well as enable more >> >> optimized special >> >>>> parsing by reading off specific lengths of chunks. These >> >>>> were >> >> things that >> >>>> people kept needing, but the old api didn't support well. >> > If you were >> >>>> writing a library for a special parser, you might write a >> > custom >> >> Writable >> >>>> stream and inside it you would be using the read(n) api to >> >> control *how* you >> >>>> read data off the socket. I hope that makes sense. >> >>>> >> >>>> :Marco >> >>>> >> >>>> On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir >> > Jonsson wrote: >> >>>>> >> >>>>> The new streams have excellent support for high/low >> > watermarks and >> >>>>> auto-pausing/resuming, but the documentation confuses me a >> > little... >> >>>>> particularly the read method. >> >>>>> >> >>>>> When I read the new docs for the first time I was under >> >>>>> the >> >> impression >> >>>>> that the optimal way to become a user of a stream is to >> >>>>> write >> >> loops around >> >>>>> the read functio. However in practice I find myself >> >>>>> simply >> >> writing custom >> >>>>> writeStreams and use the callback to control upstream >> >>>>> pressure >> >> (in addition >> >>>>> to source Watermarks if needed). Here is an example >> >>>>> where I >> >> move the >> >>>>> output to a queue that executes a custom function in >> > parallel (i.e. >> >>>>> uploading to a database) >> > https://gist.github.com/ZJONSSON/5189249 >> >>>>> >> >>>>> Are there any benefits to using the read method directly >> >>>>> on a >> >> stream vs. >> >>>>> piping to a custom Writable stream? >> >>> >> >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >>> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> > >> >> You received this message because you are subscribed to the >> > Google >> >>> Groups "nodejs" group. To post to this group, send email to >> >>> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> <mailto:nodejs@googlegroups.com >> >> <mailto:nodejs@googlegroups.com>> >> >>> To unsubscribe from this group, send email to >> >>> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/nodejs?hl=en?hl=en >> >>> >> >>> --- You received this message because you are subscribed to >> >>> the >> > Google >> >> Groups >> >>> "nodejs" group. To unsubscribe from this group and stop >> >>> receiving emails >> > from it, >> >> send an >> >>> email to nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >> >>> For more options, visit >> > https://groups.google.com/groups/opt_out. >> >>> >> >>> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> > >> > >> You received this message because you are subscribed to the >> > Google >> >> Groups "nodejs" group. To post to this group, send email to >> >> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> <mailto:nodejs@googlegroups.com <mailto:nodejs@googlegroups.com>> >> >> To unsubscribe from this group, send email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed to the >> >> Google Groups "nodejs" group. To unsubscribe from this group and >> >> stop receiving emails from it, send an email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> <mailto:nodejs%2bunsubscr...@googlegroups.com >> > <mailto:nodejs%252bunsubscr...@googlegroups.com>>. >> >> For more options, visit >> >> https://groups.google.com/groups/opt_out. >> >> >> >> >> >> >> >> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: >> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > >> >> >> >> >> > You received this message because you are subscribed to the Google >> >> Groups "nodejs" group. To post to this group, send email to >> >> nodejs@googlegroups.com >> > <mailto:nodejs@googlegroups.com> >> >> To unsubscribe from this group, send email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com> >> >> For more options, visit this group at >> >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> --- You received this message because you are subscribed to the >> >> Google Groups "nodejs" group. To unsubscribe from this group and >> >> stop receiving emails from it, send an email to >> >> nodejs+unsubscr...@googlegroups.com >> > <mailto:nodejs%2bunsubscr...@googlegroups.com>. >> >> For more options, visit >> >> https://groups.google.com/groups/opt_out. >> >> >> >> >> > >> > >> >> >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "nodejs" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/nodejs/8VGu32aczR0/unsubscribe?hl=en. >> To unsubscribe from this group and all its topics, send an email to >> nodejs+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> >> -- >> Marco Rogers >> marco.rog...@gmail.com | https://twitter.com/polotek >> >> Life is ten percent what happens to you and ninety percent how you respond >> to it. >> - Lou Holtz >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to nodejs+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to nodejs+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
