3. Make sure to not do anything even slightly wrong!
This part is really hard. I don't think I've written a single stream module
that complies with it. (and I've written 60. it's really hard)
On Mon, Apr 1, 2013 at 5:39 PM, Isaac Schlueter i...@izs.me wrote:
Compared with the stuff that you
Dude, tell me about it! Even the streams in core were a total mess!
Having a single place where most of the implementation lives is a huge win.
On Wed, Apr 3, 2013 at 11:06 AM, Jake Verbaten rayn...@gmail.com wrote:
3. Make sure to not do anything even slightly wrong!
This part is really
@mikeal when I last checked a few weeks ago there where over 350 stream
modules in npm.
On Tue, Mar 26, 2013 at 9:38 PM, Isaac Schlueter i...@izs.me wrote:
You're still doing it wrong. This is like complaining that you can't
tell a stream is ended because you waited to attach an `end`
350 is a big number until you factor out the number of those written by 3
people (you, substack and Raynos). many modules might exists but, it is my
impression, that most of them are written by about a dozen people. this
impression could be wrong but i've been around since we first started
Compared with the stuff that you needed to do with streams1, streams2
is much much easier. Basically:
1. Pick the class you're extending: Readable, Writable, Duplex, Transform
2. Implement the constructor, and one method. (Well, 2 for Duplex,
but that's to be expected.)
If you are doing a
On Mon, Mar 25, 2013 at 4:53 PM, Mikeal Rogers mikeal.rog...@gmail.comwrote:
On Mar 25, 2013, at 4:40PM, Michael Jackson mjijack...@gmail.com wrote:
(well, pausing seems to work a lot better now)
Is this true?
In request I'm just abusing the backwards compatibility mode by calling
On Tue, Mar 26, 2013 at 2:40 PM, Michael Jackson mjijack...@gmail.comwrote:
On Mon, Mar 25, 2013 at 4:53 PM, Mikeal Rogers mikeal.rog...@gmail.comwrote:
On Mar 25, 2013, at 4:40PM, Michael Jackson mjijack...@gmail.com wrote:
(well, pausing seems to work a lot better now)
Is this true?
inline
On 26/03/2013 04:13 πμ, Isaac Schlueter wrote:
If you add a `readable` handler, then `'readable'` (and
potentially `'end'`, if there's no data) will be emitted.
So the readable event is more of an advisory event. The docs
should probably say something about how you could possibly
You're still doing it wrong. This is like complaining that you can't tell
a stream is ended because you waited to attach an `end` event handler. If
you've deferred to the event loop you have to check if a stream isn't ended
first. Likewise, you should call read and, only when it's exhausted,
You're still doing it wrong. This is like complaining that you can't tell a
stream is ended because you waited to attach an `end` event handler.
But that's supported, now :) If you don't consume the stream, it
never emits 'end', even if it never has any data. You still have to
read() or
Is it correct to assume that a Readable won't emit the readable event
until you're registered for it?
Reading through the streams2 docs, I was under the impression that all
streams start out paused and don't start emitting data until you add either
a data (for old streams) or a readable listener.
readable is emitted after you've actually started reading.
In your example, you dont ever `response.read()`, so no readable event
is ever emitted.
As you said, streams start in paused state and ready to be read.
On 03/25/13 22:28, Michael Jackson wrote:
Is it correct to assume that a Readable
readable is emitted after you've actually started reading.
That's not what it says in the
docshttp://nodejs.org/api/stream.html#stream_event_readable
.
###
Event: 'readable'
When there is data ready to be consumed, this event will fire.
When this event emits, call the read() method to consume
You can always call `stream.read`, at any time. This is how data is
*pulled*off the stream (instead of it being pushed to you, whether
you're ready or
not). Because of this you won't lose any data. With new streams there's no
real notion of a paused state -- it's always paused. Once you grok that
You're right, my bad.
But still, data stay in the buffer until someone tries to `.read()`.
So, if you're being passed a stream that you dont know whether the
first `readable` event has fired, you can try to actually read from
it. If it returns null, then you wait for `readable`.
On 03/25/13
You *must* try to read from it. Otherwise it's likely to remain paused.
On Mar 25, 2013, at 1:50PM, Dan Milon danmi...@gmail.com wrote:
You're right, my bad.
But still, data stay in the buffer until someone tries to `.read()`.
So, if you're being passed a stream that you dont know whether
I haven't experimented with streams2 as much as I should have. But I
remember talking to Isaac about it early on. The way I think about it is
still the same.
It feels like the semantics of how node streams produce data is much more
consistent and predictable now. Node still starts by reading data
On Mon, Mar 25, 2013 at 5:01 PM, Marco Rogers marco.rog...@gmail.comwrote:
I haven't experimented with streams2 as much as I should have. But I
remember talking to Isaac about it early on. The way I think about it is
still the same.
It feels like the semantics of how node streams produce
If stream.read returns null that could mean one of two things:
1) the stream doesn't currently have any data, but still might have some
in the future
2) the stream is already ended
AFAICT, the only way you can know which state you're in is by checking the
stream.readable property which is
You're absolutely right. I totally forgot about that aspect of the
semantics. It changes my metaphor a little. So stream.read() isn't a blind
blocking call that can get you into lots of trouble, which I think my
previous message implied. Instead it's a peek into the underlying semantics
of how
I'm more and more convinced that need to go back and read all the available
info about streams2. Answering these detail semantics questions is pretty
important.
:Marco
On Mon, Mar 25, 2013 at 2:29 PM, Michael Jackson mjijack...@gmail.comwrote:
If stream.read returns null that could mean one of
This thread is pretty huge.
At this point, would people say there is more confusion about streams2 than old
streams? I know that some of this is a little hard to get our heads around but
i always got the feeling that only about 10 people really understood all of old
streams.
-Mikeal
On Mar
Yeah, I think the old streams still had lots of confusion around them and
the adoption wasn't great. If we can't beat that with streams2 then we are
doing a lot of work for not much gain. I think there are common themes that
barrier to understanding with both approaches. I've been trying to think
Ah, looking through the Readable code a bit more it seems that the end
event won't ever fire until read is called at least
oncehttps://github.com/joyent/node/blob/master/lib/_stream_readable.js#L52-L56
.
So I guess what I could do is call read() *and* register an end event
handler in the same
Ok, that makes sense.
So the readable event is more of an advisory event. The docs should
probably say something about how you could possibly miss the event entirely
if you're doing some other IO before you try and read from the stream.
For posterity's sake, I adjusted my previous example:
That's not guaranteed to work.
You're assuming that `stream.read()` will return the whole internal
buffer, which is not documented anywhere.
The right approach is to call `.read()` until it returns null.
Something like that:
function collectStream(stream, cb) {
var bufs = []
function
I can see what you're saying, but the node docs do
sayhttp://nodejs.org/api/stream.html#stream_readable_read_size_1that
if you don't pass a size argument to stream.read then the entire
contents of the internal buffer are returned.
In any case, this would all be a lot easier if the readable event
How did this get so bloody complicated?
On Mon, Mar 25, 2013 at 3:48 PM, Michael Jackson mjijack...@gmail.comwrote:
I can see what you're saying, but the node docs do
sayhttp://nodejs.org/api/stream.html#stream_readable_read_size_1that if you
don't pass a size argument to stream.read then
A small amendment to #3: if read returns null it could also mean that the
stream is ended. So you need to register for the end event as well. It
looks like streams internals refuse to fire end until at least one call
to stream.read has happened, so you're guaranteed to get that event.
It's the
I don't think there's necessarily any *more* confusion over streams2 than
there ever was over old streams, just a different set of questions to
answer.
Part of the reason I was so excited about streams2 is that I thought it
meant I could finally deprecate
On Mar 25, 2013, at 4:40PM, Michael Jackson mjijack...@gmail.com wrote:
I don't think there's necessarily any *more* confusion over streams2 than
there ever was over old streams, just a different set of questions to answer.
Part of the reason I was so excited about streams2 is that I
If you add a `readable` handler, then `'readable'` (and potentially
`'end'`, if there's no data) will be emitted.
So the readable event is more of an advisory event. The docs should probably
say something about how you could possibly miss the event entirely if you're
doing some other IO
If you add a `readable` handler, then `'readable'` (and potentially
`'end'`, if there's no data) will be emitted.
Then there's a bug.
So the readable event is more of an advisory event. The docs should
probably say something about how you could possibly miss the event entirely
if you're
Thanks for all the answers. I almost forgot to look back at this thread as
the custom writeStreams have exceeded the high expectation I had already
for Streams 2.
For me, the reference manual was a little confusing, as there are complete
examples on using the read method, no mention of reading
re old-mode
Yes, that's fine. If you just want to get all the data asap, use
on('data', handler). It'll work great, and it's still very fast.
pause()/resume(), the whole bit. (The difference is that it won't
emit data until you're listening, and pause() will *actually* pause.)
Re read(cb)
On Monday, March 18, 2013 11:06:48 AM UTC-7, Sigurgeir Jonsson wrote:
The new streams have excellent support for high/low watermarks and
auto-pausing/resuming, but the documentation confuses me a little...
particularly the read method.
When I read the new docs for the first time I was
@Nathan's response is right. Creating a writable stream is preferable in
most cases. But I wanted to add a little context to that. If you're dealing
with a base readable stream, it's just pushing chunks of data at you off
the wire. Your first task is to collect those chunks into meaningful
I have a question regarding the new streams to which I could not find any
answer.
*Is it okay to use streams as before (aka old mode) ?*
On Monday, 18 March 2013 19:06:48 UTC+1, Sigurgeir Jonsson wrote:
The new streams have excellent support for high/low watermarks and
The real question being: will the pain of using it go away?
On Tuesday, 19 March 2013 10:12:38 UTC+1, Floby wrote:
I have a question regarding the new streams to which I could not find any
answer.
*Is it okay to use streams as before (aka old mode) ?*
On Monday, 18 March 2013 19:06:48
Floby,
It'll generally work just fine, but please consider using streams2 in the
new code.
Cheers,
Fedor.
On Tue, Mar 19, 2013 at 1:13 PM, Floby florent.j...@gmail.com wrote:
The real question being: will the pain of using it go away?
On Tuesday, 19 March 2013 10:12:38 UTC+1, Floby wrote:
40 matches
Mail list logo
