Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
Thanks izs. I got the idea and how to do this. Can't we document this in the API doc, so it helps others. Or build a api for this kind of scenarios as well. On Tue, Dec 25, 2012 at 4:00 AM, Isaac Schlueter i...@izs.me wrote: expensive -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/arunoda -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
This is probably an API design issue with readable stream When a readable stream wraps a source where data must be pulled from then _read makes sense. When a readable stream wraps a source that pushes data then having some kind of stream._push(chunk) api makes more sense. My thin readable-stream abstraction has a push api ( https://github.com/Raynos/read-stream#example-queue ) On Fri, Dec 21, 2012 at 11:11 AM, Bradley Meck bradley.m...@gmail.comwrote: Personally I did not want to write a buffering system for in memory _read and just used a PassThrough stream for node-tart's substream parts : https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm trying as well to find a sane way to allow internal buffering from a different stream but have not thought of an elegant solution. On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote: Hi, I was looking at the new readable stream documentation and it said that we've to implement _read() method in order to implement the readable stream. But I have a problem? I want to create a readable stream where it's source is redis subscribe(). How can I implement a readable stream for this with the new API? Do I need to buffer incoming data from redis? -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/**arunoda http://www.linkedin.com/in/arunoda -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
Yes I know buffering is not a good solution. Seems like need to implement this with the old stream api. (emitting data events). On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck bradley.m...@gmail.comwrote: Personally I did not want to write a buffering system for in memory _read and just used a PassThrough stream for node-tart's substream parts : https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm trying as well to find a sane way to allow internal buffering from a different stream but have not thought of an elegant solution. On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote: Hi, I was looking at the new readable stream documentation and it said that we've to implement _read() method in order to implement the readable stream. But I have a problem? I want to create a readable stream where it's source is redis subscribe(). How can I implement a readable stream for this with the new API? Do I need to buffer incoming data from redis? -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/**arunoda http://www.linkedin.com/in/arunoda -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/arunoda -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
If you havent seen redis-stream, it might help a bit.
https://github.com/tblobaum/redis-stream
var Redis = require('redis-stream')
, client = new Redis('localhost', 6379, 0)
, subscribe = client.stream('subscribe')
subscribe.pipe(process.stdout)
subscribe.write('channel-one')
Thomas Blobaum
https://github.com/tblobaum
On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
Yes I know buffering is not a good solution.
Seems like need to implement this with the old stream api. (emitting data
events).
On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck bradley.m...@gmail.com
wrote:
Personally I did not want to write a buffering system for in memory _read
and just used a PassThrough stream for node-tart's substream parts :
https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm trying
as well to find a sane way to allow internal buffering from a different
stream but have not thought of an elegant solution.
On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote:
Hi,
I was looking at the new readable stream documentation and it said that
we've to implement _read() method in order to implement the readable stream.
But I have a problem?
I want to create a readable stream where it's source is redis
subscribe(). How can I implement a readable stream for this with the new
API?
Do I need to buffer incoming data from redis?
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
In this case, source input does not allow back pressure. We have to deal with emit data or forget situation? I hope necessary buffering will be handled by libuv (at the writable stream). On Sat, Dec 22, 2012 at 1:32 AM, Alexey Petrushin alexey.petrus...@gmail.com wrote: I don't see how old push stream can help to solve this problem. How do You implement back pressure in this case? I guess how - with storing it in node.js memory. And if so - how it's different from new passive _read stream API, if in the end both do exactly the same - store not-readed notifications in memory? -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en -- Arunoda Susiripala @arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/arunoda -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups nodejs group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
That is not the problem. My question is how to build redis-stream using
Stream2 ?
On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum tblob...@gmail.com wrote:
If you havent seen redis-stream, it might help a bit.
https://github.com/tblobaum/redis-stream
var Redis = require('redis-stream')
, client = new Redis('localhost', 6379, 0)
, subscribe = client.stream('subscribe')
subscribe.pipe(process.stdout)
subscribe.write('channel-one')
Thomas Blobaum
https://github.com/tblobaum
On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
Yes I know buffering is not a good solution.
Seems like need to implement this with the old stream api. (emitting data
events).
On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck bradley.m...@gmail.com
wrote:
Personally I did not want to write a buffering system for in memory
_read
and just used a PassThrough stream for node-tart's substream parts :
https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
trying
as well to find a sane way to allow internal buffering from a different
stream but have not thought of an elegant solution.
On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
wrote:
Hi,
I was looking at the new readable stream documentation and it said that
we've to implement _read() method in order to implement the readable
stream.
But I have a problem?
I want to create a readable stream where it's source is redis
subscribe(). How can I implement a readable stream for this with the
new
API?
Do I need to buffer incoming data from redis?
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda http://twitter.com/arunoda
http://gplus.to/arunodahttps://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
Yes, this is doable, but it's a bit odd. Right now, the Socket class
in lib/net.js is handling a similar case, since it has to consume data
from a handle object that only has readStart() and readStop() methods.
So, it's doing a lot of inappropriate touching of internal APIs that
it shouldn't need to do. There does need to be a better mechanism for
handling a pause/resume-able source, and perhaps a userland
abstraction for handling a just drop data if it comes too fast kind
of scenario. It'd be better if lib/net.js didn't have to use any
internal stream APIs to get its job done.
On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
That is not the problem. My question is how to build redis-stream using
Stream2 ?
On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum tblob...@gmail.com wrote:
If you havent seen redis-stream, it might help a bit.
https://github.com/tblobaum/redis-stream
var Redis = require('redis-stream')
, client = new Redis('localhost', 6379, 0)
, subscribe = client.stream('subscribe')
subscribe.pipe(process.stdout)
subscribe.write('channel-one')
Thomas Blobaum
https://github.com/tblobaum
On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
Yes I know buffering is not a good solution.
Seems like need to implement this with the old stream api. (emitting
data
events).
On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck bradley.m...@gmail.com
wrote:
Personally I did not want to write a buffering system for in memory
_read
and just used a PassThrough stream for node-tart's substream parts :
https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
trying
as well to find a sane way to allow internal buffering from a different
stream but have not thought of an elegant solution.
On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
wrote:
Hi,
I was looking at the new readable stream documentation and it said
that
we've to implement _read() method in order to implement the readable
stream.
But I have a problem?
I want to create a readable stream where it's source is redis
subscribe(). How can I implement a readable stream for this with the
new
API?
Do I need to buffer incoming data from redis?
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?
You have to buffer. Ideally you use read-stream's internal buffer rather
then your own buffer.
If you apply backpressure you apply it like you apply it to TCP. net.Socket
already handles back pressure.
On Fri, Dec 21, 2012 at 4:52 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
Actually i really dont need to manually work on the steam api internal
touching. What if i create an old stream and wrap it with the new one.
Is that above make sense. Or do I have to intervene manually to drop
packets.
On Saturday, December 22, 2012, Isaac Schlueter wrote:
Yes, this is doable, but it's a bit odd. Right now, the Socket class
in lib/net.js is handling a similar case, since it has to consume data
from a handle object that only has readStart() and readStop() methods.
So, it's doing a lot of inappropriate touching of internal APIs that
it shouldn't need to do. There does need to be a better mechanism for
handling a pause/resume-able source, and perhaps a userland
abstraction for handling a just drop data if it comes too fast kind
of scenario. It'd be better if lib/net.js didn't have to use any
internal stream APIs to get its job done.
On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
That is not the problem. My question is how to build redis-stream using
Stream2 ?
On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum tblob...@gmail.com
wrote:
If you havent seen redis-stream, it might help a bit.
https://github.com/tblobaum/redis-stream
var Redis = require('redis-stream')
, client = new Redis('localhost', 6379, 0)
, subscribe = client.stream('subscribe')
subscribe.pipe(process.stdout)
subscribe.write('channel-one')
Thomas Blobaum
https://github.com/tblobaum
On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
arunoda.susirip...@gmail.com wrote:
Yes I know buffering is not a good solution.
Seems like need to implement this with the old stream api. (emitting
data
events).
On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck
bradley.m...@gmail.com
wrote:
Personally I did not want to write a buffering system for in memory
_read
and just used a PassThrough stream for node-tart's substream parts :
https://github.com/bmeck/node-tart/blob/master/lib/substream.js .
I'm
trying
as well to find a sane way to allow internal buffering from a
different
stream but have not thought of an elegant solution.
On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
wrote:
Hi,
I was looking at the new readable stream documentation and it said
that
we've to implement _read() method in order to implement the
readable
stream.
But I have a problem?
I want to create a readable stream where it's source is redis
subscribe(). How can I implement a readable stream for this with
the
new
API?
Do I need to buffer incoming data from redis?
--
Arunoda Susiripala
@arunoda
https://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Arunoda Susiripala
@arunoda http://twitter.com/arunoda
http://gplus.to/arunodahttps://github.com/arunoda
http://www.linkedin.com/in/arunoda
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups nodejs group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
