[nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Bradley Meck]

Personally I did not want to write a buffering system for in memory _read 
and just used a PassThrough stream for node-tart's substream parts 
: https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm 
trying as well to find a sane way to allow internal buffering from a 
different stream but have not thought of an elegant solution.

On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote:
>
> Hi, 
>
> I was looking at the new readable stream documentation and it said that 
> we've to implement _read() method in order to implement the readable 
> stream.
> But I have a problem?
>
> I want to create a readable stream where it's source is redis subscribe(). 
> How can I implement a readable stream for this with the new API?
>
> Do I need to buffer incoming data from redis?
>
> -- 
> Arunoda Susiripala
>
> @arunoda 
> https://github.com/arunoda
> http://www.linkedin.com/in/arunoda
>

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Alexey Petrushin]

I don't see how old "push stream" can help to solve this problem. How do 
You implement  "back pressure" in this case?

I guess how - with storing it in node.js memory. And if so - how it's 
different from  new "passive" _read stream API, if in the end both do 
exactly the same - store not-readed notifications in memory?

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

Actually i really dont need to manually work on the steam api internal
touching.  What if i create an old stream and wrap it with the new one.

Is that above make sense. Or do I have to intervene manually to drop
packets.

On Saturday, December 22, 2012, Isaac Schlueter wrote:

> Yes, this is doable, but it's a bit odd.  Right now, the Socket class
> in lib/net.js is handling a similar case, since it has to consume data
> from a handle object that only has readStart() and readStop() methods.
>  So, it's doing a lot of inappropriate touching of internal APIs that
> it shouldn't need to do.  There does need to be a better mechanism for
> handling a pause/resume-able source, and perhaps a userland
> abstraction for handling a "just drop data if it comes too fast" kind
> of scenario.  It'd be better if lib/net.js didn't have to use any
> internal stream APIs to get its job done.
>
>
> On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
>  wrote:
> > That is not the problem. My question is how to build redis-stream using
> > Stream2 ?
> >
> >
> > On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum 
> wrote:
> >>
> >> If you havent seen redis-stream, it might help a bit.
> >> https://github.com/tblobaum/redis-stream
> >>
> >> var Redis = require('redis-stream')
> >>   , client = new Redis('localhost', 6379, 0)
> >>   , subscribe = client.stream('subscribe')
> >>
> >> subscribe.pipe(process.stdout)
> >> subscribe.write('channel-one')
> >>
> >>
> >> Thomas Blobaum
> >> https://github.com/tblobaum
> >>
> >>
> >> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
> >>  wrote:
> >> > Yes I know buffering is not a good solution.
> >> > Seems like need to implement this with the old stream api. (emitting
> >> > data
> >> > events).
> >> >
> >> >
> >> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck <
> bradley.m...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Personally I did not want to write a buffering system for in memory
> >> >> _read
> >> >> and just used a PassThrough stream for node-tart's substream parts :
> >> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js .
> I'm
> >> >> trying
> >> >> as well to find a sane way to allow internal buffering from a
> different
> >> >> stream but have not thought of an elegant solution.
> >> >>
> >> >>
> >> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I was looking at the new readable stream documentation and it said
> >> >>> that
> >> >>> we've to implement _read() method in order to implement the readable
> >> >>> stream.
> >> >>>
> >> >>> But I have a problem?
> >> >>>
> >> >>> I want to create a readable stream where it's source is redis
> >> >>> subscribe(). How can I implement a readable stream for this with the
> >> >>> new
> >> >>> API?
> >> >>>
> >> >>> Do I need to buffer incoming data from redis?
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Arunoda Susiripala
> >> >>>
> >> >>> @arunoda
> >> >>> https://github.com/arunoda
> >> >>> http://www.linkedin.com/in/arunoda
> >> >>
> >> >> --
> >> >> Job Board: http://jobs.nodejs.org/
> >> >> Posting guidelines:
> >> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups "nodejs" group.
> >> >> To post to this group, send email to nodejs@googlegroups.com
> >> >> To unsubscribe from this group, send email to
> >> >> nodejs+unsubscr...@googlegroups.com
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/nodejs?hl=en?hl=en
> >> >
> >> >
> >> >
> >



-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

I should look at how net module does it.

On Saturday, December 22, 2012, Jake Verbaten wrote:

> You have to buffer. Ideally you use read-stream's internal buffer rather
> then your own buffer.
>
> If you apply backpressure you apply it like you apply it to TCP.
> net.Socket already handles back pressure.
>
>
> On Fri, Dec 21, 2012 at 4:52 PM, Arunoda Susiripala <
> arunoda.susirip...@gmail.com> wrote:
>
> Actually i really dont need to manually work on the steam api internal
> touching.  What if i create an old stream and wrap it with the new one.
>
> Is that above make sense. Or do I have to intervene manually to drop
> packets.
>
>
> On Saturday, December 22, 2012, Isaac Schlueter wrote:
>
> Yes, this is doable, but it's a bit odd.  Right now, the Socket class
> in lib/net.js is handling a similar case, since it has to consume data
> from a handle object that only has readStart() and readStop() methods.
>  So, it's doing a lot of inappropriate touching of internal APIs that
> it shouldn't need to do.  There does need to be a better mechanism for
> handling a pause/resume-able source, and perhaps a userland
> abstraction for handling a "just drop data if it comes too fast" kind
> of scenario.  It'd be better if lib/net.js didn't have to use any
> internal stream APIs to get its job done.
>
>
> On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
>  wrote:
> > That is not the problem. My question is how to build redis-stream using
> > Stream2 ?
> >
> >
> > On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum 
> wrote:
> >>
> >> If you havent seen redis-stream, it might help a bit.
> >> https://github.com/tblobaum/redis-stream
> >>
> >> var Redis = require('redis-stream')
> >>   , client = new Redis('localhost', 6379, 0)
> >>   , subscribe = client.stream('subscribe')
> >>
> >> subscribe.pipe(process.stdout)
> >> subscribe.write('channel-one')
> >>
> >>
> >> Thomas Blobaum
> >> https://github.com/tblobaum
> >>
> >>
> >> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
> >>  wrote:
> >> > Yes I know buffering is not a good solution.
> >> > Seems like need to implement this with the old stream api. (emitting
> >> > data
> >> > events).
> >> >
> >> >
> >> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck <
> bradley.m...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Personally I did not want to write a buffering system for in memory
> >> >> _read
> >> >> and just used a PassThrough stream for node-tart's substream parts :
> >> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js .
> I'm
> >> >> trying
> >> >> as well to find a sane way to allow internal buffering from a
> different
> >> >> stream but have not thought of an elegant solution.
> >> >>
> >> >>
> >> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I was looking at the new readable stream documentation and it said
> >> >>> that
> >> >>> we've to implement _read() method in order to implement the readable
> >> >>> stream.
> >> >>>
> >> >>> But I have a problem?
> >> >>>
> >> >>> I want to create a readable stream where it's source is redis
> >> >>> subscribe(). How can I implement a readable stream for this with the
> >> >>> new
> >> >>> API?
> >> >>>
> >> >>> Do I need to buffer incoming data from redis?
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Arunoda Susiripala
> >> >>>
> >> >>> @arunoda
> >> >>> https://github.com/arunoda
> >> >>> http://www.linkedin.com/in/arunoda
> >> >>
> >> >> --
> >> >> Job Board: 
>
>

-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Jake Verbaten]

This is probably an API design issue with readable stream

When a readable stream wraps a source where data must be pulled from then
_read makes sense.

When a readable stream wraps a source that pushes data then having some
kind of stream._push(chunk) api makes more sense.

My thin readable-stream abstraction has a push api (
https://github.com/Raynos/read-stream#example-queue )


On Fri, Dec 21, 2012 at 11:11 AM, Bradley Meck wrote:

> Personally I did not want to write a buffering system for in memory _read
> and just used a PassThrough stream for node-tart's substream parts :
> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
> trying as well to find a sane way to allow internal buffering from a
> different stream but have not thought of an elegant solution.
>
>
> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote:
>>
>> Hi,
>>
>> I was looking at the new readable stream documentation and it said that
>> we've to implement _read() method in order to implement the readable
>> stream.
>> But I have a problem?
>>
>> I want to create a readable stream where it's source is redis subscribe().
>> How can I implement a readable stream for this with the new API?
>>
>> Do I need to buffer incoming data from redis?
>>
>> --
>> Arunoda Susiripala
>>
>> @arunoda 
>> https://github.com/arunoda
>> http://www.linkedin.com/in/**arunoda 
>>
>  --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

Yes I know buffering is not a good solution.
Seems like need to implement this with the old stream api. (emitting data
events).


On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck wrote:

> Personally I did not want to write a buffering system for in memory _read
> and just used a PassThrough stream for node-tart's substream parts :
> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
> trying as well to find a sane way to allow internal buffering from a
> different stream but have not thought of an elegant solution.
>
>
> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote:
>>
>> Hi,
>>
>> I was looking at the new readable stream documentation and it said that
>> we've to implement _read() method in order to implement the readable
>> stream.
>> But I have a problem?
>>
>> I want to create a readable stream where it's source is redis subscribe().
>> How can I implement a readable stream for this with the new API?
>>
>> Do I need to buffer incoming data from redis?
>>
>> --
>> Arunoda Susiripala
>>
>> @arunoda 
>> https://github.com/arunoda
>> http://www.linkedin.com/in/**arunoda 
>>
>  --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>



-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Thomas Blobaum]

If you havent seen redis-stream, it might help a bit.
https://github.com/tblobaum/redis-stream

var Redis = require('redis-stream')
  , client = new Redis('localhost', 6379, 0)
  , subscribe = client.stream('subscribe')

subscribe.pipe(process.stdout)
subscribe.write('channel-one')


Thomas Blobaum
https://github.com/tblobaum


On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
 wrote:
> Yes I know buffering is not a good solution.
> Seems like need to implement this with the old stream api. (emitting data
> events).
>
>
> On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck 
> wrote:
>>
>> Personally I did not want to write a buffering system for in memory _read
>> and just used a PassThrough stream for node-tart's substream parts :
>> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm trying
>> as well to find a sane way to allow internal buffering from a different
>> stream but have not thought of an elegant solution.
>>
>>
>> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala wrote:
>>>
>>> Hi,
>>>
>>> I was looking at the new readable stream documentation and it said that
>>> we've to implement _read() method in order to implement the readable stream.
>>>
>>> But I have a problem?
>>>
>>> I want to create a readable stream where it's source is redis
>>> subscribe(). How can I implement a readable stream for this with the new
>>> API?
>>>
>>> Do I need to buffer incoming data from redis?
>>>
>>>
>>> --
>>> Arunoda Susiripala
>>>
>>> @arunoda
>>> https://github.com/arunoda
>>> http://www.linkedin.com/in/arunoda
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nodejs@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
>
> --
> Arunoda Susiripala
>
> @arunoda
> https://github.com/arunoda
> http://www.linkedin.com/in/arunoda
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

In this case, source input does not allow back pressure. We have to deal
with emit data  or forget situation?
I hope necessary buffering will be handled by libuv (at the writable
stream).



On Sat, Dec 22, 2012 at 1:32 AM, Alexey Petrushin <
alexey.petrus...@gmail.com> wrote:

> I don't see how old "push stream" can help to solve this problem. How do
> You implement  "back pressure" in this case?
>
> I guess how - with storing it in node.js memory. And if so - how it's
> different from  new "passive" _read stream API, if in the end both do
> exactly the same - store not-readed notifications in memory?
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>



-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

That is not the problem. My question is how to build redis-stream using
Stream2 ?

On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum  wrote:

> If you havent seen redis-stream, it might help a bit.
> https://github.com/tblobaum/redis-stream
>
> var Redis = require('redis-stream')
>   , client = new Redis('localhost', 6379, 0)
>   , subscribe = client.stream('subscribe')
>
> subscribe.pipe(process.stdout)
> subscribe.write('channel-one')
>
>
> Thomas Blobaum
> https://github.com/tblobaum
>
>
> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
>  wrote:
> > Yes I know buffering is not a good solution.
> > Seems like need to implement this with the old stream api. (emitting data
> > events).
> >
> >
> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck 
> > wrote:
> >>
> >> Personally I did not want to write a buffering system for in memory
> _read
> >> and just used a PassThrough stream for node-tart's substream parts :
> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
> trying
> >> as well to find a sane way to allow internal buffering from a different
> >> stream but have not thought of an elegant solution.
> >>
> >>
> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I was looking at the new readable stream documentation and it said that
> >>> we've to implement _read() method in order to implement the readable
> stream.
> >>>
> >>> But I have a problem?
> >>>
> >>> I want to create a readable stream where it's source is redis
> >>> subscribe(). How can I implement a readable stream for this with the
> new
> >>> API?
> >>>
> >>> Do I need to buffer incoming data from redis?
> >>>
> >>>
> >>> --
> >>> Arunoda Susiripala
> >>>
> >>> @arunoda
> >>> https://github.com/arunoda
> >>> http://www.linkedin.com/in/arunoda
> >>
> >> --
> >> Job Board: http://jobs.nodejs.org/
> >> Posting guidelines:
> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> >> You received this message because you are subscribed to the Google
> >> Groups "nodejs" group.
> >> To post to this group, send email to nodejs@googlegroups.com
> >> To unsubscribe from this group, send email to
> >> nodejs+unsubscr...@googlegroups.com
> >> For more options, visit this group at
> >> http://groups.google.com/group/nodejs?hl=en?hl=en
> >
> >
> >
> >
> > --
> > Arunoda Susiripala
> >
> > @arunoda
> > https://github.com/arunoda
> > http://www.linkedin.com/in/arunoda
> >
> > --
> > Job Board: http://jobs.nodejs.org/
> > Posting guidelines:
> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > You received this message because you are subscribed to the Google
> > Groups "nodejs" group.
> > To post to this group, send email to nodejs@googlegroups.com
> > To unsubscribe from this group, send email to
> > nodejs+unsubscr...@googlegroups.com
> > For more options, visit this group at
> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>



-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Isaac Schlueter]

Yes, this is doable, but it's a bit odd.  Right now, the Socket class
in lib/net.js is handling a similar case, since it has to consume data
from a handle object that only has readStart() and readStop() methods.
 So, it's doing a lot of inappropriate touching of internal APIs that
it shouldn't need to do.  There does need to be a better mechanism for
handling a pause/resume-able source, and perhaps a userland
abstraction for handling a "just drop data if it comes too fast" kind
of scenario.  It'd be better if lib/net.js didn't have to use any
internal stream APIs to get its job done.


On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
 wrote:
> That is not the problem. My question is how to build redis-stream using
> Stream2 ?
>
>
> On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum  wrote:
>>
>> If you havent seen redis-stream, it might help a bit.
>> https://github.com/tblobaum/redis-stream
>>
>> var Redis = require('redis-stream')
>>   , client = new Redis('localhost', 6379, 0)
>>   , subscribe = client.stream('subscribe')
>>
>> subscribe.pipe(process.stdout)
>> subscribe.write('channel-one')
>>
>>
>> Thomas Blobaum
>> https://github.com/tblobaum
>>
>>
>> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
>>  wrote:
>> > Yes I know buffering is not a good solution.
>> > Seems like need to implement this with the old stream api. (emitting
>> > data
>> > events).
>> >
>> >
>> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck 
>> > wrote:
>> >>
>> >> Personally I did not want to write a buffering system for in memory
>> >> _read
>> >> and just used a PassThrough stream for node-tart's substream parts :
>> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js . I'm
>> >> trying
>> >> as well to find a sane way to allow internal buffering from a different
>> >> stream but have not thought of an elegant solution.
>> >>
>> >>
>> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I was looking at the new readable stream documentation and it said
>> >>> that
>> >>> we've to implement _read() method in order to implement the readable
>> >>> stream.
>> >>>
>> >>> But I have a problem?
>> >>>
>> >>> I want to create a readable stream where it's source is redis
>> >>> subscribe(). How can I implement a readable stream for this with the
>> >>> new
>> >>> API?
>> >>>
>> >>> Do I need to buffer incoming data from redis?
>> >>>
>> >>>
>> >>> --
>> >>> Arunoda Susiripala
>> >>>
>> >>> @arunoda
>> >>> https://github.com/arunoda
>> >>> http://www.linkedin.com/in/arunoda
>> >>
>> >> --
>> >> Job Board: http://jobs.nodejs.org/
>> >> Posting guidelines:
>> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> >> You received this message because you are subscribed to the Google
>> >> Groups "nodejs" group.
>> >> To post to this group, send email to nodejs@googlegroups.com
>> >> To unsubscribe from this group, send email to
>> >> nodejs+unsubscr...@googlegroups.com
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/nodejs?hl=en?hl=en
>> >
>> >
>> >
>> >
>> > --
>> > Arunoda Susiripala
>> >
>> > @arunoda
>> > https://github.com/arunoda
>> > http://www.linkedin.com/in/arunoda
>> >
>> > --
>> > Job Board: http://jobs.nodejs.org/
>> > Posting guidelines:
>> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> > You received this message because you are subscribed to the Google
>> > Groups "nodejs" group.
>> > To post to this group, send email to nodejs@googlegroups.com
>> > To unsubscribe from this group, send email to
>> > nodejs+unsubscr...@googlegroups.com
>> > For more options, visit this group at
>> > http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nodejs@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
>
> --
> Arunoda Susiripala
>
> @arunoda
> https://github.com/arunoda
> http://www.linkedin.com/in/arunoda
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Jake Verbaten]

You have to buffer. Ideally you use read-stream's internal buffer rather
then your own buffer.

If you apply backpressure you apply it like you apply it to TCP. net.Socket
already handles back pressure.


On Fri, Dec 21, 2012 at 4:52 PM, Arunoda Susiripala <
arunoda.susirip...@gmail.com> wrote:

> Actually i really dont need to manually work on the steam api internal
> touching.  What if i create an old stream and wrap it with the new one.
>
> Is that above make sense. Or do I have to intervene manually to drop
> packets.
>
>
> On Saturday, December 22, 2012, Isaac Schlueter wrote:
>
>> Yes, this is doable, but it's a bit odd.  Right now, the Socket class
>> in lib/net.js is handling a similar case, since it has to consume data
>> from a handle object that only has readStart() and readStop() methods.
>>  So, it's doing a lot of inappropriate touching of internal APIs that
>> it shouldn't need to do.  There does need to be a better mechanism for
>> handling a pause/resume-able source, and perhaps a userland
>> abstraction for handling a "just drop data if it comes too fast" kind
>> of scenario.  It'd be better if lib/net.js didn't have to use any
>> internal stream APIs to get its job done.
>>
>>
>> On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
>>  wrote:
>> > That is not the problem. My question is how to build redis-stream using
>> > Stream2 ?
>> >
>> >
>> > On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum 
>> wrote:
>> >>
>> >> If you havent seen redis-stream, it might help a bit.
>> >> https://github.com/tblobaum/redis-stream
>> >>
>> >> var Redis = require('redis-stream')
>> >>   , client = new Redis('localhost', 6379, 0)
>> >>   , subscribe = client.stream('subscribe')
>> >>
>> >> subscribe.pipe(process.stdout)
>> >> subscribe.write('channel-one')
>> >>
>> >>
>> >> Thomas Blobaum
>> >> https://github.com/tblobaum
>> >>
>> >>
>> >> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
>> >>  wrote:
>> >> > Yes I know buffering is not a good solution.
>> >> > Seems like need to implement this with the old stream api. (emitting
>> >> > data
>> >> > events).
>> >> >
>> >> >
>> >> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck <
>> bradley.m...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Personally I did not want to write a buffering system for in memory
>> >> >> _read
>> >> >> and just used a PassThrough stream for node-tart's substream parts :
>> >> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js .
>> I'm
>> >> >> trying
>> >> >> as well to find a sane way to allow internal buffering from a
>> different
>> >> >> stream but have not thought of an elegant solution.
>> >> >>
>> >> >>
>> >> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
>> >> >> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I was looking at the new readable stream documentation and it said
>> >> >>> that
>> >> >>> we've to implement _read() method in order to implement the
>> readable
>> >> >>> stream.
>> >> >>>
>> >> >>> But I have a problem?
>> >> >>>
>> >> >>> I want to create a readable stream where it's source is redis
>> >> >>> subscribe(). How can I implement a readable stream for this with
>> the
>> >> >>> new
>> >> >>> API?
>> >> >>>
>> >> >>> Do I need to buffer incoming data from redis?
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Arunoda Susiripala
>> >> >>>
>> >> >>> @arunoda
>> >> >>> https://github.com/arunoda
>> >> >>> http://www.linkedin.com/in/arunoda
>> >> >>
>> >> >> --
>> >> >> Job Board: http://jobs.nodejs.org/
>> >> >> Posting guidelines:
>> >> >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups "nodejs" group.
>> >> >> To post to this group, send email to nodejs@googlegroups.com
>> >> >> To unsubscribe from this group, send email to
>> >> >> nodejs+unsubscr...@googlegroups.com
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/nodejs?hl=en?hl=en
>> >> >
>> >> >
>> >> >
>> >
>
>
>
> --
> Arunoda Susiripala
>
> @arunoda 
> https://github.com/arunoda
> http://www.linkedin.com/in/arunoda
>
>  --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nod

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Isaac Schlueter]

Wrapping streams is doable, and can be very expedient, but it can get
expensive.  It's better to not do this if you don't have to.

The basic idea is that, every time a new chunk of data comes in, you
call stream._readableState.onread(null, chunk).  Then, if
stream._readableState.length > stream._readableState.highWaterMark,
pause the source.  Whenever some data is consumed, if the length is
less than the highWaterMark, then resume the source.

Definitely needs some API love.

On Fri, Dec 21, 2012 at 7:12 PM, Arunoda Susiripala
 wrote:
> I should look at how net module does it.
>
>
> On Saturday, December 22, 2012, Jake Verbaten wrote:
>>
>> You have to buffer. Ideally you use read-stream's internal buffer rather
>> then your own buffer.
>>
>> If you apply backpressure you apply it like you apply it to TCP.
>> net.Socket already handles back pressure.
>>
>>
>> On Fri, Dec 21, 2012 at 4:52 PM, Arunoda Susiripala
>>  wrote:
>>
>> Actually i really dont need to manually work on the steam api internal
>> touching.  What if i create an old stream and wrap it with the new one.
>>
>> Is that above make sense. Or do I have to intervene manually to drop
>> packets.
>>
>>
>> On Saturday, December 22, 2012, Isaac Schlueter wrote:
>>
>> Yes, this is doable, but it's a bit odd.  Right now, the Socket class
>> in lib/net.js is handling a similar case, since it has to consume data
>> from a handle object that only has readStart() and readStop() methods.
>>  So, it's doing a lot of inappropriate touching of internal APIs that
>> it shouldn't need to do.  There does need to be a better mechanism for
>> handling a pause/resume-able source, and perhaps a userland
>> abstraction for handling a "just drop data if it comes too fast" kind
>> of scenario.  It'd be better if lib/net.js didn't have to use any
>> internal stream APIs to get its job done.
>>
>>
>> On Fri, Dec 21, 2012 at 12:32 PM, Arunoda Susiripala
>>  wrote:
>> > That is not the problem. My question is how to build redis-stream using
>> > Stream2 ?
>> >
>> >
>> > On Sat, Dec 22, 2012 at 1:55 AM, Thomas Blobaum 
>> > wrote:
>> >>
>> >> If you havent seen redis-stream, it might help a bit.
>> >> https://github.com/tblobaum/redis-stream
>> >>
>> >> var Redis = require('redis-stream')
>> >>   , client = new Redis('localhost', 6379, 0)
>> >>   , subscribe = client.stream('subscribe')
>> >>
>> >> subscribe.pipe(process.stdout)
>> >> subscribe.write('channel-one')
>> >>
>> >>
>> >> Thomas Blobaum
>> >> https://github.com/tblobaum
>> >>
>> >>
>> >> On Fri, Dec 21, 2012 at 2:20 PM, Arunoda Susiripala
>> >>  wrote:
>> >> > Yes I know buffering is not a good solution.
>> >> > Seems like need to implement this with the old stream api. (emitting
>> >> > data
>> >> > events).
>> >> >
>> >> >
>> >> > On Sat, Dec 22, 2012 at 12:41 AM, Bradley Meck
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Personally I did not want to write a buffering system for in memory
>> >> >> _read
>> >> >> and just used a PassThrough stream for node-tart's substream parts :
>> >> >> https://github.com/bmeck/node-tart/blob/master/lib/substream.js .
>> >> >> I'm
>> >> >> trying
>> >> >> as well to find a sane way to allow internal buffering from a
>> >> >> different
>> >> >> stream but have not thought of an elegant solution.
>> >> >>
>> >> >>
>> >> >> On Friday, December 21, 2012 10:07:07 AM UTC-6, Arunoda Susiripala
>> >> >> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I was looking at the new readable stream documentation and it said
>> >> >>> that
>> >> >>> we've to implement _read() method in order to implement the
>> >> >>> readable
>> >> >>> stream.
>> >> >>>
>> >> >>> But I have a problem?
>> >> >>>
>> >> >>> I want to create a readable stream where it's source is redis
>> >> >>> subscribe(). How can I implement a readable stream for this with
>> >> >>> the
>> >> >>> new
>> >> >>> API?
>> >> >>>
>> >> >>> Do I need to buffer incoming data from redis?
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Arunoda Susiripala
>> >> >>>
>> >> >>> @arunoda
>> >> >>> https://github.com/arunoda
>> >> >>> http://www.linkedin.com/in/arunoda
>> >> >>
>> >> >> --
>> >> >> Job Board:
>
>
>
> --
> Arunoda Susiripala
>
> @arunoda
> https://github.com/arunoda
> http://www.linkedin.com/in/arunoda
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googleg

Re: [nodejs] Re: Stream2: How to create readable stream for redis subscribe?

[Arunoda Susiripala]

Thanks izs.

I got the idea and how to do this.
Can't we document this in the API doc, so it helps others.
Or build a api for this kind of scenarios as well.



On Tue, Dec 25, 2012 at 4:00 AM, Isaac Schlueter  wrote:

> expensive




-- 
Arunoda Susiripala

@arunoda 
https://github.com/arunoda
http://www.linkedin.com/in/arunoda

-- 
Job Board: http://jobs.nodejs.org/
Posting guidelines: 
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en