On Jul 28, 2012, at July 28, 201212:22 PM, Tim Caswell <t...@creationix.com> wrote:
> On Sat, Jul 28, 2012 at 2:14 PM, Mikeal Rogers <mikeal.rog...@gmail.com> > wrote: >> >> On Jul 28, 2012, at July 28, 201212:05 PM, Tim Caswell <t...@creationix.com> >> wrote: >> >>> FWIW, I actually like Bruno's proposal. It doesn't cover all the use >>> cases, but it makes backpressure enabled pumps really easy. >>> >>> One use case missing that's easy to add is when consuming a binary >>> protocol, I often only want part of the input. For example, I might >>> want to get the first 4 bytes, decode that as a uint32 length header >>> and then read n more bytes for the body. Without being able to >>> request how many bytes I want, I have to handle putting data back in >>> the stream that I don't need. That's very error prone and tedious. >>> So on the read function, add an optional "maxBytes" or "bytes" >>> parameter. The difference is in the maxBytes case, I want the data as >>> soon as there is anything, even if it's less than the number of bytes >>> I want. In the "bytes" case I want to wait till that many bytes are >>> available. Both are valid for different use cases. >> >> The early stuff I saw included a "length" option. >> >>> >>> Also streams (both readable and writable) need a configurable >>> low-water mark. I don't want to wait till the pipe is empty before I >>> start piping data again. This mark would control how soon writable >>> streams called my write callback and how much readable streams would >>> readahead from their data source before waiting for me to call read. >>> I want to keep it always full. It would be great if this was handled >>> internally in the stream and consumers of the stream simply configured >>> what the mark should be. >> >> I think you're missing how this works. Nobody automatically asks for data so >> watermarks aren't strictly necessary. You ask for data if it's available and >> you read as much as you can handle. >> >> There is no "readahead". If someone stops calling read() then the buffer >> fills and, if it's a TCP stream, it's asked to stop sending data. >> >> Remember that when the "readable" event goes off it's expected that the >> pending data is read in the same event loop cycle. >> > > In any backpressure case where the data provider is faster than the > consumer, there will be a series of pauses and resumes to slow down > the provider. That is the point of backpressure. > > So in these discussions we need to assume that the pauses and resumes > will be happening. What triggers a pause, what triggers a resume? > Will there ever be a case where the pipe is empty and stalled. This > is what I'm trying to avoid. It's kills throughput. > >> If someone stops calling read() then the buffer fills and, if it's a TCP >> stream, it's asked to stop sending data. > > When does it ask the tcp stream to stop sending data. And when do we > tell the tcp stream to start sending data? How long does it take for > the tcp stream to get this message and start sending the data? If we > wait till the buffer is 100% empty and new data isn't loaded > immedietly, then it's a stall. This is where a low-water mark keeps > things running smoothly. pause and resume at the stream layer are gone. the TCP socket will get paused if "readable" is emitted and read() is not called. it will be resumed when read() is called again. the amount of data that is available on a "readable" event is variable. the amount of data buffered is of no importance to pause/resume logic. -Mikeal
