I'm on vacation this week, but decided to see how the node.js list is going.
First I want to reply to Eldar's concerns., Yes, I'm also frustrated with
streams in node.js. But, I do *not* advocate we all stop using node
streams in node modules. Isaac is right that node has reached a level of
maturity where these kinds of changes are no longer possible if we still
call it node. Remember the python 3 fiasco. I think meteor did a great job
of making radical changes and calling it something else.
The main reason I don't use node streams in js-git is because node.js is
just one of the many platforms I support and it didn't make sense to bring
the node stream interface to all the other platforms. I wanted a simpler
common stream API to use and hence the invention of first min-streams and
then simple-streams. I absolutely think that simple-streams should stay a
userland thing in node.js.
Now what I am excited about is the idea Isaac calls "no.js". In this new
platform the core shouldn't have complex streams or even event emitters.
It should only expose platform primitives in the simplest interface
possible (probably still node-style callbacks). In that new system, new
platforms could be easily built on top including one that emulates the
"classic" node.js style APIs. They could all share the same underlying
platform code. This is the future, not jamming more things into today's
node.js. But we can't really work on "no.js" till node.js is "done".
Bruno, thanks for the constructive feedback on my interface. Ever since I
saw your post to this list about your callback based streams, I've loved
the idea. The reason I don't spec a writable interface in simple-streams
is to keep things simple and minimal. But as Jake has noticed, there are
times where a writable interface is the natural interface for a given
problem and I do use them sometimes. I think write(item, callback) is just
fine. (Especially if it's also write(item) -> (callback) if no callback is
passed in)
Regarding the item event, it can be binary data or object streams. My
interface makes no assumptions. In fact, I use object streams a *lot* in
js-git. A very common codec implemented in simple-streams is one that
accepts binary chunks from a file or network stream and emits parsed
objects (or vice-versa for the encoder). As far as "abort" vs "close", I
think it's a slight semantic difference. Close implies file descriptors to
me while "abort" is more abstract. The point of this function is to tell
upstream that you're done reading data and it can cleanup whatever
resources it has. If there is a connector in the middle of the stream
chain like a tee, then it only aborts it's upstream when both it's
downstream chains have aborted.
Feel free to create an issue in js-git to discuss simple-streams (or
continue discussing here, but know I'm too busy writing code to read all
the threads here).
-Tim Caswell
On Thu, Aug 15, 2013 at 7:23 PM, Jake Verbaten <rayn...@gmail.com> wrote:
> > Yes, but why not do things the natural way:
>
> This an imperative vs functional style mismatch. "the natural way" depends
> on your background
>
> ```js
> var fileStream = fs.read("/foo.txt")
> var result = fs.write("/bar.txt", fileStream)
> result(function (err) {
> // entire fileStream has been flushed to disk
> })
> ```
>
> vs
>
> ```
> var fileStream = fs.read("/foo.txt")
> var writeStream = fs.write("/bar.txt")
>
> function pump() {
> fileStream.read(function (err, chunk) {
> if (err) return cleanup()
> writeStream.write(chunk, function (err) {
> if (err) return cleanup()
> pump()
> })
> })
> }
> ```
>
> The functional style is a different way of structuring streaming code that
> assumes that you "never" just write data somewhere.
> You move data from some streaming input to some streaming output in the
> 95% case.
>
> I agree there's value in having a helper with a { write: (chunk, callback)
> } interface, just is not that important to spec.
>
> From looking at creationix's js-git code it appears he uses both styles.
> With `tcpSocket.write(data)` and `tcpSocket.sink(stream)`
>
>
>
> On Thu, Aug 15, 2013 at 2:59 PM, Bruno Jouhier <bjouh...@gmail.com> wrote:
>
>>
>>
>> On Thursday, August 15, 2013 7:43:30 PM UTC+2, Raynos wrote:
>>>
>>> > * Would be nice to also include the writable side in the spec, with a
>>> symmetrical API (a write / abort pair).
>>>
>>> For simplicity you can get away with a Writable stream to just be
>>> `function (simpleStream) { /* consume it */ }`
>>>
>>
>> Yes, but why not do things the natural way:
>>
>> When you need to read data, you "open" (or create) a readable stream,
>> then you read from it, then you close it.
>> When you need to write data, you "open" (or create) a writable stream,
>> then you write from it, then you close it.
>> Piping is just a loop that reads from a readable stream and writes to a
>> writable stream.
>>
>> This is how you write things in C with standard I/O libraries. Why invent
>> some weird APIs instead of following this simple model?
>>
>> And backpressure is no excuse. If your read and write functions are both
>> callback based, backpressure works naturally through the event loop.
>>
>> Bruno
>>
>>
>> Then you don't need a central `pipe()` or `pump()` implementation and
>> each writable stream can implement it's own
>>
>>> consumption & back pressure algorithm.
>>>
>>> > * Spec does not say what item is, I'm assuming it can be a Buffer or a
>>> string. Can it also be an object? why not?
>>>
>>> I believe it's an arbitrary value including another simple stream
>>>
>>>
>>> On Thu, Aug 15, 2013 at 3:05 AM, Bruno Jouhier <bjou...@gmail.com>wrote:
>>>
>>>> Isaac's message is clear. Simple streams won't make it into core.
>>>>
>>>> So the solution is probably in userland. Tim's module looks like a good
>>>> starting point and if it gets popular it could become a de facto "simple
>>>> stream" standard API.
>>>>
>>>> My comments on it:
>>>>
>>>> * Would be nice to also include the writable side in the spec, with a
>>>> symmetrical API (a write / abort pair).
>>>> * Why not rename abort close?
>>>> * Spec does not say what item is, I'm assuming it can be a Buffer or a
>>>> string. Can it also be an object? why not?
>>>> * The "Related Interfaces" part should be left out. If we want it to be
>>>> popular, we have to keep it as simple as possible.
>>>>
>>>> Streamline's streams interface is very similar and provides wrappers
>>>> around most of node's steams (HTTP and TCP, client and server). It can also
>>>> be used as source of inspiration (but I'll remap it to simple-streams if
>>>> simple-streams catches up).
>>>>
>>>> Bruno
>>>>
>>>>
>>>>
>>>> On Wednesday, August 14, 2013 7:02:19 PM UTC+2, Eldar wrote:
>>>>>
>>>>> Node completely shutdowns because emitter.emit('error') triggers
>>>>> exception when there is no listeners for error event.
>>>>> The ".pipe()" listens for errors but doesn't change that behavior.
>>>>>
>>>>> So strictly speaking above server will shutdown on file error because
>>>>> of uncaught exception
>>>>> and leak on socket hang up, because "close" not an "error" event will
>>>>> be emitted.
>>>>>
>>>>> среда, 14 августа 2013 г., 13:47:14 UTC+4 пользователь Floby написал:
>>>>>>
>>>>>> file.pipe(res) does clean the resources on error because node
>>>>>> completely shuts down, doesn't it?
>>>>>>
>>>>>> On Tuesday, 13 August 2013 12:08:30 UTC+2, Eldar wrote:
>>>>>>>
>>>>>>> file.pipe(res)
>>>>>>>
>>>>>>> On response error (say client clicked close window button) the file
>>>>>>> will remain open.
>>>>>>> And vice versa on file error response will hang. So the correct way
>>>>>>> to pipe is
>>>>>>>
>>>>>>> file.pipe(res)
>>>>>>> file.on('error', res.destroy.bind(res))
>>>>>>> res.on('error', file.destroy.bind(res))
>>>>>>>
>>>>>>> вторник, 13 августа 2013 г., 12:58:09 UTC+4 пользователь Floby
>>>>>>> написал:
>>>>>>>>
>>>>>>>> ok.
>>>>>>>> I didn't understand the part about " `.pipe()` doesn't cleanup any
>>>>>>>> resources.
>>>>>>>> Everything works fine until the streaming process completes
>>>>>>>> successfully. "
>>>>>>>>
>>>>>>>> Is there an example where that becomes obvious?
>>>>>>>>
>>>>>>>> On Monday, 12 August 2013 18:05:02 UTC+2, Eldar wrote:
>>>>>>>>>
>>>>>>>>> If you want to create some function which accepts a stream and can
>>>>>>>>> affect
>>>>>>>>> it's state you should be able to destroy the given stream on demand
>>>>>>>>> e.g. when something went wrong and/or you are not interested in
>>>>>>>>> receiving data anymore.
>>>>>>>>> Unfortunately that's impossible with current node readable streams
>>>>>>>>> because they:
>>>>>>>>>
>>>>>>>>> 1. Support multiple consumers
>>>>>>>>> 2. Don't have common cleanup interface
>>>>>>>>>
>>>>>>>>> So you can't touch another's stream because someone else may use it
>>>>>>>>> and if you could you don't know how.
>>>>>>>>>
>>>>>>>>> The absence of ability to pass streams as arguments is a huge flow.
>>>>>>>>> You can build almost nothing
>>>>>>>>> without that. Now someone may wonder: but there are so many useful
>>>>>>>>> modules about streams in npm!?
>>>>>>>>> Right. They all rely on `.pipe()`. For some reason all kind of blog
>>>>>>>>> posts, tutorials and even node's
>>>>>>>>> own documentation teach that for passing readable you can do
>>>>>>>>> `readable.pipe(writable)`.
>>>>>>>>> E.g. http file server could be:
>>>>>>>>>
>>>>>>>>> ```javascript
>>>>>>>>> var http = require('http')
>>>>>>>>> var fs = require('fs')
>>>>>>>>> http.createServer(function(**req**, res) {
>>>>>>>>> res.writeHead(200, {'Content-Type': 'text/plain'})
>>>>>>>>> fs.createReadStream(req.url.**sl**ice(1)).pipe(res)
>>>>>>>>> }).listen(3000)
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> That's a lie. Exactly for the reasons above `.pipe()` doesn't cleanup
>>>>>>>>> any resources.
>>>>>>>>> Everything works fine until the streaming process completes
>>>>>>>>> successfully.
>>>>>>>>> But if something fails you get horrible leaks and hangs.
>>>>>>>>> Almost every module doing stream processing have this flow. Obvious
>>>>>>>>> or not.
>>>>>>>>>
>>>>>>>>> That all is sad. We need another, simpler API. Although I believe
>>>>>>>>> incompatible changes should be made
>>>>>>>>> to node core that is not what I am calling about (at least not at the
>>>>>>>>> first place).
>>>>>>>>> But we definitely should stop to expose node streams in userland
>>>>>>>>> modules,
>>>>>>>>> stop to extend "stream base classes", just stop to use it in userland.
>>>>>>>>>
>>>>>>>>> Now the constructive part. Tim Caswell proposed recently
>>>>>>>>> simple-stream
>>>>>>>>> <https://github.com/creationix/js-git/blob/master/specs/simple-stream.md>
>>>>>>>>> and build many things on top of it.
>>>>>>>>>
>>>>>>>>> Let's just peek this API as a standard and start using it.
>>>>>>>>> Here is readable-to-simple-stream
>>>>>>>>> <https://github.com/eldargab/stream-simple> converter which can be
>>>>>>>>> used for dealing with node core.
>>>>>>>>>
>>>>>>>>> It also contains slightly more detailed version of original spec
>>>>>>>>> which could be discussed.
>>>>>>>>>
>>>>>>>>> That's it.
>>>>>>>>>
>>>>>>>>> --
>>>> --
>>>> Job Board: http://jobs.nodejs.org/
>>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-
>>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines>
>>>> You received this message because you are subscribed to the Google
>>>> Groups "nodejs" group.
>>>> To post to this group, send email to nod...@googlegroups.com
>>>>
>>>> To unsubscribe from this group, send email to
>>>> nodejs+un...@**googlegroups.com
>>>>
>>>> For more options, visit this group at
>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en>
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "nodejs" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to nodejs+un...@**googlegroups.com.
>>>>
>>>> For more options, visit
>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .
>>>>
>>>
>>> --
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nodejs@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "nodejs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to nodejs+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
