Wrote this at an offsite where we were discussing Hadoop as a proof of concept:
https://github.com/dshaw/reduce-stream
in conjunction with
https://github.com/dshaw/filter-stream
Was able to boil down 60+ GB of log data to weighted geodata using
these two modules.
My apologies for the complete lack of docs. Hopefully the tests shed
some light on this little mad science project.
This needs a couple more stream patterns (stream fanout) to be truly
useful, but the idea does work.
Daniel Shaw
@dshaw
On Sat, Jun 23, 2012 at 2:08 PM, José F. Romaniello
<jfromanie...@gmail.com> wrote:
> Excellent Dominic, yes! The idea of reduce ( idea, 0 code) was to make a
> readable/writeable stream. For each chunk passed to the reduce, it will
> execute the reduce function, when the input stream ends it will the reduce
> stream will emit one and only one data event with the value of the
> accumulator and it will ends. The example i wrote is totally wrong and
> should be
>
>
> var aFileStream = fs.createReadStream("foo.txt");
> aFileStream.pipe(funs.reduce( function (prev, chunk) { return prev +
> chunk.match(/o/g).length; }))
> .pipe(funs.map( function (total) { return " total amoun of 'o':
> " + total; }))
> .pipe(process.stdout);
>
> Not sure if is useful though
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
