Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Denys Khanzhyiev
I will agree that WriteStream shows some strange performance loss. It may be explained by WriteStream's overhead for buffering and eventing. But that overhead should be almost same for equal amount of write calls. But it is not so. I just have tested with much bigger DUMMY_MESSAGE - it is still 2 t

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Denys Khanzhyiev
this variant is more efficient. function benchmarkWriteStream() { const stream = fs.createWriteStream( './dummyWriteStream.log', { encoding: "utf8", mode: parseInt('0644', 8) } ); const startTime = new Date(); var i = NUM_WRITES; (function doWrite(){ if(i--) {

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Denys Khanzhyiev
Or even simplier (function doWrite(){ if(i--) stream.write(DUMMY_MESSAGE,doWrite); else stream.end(''); })(); 2014-05-20 22:45 GMT+03:00 Denys Khanzhyiev : > this variant is more efficient. > > function benchmarkWriteStream() { > const stream = fs.createWriteStream( > './dummyW

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Jeff Jolma
Hi Aria. I see similar results even after using stream.on('finish'). (I removed the async benchmark altogether.) $ node --harmony benchmarks.js Running benchmarks with 10 writes fs.writeSync took 0.32 seconds writeStream.write took 4.073 seconds $ node --harmony benchmarks.js Running ben

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Forrest Norvell
I am surprised at the performance hit this buffering causes. Should it really take 4.5 seconds to write 100,000 lines and 290 seconds to write 200,000 lines? Node 0.12 will have better support for writev / corking and uncorking, which will allow you to coalesce writes and regain some of the effici

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Jeff Jolma
Hi Denys. Thanks for taking a look. 1. Yes, that is a bug in the async benchmark. Fixing that doesn't change the timings I see, though. 2. Yup, but I am surprised at the performance hit this buffering causes. Should it really take 4.5 seconds to write 100,000 lines and 290 seconds to write 2

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Aria Stewart
On May 20, 02014, at 13:29, Jeff Jolma wrote: > Hello. > > I am working on a node app and found that its performance bottleneck is > logging. And digging deeper, it looks like the root bottleneck is on the > WriteStream. > > I ran some quick benchmarks of the WriteStream performance, and it

Re: [nodejs] Poor WriteStream Performance

2014-05-20 Thread Denys Khanzhyiev
1. Your async code is incorect as you close file before all write operation complete. 2. WriteStream buffers written data until other end is ready to consume data. So you actually first fill memory with data and then write it to file. 2014-05-20 20:29 GMT+03:00 Jeff Jolma : > Hello. > > I am wor

[nodejs] Poor WriteStream Performance

2014-05-20 Thread Jeff Jolma
Hello. I am working on a node app and found that its performance bottleneck is logging. And digging deeper, it looks like the root bottleneck is on the WriteStream. I ran some quick benchmarks of the WriteStream performance, and it seems really slow. Can someone help me understand why it is t