Try this test instead, I bet it gives you the same performance
characteristics:
var a = [];
for (i=0; i<Math.pow(2,power); i++) {
a.push(new Buffer('A'));
}
Which is the same thing you're making Stream do. I bet all the overhead is
in that.
On Tue, Mar 20, 2012 at 4:00 PM, C. Mundi <[email protected]> wrote:
> Yes. I am deliberately abusing the stream with the goal of understanding
> behavior.
>
> I would like to understand why it drains so slowly to disk, on the order
> of a few KB/sec. I would imagine that a stream created specifically with a
> file as its drain would be implemented to dump the biggest chunks it could
> on every write to the filesystem. Hence I wonder whether the low rate is
> really a node stream behavior or a virtual filesystem (buffering)
> behavior. I guess I also need to ask how nodes own streams are backed.
>
> I really want to understand just enough to predict bounds on scaling.
> On Mar 20, 2012 12:30 PM, "Matt" <[email protected]> wrote:
>
>> You're not "streaming" data into the stream. You're pumping it full as
>> quickly as possible and letting it drain in its own time. Obviously this is
>> sub-optimal.
>>
>> On Tue, Mar 20, 2012 at 3:20 PM, C. Mundi <[email protected]> wrote:
>>
>>>
>>> Hi. I am trying to learn how to use streams properly in node.
>>>
>>> The attached script stakes a single argv parameter N and writes 2^N
>>> bytes to a file via a stream.
>>>
>>> I expect time behavior O(2^N). But what I am seeing is scaling faster.
>>>
>>> I collected data like this in bash on Linux 3.0.0.-16 x64:
>>>
>>> for i in {0..20}; do time node simple.js $i; done
>>>
>>> and plotted the results in the attached spreadhseet (OpenOffice format).
>>>
>>> Watching it execute, I see that almost all of the time for N>10 is spent
>>> waiting for the stream to close. It seems to be draining very slowly! The
>>> maximum memory use by node at any time is 180 MB, less than 10% of physical
>>> memory. For N greater than about 10, node takes 100% CPU while closing the
>>> stream. The system is otherwise not under load. The system is not
>>> swapping much if at all.
>>>
>>> The straight lines in the plot are a guide to the eyes -- not a fit!
>>> You can see that the time behavior is superlinear on the log-log plot.
>>>
>>> If I were using a lot of memory, I would expect the GC to kick in and
>>> then all bets are off. But my biggest run (N=20) is only a megabyte.
>>>
>>> Could the pushback be coming from the Linux VFS? The scaling of the
>>> system time makes me wonder.
>>>
>>> Please help me understand why I should not do what I've done and where
>>> to learn about how to use streams efficiently in node.
>>>
>>> Note: The code is a demonstration. I would not typically create files
>>> this way.
>>>
>>> Thanks for cluing the noob (again).
>>>
>>>
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to [email protected]
>>> To unsubscribe from this group, send email to
>>> [email protected]
>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to [email protected]
>> To unsubscribe from this group, send email to
>> [email protected]
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to [email protected]
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
