Comment #44 on issue 847 by avi...@gmail.com: 1gb memory limit
https://code.google.com/p/v8/issues/detail?id=847
I struggled with v8 memory limits for a long time. While in some
applications I was easily using over 8gb without any issues, in others I
couldn't use more than 2, so the answer is kind of complicated.
The 2 limits I was running into I think were on a limit on the size of all
the object keys, as well as a limit on the size of the "new" space.
For example, I was creating an in memory database, and wanted a way to
index values so that retrieving all the rows where a property "age" is 20
and height is "6" could be done very fast. Essentially, I was trying to
find the intersection of 2 lists of rows. To do that I created two
objects, where each rowNumber just mapped to true. Then I would iterate
over the smaller of the objects, and see if each rownum was a defined
property on the larger object. But when the database grew, I got out of
memory errors, and I think I pinned it on the number of combined keys in
all those maps simply growing too large. I changed the design, so that
instead of objects for the rows I used sorted arrays, and finding the
intersection was just a matter of a number of binary searches (instead of
object key lookups). It was still very fast, and the memory limit
disappeared. The exact out of memory error I was getting mentioned
something about strings which led me in the right direction.
In another app, I was creating large arrays, a few hundred thousand
elements, passing them around, de-referencing them, combining them into new
arrays, etc. and my app would crash at different points, claiming it was
out of memory. It seemed I was just creating and destroying arrays too
quickly, and the "new space" was overfilling. So, when I had to make a new
array out of two existing arrays, instead of dereferencing the old arrays
and claiming memory to create a new one, I instead created a "combined
array" class. The combined array class would have a list of arrays and
when you iterated over the combined array you iterated over all the arrays
in its list. This way I wasn't constantly dereferencing memory, and
claiming new memory. As soon as I did that my problem disappeared.
I also had lots of large arrays in my app. There were lets say a million
objects, and then there were all these classes that would be interested in
a few hundred thousand of them, and they would store them in their own
array. I don't think this was what caused my issues, but back then I
didn't know what did, so I made each class, instead of storing an array of
the objects themselves, they would just store numbers that were references
to these objects (Another class was in charge of resolving the references
to the actual objects). And since I was just storing a list of numbers, I
could put them into a typed array, who's storage is stored outside the V8
heap.
Anyways, I hope that makes sense. It took me weeks to track down these
problems. This was all in node 10.32 (Older versions weren't any
better). ;(.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
--
--
v8-dev mailing list
v8-dev@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to v8-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
