[v8-users] Re: GC cost of adding to an object with 50,000+ keys

[Jonathan Otto]

I switched to a Map which looks like it may have solved the issue.

See this: https://jsperf.com/es6-map-vs-object-properties/

And, running Node.js --perf with the code from that jsperf yields the 
following (on Node v8.9.1 which, according to process.versions, is V8 
6.1.534.47):

*Map (15.7% in GC)*
 [Summary]:
   ticks  total  nonlib   name
903   68.1%   68.4%  JavaScript
395   29.8%   29.9%  C++
208   15.7%   15.7%  GC
  50.4%  Shared libraries
 231.7%  Unaccounted


 [C++]:
   ticks  total  nonlib   name
 916.9%6.9%  t v8::internal::Scavenger::ScavengeObject(v8::
internal::HeapObject**, v8::internal::HeapObject*)
 382.9%2.9%  t v8::internal::OrderedHashTable::Rehash(v8::internal::Handle, int)
 362.7%2.7%  T v8::internal::Runtime_GenerateRandomNumbers(int, 
v8::internal::Object**, v8::internal::Isolate*)
 362.7%2.7%  T v8::internal::IncrementalMarking::Step(unsigned 
long, v8::internal::IncrementalMarking::CompletionAction, v8::internal::
IncrementalMarking::ForceCompletionAction, v8::internal::StepOrigin)
 302.3%2.3%  T v8::internal::Heap::Scavenge()
 251.9%1.9%  T v8::internal::Heap::AllocateFixedArrayWithFiller(
int, v8::internal::PretenureFlag, v8::internal::Object*)
 241.8%1.8%  t node::(anonymous namespace)::ContextifyScript::
New(v8::FunctionCallbackInfo const&)
 141.1%1.1%  t v8::internal::(anonymous namespace)::
GetSimpleHash(v8::internal::Object*)


*Object (26.8% in GC)*
 [Summary]:
   ticks  total  nonlib   name
698   32.2%   32.2%  JavaScript
   1417   65.3%   65.4%  C++
581   26.8%   26.8%  GC
  20.1%  Shared libraries
 532.4%  Unaccounted


 [C++]:
   ticks  total  nonlib   name
269   12.4%   12.4%  T v8::internal::IncrementalMarking::Step(unsigned 
long, v8::internal::IncrementalMarking::CompletionAction, v8::internal::
IncrementalMarking::ForceCompletionAction, v8::internal::StepOrigin)
1486.8%6.8%  t v8::internal::HashTable::Rehash(v8::internal::
NameDictionary*)
1225.6%5.6%  T v8::internal::Heap::Scavenge()
1074.9%4.9%  t v8::internal::Scavenger::ScavengeObject(v8::
internal::HeapObject**, v8::internal::HeapObject*)
1034.7%4.8%  T v8::internal::HashTable::Rehash(v8::internal::StringTable*)
 914.2%4.2%  T v8::internal::StringTable::LookupKey(v8::internal
::Isolate*, v8::internal::StringTableKey*)
 823.8%3.8%  t v8::internal::LookupIterator::State v8::internal
::LookupIterator::LookupInRegularHolder(v8::internal::Map*, v8::
internal::JSReceiver*)
 482.2%2.2%  T v8::internal::Heap::AllocateFixedArrayWithFiller(
int, v8::internal::PretenureFlag, v8::internal::Object*)
 411.9%1.9%  T v8::internal::String::SlowEquals(v8::internal::
String*)


If I plot out the actual duration of the add/update operation to either a 
Map or Object, they're both almost equivalent as N increases, but the 
memory usage and GC activity *around* that add/update for the Object is 
much higher. In fact, depending on the length of the random string being 
created, the Object often maxes out the heap and crashes Node.js. (I 
experimented with increasing --max_old_space_size up to 1) There's also 
a point (random strings of length 5, with N of 8,000,000) where even if 
your heap is large enough to hold all the strings, the adding/appending to 
an Object just seizes up (and Map still works).

On Thursday, November 23, 2017 at 12:59:06 AM UTC-8, Camillo Bruni wrote:
>
> Hi Jonathan, 
>
> this seems suspicious. Would it be possible for you to write a standalone 
> repro?
> Without further information it is hard to asses what exactly might be 
> going wrong here.
>
> Cheers,
> Camillo
>
> On Wednesday, November 22, 2017 at 1:29:50 AM UTC+1, Jonathan Otto wrote:
>>
>> I have a plain JavaScript object that looks like:
>>
>> {
>>   sha256a: { id, fieldA, fieldB, fieldC },
>>   sha256b: { id, fieldA, fieldB, fieldC },
>>   ...
>> }
>>
>> and it seems like when I get over ~50,000 sha's then I start to have high 
>> CPU usage (~3%) when adding a key or changing a value for an existing key - 
>> note I am adding/changing keys once or twice per second. Naively I'm 
>> wondering if the GC is traversing the entire object on each addition or 
>> change and whether there's a cliff where that falls off.
>>
>> Is there a better way to manage a local store in JavaScript?
>>
>

-- 
-- 
v8-users mailing list
v8-users@googlegroups.com
http://groups.google.com/group/v8-users
--- 
You received this message because you are subscribed to the Google Groups 
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [v8-users] How to build a fixed version of V8 and its dependencies.

[Jakob Kummerow]

The depot_tools system makes this easy, actually:

git checkout 
gclient sync
ninja -C out/ d8

(If you go back far enough before ninja was supported, replace the last
command with the equivalent make/GYP based build.)


On Mon, Nov 27, 2017 at 9:23 AM Edd Barrett  wrote:

> Hi Mathias,
>
> Thanks for your swift response.
>
> On Monday, November 27, 2017 at 1:40:05 PM UTC, Mathias Bynens wrote:
>>
>> This is not a direct answer to your question, but we already host
>> prebuilt `d8` binaries for Linux on 32-bit and 64-bit architectures. We
>> plan on doing the same for other operating systems:
>> https://bugs.chromium.org/p/v8/issues/detail?id=5918 By using a
>> precompiled binary, you could bypass compiling from source together.
>>
>
>
> That's good to know, but for our purposes we really do need to build from
> source:
>
>  * We have patches to grant access to the monotonic clock and performance
> counters.
>  * The experiment should be fully repeatable from source.
>
> I think it's something the V8 team should consider for the future.
>
> Cheers
>
> --
> --
> v8-users mailing list
> v8-users@googlegroups.com
> http://groups.google.com/group/v8-users
> ---
> You received this message because you are subscribed to the Google Groups
> "v8-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to v8-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- 
v8-users mailing list
v8-users@googlegroups.com
http://groups.google.com/group/v8-users
--- 
You received this message because you are subscribed to the Google Groups 
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [v8-users] How to build a fixed version of V8 and its dependencies.

[Edd Barrett]

Hi Mathias,

Thanks for your swift response.

On Monday, November 27, 2017 at 1:40:05 PM UTC, Mathias Bynens wrote:
>
> This is not a direct answer to your question, but we already host prebuilt 
> `d8` binaries for Linux on 32-bit and 64-bit architectures. We plan on 
> doing the same for other operating systems: 
> https://bugs.chromium.org/p/v8/issues/detail?id=5918 By using a 
> precompiled binary, you could bypass compiling from source together.
>

 
That's good to know, but for our purposes we really do need to build from 
source:

 * We have patches to grant access to the monotonic clock and performance 
counters.
 * The experiment should be fully repeatable from source.

I think it's something the V8 team should consider for the future.

Cheers

-- 
-- 
v8-users mailing list
v8-users@googlegroups.com
http://groups.google.com/group/v8-users
--- 
You received this message because you are subscribed to the Google Groups 
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [v8-users] How to build a fixed version of V8 and its dependencies.

['Mathias Bynens' via v8-users]

This is not a direct answer to your question, but we already host prebuilt
`d8` binaries for Linux on 32-bit and 64-bit architectures. We plan on
doing the same for other operating systems:
https://bugs.chromium.org/p/v8/issues/detail?id=5918 By using a precompiled
binary, you could bypass compiling from source together.

On Mon, Nov 27, 2017 at 12:19 PM, Edd Barrett  wrote:

> Hi,
>
> I raised this as a bug, but was asked to re-post here.
>
> For a benchmarking experiment I need to build a frozen version of V8. For
> most VMs this means just checking out a specific git version of the code
> and building it. For V8 it is complicated by the depot_tools system. I've
> yet to find a reliable way to do this.
>
> In the past, I've gotten by with cloning a specific version of depot
> tools, fetching V8, then walking the dependencies resetting the git
> versions of everything to a fixed date (this hack is needed because `fetch`
> gets moving targets). When this stopped working (it was a hack anyway), I
> resorted to rolling my own tarballs, which is not really ideal either.
>
> Is there an official way to build an frozen version of V8? How do you guys
> build (e.g.) V8-5.8.283.32 and using the dependencies of the time of that
> release?
>
> Without the ability to do this, it's difficult to conduct repeatable
> benchmarking experiments on V8.
>
> Thanks
>
> --
> --
> v8-users mailing list
> v8-users@googlegroups.com
> http://groups.google.com/group/v8-users
> ---
> You received this message because you are subscribed to the Google Groups
> "v8-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to v8-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

Mathias Bynens |  V8 |  Google Germany GmbH |  Erika-Mann-Str. 33, 80636
München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten
haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter,
löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen,
dass die E-Mail an die falsche Person gesendet wurde.



This e-mail is confidential. If you received this communication by mistake,
please don’t forward it to anyone else, please erase all copies and
attachments, and please let me know that it has gone to the wrong person.

-- 
-- 
v8-users mailing list
v8-users@googlegroups.com
http://groups.google.com/group/v8-users
--- 
You received this message because you are subscribed to the Google Groups 
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[v8-users] How to build a fixed version of V8 and its dependencies.

[Edd Barrett]

Hi,

I raised this as a bug, but was asked to re-post here.

For a benchmarking experiment I need to build a frozen version of V8. For 
most VMs this means just checking out a specific git version of the code 
and building it. For V8 it is complicated by the depot_tools system. I've 
yet to find a reliable way to do this.

In the past, I've gotten by with cloning a specific version of depot tools, 
fetching V8, then walking the dependencies resetting the git versions of 
everything to a fixed date (this hack is needed because `fetch` gets moving 
targets). When this stopped working (it was a hack anyway), I resorted to 
rolling my own tarballs, which is not really ideal either.

Is there an official way to build an frozen version of V8? How do you guys 
build (e.g.) V8-5.8.283.32 and using the dependencies of the time of that 
release?

Without the ability to do this, it's difficult to conduct repeatable 
benchmarking experiments on V8.

Thanks

-- 
-- 
v8-users mailing list
v8-users@googlegroups.com
http://groups.google.com/group/v8-users
--- 
You received this message because you are subscribed to the Google Groups 
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.