The external memory is the one the internal heap knows about:
uint64_t Heap::external_memory() const { return external_memory_.total(); }
The following code in wasm-engine.cc:1015 attributes external memory to the
isolate, in the From() call on the second-to-last line.
Is the native_module likely to be shared between isolates here, and long
lived?
Could it be that it is gradually committing more code space, causing later
isolates to get a higher external
memory size?
(does this backquoting work in email for fixed formatting? Probably not).
```
// Use the given shared {NativeModule}, but increase its reference count
by
// allocating a new {Managed<T>} that the {Script} references.
size_t code_size_estimate = native_module->committed_code_space();
size_t memory_estimate =
code_size_estimate +
wasm::WasmCodeManager::EstimateNativeModuleMetaDataSize(module);
DirectHandle<Managed<wasm::NativeModule>> managed_native_module =
Managed<wasm::NativeModule>::From(isolate, memory_estimate,
std::move(native_module));
```
On Tue, Jan 14, 2025 at 12:59 PM Jakob Kummerow <[email protected]>
wrote:
> Erik: Shared GC is still only partially implemented and definitely not
> shipped (or usable), so that document is surely unrelated to whatever is
> going on here. All existing ways to share data between isolates (such as
> the NativeModule cache) use other mechanisms.
>
> Kenton: I can't rule out anything. We admittedly don't have much test
> coverage for thousands-of-isolates scenarios. Perhaps the
> --trace-wasm-offheap-memory flag can help narrow it down a bit. It's
> currently only hooked up with the memory measurement API, so you'll either
> have to use that, or hack some more triggers into convenient places
> (perhaps isolate shutdown or creation?), see occurrences of
> v8_flags.print_wasm_offheap_memory_size for inspiration.
>
> A few more ideas:
> - from what you describe, perhaps it would be feasible to craft a
> reproducer. It'd probably have to be a custom V8 embedder that, in a loop,
> creates many fresh isolates and instantiates/runs the same (or several?)
> demo Wasm module in them.
> - it could make sense to verify (with printfs in their destructors) that
> both Isolates and NativeModules get destroyed as expected. It's
> conceivable that the memory growth you're observing is intentional caching
> (of generated code, or something?) because the WasmEngine thinks that the
> cached data is still needed/useful.
>
> How/where exactly are you seeing this increased "external memory"?
> I.e. what reporting system are you using to get memory consumption numbers?
>
>
> On Tue, Jan 14, 2025 at 1:09 AM Kenton Varda <[email protected]>
> wrote:
>
>> To add context here:
>>
>> The problem appears to show up only after running in production for an
>> hour or two. During that time we will have created thousands of isolates to
>> handle millions of requests.
>>
>> But the problem seems to affect *new* isolates, even when those isolates
>> are loaded with applications that had been loaded into previous isolates
>> without problems. Startup of an application should be 100% deterministic
>> since we disallow any I/O during startup, but we're seeing that after the
>> host has been running a while, new isolates are showing much higher
>> "external memory" on startup. (E.g. 400MB external memory, but we enforce a
>> 128MB limit on the whole isolate.)
>>
>> We observed that the wasm native module cache causes identical wasm
>> modules to be shared across isolates, and that wasm lazy compilation causes
>> memory usage of a wasm module -- as accounted by all isolates that have
>> loaded it -- to change.
>>
>> Could it be that there is a memory leak in lazy compilation, such that
>> these shared cached modules are gradually growing over time, to the point
>> where new isolates that try to load these modules are being hit with
>> extremely high "external memory" numbers right off the bat?
>>
>> -Kenton
>>
>> On Mon, Jan 13, 2025 at 5:31 PM Erik Corry <[email protected]>
>> wrote:
>>
>>> It looks like it's related to shared objects between isolates. Is there
>>> a newer document than
>>> https://docs.google.com/document/d/18lYuaEsDSudzl2TDu-nc-0sVXW7WTGAs14k64GEhnFg/edit?usp=drivesdk
>>> that describes how this works today? In particular cross-isolate GCs?
>>>
>>> On Mon, 13 Jan 2025, 15:25 Jakob Kummerow, <[email protected]>
>>> wrote:
>>>
>>>> Sounds like a bug, but without more details (or a repro) I don't have a
>>>> more specific guess than that.
>>>>
>>>> If you're desperate, you could try to bisect it (even with a flaky
>>>> repro). Or review the ~500 changes between those branches:
>>>> https://chromium.googlesource.com/v8/v8/+log/branch-heads/13.1..branch-heads/13.2?n=10000
>>>>
>>>>
>>>> On Mon, Jan 13, 2025 at 2:48 PM 'Dan Lapid' via v8-dev <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>> In V8 13.2 and 13.3 we see wasm isolates external memory usage blowing
>>>>> up sometimes (up to gigabytes).
>>>>> Under V8 13.1 the same code would never ever use more than 80-100MB
>>>>> The issue doesn't happen every time for the same wasm bytecode. It
>>>>> doesn't even reproduce locally.
>>>>> But some significant percentage of the time it does happen.
>>>>> This has only started happening in 13.2, what are we missing? Should
>>>>> we be enabling/disabling some flags?
>>>>> It also seems that 13.3 is significantly worse in terms of error rate.
>>>>> The problem happens under "--liftoff-only".
>>>>> We use pointer compression but not sandbox.
>>>>> We've tried enabling --turboshaft-wasm in 13.1 and the problem did not
>>>>> reproduce.
>>>>> Has anything changed that we need to adapt to?
>>>>> Would really appreciate your help!
>>>>>
>>>>> --
>>>>>
>>>>
--
--
v8-dev mailing list
[email protected]
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups
"v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/v8-dev/CAHZxHpgiFg4oM1FtVoQZJ1U2qJtKWyKvrdMKJW3Tb6qb7cRfuQ%40mail.gmail.com.
