Re: Memory tracking in GJS
On Wed, Feb 19, 2014 at 8:50 PM, Cosimo Cecchi wrote: Hi all, The state of garbage collection for typical GTK/JS applications is pretty sad these days. It's worth a comparison with (C)Python. In some ways, Python's hybrid refcount + GC is the worst of both worlds - you have the constant overhead (cacheline bouncing, etc.) hit of refcounting, with the unpredictability of GC as soon as you happen to make a reference cycle. Except for the case of objects allocated in local function scope. This case is a very important one for us. Many of my applications do this a *lot* - lots of Gio.File instances for example. And stuff like Cairo surfaces. (C)Python will deterministically clean these up at the end of the function - they don't escape, their refcount drops to 0. It'd be quite nice if the Spidermonkey compiler attempted to do escape analysis and deterministic cleanup when it could. The problem is that for a native GObject wrapped into a JSObject there are effectively two sets of allocations: those internal to GObject and those needed to wrap it into a JSObject. While in the latter case the garbage collector will keep track of those segments, we currently never forward the GObject allocations/payload size to the garbage collector. I don't think we can ever do that in a reliable way. At least with the current design of GObject. Now...the Samba guys are heavily invested in talloc ( http://talloc.samba.org/talloc/doc/html/index.html ) which gives this kind of information for effectively free ( talloc_get_size () ). I've thought sometimes about having an optional talloc-like API for GLib, but it would be really invasive. The result is that, especially with large objects (e.g. pixbufs, cairo surfaces, ...) there's a huge difference between what mozjs thinks is the memory situation and the reality, and an application can effectively run out of system memory before the garbage collector kicks in. Firefox gets away with it because JS code only needs to allocate memory outside of the garbage collector very rarely, and those cases are handled internally by manually updating the counter. It depends on the JS code, but certainly a lot of real-world web pages do heavy DOM manipulation which does involve lots of nontrivially-sized native objects. This is the case that every web browser is optimized for, because they have to. Some ideas that have been floating around: * have special methods on "interesting" objects to explicitly release memory. gnome-shell for instance does this already when using Cairo from within JS, for exactly the same reason. I think there's no question we should allow this. It's a bit ugly, but * override the GLib memory allocator with e.g. g_mem_set_vtable() to intercept allocations and signal the garbage collector. Problems: doesn't work for things that use e.g. static constructors, is tricky for re-entrancy, libraries like GdkPixbuf might not use the GLib allocator for image payloads, doesn't work with the slice allocator. I'd really rather investigate something like talloc-for-glib if we were to try going down this route. It would have the potential to improve performance of pure C apps as well. * keep track of memory allocated in GJS itself. We can use GType to query the size of the instance, class and even the size of private data. This still doesn't include additional allocations inside the object itself, such as strings, buffers and payloads. We could track those case-by-case by e.g. overriding constructors in the binding for "interesting" pixbuf-like types (ugh), or have a way to query that information from the type system/introspection itself. I just don't think this one is going to scale. Any other ideas? Beyond the above, I think ultimately this needs help from the kernel. That's where I was going with https://git.gnome.org/browse/gjs/commit/?id=7aae32d1df14af87e9a31c785447b27012b64af9 Here the kernel is keeping track of our total memory usage, both JS and C. I feel this is a fundamental enough issue that there must be a good way to design a generic solution. It all gets a lot more interesting when you consider *multiple* garbage collected processes on the system. I think this is still an open research area. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Memory tracking in GJS
hi, On Thu, Feb 20, 2014, at 7:53, Giovanni Campagna wrote: > It's worth pointing out that mozjs will *force* a garbage collect if > you go over the malloc limit (a dynamic value between 30 and 90 MB), > but will garbage collect a lot more often if you call MaybeGC. My preferred solution to the problem involves doing this, perhaps from a very low priority idle, once the mainloop has stopped spinning. I think any proposal that involves a sort of 'information API' where 'well behaved' classes are supposed to provide memory usage information is probably going to be very burdensome. Just to explore an example, a bit: let's say that we really strictly wanted to limit our scope and apply this logic *only* to pixbufs, because we identified those to be a problematic area. We now either need to do one of two things: - keep track of all pixbuf allocations in the program so that gjs can query a list and use this information to decide to do more GC - wire this interface through to the public API of any object that holds a pixbuf within itself, and through to the APIs of objects that hold those objects (etc.) in order to make this information visible on the actual objects held by gjs I consider the first alternative to be excessively evil. It's also unlikely to be able to give you the information you really need -- sure, we have 100MB of pixbufs allocated, but they might be in places that GC wouldn't help you. The second alternative is just entirely too much work to expect of every class that might stand between a pixbuf and a public API -- and we've only limited our scope to pixbufs already. It's also problematic from a (perhaps pedantic) accounting standpoint: if two objects reference a shared pixbuf (or intermediate object) then who gets to claim the memory use as their own? The other allocation-based alternatives sound like possibilities, but perhaps are very much too brittle. For what it's worth, I'm happy to try to find a way to make disabling the slice allocator and adding a memory vtable possible again... Is Giovanni's proposal workable? Cheers ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Memory tracking in GJS
On Thu, Feb 20, 2014 at 7:53 AM, Giovanni Campagna wrote: We can probably look into mallinfo() as well, Been there, done that: https://git.gnome.org/browse/gjs/commit/?id=7aae32d1df14af87e9a31c785447b27012b64af9 ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Memory tracking in GJS
2014-02-20 2:50 GMT+01:00 Cosimo Cecchi : > Hi all, > > The state of garbage collection for typical GTK/JS applications is pretty > sad these days. mozjs will execute a garbage collection run when it thinks > around 30MB of allocations have been done. It's worth pointing out that mozjs will *force* a garbage collect if you go over the malloc limit (a dynamic value between 30 and 90 MB), but will garbage collect a lot more often if you call MaybeGC. Firefox does that at the end of the every user script (with a comment saying it's for sunspider), maybe we can do the same in gjs? Additionally, gjs does not support the more advanced forms of GC, so they trigger less often and block longer. See #724797 for incremental GC for example. > Any other ideas? I feel this is a fundamental enough issue that there must > be a good way to design a generic solution. We can probably look into mallinfo() as well, it offers a reasonable estimate of the memory usage for a single threaded application (because most allocations happen from the main arena). Giovanni ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: Memory tracking in GJS
Hey Cosimo, Nice to see someone looking at this issue. On 20/02/14 01:50, Cosimo Cecchi wrote: * keep track of memory allocated in GJS itself. We can use GType to query the size of the instance, class and even the size of private data. This still doesn't include additional allocations inside the object itself, such as strings, buffers and payloads. We could track those case-by-case by e.g. overriding constructors in the binding for "interesting" pixbuf-like types (ugh), or have a way to query that information from the type system/introspection itself. Adding on this, I have to mention my experience with OpenGL allocated textures. These often end up being completely outside of the process' address space (so completely invisible) and only represented by a very thin object in GObject/Gjs. - Lionel ___ gtk-devel-list mailing list gtk-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/gtk-devel-list
