On Fri, 5 Jun 2009 02:18:53 -0700 (PDT) Robert Lehr <robert.l...@gmail.com> wrote: > > The problem was that it was not as fast as I expected it should be > given that it was using no less than 100% of the CPU on my system. > (two 3GHz Xeon CPUs [1 core]; 3GB RAM; a beefy workstation). That > this was visible in the GUI shows how slow it appeared to me. Also, > it was using 700MB RAM (VIRT in top). Sure - it was "swapped" (I'm > familiar w/ some of the interpretations of these memory issues) except > that my system has ZERO swap space. PMAP showed that 400MB of it was > heap, too, not libraries or binaries or anything else that we can > safely ignore. This was (apparently) real, allocated heap, private to > the process's address space. > > Additionally, as the simulation ran, the initial RSS of 60MB rose to > 130MB then stopped. The VIRT remained constant. I had expected that > - however I remained concerned. >
Just wanted to add some personal notes about the VMM behavior of Linux in particular that I've observed while tinkering with JVM and PyPy's garbage collectors and allocators. Regarding the VIRT and SWAP numbers, it should be noted that, on Linux, VIRT is a rough estimate of the amount of virtual *address space* that the process has actually allocated. Not all of this address space necessarily has corresponding storage on RAM or swap. The reason revolves mostly around mmap(). Large allocations (of the sort that GC arenas and pool allocators typically make) are usually serviced by mmap(), which is very lazy in Linux. As it has been described to me, what this means is that in the kernel there is a reserved "dummy" physical page that's filled with zeros that all pages in the newly allocated virtual address space are mapped to. When the application actually faults in the page, only then is it actually allocated in physical memory by the kernel. The kernel also "resets" a page into this lazy state (deallocating its physical memory) when a process copies /dev/zero into it. Many applications incorporate this knowledge so they can take advantage of this lazy allocation behavior when running on Linux (several of PyPy's allocators definitely do). I would be extremely surprised if the JVM didn't do this as well, because it allows a process to "cheat" on RAM usage by allocating a big arena of memory up front without actually consuming that amount of RAM. Then, the app just uses the memory as needed and lets the kernel/VMM worry about allocation. The side effect of this is that stuff like top and ps report these allocated-but-unbacked pages as "swapped" even though they aren't actually on disk anywhere. When mmap() is used for loading real files off the filesystem (also a lazy operation on Linux), I believe pages of the file that have not been faulted into RAM are also reported as swapped. Executable images are AFAIK loaded by the kernel with mmap(), and large images like the JVM almost certainly have many pages that may never be faulted into RAM when running some kinds of apps. It would come as no surprise to me if the JVM does some mmap()ing of portions of its own runtime or other things that may also never get referenced. As an aside, RSS (resident set size) is an estimate of the size of all pages available to the process that are currently resident in RAM. This area I am a little less clear on, but I *believe* this number is also a bit inflated because it includes pages that are shared behind the scenes from mmap()ed files. What top reports as "shared" (I *think*) is actually restricted to application-requested POSIX shared memory and doesn't count mmap()'s copy-on-write, sharing-as-an-optimization shared memory. Sorry for the length, but I hope this may give some insight--hopefully I'm not telling you things you already know :) -Kyle --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
