Re: What's the memory footprint of a set of processes?
Thus spake Matthew Dillon [EMAIL PROTECTED]: Generally speaking I don't think you have to go into this level of detail to figure out approximate real memory use. Just look at the output of the /proc/PID/map and pull out the 'default' or 'swap' lines. Ignore the 'vnode' lines. ... Now you have two metrics. You can calculate the size based on the range (the first two arguments). (addressB - addressA), and you can get the total number of resident pages from the third argument. What you can't do easily is figure out the total allocation because that is actually going to be resident pages + swapped pages. swapped pages is not included in the proc output. I think the original poster wanted to know the real memory use of a set of processes, taking sharing into account. I don't see how your approach could do that. Even if you knew the structure of the shadow chain, you would have to know specifically which pages had been COWed, no? I thought counting vm_page structures would be the way to go... I really want to find the time to learn all this stuff better. I still don't know the difference between COW and NEEDS_COPY, or why the pageout daemon seems to be biased against shared pages, or a lot of things about pv_entry structures. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Matthew Dillon [EMAIL PROTECTED] writes: You can also theoretically push into shadow VM objects to locate pages from the parent process that have not yet been COW'd into the child (in the case of a fork()), noting also that these shadow objects might be shared with other children of the parent and by the parent process itself, but under most conditions this information will not be significant and can be ignored. Actually, that's just the sort of thing I'm looking for. The shared case may be relatively rare, but it tends to be extreme: the processes that use the most memory seem to be the ones that fork the most - database servers, java, etc. The point of this whole thing is an attempt to limit the memory use of a user (instead of a process), but I don't want to penalize such sharing. Any vnode object is always shared with other processes mapping the same vnode. Since this information is backed by a file, figuring out how much 'memory' it represents by any reasonable definition is guesswork. The resident page count will represent how much of the vnode is cached, but not how much of the vnode is actually being accessed by the process. That's OK - resident count is what I'm interested in. That, and the swap approximation (which should suffice) you mentioned for non-vnode objects. - Jamie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
:I think the original poster wanted to know the real memory use of :a set of processes, taking sharing into account. I don't see how :your approach could do that. Even if you knew the structure of :the shadow chain, you would have to know specifically which pages :had been COWed, no? : :I thought counting vm_page structures would be the way to go... :I really want to find the time to learn all this stuff better. :I still don't know the difference between COW and NEEDS_COPY, :or why the pageout daemon seems to be biased against shared :pages, or a lot of things about pv_entry structures. It's not possible to get a wholely accurate count no matter what you do. For example, to truely know whether a process is using a page you have to run through the process's page table (PMAP), access the vm_page, then locate where in the shadow chain the VM object the vm_page belongs to resides. But since hardware page tables are throw-away, the system could very well have thrown away whole page tables so this method is no more accurate then any other. Shadow VM objects are optimized on the fly. When a shadow object representing data shared by multiple processes is completely covered (due to COW faults) the system will delete the shadow object. So you can get a pretty good idea in regards to shared pages simply by noting the existance of the shadow object and the number of resident or swap pages residing in that object. MAP_ENTRY_NEEDS_COPY is a vm_map_entry optimization that allows two vm_map_entry's (for two different processes) to share the same vm_object even though they are copy-on-write. This allows us to defer allocating new VM objects (defer creating the shadow structures) for anonymous area of memory when a process fork()s, until an actual copy-on-write occurs. Being able to defer this allocation greatly reduces the time and memory required to fork() and gives us a flatter, more easily optimized VM object structure. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
I just had another idea on how you could do this. It's actually not all that complex. All you do is collect statistics on each VM object individually (whether shadow or top level or whatever). e.g. Resident page count, swap usage. Then you collect a list of VM objects associated with a process, regardless of HOW they are associated. Recurse through the vm_map_entry's and the shadow chains to get the list. Then you simply group all the processes which share VM objects together and report statistics on a group-by-group basis rather then on a process-by-process basis. You won't know what an individual process uses but you know exactly what the group of processes use in aggregate. Groups of processes will tend to form due to fork()ing, processes sharing memory via SysV shared memory, and processes that happen to be mapping the same file (vnode). You will get accurate memory use for each group. -Matt : Actually, that's just the sort of thing I'm looking for. The shared case :may be relatively rare, but it tends to be extreme: the processes that use :the most memory seem to be the ones that fork the most - database servers, :java, etc. The point of this whole thing is an attempt to limit the memory :use of a user (instead of a process), but I don't want to penalize such :sharing. : :.. : That's OK - resident count is what I'm interested in. That, and the swap :approximation (which should suffice) you mentioned for non-vnode objects. : :- Jamie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Matthew Dillon [EMAIL PROTECTED] writes: Then you simply group all the processes which share VM objects together and report statistics on a group-by-group basis rather then on a process-by-process basis. You won't know what an individual process uses but you know exactly what the group of processes use in aggregate. Yes, that sounds good. I've got a jail-like setup, so the process groups I'm interested in should be nearly identical to those sharing objects. I don't need to know individual process usage, so this seems to be just what I'm looking for. Now on to some actual coding :-). - Jamie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Thus spake Matthew Dillon [EMAIL PROTECTED]: It's not possible to get a wholely accurate count no matter what you do. For example, to truely know whether a process is using a page you have to run through the process's page table (PMAP), access the vm_page, then locate where in the shadow chain the VM object the vm_page belongs to resides. But since hardware page tables are throw-away, the system could very well have thrown away whole page tables so this method is no more accurate then any other. Thanks for the explanations! I still don't understand why this doesn't work, assuming you don't care about nonresident pages: for each process p in the set for each map entry e in p-vmspace-vm_map for each page m in e-object.vm_object-memq if I haven't seen this m.phys_addr yet in the scan resident_pages++ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
: :Thus spake Matthew Dillon [EMAIL PROTECTED]: : It's not possible to get a wholely accurate count no matter what : you do. For example, to truely know whether a process is using : a page you have to run through the process's page table (PMAP), : access the vm_page, then locate where in the shadow chain the VM object : the vm_page belongs to resides. But since hardware page tables are : throw-away, the system could very well have thrown away whole page : tables so this method is no more accurate then any other. : :Thanks for the explanations! I still don't understand why this :doesn't work, assuming you don't care about nonresident pages: : :for each process p in the set : for each map entry e in p-vmspace-vm_map : for each page m in e-object.vm_object-memq : if I haven't seen this m.phys_addr yet in the scan : resident_pages++ That would get close, as long as the machine is not paging heavily. Think of it this way: If you have a lot of ram the above calculation will give you an upper bound on memory use, but some of the pages in the VM object's may be very old and not actually under active access by the process (for example, the pages might represent part of the program that was used during initialization and then never used again). If you do not have so much memory older pages will get flushed out or flushed to swap and the above calculation will represent more of a lower bound on the memory used by the group of processes. Consider a database. A database might be mapping a large table file and, if you have lots of memory, most of that table file might be cached (in the VM object's memq), but that doesn't mean the database has necessarily needed all that data recently. The database might run just as well with less memory available. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Thus spake Matthew Dillon [EMAIL PROTECTED]: :Thanks for the explanations! I still don't understand why this :doesn't work, assuming you don't care about nonresident pages: : :for each process p in the set : for each map entry e in p-vmspace-vm_map : for each page m in e-object.vm_object-memq : if I haven't seen this m.phys_addr yet in the scan : resident_pages++ That would get close, as long as the machine is not paging heavily. Think of it this way: If you have a lot of ram the above calculation will give you an upper bound on memory use, but some of the pages in the VM object's may be very old and not actually under active access by the process (for example, the pages might represent part of the program that was used during initialization and then never used again). If you do not have so much memory older pages will get flushed out or flushed to swap and the above calculation will represent more of a lower bound on the memory used by the group of processes. Yes, I understand this; that's why I said ``assuming you don't care about nonresident pages'' (a pretty big assumption, mind you.) I was just thinking about essentially calculating the physical memory usage for a set of processes, taking sharing into account, and I take it you were talking about calculating the total amount mapped. I imagine both metrics would be useful. For instance, a database might map a huge file but have a very small resident set. I don't know what the OP intended... To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
What's the memory footprint of a set of processes?
How do I find how much memory (real and/or virtual) is being used by a set of processes, taking shared pages into account? I see per-process numbers I can use (vmspace_resident_count and vmspace_swap_count), and overall usage numbers exist, but I can't find a better way of measuring multiple processes than adding their individual totals together. This can lead to wild inaccuracies if a few processes share a lot of pages (Java comes to mind, as do database servers). Is there some kind of reference count I can access per page, or some not-too-expensive way to see what processes are using a page? I see some things in the VM code that look like recerence counts (such as act_count in struct vm_page), but they don't seem to really be such, or at least they don't count what I'm expecting them to. This is on 4.7. I haven't really looked at 5, so I don't know how different it is. - James Gritton [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
check out /proc/PID/map for a really detailed map of the process. On Wed, 29 Jan 2003, James Gritton wrote: How do I find how much memory (real and/or virtual) is being used by a set of processes, taking shared pages into account? I see per-process numbers I can use (vmspace_resident_count and vmspace_swap_count), and overall usage numbers exist, but I can't find a better way of measuring multiple processes than adding their individual totals together. This can lead to wild inaccuracies if a few processes share a lot of pages (Java comes to mind, as do database servers). Is there some kind of reference count I can access per page, or some not-too-expensive way to see what processes are using a page? I see some things in the VM code that look like recerence counts (such as act_count in struct vm_page), but they don't seem to really be such, or at least they don't count what I'm expecting them to. This is on 4.7. I haven't really looked at 5, so I don't know how different it is. - James Gritton [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Julian Elischer [EMAIL PROTECTED] writes: check out /proc/PID/map for a really detailed map of the process. That looks good for a single process, suffers from the problem I'm having. For example, if I run a program that simply mallocs a chumk of memory and reads through it (to map it all in), it's pretty easy to identify the affected object: guppy% cat /proc/23072/map 0x8048000 0x8049000 1 0 0xcdd610c0 r-x 1 0 0x0 COW NC vnode 0x8049000 0x804a000 1 0 0xce619060 rw- 2 0 0x2180 NCOW NNC default 0x804a000 0xc04b000 16385 0 0xce619060 rwx 2 0 0x2180 NCOW NNC default ... But after sleeping for a bit, the program then forks, and I have two processes sharing the same memory. I hope to see something that reflects this, but: guppy% cat /proc/23072/map 0x8048000 0x8049000 1 0 0xcdd610c0 r-x 2 0 0x0 COW NC vnode 0x8049000 0x804a000 1 0 0xcdc84360 rw- 2 0 0x180 COW NC default 0x804a000 0xc04b000 16385 0 0xce619060 rwx 2 0 0x180 COW NC default ... guppy% cat /proc/23074/map 0x8048000 0x8049000 1 0 0xcdd610c0 r-x 2 0 0x0 COW NC vnode 0x8049000 0x804a000 1 0 0xcdc84360 rw- 2 0 0x180 COW NC default 0x804a000 0xc04b000 16385 0 0xce619060 rwx 2 0 0x180 COW NC default ... The object's ref_count hasn't changed, which is what I meant about seeing reference counts in the kernel that were apparently not counting what I'm looking for. I did see a ref_count increase on the first object (presumably the text image), but nothing on the allocated memory. It seems the object level isn't fine enough, but the deeper I go into the VM code, the more confused I become. In this forked process example, what happens when I alter a few COW pages in the currently-shared object? Apparently a shadow object is created, but it claims to be the same size as the original object. True, but I know it's not actually using that many pages, since most of them are still validly shared. System usage numbers tell me this is true, but I can't find what in the process or object data structures reflect this fact. - Jamie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
Thus spake James Gritton [EMAIL PROTECTED]: The object's ref_count hasn't changed, which is what I meant about seeing reference counts in the kernel that were apparently not counting what I'm looking for. I did see a ref_count increase on the first object (presumably the text image), but nothing on the allocated memory. It seems the object level isn't fine enough, but the deeper I go into the VM code, the more confused I become. In this forked process example, what happens when I alter a few COW pages in the currently-shared object? Apparently a shadow object is created, but it claims to be the same size as the original object. True, but I know it's not actually using that many pages, since most of them are still validly shared. System usage numbers tell me this is true, but I can't find what in the process or object data structures reflect this fact. No, you don't have enough information. Even if you knew which objects shadowed which, I still don't think you would have enough information. You want to account for physical pages, so you should be looking at vm_page structures. AFAIK, there isn't an interface to do that, but one shouldn't be too hard to implement. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: What's the memory footprint of a set of processes?
:Thus spake James Gritton [EMAIL PROTECTED]: : The object's ref_count hasn't changed, which is what I meant about seeing : reference counts in the kernel that were apparently not counting what I'm : looking for. I did see a ref_count increase on the first object : (presumably the text image), but nothing on the allocated memory. : :It seems the object level isn't fine enough, but the deeper I go into the : VM code, the more confused I become. In this forked process example, what : happens when I alter a few COW pages in the currently-shared object? : Apparently a shadow object is created, but it claims to be the same size as : the original object. True, but I know it's not actually using that many : pages, since most of them are still validly shared. System usage numbers : tell me this is true, but I can't find what in the process or object data : structures reflect this fact. : :No, you don't have enough information. Even if you knew which :objects shadowed which, I still don't think you would have enough :information. You want to account for physical pages, so you :should be looking at vm_page structures. AFAIK, there isn't an :interface to do that, but one shouldn't be too hard to implement. Well, first he should read my DaemonNews article. http://www.daemonnews.org/21/freebsd_vm.html Now, in regards to COW faults and shadow pages, it basically comes down to the process's VM map (struct vm_map) which contains a list of vm_map_entry structures which tell the VM system what VM address ranges correspond to what VM objects. For all intents and purposes, the size of a VM object (struct vm_object) which represents anonymous memory is irrelevant. What is relevant is the size of the 'window' into the VM object defined by the vm_map_entry. You can get a list of vm_map_entry's associated with a process using the /proc filesystem. e.g. dd if=/proc/PID/map bs=256k dd if=/proc/86006/map bs=256k Shadow objects are basically just a way for the VM system to be able to mix pages that are COW faulted and pages that have not yet been COW faulted together without having to create independant little VM objects for each faulted page. The shadowing is basically just layering two or more VM objects on top of each other. If the top VM object doesn't have the page the system recurses into deeper VM objects to find the page. If the access is a read the deeper page is simply used straight out. If the access is a write the deeper page is COW copied into the top level VM object. Generally speaking I don't think you have to go into this level of detail to figure out approximate real memory use. Just look at the output of the /proc/PID/map and pull out the 'default' or 'swap' lines. Ignore the 'vnode' lines. apollo:/home/dillon# dd if=/proc/85965/map bs=256k 0x8048000 0x805b000 14 15 0xd2482600 r-x 2 1 0x0 COW NC vnode 0x805b000 0x805c000 1 0 0xd2830960 rw- 1 0 0x2180 COW NNC vnode 0x805c000 0x8061000 5 0 0xd29ea360 rw- 2 0 0x2180 NCOW NNC default 0x8061000 0x8156000 237 0 0xd29ea360 rwx 2 0 0x2180 NCOW NNC default 0x2805b000 0x2806d000 17 0 0xc031df60 r-x 77 34 0x4 COW NC vnode 0x2806d000 0x2806e000 1 0 0xd2c12d80 rw- 1 0 0x2180 COW NNC vnode 0x2806e000 0x2807 2 0 0xd2c84c60 rw- 2 0 0x2180 NCOW NNC default 0x2807 0x28078000 6 0 0xd2c84c60 rwx 2 0 0x2180 NCOW NNC default 0x28078000 0x280f8000 90 0 0xc031e2c0 r-x 100 58 0x4 COW NC vnode 0x280f8000 0x280f9000 1 0 0xd2d5ad20 r-x 1 0 0x2180 COW NNC vnode 0x280f9000 0x280fe000 5 0 0xd34e2f00 rwx 1 0 0x2180 COW NNC vnode 0x280fe000 0x28112000 7 0 0xd34100c0 rwx 1 0 0x2180 NCOW NNC default 0x2a07b000 0x2bfe4000 8041 0 0xd342e4e0 r-x 1 0 0x0 NCOW NNC vnode 0xbfbe 0xbfc0 2 0 0xd2730300 rwx 1 0 0x2180 NCOW NNC default Now you have two metrics. You can calculate the size based on the range (the first two arguments). (addressB - addressA), and you can get the total number of resident pages from the third argument. What you can't do easily is figure out the total allocation because that is actually going to be resident pages + swapped pages. swapped pages is not included in the proc output. The fifth argument is the VM object. You could garner more information from the VM object by accessing it via /dev/kvm. You can get an approximate number of swapped out pages using: (vm_object-un_pager.swp.swp_bcount * 16) (only from VM objects of type OBJT_SWAP). You can get an exact count by delving into the swap array and locating all the swblock meta structures associated with the object used to store swap information for the object. That gets rather involved, I wouldn't bother. You can also theoretically push into shadow VM objects to locate pages from the parent process that have not yet been COW'd into the child (in the case of a
