Re: Linux memory monitoring compared to MS-Windows
On Dec 18, 2007 2:36 AM, Oded Arbel [EMAIL PROTECTED] wrote: how much of that virtual memory the process actually tries to use but can't get it all in physical RAM because other processes are also hogging the memory. Does such a thing exist in Linux? To phrase it differently: In the last second (your question inherently calls for a measurement period to be specified), how many pages/megs of memory were touched by this process?
Re: Linux memory monitoring compared to MS-Windows
On Tue, 2007-12-18 at 12:47 +0200, Ilya Konstantinov wrote: On Dec 18, 2007 2:36 AM, Oded Arbel [EMAIL PROTECTED] wrote: how much of that virtual memory the process actually tries to use but can't get it all in physical RAM because other processes are also hogging the memory. Does such a thing exist in Linux? To phrase it differently: In the last second (your question inherently calls for a measurement period to be specified), how many pages/megs of memory were touched by this process? Yes, or at least that's how I understand it. I'm not sure about the time frame as the MSDN documentation does not specify. I would expect it to be much larger then a second - maybe a minute or a few. -- Oded = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
There's something in your question I don't understand: If a process has 1gb in virtual memory, of which 500mb in physical, then this means that it has 500mb in swap. Or in other words: 500mb that the process wanted in physical but couldn't. (isn't that what you asked to know) Obviously I'm missing something here. Anyway, few notes: - MM starts swapping *EVEN BEFORE* all processes hog the whole physical memory. That's because the kernel wants to leave some memory space for buffers/cache. So you may find yourself swapping idle pages while theoretically some physical memory might be freed. This behavior tunable via /proc/sys/vm/swappiness - free command has an interesting 2nd line, which shows the net memory usage (i.e. without buffers/cache) - The amazing top command can add a swap column; press O, then p and enter. (If 'top' can do it, the per-process swap-vs.-physical data can probably be fetched from /proc/ some way, I don't know how, though) On Tuesday 18 December 2007 02:36, Oded Arbel wrote: Hi List. I heard (but haven't actually seen) that in MS-Windows the system keeps track of some notion of working set, which is supposedly (if I understand correctly) the total size of pages that an application referenced recently - whether these are currently resident or swapped out (see http://msdn2.microsoft.com/en-us/library/ms684891.aspx which is an MSDN article I found on the subject). The way I understand processes normally work (in Linux anyway) is that as long as there is enough memory available the memory manager keeps all pages that an application constantly references in physical RAM, and pages that are not references are swapped out after a while. A good example of such is a long running Java virtual machine process (at least the Sun implementation anyway) that doesn't return unused memory to the operating system letting it being swapped out until its needed again - so I have some jvm process which takes up some 1.5GB of virtual but less then 150MB resident: it was processing a lot of data some time in the past but now its idling. Now (again - according to my understanding) under contention - i.e. when processes need to use more physical memory then what is available - the memory manager keeps swapping stuff in and out of memory in an attempt to satisfy all requests. Under such conditions its might be useful to know - for each process - the amount of physical memory in use, the amount of virtual mapped to the process, but also how much of that virtual memory the process actually tries to use but can't get it all in physical RAM because other processes are also hogging the memory. Does such a thing exist in Linux? Thanks in advance -- Oded = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED] = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
On Dec 18, 2007 2:47 PM, Oren Held [EMAIL PROTECTED] wrote: If a process has 1gb in virtual memory, of which 500mb in physical, then this means that it has 500mb in swap. Or in other words: 500mb that the process wanted in physical but couldn't. (isn't that what you asked to know) BTW, are you sure the Virtual column indicates real+swapped pages? I'm not sure what the terminology really refers to, but something called virtual is likely to refer to all pages, even totally non-committed ones -- i.e. when you malloc 1GB, your address space is expanded but until you touch those pages, the kernel wouldn't bother allotting real nor swap memory for it.
Re: Linux memory monitoring compared to MS-Windows
Oren Held wrote: There's something in your question I don't understand: If a process has 1gb in virtual memory, of which 500mb in physical, then this means that it has 500mb in swap. No, that is not what it means. Virtual memory amount might be different then physical memory amount due a whole bunch of different reason, swapping being just a single instance and not even the common one. Some other reasons are: - Lazy allocation. Malloc allocates virtual memory addresses, but only use does physical memory allocation. - Copy on write semantics, as is done with fork() for example. - Shared memory, including that of shared libraries text and data sections. - Memory mapped IO. And so on... In addition, memory may exist in both physical memory and swap at the same time, possibly in different versions (e.g. dirty anonymous page which has been previously swapped). In short, please ask your question again using more exact terms so that we understand what you want to ask. Gilad -- Gilad Ben-Yossef [EMAIL PROTECTED] Codefidence. A name you can trust(tm) Web: http://codefidence.com | Cel: +972.52.8260388 IL: +972.3.7515563 ext. 201 | Fax:+972.3.7515503 = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
On Tue, 2007-12-18 at 17:03 +0200, Gilad Ben-Yossef wrote: Oren Held wrote: There's something in your question I don't understand: If a process has 1gb in virtual memory, of which 500mb in physical, then this means that it has 500mb in swap. No, that is not what it means. Virtual memory amount might be different then physical memory amount due a whole bunch of different reason, swapping being just a single instance and not even the common one. In short, please ask your question again using more exact terms so that we understand what you want to ask. I was using the 1.5GB process only as a (bad) example. If you really want specifics, then said process has about 1.5GB under the virtual column (which can indeed be a lot of different things other then just real + swap, although interestingly this is exactly how 'man top' defines VIRT), about 200MB under resident column, some 8MB under shared, and 1.2GB under SWAP (according to top, not htop - I couldn't get htop to list this). Now according to 'free' only 250MB of swap are in actual use. The way I see it, 'top's SWAP is computed from virtual - resident regardless of how much swap space the process actually uses - so as Gilad said - talking about the virtual image size is next to useless. Anyway, the original question wasn't really about that specific process - I simply figured it as a good example. Obviously I was wrong, but that's ok - I was expecting something like this :-) The real question - as emphasized by all the comments I received - is: can I know how much memory the process is accessing (within some time period) specifically when its more then the total of pages actually held in physical memory. *accessing = reading to or writing from, not just having them assigned to the process. -- Oded = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
On Tue, Dec 18, 2007 at 05:54:56PM +0200, Oded Arbel wrote: The real question - as emphasized by all the comments I received - is: can I know how much memory the process is accessing (within some time period) specifically when its more then the total of pages actually held in physical memory. *accessing = reading to or writing from, not just having them assigned to the process. Look at the pagemap patches, which will give you the raw info you need to calculate this. See http://lwn.net/Articles/230975/ for an introduction. Cheers, Muli = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
Oded Arbel [EMAIL PROTECTED] writes: Now (again - according to my understanding) under contention - i.e. when processes need to use more physical memory then what is available - the memory manager keeps swapping stuff in and out of memory in an attempt to satisfy all requests. Under such conditions its might be useful to know - for each process - the amount of physical memory in use, the amount of virtual mapped to the process, but also how much of that virtual memory the process actually tries to use but can't get it all in physical RAM because other processes are also hogging the memory. All of the above is basically correct. Does such a thing exist in Linux? Internally to the kernel, look at the active_list and inactive_list members of struct zone. I have no idea whether it is exposed to userspace in any way. You can find out, I suppose. So the kernel knows it, what would you do with it as a user? If you need it and the kernel does not allow you, I suppose you can write your own /proc file module... May be a nice exercise for students... -- Oleg Goldshmidt | [EMAIL PROTECTED] | http://www.goldshmidt.org = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Re: Linux memory monitoring compared to MS-Windows
On Tue, 2007-12-18 at 18:36 +0200, Muli Ben-Yehuda wrote: On Tue, Dec 18, 2007 at 05:54:56PM +0200, Oded Arbel wrote: The real question - as emphasized by all the comments I received - is: can I know how much memory the process is accessing (within some time period) specifically when its more then the total of pages actually held in physical memory. *accessing = reading to or writing from, not just having them assigned to the process. Look at the pagemap patches, which will give you the raw info you need to calculate this. See http://lwn.net/Articles/230975/ for an introduction. Thanks ! this is really interesting. Using the clear_refs item one can easily implement the supposed working set functionality. I'll see if I can compile it for my kernel. -- Oded = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
Linux memory monitoring compared to MS-Windows
Hi List. I heard (but haven't actually seen) that in MS-Windows the system keeps track of some notion of working set, which is supposedly (if I understand correctly) the total size of pages that an application referenced recently - whether these are currently resident or swapped out (see http://msdn2.microsoft.com/en-us/library/ms684891.aspx which is an MSDN article I found on the subject). The way I understand processes normally work (in Linux anyway) is that as long as there is enough memory available the memory manager keeps all pages that an application constantly references in physical RAM, and pages that are not references are swapped out after a while. A good example of such is a long running Java virtual machine process (at least the Sun implementation anyway) that doesn't return unused memory to the operating system letting it being swapped out until its needed again - so I have some jvm process which takes up some 1.5GB of virtual but less then 150MB resident: it was processing a lot of data some time in the past but now its idling. Now (again - according to my understanding) under contention - i.e. when processes need to use more physical memory then what is available - the memory manager keeps swapping stuff in and out of memory in an attempt to satisfy all requests. Under such conditions its might be useful to know - for each process - the amount of physical memory in use, the amount of virtual mapped to the process, but also how much of that virtual memory the process actually tries to use but can't get it all in physical RAM because other processes are also hogging the memory. Does such a thing exist in Linux? Thanks in advance -- Oded = To unsubscribe, send mail to [EMAIL PROTECTED] with the word unsubscribe in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
