[chromium-dev] Re: Memory usage in chrome
On Wed, Jun 24, 2009 at 8:19 AM, Mike Belshembel...@google.com wrote: Anecdotally, a couple of people have said it works and a couple of people have said it makes no difference. I do believe that people doing compiles could see a difference. To determine if it was real, we did an experiment with memory model high and memory model medium and measured the whiteout duration. We did not see any statistical difference between the two. It could be that that was not a good test, but we don't know of a better test. FWIW, it makes a huge difference on my machine, too. The main competition for RAM is Thunderbird (VM 1.3GB right now) and Firefox (480 MB). The biggest Chrome process is 400 MB. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Actually, I had done testing like that in a VM to get a sense of raw process/threads limits on a low memory system. It's relatively easy to achieve a fair level of confidence with a live VM snapshot and making sure to reuse the same timings for measurement. The measurement timing is important since the OS will trigger page flush after X number of seconds. If you are not in a hurry, usually taking measurements 60 seconds after your test is idle will be quite stable. M-A On Sat, Jun 27, 2009 at 7:39 PM, Mike Belshe mbel...@google.com wrote: This one is the hardest to test, you need to run a pristinely clean system to execute. Also - don't forget to make the browser window sizes the same (and with the same amount of visible window) for all browsers under test, because if the kernel can't offload to the graphics card, the display memory will be counted here. But yeah, if you can make all that work, then it is a good test! mike On Sat, Jun 27, 2009 at 2:50 PM, Linus Upson li...@google.com wrote: If I recall correctly, the best way we found to measure the total memory usage of a multi-process system like chrome was to measure the total commit charge of windows as you run the test. This will correctly account for shared memory, mapped pages that have been touched, kernel memory, etc. I don't recall if it includes virtual alloced paged that haven't been made real. The big limitation is that your test needs to be the only thing running on the machine. Linus On Thu, Jun 25, 2009 at 4:11 PM, Mike Beltzner beltz...@mozilla.comwrote: On 25-Jun-09, at 7:02 PM, Mike Belshe wrote: This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. Cool! Good to know. I'll take a peek into that code you mentioned to see what the heuristic is that you're using. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. I'm basically being lazy. I'd like to not have to make my own counter for Private Working Set, so I watched the values of Memory (Private Working Set) and Commit Size in the Task Manager as the test ran, and noticed that they increased/decreased at the same time, and the delta between them was a near constant 2x. Since my interest here is developing a metric that can help us understand when we're regressing/ improving memory usage, the exact value isn't as important to me as the delta. If the deltas are simply off by a constant factor, I could live with that. As I said: lazy! The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. Someone just did in IRC, actually. Apparently in addition to what you said, as soon as a page is in cache, processes get pooled. I clear caches between test runs, but it sounds like since we're calling these with window.open() in our test, they all get placed in the same process. Overall, though,
[chromium-dev] Re: Memory usage in chrome
If I recall correctly, the best way we found to measure the total memory usage of a multi-process system like chrome was to measure the total commit charge of windows as you run the test. This will correctly account for shared memory, mapped pages that have been touched, kernel memory, etc. I don't recall if it includes virtual alloced paged that haven't been made real. The big limitation is that your test needs to be the only thing running on the machine. Linus On Thu, Jun 25, 2009 at 4:11 PM, Mike Beltzner beltz...@mozilla.com wrote: On 25-Jun-09, at 7:02 PM, Mike Belshe wrote: This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. Cool! Good to know. I'll take a peek into that code you mentioned to see what the heuristic is that you're using. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. I'm basically being lazy. I'd like to not have to make my own counter for Private Working Set, so I watched the values of Memory (Private Working Set) and Commit Size in the Task Manager as the test ran, and noticed that they increased/decreased at the same time, and the delta between them was a near constant 2x. Since my interest here is developing a metric that can help us understand when we're regressing/ improving memory usage, the exact value isn't as important to me as the delta. If the deltas are simply off by a constant factor, I could live with that. As I said: lazy! The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. Someone just did in IRC, actually. Apparently in addition to what you said, as soon as a page is in cache, processes get pooled. I clear caches between test runs, but it sounds like since we're calling these with window.open() in our test, they all get placed in the same process. Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/Flash. Good news, I guess, in terms of reporting accurately! OK - I think this might basically use one renderer process in chrome? Because of the new-process creation policy, it may not be representative of real world usage. Darin? Right, but AIUI, it's an erring on the side of reporting less, not more. If there's a better way to automate pageloads that represents real world usage, please let me know. The whole while, we measure the amount of memory taken using the PROCESS_MEMORY_COUNTERS structure, summating over processes when multiple exist (as they do in the case of Internet Explorer 8 and Chrome 2) Ok - that will double count shared memory. I'd estimate 3-5MB per process. So we're talking about over-reporting by 9-15MB across the
[chromium-dev] Re: Memory usage in chrome
On Sat, Jun 27, 2009 at 2:50 PM, Linus Upsonli...@google.com wrote: If I recall correctly, the best way we found to measure the total memory usage of a multi-process system like chrome was to measure the total commit charge of windows as you run the test. My favorite test is to plot the performance of the app on some benchmark as a function of the amount of memory the computer is booted with. Comparing that plot for two different apps gives you a pretty clear picture of relative RAM requirements. Chrome kicked the bejeesus out of Firefox last time I tried that, but I think that was before Firefox 3.5. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
This one is the hardest to test, you need to run a pristinely clean system to execute. Also - don't forget to make the browser window sizes the same (and with the same amount of visible window) for all browsers under test, because if the kernel can't offload to the graphics card, the display memory will be counted here. But yeah, if you can make all that work, then it is a good test! mike On Sat, Jun 27, 2009 at 2:50 PM, Linus Upson li...@google.com wrote: If I recall correctly, the best way we found to measure the total memory usage of a multi-process system like chrome was to measure the total commit charge of windows as you run the test. This will correctly account for shared memory, mapped pages that have been touched, kernel memory, etc. I don't recall if it includes virtual alloced paged that haven't been made real. The big limitation is that your test needs to be the only thing running on the machine. Linus On Thu, Jun 25, 2009 at 4:11 PM, Mike Beltzner beltz...@mozilla.comwrote: On 25-Jun-09, at 7:02 PM, Mike Belshe wrote: This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. Cool! Good to know. I'll take a peek into that code you mentioned to see what the heuristic is that you're using. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. I'm basically being lazy. I'd like to not have to make my own counter for Private Working Set, so I watched the values of Memory (Private Working Set) and Commit Size in the Task Manager as the test ran, and noticed that they increased/decreased at the same time, and the delta between them was a near constant 2x. Since my interest here is developing a metric that can help us understand when we're regressing/ improving memory usage, the exact value isn't as important to me as the delta. If the deltas are simply off by a constant factor, I could live with that. As I said: lazy! The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. Someone just did in IRC, actually. Apparently in addition to what you said, as soon as a page is in cache, processes get pooled. I clear caches between test runs, but it sounds like since we're calling these with window.open() in our test, they all get placed in the same process. Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/Flash. Good news, I guess, in terms of reporting accurately! OK - I think this might basically use one renderer process in chrome? Because of the new-process creation policy, it may not be representative of real world usage. Darin? Right, but AIUI, it's an erring on the side of reporting less,
[chromium-dev] Re: Memory usage in chrome
On Thu, Jun 25, 2009 at 4:11 PM, Mike Beltzner beltz...@mozilla.com wrote: On 25-Jun-09, at 7:02 PM, Mike Belshe wrote: This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. Cool! Good to know. I'll take a peek into that code you mentioned to see what the heuristic is that you're using. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. I'm basically being lazy. I'd like to not have to make my own counter for Private Working Set, so I watched the values of Memory (Private Working Set) and Commit Size in the Task Manager as the test ran, and noticed that they increased/decreased at the same time, and the delta between them was a near constant 2x. Since my interest here is developing a metric that can help us understand when we're regressing/improving memory usage, the exact value isn't as important to me as the delta. If the deltas are simply off by a constant factor, I could live with that. As I said: lazy! understood. Just in case, here is the code for you to take: http://src.chromium.org/viewvc/chrome/trunk/src/base/process_util_win.cc?view=markuppathrev=14705 The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. Someone just did in IRC, actually. Apparently in addition to what you said, as soon as a page is in cache, processes get pooled. I clear caches between test runs, but it sounds like since we're calling these with window.open() in our test, they all get placed in the same process. Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/Flash. Good news, I guess, in terms of reporting accurately! Good news and bad news :-) If you publish results saying how Chrome did, Chrome doesn't get to cleanup as cleanly in this case. It still *should*, but it's not what users do :-) OK - I think this might basically use one renderer process in chrome? Because of the new-process creation policy, it may not be representative of real world usage. Darin? Right, but AIUI, it's an erring on the side of reporting less, not more. If there's a better way to automate pageloads that represents real world usage, please let me know. I don't know of cross-browser code that will accomplish what you want. Maybe we should add that. The whole while, we measure the amount of memory taken using the PROCESS_MEMORY_COUNTERS structure, summating over processes when multiple exist (as they do in the case of Internet Explorer 8 and Chrome 2) Ok - that will double count shared memory. I'd estimate 3-5MB per process. So we're talking about over-reporting by 9-15MB across the test. Again, good to know. I'll try to take a closer look at your
[chromium-dev] Re: Memory usage in chrome
On 26-Jun-09, at 12:59 PM, Mike Belshe wrote: Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/ Flash. Good news, I guess, in terms of reporting accurately! Good news and bad news :-) If you publish results saying how Chrome did, Chrome doesn't get to cleanup as cleanly in this case. It still *should*, but it's not what users do :-) Hm, so the thinking is that since users will close tabs and open new ones and navigate from the Omnibox that will close and open processes, thus freeing memory? I don't think I'm going to publish results, per se, but if I do, I'll point this part out for sure. Right, but AIUI, it's an erring on the side of reporting less, not more. If there's a better way to automate pageloads that represents real world usage, please let me know. I don't know of cross-browser code that will accomplish what you want. Maybe we should add that. What about command line invocation? Does that spawn new processes? cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Fri, Jun 26, 2009 at 10:11 AM, Mike Beltzner beltz...@mozilla.comwrote: On 26-Jun-09, at 12:59 PM, Mike Belshe wrote: Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/Flash. Good news, I guess, in terms of reporting accurately! Good news and bad news :-) If you publish results saying how Chrome did, Chrome doesn't get to cleanup as cleanly in this case. It still *should*, but it's not what users do :-) Hm, so the thinking is that since users will close tabs and open new ones and navigate from the Omnibox that will close and open processes, thus freeing memory? I don't think I'm going to publish results, per se, but if I do, I'll point this part out for sure. Right, but AIUI, it's an erring on the side of reporting less, not more. If there's a better way to automate pageloads that represents real world usage, please let me know. I don't know of cross-browser code that will accomplish what you want. Maybe we should add that. What about command line invocation? Does that spawn new processes? Oh - good call - yes - that should work. cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Thu, Jun 25, 2009 at 9:40 AM, Mike Beltzner beltz...@mozilla.com wrote: On 22-Jun-09, at 12:57 PM, Mike Belshe wrote: Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. Figuring out how to track this is a bit of a terminology mess ;) Referring to the following document: http://msdn.microsoft.com/en-us/library/aa965225(VS.85).aspx Yeah, the APIs all have constraints. We end up walking the pages and adding them up. See process_util_win.cc in the chromium tree. Be sure to check about:memory and hover over the little ? icons to see what we measure. http://msdn.microsoft.com/en-us/library/aa965225(VS.85).aspx If someone measures the Private Bytes counter, which uses the PROCESS_MEMORY_COUNTERS_EX.PrivateUsage structure, that seems to map to the Task Manager Commit Size which isn't the thing I believe we want to measure. Right. The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. I wish it were easier too. In Vista's task manager, the primary metric is Memory (Private Working Set), and I believe the only way to get this number is to call QueryWorkingSet, walk the pages, and add up the ones which are marked as private. If you come up with a better way, please let me know! Mike Would you agree that summating the data from the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure would give a fair indication of memory usage? cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On 25-Jun-09, at 12:52 PM, Mike Belshe wrote: Yeah, the APIs all have constraints. We end up walking the pages and adding them up. See process_util_win.cc in the chromium tree. Be sure to check about:memory and hover over the little ? icons to see what we measure. This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? If so, is there any command-line switch that will output those summary stats to console on a regular basis? If someone measures the Private Bytes counter, which uses the PROCESS_MEMORY_COUNTERS_EX.PrivateUsage structure, that seems to map to the Task Manager Commit Size which isn't the thing I believe we want to measure. Right. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? If you come up with a better way, please let me know! Well, I can tell you what we've done and what we're doing, just trying to get confidence on various metrics. We've replicated the membuster test we used around the release of Firefox 3 (see http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ ) as a way of measuring Firefox's ability to hold a steady state of memory across a browsing session (ie: not leak) and then release that memory when the session ends (ie: not bogart system resources). The test: - runs on Windows 7, using Python and our Standalone Talos code to make measurements - uses a local web proxy to ensure we're always viewing the same web content - uses the JS at view-source:http://random.pavlov.net/membuster/index.html to: --- open 30 pages using window.open() calls --- load ~300 pages, opening and closing windows with each load (always 30 open) --- close all 30 windows when done cycling The whole while, we measure the amount of memory taken using the PROCESS_MEMORY_COUNTERS structure, summating over processes when multiple exist (as they do in the case of Internet Explorer 8 and Chrome 2) The results can be seen here, using both the PrivateUsage (shows as Commit Size in Windows 7 Task Manager) and WorkingSet (shows as Working Set (Memory) in Windows 7 Task Manager) counters: http://people.mozilla.com/~beltzner/images/private-usage.png http://people.mozilla.com/~beltzner/images/working-set.png The datapoint we wanted to get out of this was primarily about Firefox 3.5 vs Firefox 3.0.11, and as you can see we actually take slightly (7-10MB) more memory during the page cycling, but manage to release more when done. Safari had previously been unable to complete this test, but now does. I'd love to get your feedback on how we can improve our recording here. I think we're going to try and bundle together the test and files into an easier-to-use-tool than our delicately cobbled together solution, but if you wanted to try it yourself, here's a ZIP with the required code (needs Python 2.5 or later to be installed on your system): Once unzipped, enter the directory and run the following commands: \path\to\python\python.exe proxyserver.py -v -l -u proxy-cache.db \path\to\python\python.exe measure_any_windows_app.py \path\to \resultsfile\resultsfile.txt You'll want to edit measure_any_windows_app.py to have the correct paths to the various applications, and you'll also want to make sure your browsers are set to use 127.0.0.1 port 8000 as a proxy (under Advanced Network Settings in Firefox, Internet Properties Connections LAN Settings for Chrome/IE/Safari) (Thanks to David Dahl for modifying the required files to get at those measurements) cheers,
[chromium-dev] Re: Memory usage in chrome
On 25-Jun-09, at 6:26 PM, Mike Beltzner wrote: here's a ZIP with the required code (needs Python 2.5 or later to be installed on your system): Oops, forgot the link! http://people.mozilla.org/~beltzner/membuster-talos.rar cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Thu, Jun 25, 2009 at 3:26 PM, Mike Beltzner beltz...@mozilla.com wrote: On 25-Jun-09, at 12:52 PM, Mike Belshe wrote: Yeah, the APIs all have constraints. We end up walking the pages and adding them up. See process_util_win.cc in the chromium tree. Be sure to check about:memory and hover over the little ? icons to see what we measure. This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. If so, is there any command-line switch that will output those summary stats to console on a regular basis? You can use --enable-logging --log-level=0 - each time you refresh the about:memory page, it will drop a line to the log file. This can be useful for plots over time. If someone measures the Private Bytes counter, which uses the PROCESS_MEMORY_COUNTERS_EX.PrivateUsage structure, that seems to map to the Task Manager Commit Size which isn't the thing I believe we want to measure. Right. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. But of course, as you load more tabs, you'll see the memory use go up. The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. If you come up with a better way, please let me know! Well, I can tell you what we've done and what we're doing, just trying to get confidence on various metrics. We've replicated the membuster test we used around the release of Firefox 3 (see http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ ) as a way of measuring Firefox's ability to hold a steady state of memory across a browsing session (ie: not leak) and then release that memory when the session ends (ie: not bogart system resources). The test: - runs on Windows 7, using Python and our Standalone Talos code to make measurements - uses a local web proxy to ensure we're always viewing the same web content - uses the JS at view-source: http://random.pavlov.net/membuster/index.html to: --- open 30 pages using window.open() calls --- load ~300 pages, opening and closing windows with each load (always 30 open) --- close all 30 windows when done cycling OK - I think this might basically use one renderer process in chrome? Because of the new-process creation policy, it may not be representative of real world usage. Darin? The whole while, we measure the amount of memory taken using the PROCESS_MEMORY_COUNTERS structure, summating over processes when multiple exist (as they do in the case of Internet Explorer 8 and Chrome 2) Ok - that will double count shared memory. I'd estimate 3-5MB per process. The results can be seen here, using both the PrivateUsage (shows as Commit Size in Windows 7 Task Manager) and WorkingSet (shows as Working Set (Memory) in Windows 7 Task Manager) counters: http://people.mozilla.com/~beltzner/images/private-usage.png http://people.mozilla.com/~beltzner/images/working-set.png
[chromium-dev] Re: Memory usage in chrome
On 25-Jun-09, at 7:02 PM, Mike Belshe wrote: This screen actually confuses me a little, as the Summary statistics don't match the summation of the process based statistics. Do you mean to say your summary statistics take into account the memory that's being shared across the various processes? Correct. The shared across all processes is a bit of a hack, because you can't know exactly which pages are shared across every single process. We do a heuristic. Cool! Good to know. I'll take a peek into that code you mentioned to see what the heuristic is that you're using. Interestingly, as I watched this value change while webpages were loading, it tracked the same pattern of growth/decline as Memory (Private Working Set) in the Task Manager, though the values were usually about 2x or so more. I suppose this is due to the heap sharing you were speaking of earlier? I'm not quite sure what you mean. I'm basically being lazy. I'd like to not have to make my own counter for Private Working Set, so I watched the values of Memory (Private Working Set) and Commit Size in the Task Manager as the test ran, and noticed that they increased/decreased at the same time, and the delta between them was a near constant 2x. Since my interest here is developing a metric that can help us understand when we're regressing/ improving memory usage, the exact value isn't as important to me as the delta. If the deltas are simply off by a constant factor, I could live with that. As I said: lazy! The Working Set - Private counter doesn't seem to have a structure according to the MSDN document; that's what maps to the Memory (Private Working Set) column in the TaskManager. Right, I think you have to use QueryWorkingSet, walk the pages and categorize them yourself. OK, I can look into trying that. Though I'm wondering if it's worth the bother, as the meta-pattern, to me, is more interesting than the precise megabyte count. For a single process browser, it's not worth the effort; I think it's the only way to know how to account for shared memory. The closest thing I can find is the Working Set counter, which uses the PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize structure and shows up in the Vista Task Manager as Working Set (Memory) For multi-proc browsers like chrome, this will way overstate RAM; there is a good 5-6MB of shared working set in each process. So for 10 tabs, you'd could an extra 50MB for Chrome if you do it this way. Looking both in Task Manager and about:memory, when I have 30 tabs open I'm not seeing 30 processes. Are you sure you're right about this point? You don't always get a new process for every tab. If two tabs are connected via javascript, then they'll be in the same process (the about:memory shows which tabs are in the same process). So, clicking a link, for example, will open in the same tab, but typing the URL in the omnibox will create a new process. Others could tell you more about the exact policy for when you get a new process and when you don't. Someone just did in IRC, actually. Apparently in addition to what you said, as soon as a page is in cache, processes get pooled. I clear caches between test runs, but it sounds like since we're calling these with window.open() in our test, they all get placed in the same process. Overall, though, that should mean that we're *not* double counting memory. In fact, when I observed as the test ran, there were only three processes: one for the browser, one for the single content process from which all tabs were spawned, and one for Shockwave/Flash. Good news, I guess, in terms of reporting accurately! OK - I think this might basically use one renderer process in chrome? Because of the new-process creation policy, it may not be representative of real world usage. Darin? Right, but AIUI, it's an erring on the side of reporting less, not more. If there's a better way to automate pageloads that represents real world usage, please let me know. The whole while, we measure the amount of memory taken using the PROCESS_MEMORY_COUNTERS structure, summating over processes when multiple exist (as they do in the case of Internet Explorer 8 and Chrome 2) Ok - that will double count shared memory. I'd estimate 3-5MB per process. So we're talking about over-reporting by 9-15MB across the test. Again, good to know. I'll try to take a closer look at your test, but I'm not sure when I'll have time :-( No rush here, and I appreciate your time and candor to date! cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
FWIW, I strongly believe we should move the default to --memory-model=high. This is what pretty much every other app in the world does, and we mostly penalize ourselves when the OS aggressively swaps us out for a dumb reason (which yes, Windows does do). We have a lot of complaints of I came back the next hour/day/whatever and everything was unresponsive. I don't think our current tradeoff is the right one. I know Mike wants to be a good citizen and feels like if the OS swapped you out it really needed that RAM, but in my own observations of my machine the OS swaps for retarded reasons and I gain nothing but headaches. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
+1. Most people are not doing compiles, we're trying to say that people live in the web and in their browser, and that their browser is the primary application. For me at least, that is true. The browser is the app I use the most -- the only other app I use regularly is an ssh client, which can usually fit in memory or get swapped back in much quicker than Chrome. 2009/6/23 Peter Kasting pkast...@chromium.org FWIW, I strongly believe we should move the default to --memory-model=high. This is what pretty much every other app in the world does, and we mostly penalize ourselves when the OS aggressively swaps us out for a dumb reason (which yes, Windows does do). We have a lot of complaints of I came back the next hour/day/whatever and everything was unresponsive. I don't think our current tradeoff is the right one. I know Mike wants to be a good citizen and feels like if the OS swapped you out it really needed that RAM, but in my own observations of my machine the OS swaps for retarded reasons and I gain nothing but headaches. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Could we stat at memory-model=hight and then change our memory model mid-flight if there are any large, non-chrome, memory hungry processes? -- Elliot On Wed, Jun 24, 2009 at 9:39 AM, Ian Fettei...@chromium.org wrote: +1. Most people are not doing compiles, we're trying to say that people live in the web and in their browser, and that their browser is the primary application. For me at least, that is true. The browser is the app I use the most -- the only other app I use regularly is an ssh client, which can usually fit in memory or get swapped back in much quicker than Chrome. 2009/6/23 Peter Kasting pkast...@chromium.org FWIW, I strongly believe we should move the default to --memory-model=high. This is what pretty much every other app in the world does, and we mostly penalize ourselves when the OS aggressively swaps us out for a dumb reason (which yes, Windows does do). We have a lot of complaints of I came back the next hour/day/whatever and everything was unresponsive. I don't think our current tradeoff is the right one. I know Mike wants to be a good citizen and feels like if the OS swapped you out it really needed that RAM, but in my own observations of my machine the OS swaps for retarded reasons and I gain nothing but headaches. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Wed, Jun 24, 2009 at 9:39 AM, Ian Fette i...@chromium.org wrote: +1. Most people are not doing compiles, we're trying to say that people live in the web and in their browser, and that their browser is the primary application. For me at least, that is true. The browser is the app I use the most -- the only other app I use regularly is an ssh client, which can usually fit in memory or get swapped back in much quicker than Chrome. This doesn't make sense as a use case, because if there is nothing else eating your memory, then the memory doesn't have to swap in. It's already there! :-) For users with plenty of RAM like you, memory-model=medium should work just fine. The only people that might be effected by this is people that do have something else competing for the memory (like a compile, or they are on a low-memory box and outlook is eating it). I do worry that virus scanners could churn through memory causing similar effects, but again, we measured for this and so far have been unable to detect any difference. At this point we could change to memory-model=high, despite having no real-world data that this is a problem. Chrome would be perceived to use 25% more memory, and the folks on this thread that have compiles going might feel a better experience. If we don't care about a 25% jump in memory use then we should switch the default even though data suggests it won't help. One more possibility: maybe there is a bug we haven't yet identified. Mike 2009/6/23 Peter Kasting pkast...@chromium.org FWIW, I strongly believe we should move the default to --memory-model=high. This is what pretty much every other app in the world does, and we mostly penalize ourselves when the OS aggressively swaps us out for a dumb reason (which yes, Windows does do). We have a lot of complaints of I came back the next hour/day/whatever and everything was unresponsive. I don't think our current tradeoff is the right one. I know Mike wants to be a good citizen and feels like if the OS swapped you out it really needed that RAM, but in my own observations of my machine the OS swaps for retarded reasons and I gain nothing but headaches. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
I am willing to volunteer to be a test case if you need one, tell me what to do so you will be able to monitor.Before I added that switch, it was pretty horrible. Though I am building stuff with Java, XSLT, JavaScript that process a lot of files (HTMLs) and viewing and refreshing a lot of pages (in Chrome, mostly, but sometimes in other browsers for cross browser support), plus Eclipse and Outlook, if it matters. ☆PhistucK On Wed, Jun 24, 2009 at 19:46, Mike Belshe mbel...@google.com wrote: On Wed, Jun 24, 2009 at 9:39 AM, Ian Fette i...@chromium.org wrote: +1. Most people are not doing compiles, we're trying to say that people live in the web and in their browser, and that their browser is the primary application. For me at least, that is true. The browser is the app I use the most -- the only other app I use regularly is an ssh client, which can usually fit in memory or get swapped back in much quicker than Chrome. This doesn't make sense as a use case, because if there is nothing else eating your memory, then the memory doesn't have to swap in. It's already there! :-) For users with plenty of RAM like you, memory-model=medium should work just fine. The only people that might be effected by this is people that do have something else competing for the memory (like a compile, or they are on a low-memory box and outlook is eating it). I do worry that virus scanners could churn through memory causing similar effects, but again, we measured for this and so far have been unable to detect any difference. At this point we could change to memory-model=high, despite having no real-world data that this is a problem. Chrome would be perceived to use 25% more memory, and the folks on this thread that have compiles going might feel a better experience. If we don't care about a 25% jump in memory use then we should switch the default even though data suggests it won't help. One more possibility: maybe there is a bug we haven't yet identified. Mike 2009/6/23 Peter Kasting pkast...@chromium.org FWIW, I strongly believe we should move the default to --memory-model=high. This is what pretty much every other app in the world does, and we mostly penalize ourselves when the OS aggressively swaps us out for a dumb reason (which yes, Windows does do). We have a lot of complaints of I came back the next hour/day/whatever and everything was unresponsive. I don't think our current tradeoff is the right one. I know Mike wants to be a good citizen and feels like if the OS swapped you out it really needed that RAM, but in my own observations of my machine the OS swaps for retarded reasons and I gain nothing but headaches. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
This explanation actually shows me the source of this serious jank (I hope I am using the term in the right context) I am having all of the time.I am getting back to Chrome after a few minutes of dealing with some other application and I have to wait, sometimes even for twenty seconds or more, until I get the control back on the tab contents. Sometimes I am not using a tab for a few minutes (or more) and when I switch back to it, it takes it twenty seconds or more until I get the control back of the tab contents. :( ☆PhistucK On Mon, Jun 22, 2009 at 19:57, Mike Belshe mbel...@google.com wrote: On Mon, Jun 22, 2009 at 9:29 AM, Mike Beltzner beltz...@mozilla.comwrote: On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. For most applications, there isn't much difference between private bytes and working set private bytes. However, because of Chrome's multi-proc architecture, there is a big difference. The reason is because Chrome intentionally gives memory back to the OS. For instance, on my current instance of Chrome, I'm using 16 tabs. The sum of the private bytes is 514408. The sum of the private working set bytes is 275040, nearly half of the private bytes number. This is on a machine with 8GB of RAM, so there is plenty of memory to go around. But if some other app wants the memory, Chrome gave it back to the OS and will suffer the page faults to get it back. Since Chrome has given it back to the OS (and has volunteered to take the performance consequences of getting it back), I don't think it should be counted as Chrome usage. Others may disagree. But Windows uses working set as the primary metric for all applications the OS folks agree that working set is the right way to account for memory usage. Single process browsers have a hard time giving memory back, because they can't differentiate which pages are accounted to unused, background tabs and which pages are accounted to the active, in-use tabs. Finally, the common metric which definitely doesn't work well is Windows XP's default metric: working set (private + shared). Because of shared memory between processes, simply summing the working set will far overstate the actual RAM used. Part of the motivation with Vista changing the default metric from working set to private working set was precisely to deal with the issue of better accounting of shared memory. Mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Have you tried running with --memory-model=high ? 2009/6/23 PhistucK phist...@gmail.com: This explanation actually shows me the source of this serious jank (I hope I am using the term in the right context) I am having all of the time. I am getting back to Chrome after a few minutes of dealing with some other application and I have to wait, sometimes even for twenty seconds or more, until I get the control back on the tab contents. Sometimes I am not using a tab for a few minutes (or more) and when I switch back to it, it takes it twenty seconds or more until I get the control back of the tab contents. :( ☆PhistucK On Mon, Jun 22, 2009 at 19:57, Mike Belshe mbel...@google.com wrote: On Mon, Jun 22, 2009 at 9:29 AM, Mike Beltzner beltz...@mozilla.com wrote: On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. For most applications, there isn't much difference between private bytes and working set private bytes. However, because of Chrome's multi-proc architecture, there is a big difference. The reason is because Chrome intentionally gives memory back to the OS. For instance, on my current instance of Chrome, I'm using 16 tabs. The sum of the private bytes is 514408. The sum of the private working set bytes is 275040, nearly half of the private bytes number. This is on a machine with 8GB of RAM, so there is plenty of memory to go around. But if some other app wants the memory, Chrome gave it back to the OS and will suffer the page faults to get it back. Since Chrome has given it back to the OS (and has volunteered to take the performance consequences of getting it back), I don't think it should be counted as Chrome usage. Others may disagree. But Windows uses working set as the primary metric for all applications the OS folks agree that working set is the right way to account for memory usage. Single process browsers have a hard time giving memory back, because they can't differentiate which pages are accounted to unused, background tabs and which pages are accounted to the active, in-use tabs. Finally, the common metric which definitely doesn't work well is Windows XP's default metric: working set (private + shared). Because of shared memory between processes, simply summing the working set will far overstate the actual RAM used. Part of the motivation with Vista changing the default metric from working set to private working set was precisely to deal with the issue of better accounting of shared memory. Mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
No. I added it now and I will post the results later. Thank you! I really hope it will help... ☆PhistucK 2009/6/23 Dean McNamee de...@chromium.org Have you tried running with --memory-model=high ? 2009/6/23 PhistucK phist...@gmail.com: This explanation actually shows me the source of this serious jank (I hope I am using the term in the right context) I am having all of the time. I am getting back to Chrome after a few minutes of dealing with some other application and I have to wait, sometimes even for twenty seconds or more, until I get the control back on the tab contents. Sometimes I am not using a tab for a few minutes (or more) and when I switch back to it, it takes it twenty seconds or more until I get the control back of the tab contents. :( ☆PhistucK On Mon, Jun 22, 2009 at 19:57, Mike Belshe mbel...@google.com wrote: On Mon, Jun 22, 2009 at 9:29 AM, Mike Beltzner beltz...@mozilla.com wrote: On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. For most applications, there isn't much difference between private bytes and working set private bytes. However, because of Chrome's multi-proc architecture, there is a big difference. The reason is because Chrome intentionally gives memory back to the OS. For instance, on my current instance of Chrome, I'm using 16 tabs. The sum of the private bytes is 514408. The sum of the private working set bytes is 275040, nearly half of the private bytes number. This is on a machine with 8GB of RAM, so there is plenty of memory to go around. But if some other app wants the memory, Chrome gave it back to the OS and will suffer the page faults to get it back. Since Chrome has given it back to the OS (and has volunteered to take the performance consequences of getting it back), I don't think it should be counted as Chrome usage. Others may disagree. But Windows uses working set as the primary metric for all applications the OS folks agree that working set is the right way to account for memory usage. Single process browsers have a hard time giving memory back, because they can't differentiate which pages are accounted to unused, background tabs and which pages are accounted to the active, in-use tabs. Finally, the common metric which definitely doesn't work well is Windows XP's default metric: working set (private + shared). Because of shared memory between processes, simply summing the working set will far overstate the actual RAM used. Part of the motivation with Vista changing the default metric from working set to private working set was precisely to deal with the issue of better accounting of shared memory. Mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Mike, yes we (I) increased the number of renderer processes for machines with lots of ram. I think it tops now to 40 processes. Our previous limit was not based on calculation but because we had WaitForMultipleObjects(..) which has a 64 objects maximum and we had 2 objects per process so our limit was 31 so to give us some slack we settled on 20 --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Wow, that changed my whole browsing experience.I got back to the computer after it has been on all of the night, with Chrome (and others) running and when I got back to Chrome, I did not even have to wait for a bit!!! Thank you very much for pointing it out. It was quite frustrating before (but I liked everything else about Chrome, so I kept on using it). What are the implications, though? Not giving up the space to other applications and that is it? ☆PhistucK 2009/6/23 Dean McNamee de...@chromium.org Have you tried running with --memory-model=high ? 2009/6/23 PhistucK phist...@gmail.com: This explanation actually shows me the source of this serious jank (I hope I am using the term in the right context) I am having all of the time. I am getting back to Chrome after a few minutes of dealing with some other application and I have to wait, sometimes even for twenty seconds or more, until I get the control back on the tab contents. Sometimes I am not using a tab for a few minutes (or more) and when I switch back to it, it takes it twenty seconds or more until I get the control back of the tab contents. :( ☆PhistucK On Mon, Jun 22, 2009 at 19:57, Mike Belshe mbel...@google.com wrote: On Mon, Jun 22, 2009 at 9:29 AM, Mike Beltzner beltz...@mozilla.com wrote: On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. For most applications, there isn't much difference between private bytes and working set private bytes. However, because of Chrome's multi-proc architecture, there is a big difference. The reason is because Chrome intentionally gives memory back to the OS. For instance, on my current instance of Chrome, I'm using 16 tabs. The sum of the private bytes is 514408. The sum of the private working set bytes is 275040, nearly half of the private bytes number. This is on a machine with 8GB of RAM, so there is plenty of memory to go around. But if some other app wants the memory, Chrome gave it back to the OS and will suffer the page faults to get it back. Since Chrome has given it back to the OS (and has volunteered to take the performance consequences of getting it back), I don't think it should be counted as Chrome usage. Others may disagree. But Windows uses working set as the primary metric for all applications the OS folks agree that working set is the right way to account for memory usage. Single process browsers have a hard time giving memory back, because they can't differentiate which pages are accounted to unused, background tabs and which pages are accounted to the active, in-use tabs. Finally, the common metric which definitely doesn't work well is Windows XP's default metric: working set (private + shared). Because of shared memory between processes, simply summing the working set will far overstate the actual RAM used. Part of the motivation with Vista changing the default metric from working set to private working set was precisely to deal with the issue of better accounting of shared memory. Mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
2009/6/21 PhistucK phist...@gmail.com Really? the statistics show that many people are using the app mode?Or did you mean web apps, as in web application websites? I mean web applications like gmail, hotmail, zoho, shopping carts, etc etc. But it's an unsubstantiated claim. Mike ☆PhistucK On Sun, Jun 21, 2009 at 21:03, Mike Belshe mbel...@google.com wrote: I assume he's not a benchmark pro, but he did a decent job already. We can nitpick his sampling methodology - but it won't change the result. He is correct that many procs is far more memory consuming than single proc, and we already knew this. This is a tradeoff we made consciously and deliberately. When firefox crashes, all tabs go down. When firefox memory is compromised (security), all tabs are compromised. In chrome, we don't have those problems, but instead use more RAM. Further, Chrome is also able to implement per-tab prioritization, so that background tabs don't make foreground tabs go slow. Firefox can't do that. Lastly, lets bring the test back to reality. People don't visit 150 random home pages. They may have 20-30 tabs open, but many are applications, with cookies, javascript state and much more than just the home page. When apps are in use, the memory gap between chrome and FF shrinks a lot. Mike On Sun, Jun 21, 2009 at 10:46 AM, Dan Kegel d...@kegel.com wrote: On Sun, Jun 21, 2009 at 10:22 AM, Mike Belshembel...@google.com wrote: First off - kudos to the author for posting the source and steps to reproduce! Most don't do that! Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Wait, why doesn't his program itself do the summing? (I don't see it in there.) Wouldn't that get rid of the ambiguity? How hard would it be to add that and repost? - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. cheers, mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
I built a web based application in php to upload dvd vob files to a server via the browser. The only browser that makes this project a non failure is chrome. Forget trying to upload 4+ gigs with Internet Explorer. It can not handle the memory addressing. Firefox fails 90% of the time. However chrome always gets the 4 gigs over the web interface. Chrome, an actual browser that doesnt need compatibility buttons, and also can handle moving 4+gigs with the browser. Chrome is kick ass. On Jun 22, 12:21 am, PhistucK phist...@gmail.com wrote: Really? the statistics show that many people are using the app mode?Or did you mean web apps, as in web application websites? ☆PhistucK On Sun, Jun 21, 2009 at 21:03, Mike Belshe mbel...@google.com wrote: I assume he's not a benchmark pro, but he did a decent job already. We can nitpick his sampling methodology - but it won't change the result. He is correct that many procs is far more memory consuming than single proc, and we already knew this. This is a tradeoff we made consciously and deliberately. When firefox crashes, all tabs go down. When firefox memory is compromised (security), all tabs are compromised. In chrome, we don't have those problems, but instead use more RAM. Further, Chrome is also able to implement per-tab prioritization, so that background tabs don't make foreground tabs go slow. Firefox can't do that. Lastly, lets bring the test back to reality. People don't visit 150 random home pages. They may have 20-30 tabs open, but many are applications, with cookies, javascript state and much more than just the home page. When apps are in use, the memory gap between chrome and FF shrinks a lot. Mike On Sun, Jun 21, 2009 at 10:46 AM, Dan Kegel d...@kegel.com wrote: On Sun, Jun 21, 2009 at 10:22 AM, Mike Belshembel...@google.com wrote: First off - kudos to the author for posting the source and steps to reproduce! Most don't do that! Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Wait, why doesn't his program itself do the summing? (I don't see it in there.) Wouldn't that get rid of the ambiguity? How hard would it be to add that and repost? - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Jun 21, 3:37 am, n179911 n179...@gmail.com wrote: Hi, There is a test which compares memory usage among rendering engineshttp://dotnetperls.com/chrome-memory From the site, it shows the maximum memory usage of Chrome is more than Safari is 2 times. Since both of them are Webkit base, does that mean the V8 engine uses twice as much memory as squirrel fish? --- Maximum memory used --- Peak memory usage measured during experiment. Chrome: 1216.16 MB [Largest] Firefox: 327.65 MB [Smallest] Opera: 554.11 MB Safari: 517.00 MB I find these results somewhat disingenuous. I left both Firefox and Chrome open over the weekend (both on Linux, with the Chromium daily build), Firefox with 1 tab, Chrome with 4 (one of them being Gmail). Chrome's memory usage didn't move while I had to kill Firefox for eating up 1.5GB of RAM and being generally unresponsive. Maybe Chrome uses more memory to do its job initially, but it certainly doesn't have the leaks that Firefox does which cause me to have to restart it several times a day. Keep up the awesome work, Chrome peeps. I can't wait until I'm using Chrome as my default browser on all my platforms. -- Toby DiPasquale --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Mon, Jun 22, 2009 at 9:29 AM, Mike Beltzner beltz...@mozilla.com wrote: On 21-Jun-09, at 10:22 AM, Mike Belshe wrote: Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Doesn't Private Bytes accurately represent the private memory for a given process? I thought the whole point of that was that it didn't measure any shared memory pools. Yes, that accurately represents the private memory for a process, but it doesn't reflect the user's experience. Windows generally tracks working set. Why? Because the working set is the amount of memory *not available to other apps*. If other apps can have the memory, then using the bytes is inconsequential. For most applications, there isn't much difference between private bytes and working set private bytes. However, because of Chrome's multi-proc architecture, there is a big difference. The reason is because Chrome intentionally gives memory back to the OS. For instance, on my current instance of Chrome, I'm using 16 tabs. The sum of the private bytes is 514408. The sum of the private working set bytes is 275040, nearly half of the private bytes number. This is on a machine with 8GB of RAM, so there is plenty of memory to go around. But if some other app wants the memory, Chrome gave it back to the OS and will suffer the page faults to get it back. Since Chrome has given it back to the OS (and has volunteered to take the performance consequences of getting it back), I don't think it should be counted as Chrome usage. Others may disagree. But Windows uses working set as the primary metric for all applications the OS folks agree that working set is the right way to account for memory usage. Single process browsers have a hard time giving memory back, because they can't differentiate which pages are accounted to unused, background tabs and which pages are accounted to the active, in-use tabs. Finally, the common metric which definitely doesn't work well is Windows XP's default metric: working set (private + shared). Because of shared memory between processes, simply summing the working set will far overstate the actual RAM used. Part of the motivation with Vista changing the default metric from working set to private working set was precisely to deal with the issue of better accounting of shared memory. Mike --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Sun, Jun 21, 2009 at 12:37 AM, n179911n179...@gmail.com wrote: There is a test which compares memory usage among rendering engines http://dotnetperls.com/chrome-memory From the site, it shows the maximum memory usage of Chrome is more than Safari is 2 times. From TFA: Google Chrome posted the highest maximum memory usage ***when all chrome.exe processes were summed***, reaching 1.18 gigabytes, while Firefox posted the lowest maximum memory levels of 327.65 megabytes. [Emphasis mine] Doing any memory test on a multi-process program where naive summation is used will yield inaccurate results. We wrote a blog post about this problem back when we launched in September: http://blog.chromium.org/2008/09/google-chrome-memory-usage-good-and-bad.html (I'm not claiming that Google Chrome uses the least memory; I'm just pointing out that the above experiment's methodology is invalid. Calculating actual memory consumption is really hard.) -- Elliot --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Sun, Jun 21, 2009 at 1:25 AM, Elliot Glaysher (Chromium) e...@chromium.org wrote: Google Chrome posted the highest maximum memory usage ***when all chrome.exe processes were summed***, reaching 1.18 gigabytes, while Firefox posted the lowest maximum memory levels of 327.65 megabytes. [Emphasis mine] Doing any memory test on a multi-process program where naive summation is used will yield inaccurate results. I have just mailed the article's author pointing out this flaw and mentioning both the blog post Elliot refers to and our about:memory page. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
On Sun, Jun 21, 2009 at 12:37 AM, n179911 n179...@gmail.com wrote: Hi, There is a test which compares memory usage among rendering engines http://dotnetperls.com/chrome-memory From the site, it shows the maximum memory usage of Chrome is more than Safari is 2 times. Since both of them are Webkit base, does that mean the V8 engine uses twice as much memory as squirrel fish? No. Chrome's multi-process architecture means that each renderer runs in its own Javascript heap. When you start up any browser, the JS engine will allocate some amount of memory to its heap. It will garbage collect under some policy (such as when it reaches certain size limits). In this case, chrome has 30 processes, and thus 30 javascript heaps. Since the test just hits home pages, each heap is probably fairly small. If you wanted to compare Squirrelfish to this test, you'd have to create 30 instances of it :-) Anyway, if you want to compare V8 to SquirrelFish, there are better ways to do it (you can take the browser out completely and run tests using each JS engine's command line shell). You'll find that V8's heap is very compact. The problem demonstrated in this site's test has more to do with how Chrome behaves with multiple processes than it does with Javascript. I also think that the overall performance and reliability benefits of the multi-process architecture far outweigh the memory impacts. And this test, IMHO, is not very close to a realistic test of what users do. Mike --- Maximum memory used --- Peak memory usage measured during experiment. Chrome: 1216.16 MB [Largest] Firefox: 327.65 MB [Smallest] Opera:554.11 MB Safari: 517.00 MB --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
I assume he's not a benchmark pro, but he did a decent job already. We can nitpick his sampling methodology - but it won't change the result. He is correct that many procs is far more memory consuming than single proc, and we already knew this. This is a tradeoff we made consciously and deliberately. When firefox crashes, all tabs go down. When firefox memory is compromised (security), all tabs are compromised. In chrome, we don't have those problems, but instead use more RAM. Further, Chrome is also able to implement per-tab prioritization, so that background tabs don't make foreground tabs go slow. Firefox can't do that. Lastly, lets bring the test back to reality. People don't visit 150 random home pages. They may have 20-30 tabs open, but many are applications, with cookies, javascript state and much more than just the home page. When apps are in use, the memory gap between chrome and FF shrinks a lot. Mike On Sun, Jun 21, 2009 at 10:46 AM, Dan Kegel d...@kegel.com wrote: On Sun, Jun 21, 2009 at 10:22 AM, Mike Belshembel...@google.com wrote: First off - kudos to the author for posting the source and steps to reproduce! Most don't do that! Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Wait, why doesn't his program itself do the summing? (I don't see it in there.) Wouldn't that get rid of the ambiguity? How hard would it be to add that and repost? - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
use --single-process On Jun 21, 3:37 pm, n179911 n179...@gmail.com wrote: Hi, There is a test which compares memory usage among rendering engineshttp://dotnetperls.com/chrome-memory From the site, it shows the maximum memory usage of Chrome is more than Safari is 2 times. Since both of them are Webkit base, does that mean the V8 engine uses twice as much memory as squirrel fish? --- Maximum memory used --- Peak memory usage measured during experiment. Chrome: 1216.16 MB [Largest] Firefox: 327.65 MB [Smallest] Opera: 554.11 MB Safari: 517.00 MB --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
FYI, --single-process only works in Chromium. It's disabled in Google Chrome. Search for switches::kSingleProcess in http://src.chromium.org/viewvc/chrome/trunk/src/chrome/app/chrome_dll_main.cc?revision=18801view=markup On Sun, Jun 21, 2009 at 4:22 PM, estelectronix...@gmail.com wrote: use --single-process On Jun 21, 3:37 pm, n179911 n179...@gmail.com wrote: Hi, There is a test which compares memory usage among rendering engineshttp://dotnetperls.com/chrome-memory From the site, it shows the maximum memory usage of Chrome is more than Safari is 2 times. Since both of them are Webkit base, does that mean the V8 engine uses twice as much memory as squirrel fish? --- Maximum memory used --- Peak memory usage measured during experiment. Chrome: 1216.16 MB [Largest] Firefox: 327.65 MB [Smallest] Opera: 554.11 MB Safari: 517.00 MB --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Memory usage in chrome
Really? the statistics show that many people are using the app mode?Or did you mean web apps, as in web application websites? ☆PhistucK On Sun, Jun 21, 2009 at 21:03, Mike Belshe mbel...@google.com wrote: I assume he's not a benchmark pro, but he did a decent job already. We can nitpick his sampling methodology - but it won't change the result. He is correct that many procs is far more memory consuming than single proc, and we already knew this. This is a tradeoff we made consciously and deliberately. When firefox crashes, all tabs go down. When firefox memory is compromised (security), all tabs are compromised. In chrome, we don't have those problems, but instead use more RAM. Further, Chrome is also able to implement per-tab prioritization, so that background tabs don't make foreground tabs go slow. Firefox can't do that. Lastly, lets bring the test back to reality. People don't visit 150 random home pages. They may have 20-30 tabs open, but many are applications, with cookies, javascript state and much more than just the home page. When apps are in use, the memory gap between chrome and FF shrinks a lot. Mike On Sun, Jun 21, 2009 at 10:46 AM, Dan Kegel d...@kegel.com wrote: On Sun, Jun 21, 2009 at 10:22 AM, Mike Belshembel...@google.com wrote: First off - kudos to the author for posting the source and steps to reproduce! Most don't do that! Second, the author is basically right. Since he's running on Vista, its a bit hard to tell whether his stats included shared memory or not; using the default memory statistic (Memory (Private Working Set)) is actually a pretty good measure to just sum. But he doesn't say which measurement he used. Wait, why doesn't his program itself do the summing? (I don't see it in there.) Wouldn't that get rid of the ambiguity? How hard would it be to add that and repost? - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
