[chromium-dev] Re: Temporarily disable tcmalloc in build
We need a better way to talk about this perf gain. I agree is 12% ops/ second in that particular set of benchmarks. My recollection is that we removed LFH because it was using too much memory. We need some form of normalized score based on memory usage. In other words 12% with 25% more memory usage might not be in practice a gain at all in machines with 1GB. I do agree that with all other things equal Vista is faster than XP and the memory usage of both is quite different as well. We have lots of diagnostics. How do we use them? is there a document? I am getting conflicting information here. The problem we're having here is a crash which is undetected by all tools with *or without* tcmalloc. Neither purify nor page heap have been able to detect this problem so far. I am hearing the opposite here as well. Some heap corruption problems are hard to track - but blaming tcmalloc for them doesn't make any sense. No, no, tcmalloc is awesome and certainly the bugs exist with out without it. We need purify or purify like, or valgrind like runs at check-in time. That is what I don't like. Please set me straight if this is incorrect. The heap-checker tools you mention can probably be enabled with some amount of work - it's just work. There are other portions of tcmalloc which need still to be ported to windows. Whether they would help with locating this particular crash or not - I don't know. Given that none of our other tools are helping Not sure about that. Lets wait for the outcome of issues that purify found this time. I am hearing that we got at least one good lead on a heap corruption. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Temporarily disable tcmalloc in build
On Sat, Aug 1, 2009 at 10:53 PM, Huan Ren hu...@google.com wrote: - The crash rates on chromebot are comparable for builds with and without tcmalloc. The crash rate for tcmalloc build is higher but it seems within error margin. - Wtih tcmalloc disabled, there are some promising data from Purify test run and ChromeBot run with full page heap enabled. These are still under further investigation. BTW, all the server guys have tcmalloc heap profiling enabled by default; they can pinpoint leaks at process exit by default on their servers (when I saw the tool recently, I was very impressed!) Jim and Will investigated with CraigS - turns out that this portion of tcmalloc isn't in the public-domain version of tcmalloc which we use. If we help Craig get this part of tcmalloc open sourced, we can get much of purify in 'by default'. - The performance results are same as last time we compared. With tcmalloc, the most perf gain is from page-cycler-moz: 15% improvement (with 25% more memory usage). The perfor gain on page-cycler-more-js is 4%, as we see tcmalloc has no perf impact on V8 benchmark. The performance gain on DOM is significant: from score 325 to 414. I thought it was also interesting to look at the tcmalloc benefits on XP vs Vista (we had observed this before too). It's clear that Microsoft substantially improved the heap on Vista - while the gains on XP were 8-9% for the intl page cyclers on XP, the gains were only 1-2% on Vista. Similarly for the DOM and other tests, Vista gains for tcmalloc are always substantially less than on XP. We might want to look more closely at Win7; I don't think the heap changed much from Vista to Win7, but maybe eventually tcmalloc won't be interesting. Or - if we can reduce webkit heap usage, maybe we can make it so that tcmalloc on vista doesn't improve anything. Mike r22251 re-enables tcmalloc on trunk. We can merge 22080 to dev branch if needed. Huan On Fri, Jul 31, 2009 at 6:51 PM, cpuc...@chromium.org wrote: What are the results of this experiment? On Jul 30, 12:15 pm, Huan Ren hu...@chromium.org wrote: I just submitted a change (22080) that disables tcmalloc used on Windows platform. The plan is keeping it in trunk for 24 hours and then reverting it. The intentions are - Having another round of performance comparison between build with and w/o tcmalloc. - Having a full run of UI test under purify with tcmalloc disabled. - Getting a verified CL in case we'd like to build an alternative dev build w/o tcmalloc for A/B test. As a head up, the performance, stability, and purify test results could be different during the period. Huan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Temporarily disable tcmalloc in build
On Sun, Aug 2, 2009 at 11:12 AM, Carlos Pizano c...@google.com wrote: On Sun, Aug 2, 2009 at 10:43 AM, Mike Belshe mbel...@google.com wrote: On Sat, Aug 1, 2009 at 10:53 PM, Huan Ren hu...@google.com wrote: - The crash rates on chromebot are comparable for builds with and without tcmalloc. The crash rate for tcmalloc build is higher but it seems within error margin. - Wtih tcmalloc disabled, there are some promising data from Purify test run and ChromeBot run with full page heap enabled. These are still under further investigation. BTW, all the server guys have tcmalloc heap profiling enabled by default; they can pinpoint leaks at process exit by default on their servers (when I saw the tool recently, I was very impressed!) Jim and Will investigated with CraigS - turns out that this portion of tcmalloc isn't in the public-domain version of tcmalloc which we use. If we help Craig get this part of tcmalloc open sourced, we can get much of purify in 'by default'. - The performance results are same as last time we compared. With tcmalloc, the most perf gain is from page-cycler-moz: 15% improvement (with 25% more memory usage). The perfor gain on page-cycler-more-js is 4%, as we see tcmalloc has no perf impact on V8 benchmark. The performance gain on DOM is significant: from score 325 to 414. I thought it was also interesting to look at the tcmalloc benefits on XP vs Vista (we had observed this before too). It's clear that Microsoft substantially improved the heap on Vista - while the gains on XP were 8-9% for the intl page cyclers on XP, the gains were only 1-2% on Vista. Similarly for the DOM and other tests, Vista gains for tcmalloc are always substantially less than on XP. I have said this several times. Here it goes again. You can get the Vista heap goodness in XP + SP2. It is the LFH option which was (or is) in our code already. Both the extra speed boost and the extra memory bloat. The tricks are the same; we don't have a monopoly on cool heap tricks. Carlos - I believe your claims about LFH are false. Microsoft has documented Vista-specific heap improvements above and beyond using LFH for SMP scaling. LFH was tested. But it is slower than tcmalloc by ~12%. Here are the results for default heaps, lfh heaps, and tcmalloc heaps on XPSP2: http://dom-perf-backend.prom.corp.google.com/compare?ids=1058004,1056005,1058002 tcmalloc also has other benefits: - it's portable to other platforms - we have full source and can improve it The only reason I haven't opposed more strongly tcmalloc is because I hope it also helps the other platforms be fast. The thing I did not know and that floors me is that since then we have no diagnostics. I did know that we lost pageheap but I assumed all along we had the diagnostics of this page ( http://goog-perftools.sourceforge.net/doc/heap_checker.html). My current understanding is we have none. We have lots of diagnostics. The problem we're having here is a crash which is undetected by all tools with *or without* tcmalloc. Neither purify nor page heap have been able to detect this problem so far. Some heap corruption problems are hard to track - but blaming tcmalloc for them doesn't make any sense. The heap-checker tools you mention can probably be enabled with some amount of work - it's just work. There are other portions of tcmalloc which need still to be ported to windows. Whether they would help with locating this particular crash or not - I don't know. Given that none of our other tools are helping - I'm guessing the problem is that we're not testing the right things - not that our diagnostics don't work. I don't know what decision process we go for choosing the allocator but lack of diagnostics is a deal breaker to me. I hope we can go back to LFH on windows until such time the diagnostics are there. Maybe it is too much of a pain to have this difference. If you have specific examples of where tcmalloc has been hurting us on this front, please send me the examples. So far, while we've hoped that turning off tcmalloc would help with our diagnostics - it hasn't. We also run many of our tests without tcmalloc - so we're now getting the benefit of having different allocators test our code. I think this is good. Don't take this as a conclusion on my part that tcmalloc is perfect - it needs lots more work, and I'm certain we can still make huge memory improvements by continuing work on our allocator. We can also add more diagnostics support. But at this point it is faster and has not sacrificed debugging tools. Mike We might want to look more closely at Win7; I don't think the heap changed much from Vista to Win7, but maybe eventually tcmalloc won't be interesting. Or - if we can reduce webkit heap usage, maybe we can make it so that tcmalloc on vista doesn't improve anything. Mike r22251 re-enables tcmalloc on trunk. We can merge 22080 to dev branch if needed.
[chromium-dev] Re: Temporarily disable tcmalloc in build
What are the results of this experiment? On Jul 30, 12:15 pm, Huan Ren hu...@chromium.org wrote: I just submitted a change (22080) that disables tcmalloc used on Windows platform. The plan is keeping it in trunk for 24 hours and then reverting it. The intentions are - Having another round of performance comparison between build with and w/o tcmalloc. - Having a full run of UI test under purify with tcmalloc disabled. - Getting a verified CL in case we'd like to build an alternative dev build w/o tcmalloc for A/B test. As a head up, the performance, stability, and purify test results could be different during the period. Huan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
