Re: Virtual Memory fragmentation issues
This is one of the problems Heap Profiler was written to solve: https://github.com/kevingadd/HeapProfiler It enables stack trace captures at allocation time, so at any given point in the capture you can open the heap and see who allocated a given block of memory. It can also display a fragmentation summary. It would probably be trivial to extend it to do a fragmentation analysis and generate a summary of who's causing the most fragmentation (by module or function, most likely). See http://www.luminance.org/blog/code/2011/06/03/diving-into-memory-usage-with-heap-profilerand http://www.luminance.org/blog/code/2011/08/01/a-journey-into-linker-hell-and-a-mistakefor some examples of usage. On Mon, Apr 8, 2013 at 11:39 PM, Mike Hommey m...@glandium.org wrote: On Mon, Apr 08, 2013 at 08:09:08PM -0400, Justin Lebar wrote: AIUI, on Windows the smallest block you can ask for with VirtualAlloc is 4 KiB. However, no more than one VirtualAlloc block can exist per 64 KiB chunk. So if you ask for 4 KiB you'll end up wasting the remaining 60 KiB of address space in the 64 KiB chunk. Awesome memory, Nick. MSDN seems to confirm this. The second parameter [to VirtualAlloc] indicates the range of addresses the function should allocate. This value can be anywhere from one page to 2 GB in size, but VirtualAlloc is actually constrained to a smaller range than that. The minimum size that can be reserved is 64K, and the maximum that can be reserved is the largest contiguous range of free addresses in the process. Requesting one page of reserved addresses results in a 64K address range. http://msdn.microsoft.com/en-us/library/ms810627.aspx We have the ability to intercept VirtualAlloc calls on Windows, so assuming that works for these particular VirtualAlloc calls (it should?) perhaps we can take backtraces and write them to a file. We're already intercepting VirtualAlloc in xpcom/base/AvailableMemoryTracker.cpp Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform -- -kg ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
On 4/9/13 7:12 AM, Kevin Gadd wrote: This is one of the problems Heap Profiler was written to solve: https://github.com/kevingadd/HeapProfiler It enables stack trace captures at allocation time, so at any given point in the capture you can open the heap and see who allocated a given block of memory. It can also display a fragmentation summary. It would probably be trivial to extend it to do a fragmentation analysis and generate a summary of who's causing the most fragmentation (by module or function, most likely). As far as I understand it, this tool profiles heap allocations like malloc. I am not concerned with those allocations, but only with page allocations. Can this tool intercept calls to VirtualAlloc and especially MapViewOfFile so that we can see the stack at the point the large memory mappings are created? Note that although it's interesting that we're fragmenting the page because of the 4k/64k behavior, I am much less concerned with that behavior than I am with the leaked memory mappings. --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
I see, so the hypothesis is that 100% of the fragmentation is coming from VirtualAlloc/MapViewOfFile, not from allocations in general? I believe the kernel instrumentation tools could give you data on this but it would probably require modifications to the profiler tool, so maybe it can't be used to answer questions about 'who allocated this'. It does appear that MS offers a utility for this, though, called LeakDiag: http://blogs.jetbrains.com/yole/archives/34.html It appears to instrument VirtualAlloc and record tracebacks. Maybe worth a look. On Tue, Apr 9, 2013 at 6:07 AM, Benjamin Smedberg benja...@smedbergs.uswrote: On 4/9/13 7:12 AM, Kevin Gadd wrote: This is one of the problems Heap Profiler was written to solve: https://github.com/kevingadd/**HeapProfilerhttps://github.com/kevingadd/HeapProfiler It enables stack trace captures at allocation time, so at any given point in the capture you can open the heap and see who allocated a given block of memory. It can also display a fragmentation summary. It would probably be trivial to extend it to do a fragmentation analysis and generate a summary of who's causing the most fragmentation (by module or function, most likely). As far as I understand it, this tool profiles heap allocations like malloc. I am not concerned with those allocations, but only with page allocations. Can this tool intercept calls to VirtualAlloc and especially MapViewOfFile so that we can see the stack at the point the large memory mappings are created? Note that although it's interesting that we're fragmenting the page because of the 4k/64k behavior, I am much less concerned with that behavior than I am with the leaked memory mappings. --BDS -- -kg ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
I see, so the hypothesis is that 100% of the fragmentation is coming from VirtualAlloc/MapViewOfFile, not from allocations in general? jemalloc does not make 4kb allocations, I think ever. So yes. On Tue, Apr 9, 2013 at 9:23 AM, Kevin Gadd kevin.g...@gmail.com wrote: I see, so the hypothesis is that 100% of the fragmentation is coming from VirtualAlloc/MapViewOfFile, not from allocations in general? I believe the kernel instrumentation tools could give you data on this but it would probably require modifications to the profiler tool, so maybe it can't be used to answer questions about 'who allocated this'. It does appear that MS offers a utility for this, though, called LeakDiag: http://blogs.jetbrains.com/yole/archives/34.html It appears to instrument VirtualAlloc and record tracebacks. Maybe worth a look. On Tue, Apr 9, 2013 at 6:07 AM, Benjamin Smedberg benja...@smedbergs.us wrote: On 4/9/13 7:12 AM, Kevin Gadd wrote: This is one of the problems Heap Profiler was written to solve: https://github.com/kevingadd/HeapProfiler It enables stack trace captures at allocation time, so at any given point in the capture you can open the heap and see who allocated a given block of memory. It can also display a fragmentation summary. It would probably be trivial to extend it to do a fragmentation analysis and generate a summary of who's causing the most fragmentation (by module or function, most likely). As far as I understand it, this tool profiles heap allocations like malloc. I am not concerned with those allocations, but only with page allocations. Can this tool intercept calls to VirtualAlloc and especially MapViewOfFile so that we can see the stack at the point the large memory mappings are created? Note that although it's interesting that we're fragmenting the page because of the 4k/64k behavior, I am much less concerned with that behavior than I am with the leaked memory mappings. --BDS -- -kg ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
On 2013-04-08, at 7:46 PM, Benjamin Smedberg wrote: In stability-land we're starting to see some interesting problems relating to virtual memory usage in Firefox on Windows. Either our code or the ATI driver is leaking mapped memory references in a way that chews up VM space without actually using memory. We need to figure out why and fix our work around the problem. I'm hoping that one of you on the To of this email may be able to help figure out a way to either patch Firefox or run an external debugging tool which can help record information about the stacks when these mappings are typically created and destroyed, and help create a detailed understanding of the problem so that we can either fix our code or hand the problem to the nvidia developers with the most reproducible/detailed report possible One interesting phenomenon that I have seen is pages being mysteriously added to firefox's address space. I believe this can happen when graphics memory is paged into system memory and is added to our address space. I'm not sure what OS accounting all knows about this and what doesn't. -Jeff ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
On 4/8/2013 7:46 PM, Benjamin Smedberg wrote: In this case, the crash report shows Available Virtual Memory as 303558656, but in fact the largest contiguous block of available VM is 64k. Correct, I had wrong math. The largest block available is about 3MB, and the allocation being requested from graphics is 9MB. --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
On Mon, Apr 8, 2013 at 4:46 PM, Benjamin Smedberg benja...@smedbergs.us wrote: 1) early in the memory info, there appears to be a common pattern of a committed block and then 15 free blocks. AIUI, on Windows the smallest block you can ask for with VirtualAlloc is 4 KiB. However, no more than one VirtualAlloc block can exist per 64 KiB chunk. So if you ask for 4 KiB you'll end up wasting the remaining 60 KiB of address space in the 64 KiB chunk. Assuming I'm right about this (I'm fairly certain, but not 100%), it sounds like some code (be it Mozilla or driver code) is VirtualAlloc'ing lots of 4 KiB blocks, and thus wasting lots of address space. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Virtual Memory fragmentation issues
AIUI, on Windows the smallest block you can ask for with VirtualAlloc is 4 KiB. However, no more than one VirtualAlloc block can exist per 64 KiB chunk. So if you ask for 4 KiB you'll end up wasting the remaining 60 KiB of address space in the 64 KiB chunk. Awesome memory, Nick. MSDN seems to confirm this. The second parameter [to VirtualAlloc] indicates the range of addresses the function should allocate. This value can be anywhere from one page to 2 GB in size, but VirtualAlloc is actually constrained to a smaller range than that. The minimum size that can be reserved is 64K, and the maximum that can be reserved is the largest contiguous range of free addresses in the process. Requesting one page of reserved addresses results in a 64K address range. http://msdn.microsoft.com/en-us/library/ms810627.aspx We have the ability to intercept VirtualAlloc calls on Windows, so assuming that works for these particular VirtualAlloc calls (it should?) perhaps we can take backtraces and write them to a file. On Mon, Apr 8, 2013 at 7:57 PM, Nicholas Nethercote n.netherc...@gmail.com wrote: On Mon, Apr 8, 2013 at 4:46 PM, Benjamin Smedberg benja...@smedbergs.us wrote: 1) early in the memory info, there appears to be a common pattern of a committed block and then 15 free blocks. AIUI, on Windows the smallest block you can ask for with VirtualAlloc is 4 KiB. However, no more than one VirtualAlloc block can exist per 64 KiB chunk. So if you ask for 4 KiB you'll end up wasting the remaining 60 KiB of address space in the 64 KiB chunk. Assuming I'm right about this (I'm fairly certain, but not 100%), it sounds like some code (be it Mozilla or driver code) is VirtualAlloc'ing lots of 4 KiB blocks, and thus wasting lots of address space. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
