On 21/09/2012 04:07, John Lato wrote:
Yes, that's my current understanding. I see this with ByteString and
Data.Vector.Storable, but not
Data.Vector/Data.Vector.Unboxed/Data.Text. As ByteStrings are pretty
widely used for IO, I expected that somebody else would have
experienced this too.
I would expect some memory fragmentation with pinned memory, but the
change from ghc-7.4 to ghc-7.6 is rather extreme (no fragmentation to
several GB).
This was a side-effect of the improvements we made to the allocation of
pinned objects, which ironically was made to avoid fragmentation of a
different kind. What is happening is that the memory for the pinned
objects is now taken from the nursery, and so the nursery has to be
replenished after GC. When we allocate memory for the nursery we like
to allocate it in big contiguous chunks, because that works better with
automatic prefecthing, but the memory is horribly fragmented due to all
the pinned objects, so the large allocation has to be satisfied from the OS.
The fix is not to allocate large chunks for the nursery unless there are
no small chunks to use up, so I've implemented that.
Happily I also found two other bugs while looking for this one, one of
which was a performance bug which caused this benchmark to run 10x
slower than it should have been! The other bug was a recent regression
causing it to misreport the amount of allocated memory.
Thanks for the report.
Cheers,
Simon
John L.
On Fri, Sep 21, 2012 at 10:53 AM, Carter Schonwald
<[email protected]> wrote:
So the problem is only with the data structures on the heap that are pinned
in place to play nice with C?
I'd be curious to understand the change too, though per se pinned memory (a
la storable or or bytestring) will by definition cause memory fragmentation
in a gc'd lang as a rule, (or at least one like Haskell).
-Carter
On Thu, Sep 20, 2012 at 8:59 PM, John Lato <[email protected]> wrote:
Hello,
We've noticed that some applications exhibit significantly worse
memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading
to out of memory errors in some cases. Running one app with +RTS -s,
I see this:
ghc-7.4
525,451,699,736 bytes allocated in the heap
53,404,833,048 bytes copied during GC
39,097,600 bytes maximum residency (2439 sample(s))
1,547,040 bytes maximum slop
628 MB total memory in use (0 MB lost due to fragmentation)
ghc-7.6
512,535,907,752 bytes allocated in the heap
53,327,184,712 bytes copied during GC
40,038,584 bytes maximum residency (2391 sample(s))
1,456,472 bytes maximum slop
3414 MB total memory in use (2744 MB lost due to
fragmentation)
The total memory in use (consistent with 'top's output) is much higher
when built with ghc-7.6, due entirely to fragmentation.
I've filed a bug report
(http://hackage.haskell.org/trac/ghc/ticket/7257,
http://hpaste.org/74987), but I was wondering if anyone else has
noticed this? I'm not entirely sure what's triggering this behavior
(some applications work fine), although I suspect it has to do with
allocation of pinned memory.
John L.
_______________________________________________
Glasgow-haskell-users mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
_______________________________________________
Glasgow-haskell-users mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
_______________________________________________
Glasgow-haskell-users mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
