A good starting point is to estimate how much space you think the data should take using e.g.
http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html If you do that, is the actual space usage close to what you expected? On Thu, Aug 29, 2013 at 5:35 PM, Kyle Hanson <hanoo...@gmail.com> wrote: > OK > > I have a bunch of BSON documents that I convert to ByteStrings, put in a > Map, and write to a socket based on the response. I noticed some high > memory usage (in the GBs) so I decided to investigate. I simplified my > problem into a small program that demonstrates clearer what is happening. > > I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one > with a Strict Map and Strict ByteStrings. Both share the same memory > behavior (except the lazy BS one is faster) > > Here is the strict version: > > http://lpaste.net/92298 > > And here is the lazy version: > > http://lpaste.net/92299 > > I wrote this and compared the memory and speed behavior of ByteStrings > generated by converting it from a BSON document and ByteStrings generated > more purely. > > The length of the ByteString from a BSON document is 68k and the length of > the "pure" BS is 70k. > > This is my weird memory behavior, both BSON and "pure" methods use the > same amount of memory after inserting 10k of them (90mb) > > However when I go to lookup a value, the BSON Map explodes the memory to > over 250mb. Even if I lookup just 1 value. Looking up any number of values > in the "pure BS" keeps the memory usage stable (90mb). > > I am hoping someone can help me understand this. I have read some posts > about Temporary ByteStrings causing memory issues but I don't know how to > get started debugging. > > -- > Kyle Hanson > > _______________________________________________ > Haskell-Cafe mailing list > Haskell-Cafe@haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe > >
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe