Re: [Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-11 Thread Johan Tibell
On Fri, Jun 8, 2012 at 1:40 PM, Johan Tibell wrote: > GHC used to complain when you use UNPACK with something that can't be > unpacked, but that warning seems to have been (accidentally) removed > in 7.4.1. Turns out the warning is only on if you compile with -O or higher. -- Johan

Re: [Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-08 Thread Andrew Myers
Thanks for the responses everyone, I'll try them out and see what happens :) Andrew On Fri, Jun 8, 2012 at 4:40 PM, Johan Tibell wrote: > Hi Andrew, > > On Thu, Jun 7, 2012 at 5:39 PM, Andrew Myers wrote: > > Hi Cafe, > > I'm working on inspecting some data that I'm trying to represent as > rec

Re: [Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-08 Thread Johan Tibell
Hi Andrew, On Thu, Jun 7, 2012 at 5:39 PM, Andrew Myers wrote: > Hi Cafe, > I'm working on inspecting some data that I'm trying to represent as records > in Haskell and seeing about twice the memory footprint than I was > expecting.  I've got roughly 1.4 million records in a CSV file (400M on > d

Re: [Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-08 Thread Thomas Schilling
On 8 June 2012 01:39, Andrew Myers wrote: > Hi Cafe, > I'm working on inspecting some data that I'm trying to represent as records > in Haskell and seeing about twice the memory footprint than I was > expecting. That is to be expected in a garbage-collected language. If your program requires X by

Re: [Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-07 Thread Roman Cheplyaka
* Andrew Myers [2012-06-07 20:39:50-0400] > I've written a small driver test program that just parses the CSV, finds > the minimum value for a couple of the Float fields, and exits. In the > process monitor the memory usage is 6.9G before the program exits. I've > tried profiling with +RTS -hc b

[Haskell-cafe] High memory usage with 1.4 Million records?

2012-06-07 Thread Andrew Myers
Hi Cafe, I'm working on inspecting some data that I'm trying to represent as records in Haskell and seeing about twice the memory footprint than I was expecting. I've got roughly 1.4 million records in a CSV file (400M on disk) that I parse in using bytestring-csv. bytestring-csv returns a [[Byte