Re: [Haskell-cafe] Memory-aware Haskell?
On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka r...@ro-che.info wrote: So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection This is already implemented. See the -M RTS option. 2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size For example, by un-evaluating them. It would often be possible to reduce size like that, just so long as they aren't evaluated again the next second. Well, this one would be hard. 3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user. That should be relative simple. Get implementing, will you?:D (And there still is a problem of foreign code whose memory consumption we know nothing about...) In theory, you could deal with that by hooking malloc. -- Svein Ove Aas ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Memory-aware Haskell?
On Fri, Dec 25, 2009 at 5:14 AM, Svein Ove Aas svein@aas.no wrote: On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka r...@ro-che.info wrote: So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection This is already implemented. See the -M RTS option. Correct me if I'm wrong - it's been a while since I tried to use the -M option to deal with Gitit memory usage on darcs.net - but doesn't -M just kill the program after it reaches a set RAM, and doesn't trigger any GCs? -- gwern ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Memory-aware Haskell?
Am Freitag 25 Dezember 2009 15:45:29 schrieb Gwern Branwen: On Fri, Dec 25, 2009 at 5:14 AM, Svein Ove Aas svein@aas.no wrote: On Thu, Dec 24, 2009 at 11:38 PM, Roman Cheplyaka r...@ro-che.info wrote: So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection This is already implemented. See the -M RTS option. Correct me if I'm wrong - it's been a while since I tried to use the -M option to deal with Gitit memory usage on darcs.net - but doesn't -M just kill the program after it reaches a set RAM, and doesn't trigger any GCs? It does affect GC behaviour. According to the user's guide: -Msize [Default: unlimited] Set the maximum heap size to size bytes. The heap normally grows and shrinks according to the memory requirements of the program. The only reason for having this option is to stop the heap growing without bound and filling up all the available swap space, which at the least will result in the program being summarily killed by the operating system. The maximum heap size also affects other garbage collection parameters: when the amount of live data in the heap exceeds a certain fraction of the maximum heap size, compacting collection will be automatically enabled for the oldest generation, and the -F parameter will be reduced in order to avoid exceeding the maximum heap size. Reducing the -F parameter (amount of memory reserved for the older generations as a factor of the amount of live data) triggers more GCs. So it tries to not exceed the limit you've given. You can indeed reduce the memory used by your programme via the -M option, but only to some extent. As far as I know it will not collect data held on to by live data to be reconstructed later if needed again. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Memory-aware Haskell?
Imagine some system with hard memory bound (e.g. 64M of physical memory, no swap). I don't want some accidental laziness (maybe even not in my code, but in some used package) to crash my program. So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection 2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size 3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user. (And there still is a problem of foreign code whose memory consumption we know nothing about...) What do you think? -- Roman I. Cheplyaka :: http://ro-che.info/ Don't let school get in the way of your education. - Mark Twain ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Memory-aware Haskell?
This is a problem with partitioned operating systems used in avionics. The airplane computers require certain partitions to exist between programs in both time and space. The space guarantees are most easily enforced by eliminating any dynamic memory allocation once the operating system enters a 'normal mode'. This means that all memory needs to be allocated on the stack and ahead of time. This reduces the calculation of the upper memory bound to checking what the largest possible stack a process could grow to is. Of course, calculating this maximum stack is difficult in its own right. Unfortunately, nothing I've just discussed seems to port well to Haskell (I could be wrong!). Given this, I think you're looking at a very hard problem (but one I'd be incredibly interested in discovering a solution for!). /jve On Thu, Dec 24, 2009 at 5:38 PM, Roman Cheplyaka r...@ro-che.info wrote: Imagine some system with hard memory bound (e.g. 64M of physical memory, no swap). I don't want some accidental laziness (maybe even not in my code, but in some used package) to crash my program. So, let's think what we can do at runtime. Suppose RTS takes the parameter -- upper limit of consumed memory. When it sees that memory consumption is close to upper bound, it can: 1. force garbage collection 2. apply some heuristics to find and reduce some chunks which will benefit from reduction in terms of size 3. if nothing helps, throw an exeption. It can be caught in IO and memory-aware program can make apropriate decision -- e.g. abort opening a large file and gracefully warn the user. (And there still is a problem of foreign code whose memory consumption we know nothing about...) What do you think? -- Roman I. Cheplyaka :: http://ro-che.info/ Don't let school get in the way of your education. - Mark Twain ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
