On 12/13/2013 09:10 AM, Zé Loff wrote:
On Fri, Dec 13, 2013 at 02:44:26PM +0100, Peter Hessler wrote:
On 2013 Dec 13 (Fri) at 13:24:41 +0000 (+0000), Zé Loff wrote:
:On Fri, Dec 13, 2013 at 07:16:06AM -0600, Shawn K. Quinn wrote:
:>
:> I think R is using virtual memory as best it can, and I seriously doubt
:> you will get anything resembling satisfactory performance without
:> upgrading the RAM (memory) to 8Gb.
[snip]
:> So: 1) segment this problem such that R never needs more than about 3Gb
:> of RAM in one run if possible, 2) upgrade the RAM, or 3) give R a very
:> long time to complete the task at hand and back up your hard disk
:> regularly because it will get a workout.
[snip]
Using swap is a bug. Buy more ram.
Thanks for your answers (and Marc's too, BTW). I never meant swapping to
be more than a workaround, I wasn't expecting good performance. But I
never expected it to render the machine virtually useless like it does,
hence the first post. Off to the shop, then.
swap is intended for things that are not currently being used much to be
pushed out of the way "for now" until they are needed again, presumably
much later (relatively speaking)
It works great (relatively) when you have lots of stuff loaded and
running but are using only little parts at a time, when you can dump a
big chunk of unused RAM to disk, and bring in a big chunk of now desired
data from disk into RAM.
That's not what you are doing.
You have ONE application which is using huge amounts of data, that it is
thrashing all over. Odds are, if it was able to chunk the data up so it
could work on one little part, then another little part, then another
little part, 1) it would probably work great for you. 2) it would
probably just do this, keeping most of the data on disk, rather than
sucking it all into RAM.
If your app wants one "number" off something that is swapped out, it has
to bring in the whole swapped out page just to read or write that one value.
You are running into the fact that memory is accessed on the order of
nanoseconds, and disk is accessed on the order of milliseconds, TIMES
the fact that any one location in RAM can be accessed almost as quickly
as any other location in RAM, but to get data swapped to disk requires a
painfully slow swap process of (relatively) huge blocks of data.
you could be looking at million-to-one performance ratio here.
Something that could run in a minute in RAM might run for years in swap
(that messes up your upgrade plans :).
Your application is a textbook example of "When swap fails". OpenBSD
might be able to manage its swap use better, but nothing will save you
from what you are trying to do. (well... ok, long, long ago... I've
seen some mainframes which, after you hit their physical RAM limits
(16MB, iirc), swapped to ... huge (for the day) RAM disks. But even
then, the act of swapping big pages of data out to get access to
individal values of data would be several orders of magnitude slower
than a direct RAM access),
Nick.
