[R] Memory in R on windows running 64bit XP
Hi I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states: If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB. So, using the help which states the size parameter can go up to 4095: memory.limit(size=4095) When I run mclust I get: Error: cannot allocate vector of size 1.3 Gb So, now I set the max-mem-size flag: --max-mem-size=3500M Repeat and I still get the same error: Error: cannot allocate vector of size 1.3 Gb So, can anyone help? 32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb. sessionInfo() R version 2.9.2 (2009-08-24) i386-pc-mingw32 locale: LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] mclust_3.3.1 Thanks Mick __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Memory in R on windows running 64bit XP
On 9/16/2009 10:16 AM, michael watson (IAH-C) wrote: Hi I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states: If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB. So, using the help which states the size parameter can go up to 4095: memory.limit(size=4095) When I run mclust I get: Error: cannot allocate vector of size 1.3 Gb So, now I set the max-mem-size flag: --max-mem-size=3500M Repeat and I still get the same error: Error: cannot allocate vector of size 1.3 Gb So, can anyone help? 32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb. You are misreading things in a couple of places. The most important one is the error message: it says allocation of a 1.3 Gb object failed. You may have 4 Gb available in total, but have used so much of it already that Windows won't allocate a 1.3 Gb piece to R. Using Rprofmem() might show how the allocations are happening. Another possibility is that your particular version of Windows doesn't support giving 4 Gb to R. Run memory.size(NA) to find out the limit, memory.size() to find out the current amount in use. And even if the limit is close to 4 Gb and you have more than 1.3 Gb free, the allocation may fail because there is no sufficiently large contiguous block available. Once memory is allocated, that address is in use and is unavailable. Since the 32 bit address space only covers 4 Gb, you can fairly easily end up with allocations that are spread out across it leaving no large blocks available. That's the advantage of switching to 64 bit R: even if you still only had 4 Gb of memory available, the address space is so much bigger that fragmentation probably won't be a problem. Duncan Murdoch sessionInfo() R version 2.9.2 (2009-08-24) i386-pc-mingw32 locale: LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] mclust_3.3.1 Thanks Mick __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Memory in R on windows running 64bit XP
Hi Duncan Many thanks for the reply. Here are the results of the commands: Rprofmem() Error in Rprofmem() : memory profiling is not available on this system memory.size(NA) [1] 3500 memory.size() [1] 16.19 Clearly Rprofmem() is a dead end, though if there is some way of enabling it, I will be happy to. I am only actually using 16.19Mb of memory, as all I do is open R and read in some data (a 2.7Mb text file). I am not sure how to guard against the memory fragmentation problem. The machine has 16Gb of RAM and I am only running R and Internet Explorer, but I don't know if that helps. Is there a way to deliver contiguous memory to R? Is 64bit R available for Windows? Sorry to bang on about Windows, I'm waiting for my IT department to build Linux machines, and this is what I have in the meantime. Thanks Mick From: Duncan Murdoch [murd...@stats.uwo.ca] Sent: 16 September 2009 15:43 To: michael watson (IAH-C) Cc: r-help@r-project.org Subject: Re: [R] Memory in R on windows running 64bit XP On 9/16/2009 10:16 AM, michael watson (IAH-C) wrote: Hi I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states: If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB. So, using the help which states the size parameter can go up to 4095: memory.limit(size=4095) When I run mclust I get: Error: cannot allocate vector of size 1.3 Gb So, now I set the max-mem-size flag: --max-mem-size=3500M Repeat and I still get the same error: Error: cannot allocate vector of size 1.3 Gb So, can anyone help? 32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb. You are misreading things in a couple of places. The most important one is the error message: it says allocation of a 1.3 Gb object failed. You may have 4 Gb available in total, but have used so much of it already that Windows won't allocate a 1.3 Gb piece to R. Using Rprofmem() might show how the allocations are happening. Another possibility is that your particular version of Windows doesn't support giving 4 Gb to R. Run memory.size(NA) to find out the limit, memory.size() to find out the current amount in use. And even if the limit is close to 4 Gb and you have more than 1.3 Gb free, the allocation may fail because there is no sufficiently large contiguous block available. Once memory is allocated, that address is in use and is unavailable. Since the 32 bit address space only covers 4 Gb, you can fairly easily end up with allocations that are spread out across it leaving no large blocks available. That's the advantage of switching to 64 bit R: even if you still only had 4 Gb of memory available, the address space is so much bigger that fragmentation probably won't be a problem. Duncan Murdoch sessionInfo() R version 2.9.2 (2009-08-24) i386-pc-mingw32 locale: LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] mclust_3.3.1 Thanks Mick __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Memory in R on windows running 64bit XP
On 9/16/2009 10:54 AM, michael watson (IAH-C) wrote: Hi Duncan Many thanks for the reply. Here are the results of the commands: Rprofmem() Error in Rprofmem() : memory profiling is not available on this system memory.size(NA) [1] 3500 memory.size() [1] 16.19 Clearly Rprofmem() is a dead end, though if there is some way of enabling it, I will be happy to. I forgot that you need to recompile R to enable it. Not impossible, but probably not worth the trouble just for this problem. Using gcinfo(TRUE) may give you enough information: it will report whenever there's a garbage collection. I am only actually using 16.19Mb of memory, as all I do is open R and read in some data (a 2.7Mb text file). I would guess that mclust does a lot of other allocations before it fails. I am not sure how to guard against the memory fragmentation problem. The machine has 16Gb of RAM and I am only running R and Internet Explorer, but I don't know if that helps. Is there a way to deliver contiguous memory to R? R doesn't give you any real control over that. I would guess that R is getting 3.5 Gb in a contiguous chunk based on your memory.size report, but then it is fragmenting it. Is 64bit R available for Windows? Sorry to bang on about Windows, I'm waiting for my IT department to build Linux machines, and this is what I have in the meantime. Revolution Computing sells one. There's no (reliable) 64 bit version of gcc for Windows, so we don't distribute one. Duncan Murdoch Thanks Mick From: Duncan Murdoch [murd...@stats.uwo.ca] Sent: 16 September 2009 15:43 To: michael watson (IAH-C) Cc: r-help@r-project.org Subject: Re: [R] Memory in R on windows running 64bit XP On 9/16/2009 10:16 AM, michael watson (IAH-C) wrote: Hi I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states: If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB. So, using the help which states the size parameter can go up to 4095: memory.limit(size=4095) When I run mclust I get: Error: cannot allocate vector of size 1.3 Gb So, now I set the max-mem-size flag: --max-mem-size=3500M Repeat and I still get the same error: Error: cannot allocate vector of size 1.3 Gb So, can anyone help? 32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb. You are misreading things in a couple of places. The most important one is the error message: it says allocation of a 1.3 Gb object failed. You may have 4 Gb available in total, but have used so much of it already that Windows won't allocate a 1.3 Gb piece to R. Using Rprofmem() might show how the allocations are happening. Another possibility is that your particular version of Windows doesn't support giving 4 Gb to R. Run memory.size(NA) to find out the limit, memory.size() to find out the current amount in use. And even if the limit is close to 4 Gb and you have more than 1.3 Gb free, the allocation may fail because there is no sufficiently large contiguous block available. Once memory is allocated, that address is in use and is unavailable. Since the 32 bit address space only covers 4 Gb, you can fairly easily end up with allocations that are spread out across it leaving no large blocks available. That's the advantage of switching to 64 bit R: even if you still only had 4 Gb of memory available, the address space is so much bigger that fragmentation probably won't be a problem. Duncan Murdoch sessionInfo() R version 2.9.2 (2009-08-24) i386-pc-mingw32 locale: LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] mclust_3.3.1 Thanks Mick __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Memory in R on windows running 64bit XP
Fair enough, thanks for your time. I will wait for my Linux PCs! -Original Message- From: Duncan Murdoch [mailto:murd...@stats.uwo.ca] Sent: 16 September 2009 16:42 To: michael watson (IAH-C) Cc: r-help@r-project.org Subject: Re: [R] Memory in R on windows running 64bit XP On 9/16/2009 10:54 AM, michael watson (IAH-C) wrote: Hi Duncan Many thanks for the reply. Here are the results of the commands: Rprofmem() Error in Rprofmem() : memory profiling is not available on this system memory.size(NA) [1] 3500 memory.size() [1] 16.19 Clearly Rprofmem() is a dead end, though if there is some way of enabling it, I will be happy to. I forgot that you need to recompile R to enable it. Not impossible, but probably not worth the trouble just for this problem. Using gcinfo(TRUE) may give you enough information: it will report whenever there's a garbage collection. I am only actually using 16.19Mb of memory, as all I do is open R and read in some data (a 2.7Mb text file). I would guess that mclust does a lot of other allocations before it fails. I am not sure how to guard against the memory fragmentation problem. The machine has 16Gb of RAM and I am only running R and Internet Explorer, but I don't know if that helps. Is there a way to deliver contiguous memory to R? R doesn't give you any real control over that. I would guess that R is getting 3.5 Gb in a contiguous chunk based on your memory.size report, but then it is fragmenting it. Is 64bit R available for Windows? Sorry to bang on about Windows, I'm waiting for my IT department to build Linux machines, and this is what I have in the meantime. Revolution Computing sells one. There's no (reliable) 64 bit version of gcc for Windows, so we don't distribute one. Duncan Murdoch Thanks Mick From: Duncan Murdoch [murd...@stats.uwo.ca] Sent: 16 September 2009 15:43 To: michael watson (IAH-C) Cc: r-help@r-project.org Subject: Re: [R] Memory in R on windows running 64bit XP On 9/16/2009 10:16 AM, michael watson (IAH-C) wrote: Hi I'm running 32-bit R on Windows XP 64bit and the machine has 16Gb of RAM. The help for memory.limit states: If 32-bit R is run on some 64-bit versions of Windows the maximum value of obtainable memory is just under 4GB. So, using the help which states the size parameter can go up to 4095: memory.limit(size=4095) When I run mclust I get: Error: cannot allocate vector of size 1.3 Gb So, now I set the max-mem-size flag: --max-mem-size=3500M Repeat and I still get the same error: Error: cannot allocate vector of size 1.3 Gb So, can anyone help? 32 bit R is reported to be able to use up to 4Gb of RAM on 64 bit windows, yet mine croaks at 1.3Gb. You are misreading things in a couple of places. The most important one is the error message: it says allocation of a 1.3 Gb object failed. You may have 4 Gb available in total, but have used so much of it already that Windows won't allocate a 1.3 Gb piece to R. Using Rprofmem() might show how the allocations are happening. Another possibility is that your particular version of Windows doesn't support giving 4 Gb to R. Run memory.size(NA) to find out the limit, memory.size() to find out the current amount in use. And even if the limit is close to 4 Gb and you have more than 1.3 Gb free, the allocation may fail because there is no sufficiently large contiguous block available. Once memory is allocated, that address is in use and is unavailable. Since the 32 bit address space only covers 4 Gb, you can fairly easily end up with allocations that are spread out across it leaving no large blocks available. That's the advantage of switching to 64 bit R: even if you still only had 4 Gb of memory available, the address space is so much bigger that fragmentation probably won't be a problem. Duncan Murdoch sessionInfo() R version 2.9.2 (2009-08-24) i386-pc-mingw32 locale: LC_COLLATE=English_United Kingdom.1252;LC_CTYPE=English_United Kingdom.1252;LC_MONETARY=English_United Kingdom.1252;LC_NUMERIC=C;LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] mclust_3.3.1 Thanks Mick __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
