Re: [R] allocMatrix limits
On Thu, 31 Jul 2008, Vadim Kutsyy wrote: I am getting an error "allocMatrix: too many elements specified" when I am trying to create large matrix or vector (about 1 billion elements). How can I find out limits on allocMatrix? Can I increase them? ?"Memory-limits", and you cannot increase them unless you have a system which has larger signed integers. Thanks, Vadim. PS: I am running R on SUSE10 on __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] allocMatrix limits
I am getting an error "allocMatrix: too many elements specified" when I am trying to create large matrix or vector (about 1 billion elements). How can I find out limits on allocMatrix? Can I increase them? ?"Memory-limits", and you cannot increase them unless you have a system which has larger signed integers. Thank you for pointing out this. The problem is in array.c, where allocMatrix check for "if ((double)nrow * (double)ncol > INT_MAX)". But why itn is used and not long int for indexing? (max int is 2147483647, max long int is 9223372036854775807) Thanks, Vadim __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] allocMatrix limits
> "VK" == Vadim Kutsyy <[EMAIL PROTECTED]> > on Thu, 31 Jul 2008 15:43:56 -0700 writes: >>> I am getting an error "allocMatrix: too many elements >>> specified" when I am trying to create large matrix or >>> vector (about 1 billion elements). >>> >>> How can I find out limits on allocMatrix? Can I increase >>> them? >> >> ?"Memory-limits", and you cannot increase them unless you >> have a system which has larger signed integers. >> VK> Thank you for pointing out this. VK> The problem is in array.c, where allocMatrix check for VK> "if ((double)nrow * (double)ncol > INT_MAX)". But why VK> itn is used and not long int for indexing? (max int is VK> 2147483647, max long int is 9223372036854775807) Well, Brian gave you all info: Did you really carefully read ?Memory-limits ?? BTW: The package 'Matrix' has many facilities to work with sparse matrices; and these facilities are used in "lme4" and more than a dozen other CRAN packages to work with sparse matrices in the order of 10^5 x 10^5, sometimes even 10^6 x 10^6, *but* with sparse content. Martin Maechler, ETH Zurich __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] allocMatrix limits
Martin Maechler wrote: VK> The problem is in array.c, where allocMatrix check for VK> "if ((double)nrow * (double)ncol > INT_MAX)". But why VK> itn is used and not long int for indexing? (max int is VK> 2147483647, max long int is 9223372036854775807) Well, Brian gave you all info: exactly, and given that most modern system used for computations (i.e. 64bit system) have long int which is much larger than int, I am wondering why long int is not used for indexing (I don't think that 4 bit vs 8 bit storage is an issue). Did you really carefully read ?Memory-limits ?? Yes, it is specify that 4 bit int is used for indexing in all version of R, but why? I think 2147483647 elements for a single vector is OK, but not as total number of elements for the matrix. I am running out of indexing at mere 10% memory consumption. BTW: The package 'Matrix' has many facilities to work with sparse matrices; Is is a very good package, but I don't see a relation to a limitation in array.c (and I don't have scarce matrices). Thanks, Vadim PS: I have no problem to go and modify C code, but I am just wondering what are the reasons for having such limitation. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] allocMatrix limits
> "VK" == Vadim Kutsyy <[EMAIL PROTECTED]> > on Fri, 01 Aug 2008 07:35:01 -0700 writes: VK> Martin Maechler wrote: >> VK> The problem is in array.c, where allocMatrix check for VK> "if ((double)nrow * (double)ncol > INT_MAX)". But why VK> itn is used and not long int for indexing? (max int is VK> 2147483647, max long int is 9223372036854775807) >> >> Well, Brian gave you all info: >> VK> exactly, and given that most modern system used for VK> computations (i.e. 64bit system) have long int which is VK> much larger than int, I am wondering why long int is not VK> used for indexing (I don't think that 4 bit vs 8 bit VK> storage is an issue). >> Did you really carefully read ?Memory-limits ?? >> VK> Yes, it is specify that 4 bit int is used for indexing VK> in all version of R, but why? I think 2147483647 VK> elements for a single vector is OK, but not as total VK> number of elements for the matrix. I am running out of VK> indexing at mere 10% memory consumption. Hmm, do you have 160 GBytes of RAM? But anyway, let's move this topic from R-help to R-devel. [...] VK> PS: I have no problem to go and modify C code, but I am VK> just wondering what are the reasons for having such VK> limitation. This limitation and its possible remedies are an interesting topic, but really not for R-help: It will be a lot about C programming the internal represenation of R objects, etc. Very fascinating but for R-devel. "See you there!" Martin __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
