The ratio of object size to rds file size depends on the object. Some
variation is due to how header information is stored in memory and in the
file but I suspect most is due to how compression works (e.g., a vector of
repeated values can be compressed into a smaller file than a bunch of
random bytes).
f <- function (data, ...) {
force(data)
tf <- tempfile()
on.exit(unlink(tf))
save(data, file = tf)
c(`obj/file size` = as.numeric(object.size(data)/file.size(tf)))
}
> f(rep(0,1e6))
obj/file size
1021.456
> f(rep(0,1e6), compress=FALSE)
obj/file size
0.9999986
> f(rep(89.7,1e6))
obj/file size
682.6555
> f(log(1:1e6))
obj/file size
1.309126
> f(vector("list",1e6))
obj/file size
2021.744
> f(as.list(log(1:1e6)))
obj/file size
8.907579
> f(sample(as.raw(0:255),size=8e6,replace=TRUE))
obj/file size
0.9998433
> f(rep(as.raw(0:255),length=8e6))
obj/file size
254.5595
> f(as.character(1:1e6))
obj/file size
23.5567
Bill Dunlap
TIBCO Software
wdunlap tibco.com
On Tue, Sep 18, 2018 at 8:28 AM, Joris Meys <jorism...@gmail.com> wrote:
> Dear all,
>
> I tried to read in a 3.8Gb RDS file on a computer with 16Gb available
> memory. To my astonishment, the memory footprint of R rises quickly to over
> 13Gb and the attempt ends with an error that says "cannot allocate vector
> of size 5.8Gb".
>
> I would expect that 3 times the memory would be enough to read in that
> file, but apparently I was wrong. I checked the memory.limit() and that one
> gave me a value of more than 13Gb. So I wondered if this was to be
> expected, or if there could be an underlying reason why this file doesn't
> want to open.
>
> Thank you in advance
> Joris
>
> --
> Joris Meys
> Statistical consultant
>
> Department of Data Analysis and Mathematical Modelling
> Ghent University
> Coupure Links 653, B-9000 Gent (Belgium)
> <https://maps.google.com/?q=Coupure+links+653,%C2%A0B-
> 9000+Gent,%C2%A0Belgium&entry=gmail&source=g>
>
> -----------
> Biowiskundedagen 2017-2018
> http://www.biowiskundedagen.ugent.be/
>
> -------------------------------
> Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
[[alternative HTML version deleted]]
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
