Re: [R] Out of memory issue
On Tue, 28 Apr 2009, Neotropical bat risk assessments wrote: Hi again all, I can read all of the data is in CSV format with a header row and with 1,200,240 rows. I can do all the plots and ggplot2 has no problems. Seems to be a problem with the MASS package??? It seems that the contour plots are the problem with the memory error. Error: cannot allocate vector of size 228.9 Mb I can run smaller data sets and do the contour plots w/o problems. The data can not really be summarized as I need to plot the actual values to get the insight into the large data sets. Why can't you divide the plot into, say, four regions with a quarter of the data and draw contours for each region, and then overlay the printed output? -thomas __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Out of memory issue
Neotropical bat risk assessments wrote:
Hi again all,
I can read all of the data is in CSV format with a header row and with
1,200,240 rows.
I can do all the plots and ggplot2 has no problems.
Seems to be a problem with the MASS package???
It seems that the contour plots are the problem with the memory error.
Error: cannot allocate vector of size 228.9 Mb
I can run smaller data sets and do the contour plots w/o problems.
The data can not really be summarized as I need to plot the actual values to
get the insight into the large data sets.
I am using latest R version 2.9.0 (2009-04-17)
My alternative is to bump up the memory of my one Dell workstation to 16 GB
as it runs with Ubuntu and can access that amount if I figure out how to
install R on a Linux box.
It may not truly be a memory issue but a limit to how MASS does the contour
plots?
This works 100% with the large data sets.
Bodega <- read.csv ("C:/R-Stats/Bat calls/Bodega.csv")
attach(Bodega)
library(ggplot2)
f=jpeg(file="Bodega Sc_Fc plot.jpg")
print(qplot(Sc, Fc))
dev.off()
f=jpeg(file="Bodega Fc Dens plot.jpg")
print(qplot(Fc,geom="density"))
dev.off()
This does not.and I get the vector size error.
library(MASS)
library(batcalls)
BRC<-kde2d(Sc,Fc)
f=jpeg(file="Rainey contour plot.jpg")
filled.contour(BRC)
dev.off()
Thanks again for in sight as I trundle up the learn curve for R.
You need to think about how many points you are plotting into a filled
contour plot (i.e. estimate what the memory is comsumed for just holding
the image in memory) and how much memory has been wasted already. You
may want to reduce the size of the stuff you are going to plot.
Uwev Ligges
Bruce
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Out of memory issue
Hi again all,
I can read all of the data is in CSV format with a header row and with
1,200,240 rows.
I can do all the plots and ggplot2 has no problems.
Seems to be a problem with the MASS package???
It seems that the contour plots are the problem with the memory error.
Error: cannot allocate vector of size 228.9 Mb
I can run smaller data sets and do the contour plots w/o problems.
The data can not really be summarized as I need to plot the actual values to
get the insight into the large data sets.
I am using latest R version 2.9.0 (2009-04-17)
My alternative is to bump up the memory of my one Dell workstation to 16 GB
as it runs with Ubuntu and can access that amount if I figure out how to
install R on a Linux box.
It may not truly be a memory issue but a limit to how MASS does the contour
plots?
This works 100% with the large data sets.
Bodega <- read.csv ("C:/R-Stats/Bat calls/Bodega.csv")
attach(Bodega)
library(ggplot2)
f=jpeg(file="Bodega Sc_Fc plot.jpg")
print(qplot(Sc, Fc))
dev.off()
f=jpeg(file="Bodega Fc Dens plot.jpg")
print(qplot(Fc,geom="density"))
dev.off()
This does not.and I get the vector size error.
library(MASS)
library(batcalls)
BRC<-kde2d(Sc,Fc)
f=jpeg(file="Rainey contour plot.jpg")
filled.contour(BRC)
dev.off()
Thanks again for in sight as I trundle up the learn curve for R.
Bruce
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Out of memory issue
On Apr 24, 2009, at 10:40 PM, David Winsemius wrote:
On Apr 24, 2009, at 9:06 PM, Neotropical bat risk assessments wrote:
Hi all,
Julian just reminded me I need to include the lines of code so
here it is
below.
I am trying to run some plots on data, but when loading the CSV
data file it
doss that but in ggplot R is stopping and I am getting an out of
memory
error.
d <- read.csv batcalls.rda
Error: unexpected symbol in "d <- read.csv batcalls.rda"
If you really do have a data file called batcalls.rda, then the
proper way to get it into memory is with:
load("C:/Rainey//batcalls.rda") # replace path with the actual path
on your system and remember not to use "\"'s
# .. and don't ^^ use // where only / is needed
attach(d)
Error in attach(d) : object "d" not found
Because of the error ... nothing got done.
library(ggplot2)
print(qplot(Sc, Fc))
Error in eval(expr, envir, enclos) : object "Sc" not found
You cannot do anything on something that isn't there
data(batcalls)
attach(batcalls)
print(qplot(Fc,geom="density"))
b<-kmeans(Fc,c(10,15,18,30))
print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
print(qplot(Dur,geom="density"))
b<-kmeans(Dur,c(10,15,18,30))
print(qplot(Dur,Fc,colour=as.factor(b$cluster)))
d <- read.csv ("C:/Rainey/RainyAllbats.csv")
OK, that appears to have run without error, so it fits into memory.
Now you should run:
str(d) # and report the results
attach(d)
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
library(ggplot2)
print(qplot(Sc, Fc))
Warning messages:
1: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
2: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
3: Removed 1200249 rows containing missing values (geom_point).
data(d)
Warning message:
In data(d) : data set 'd' not found
attach(d)
The following object(s) are masked from d ( position 3 ) :
Dc Dur Fc Filename Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC
Tc Tk
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
print(qplot(Fc,geom="density"))
There were 15 warnings (use warnings() to see them)
b<-kmeans(Fc,c(10,15,18,30))
Error in switch(nmeth, { : NA/NaN/Inf in foreign function call
(arg 1)
f=jpeg(file="Sc x Fc.jpg")
print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
Error in data.frame(colour = c(3L, 4L, 4L, 4L, 4L, 1L, 4L, 2L, 1L,
1L, :
arguments imply differing number of rows: 9404, 2400489
You seem to be ignoring all the informative error message.
dev.off()
Anyway to tweak this somehow to get it to run?
Using WinXP with 4 GB RAM
WinXP in its default state can only use around 2.5GB and that may
require upping the memory limits:
And read the material in the links from that page as well.
--
David Winsemius, MD
Heritage Laboratories
West Hartford, CT
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
David Winsemius, MD
Heritage Laboratories
West Hartford, CT
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Out of memory issue
On Apr 24, 2009, at 9:06 PM, Neotropical bat risk assessments wrote:
Hi all,
Julian just reminded me I need to include the lines of code so
here it is
below.
I am trying to run some plots on data, but when loading the CSV
data file it
doss that but in ggplot R is stopping and I am getting an out of
memory
error.
d <- read.csv batcalls.rda
Error: unexpected symbol in "d <- read.csv batcalls.rda"
If you really do have a data file called batcalls.rda, then the proper
way to get it into memory is with:
load("C:/Rainey//batcalls.rda") # replace path with the actual path
on your system and remember not to use "\"'s
attach(d)
Error in attach(d) : object "d" not found
Because of the error ... nothing got done.
library(ggplot2)
print(qplot(Sc, Fc))
Error in eval(expr, envir, enclos) : object "Sc" not found
You cannot do anything on something that isn't there
data(batcalls)
attach(batcalls)
print(qplot(Fc,geom="density"))
b<-kmeans(Fc,c(10,15,18,30))
print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
print(qplot(Dur,geom="density"))
b<-kmeans(Dur,c(10,15,18,30))
print(qplot(Dur,Fc,colour=as.factor(b$cluster)))
d <- read.csv ("C:/Rainey/RainyAllbats.csv")
OK, that appears to have run without error, so it fits into memory.
Now you should run:
str(d) # and report the results
attach(d)
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
library(ggplot2)
print(qplot(Sc, Fc))
Warning messages:
1: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
2: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
3: Removed 1200249 rows containing missing values (geom_point).
data(d)
Warning message:
In data(d) : data set 'd' not found
attach(d)
The following object(s) are masked from d ( position 3 ) :
Dc Dur Fc Filename Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC
Tc Tk
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
print(qplot(Fc,geom="density"))
There were 15 warnings (use warnings() to see them)
b<-kmeans(Fc,c(10,15,18,30))
Error in switch(nmeth, { : NA/NaN/Inf in foreign function call
(arg 1)
f=jpeg(file="Sc x Fc.jpg")
print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
Error in data.frame(colour = c(3L, 4L, 4L, 4L, 4L, 1L, 4L, 2L, 1L,
1L, :
arguments imply differing number of rows: 9404, 2400489
You seem to be ignoring all the informative error message.
dev.off()
Anyway to tweak this somehow to get it to run?
Using WinXP with 4 GB RAM
WinXP in its default state can only use around 2.5GB and that may
require upping the memory limits:
And read the material in the links from that page as well.
--
David Winsemius, MD
Heritage Laboratories
West Hartford, CT
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Out of memory issue
Hi all,
Julian just reminded me I need to include the lines of code so here it is
below.
I am trying to run some plots on data, but when loading the CSV data file it
doss that but in ggplot R is stopping and I am getting an out of memory
error.
>
> d <- read.csv batcalls.rda
Error: unexpected symbol in "d <- read.csv batcalls.rda"
> attach(d)
Error in attach(d) : object "d" not found
> library(ggplot2)
> print(qplot(Sc, Fc))
Error in eval(expr, envir, enclos) : object "Sc" not found
> data(batcalls)
> attach(batcalls)
> print(qplot(Fc,geom="density"))
> b<-kmeans(Fc,c(10,15,18,30))
>print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
>
> print(qplot(Dur,geom="density"))
> b<-kmeans(Dur,c(10,15,18,30))
>print(qplot(Dur,Fc,colour=as.factor(b$cluster)))
> d <- read.csv ("C:/Rainey/RainyAllbats.csv")
> attach(d)
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
> library(ggplot2)
> print(qplot(Sc, Fc))
Warning messages:
1: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
2: In data.frame(..., check.names = FALSE) :
Reached total allocation of 1535Mb: see help(memory.size)
3: Removed 1200249 rows containing missing values (geom_point).
> data(d)
Warning message:
In data(d) : data set 'd' not found
> attach(d)
The following object(s) are masked from d ( position 3 ) :
Dc Dur Fc Filename Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
The following object(s) are masked from batcalls :
Dc Dur Fc Fk Fmax Fmean Fmin Qk Qual S1 Sc st TBC Tc Tk
> print(qplot(Fc,geom="density"))
There were 15 warnings (use warnings() to see them)
> b<-kmeans(Fc,c(10,15,18,30))
Error in switch(nmeth, { : NA/NaN/Inf in foreign function call (arg 1)
> f=jpeg(file="Sc x Fc.jpg")
>print(qplot(Sc,Fc,colour=as.factor(b$cluster)))
Error in data.frame(colour = c(3L, 4L, 4L, 4L, 4L, 1L, 4L, 2L, 1L, 1L, :
arguments imply differing number of rows: 9404, 2400489
> dev.off()
Anyway to tweak this somehow to get it to run?
Using WinXP with 4 GB RAM
Tnx
Bruce
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Out of memory issue
Is your CSV file really that big? Have you passed the correct parameters to read.table()? For example, specified the correct sep="," parameter? If you've done everything correctly and are still running out of memory, maybe you need to go to 64-bit. R works great under 64-bit Ubuntu (8GB on my machine). - Ken Neotropical bat risk assessments wrote: > > Hi all, > > I am trying to run some plots on data, but when loading he CSV data > file R is stopping and I am getting an out of memory error. > > Anyway to tweak this somehow to get it to run? > > Using WinXP with 4 GB RAM > > Tnx > > Bruce > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > -- View this message in context: http://www.nabble.com/Out-of-memory-issue-tp23226922p23227138.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Out of memory issue
"PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code." Otherwise...the only other possible suggestion is to use a smaller file. Julian Neotropical bat risk assessments wrote: Hi all, I am trying to run some plots on data, but when loading he CSV data file R is stopping and I am getting an out of memory error. Anyway to tweak this somehow to get it to run? Using WinXP with 4 GB RAM Tnx Bruce __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Out of memory issue
Hi all, I am trying to run some plots on data, but when loading he CSV data file R is stopping and I am getting an out of memory error. Anyway to tweak this somehow to get it to run? Using WinXP with 4 GB RAM Tnx Bruce __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
