Hi,
I'm using R to collect data for a number of exchanges through a socket
connection and constantly running into memory problems even though task I
believe is not that memory consuming. I guess there is a miscommunication
between R and WinXP about freeing up memory.
So this is the code:
for (x in 1:length(exchanges.to.get)) {
tickers<-sqlQuery(channel,paste("SELECT Symbol FROM symbols_list WHERE
Exchange='",exchanges.to.get[x],"';",sep=''))[,1]
dir.create(paste(Working.dir,exchanges.to.get[x],'/',sep=''))
for (y in 1:length(tickers)) {
con2 <- socketConnection(Sys.info()["nodename"], port = ****) #open
socket connection to get data
writeLines(paste(command,',',tickers[y],',',interval,';',sep=''), con2)
data.<-readLines(con2)
end.of.data<-sum(c(data.=="!ENDMSG!",data.=="!SYNTAX_ERROR!"))
while(end.of.data!=1)
{new.data<-readLines(con2);end.of.data<-sum(new.data=="!ENDMSG!");
data.<-c(data.,new.data)}
if (length(data.)>3)
write.table(data.[1:(length(data.)-2)],paste(Working.dir,exchanges.to.get[x]
,'/',sub('\\*','\+',tickers[y]),'_.csv',sep=''),quote=F,col.names =
F,row.names=F)
close(con2)
}
rm(tickers)
gc()
With command gcinfo(TRUE) I got the following info (some examples) :
Garbage collection 16362 = 15411+754+197 (level 0) ...
6.3 Mbytes of cons cells used (22%)
2.2 Mbytes of vectors used (8%)
Garbage collection 16407 = 15454+756+197 (level 0) ...
13.1 Mbytes of cons cells used (46%)
10.4 Mbytes of vectors used (39%)
Garbage collection 16410 = 15456+756+198 (level 2) ...
4.9 Mbytes of cons cells used (21%)
0.9 Mbytes of vectors used (4%)
Garbage collection 16679 = 15634+796+249 (level 0) ...
150.7 Mbytes of cons cells used (95%)
203.9 Mbytes of vectors used (75%)
Garbage collection 16680 = 15634+796+250 (level 2) ...
4.9 Mbytes of cons cells used (4%)
0.9 Mbytes of vectors used (0%)
Garbage collection 16808 = 15754+802+252 (level 0) ...
6.1 Mbytes of cons cells used (7%)
1.8 Mbytes of vectors used (1%)
But the end result is in Task Manager:
RGui.exe Mem Usage 470,472K VM Size 541,988K
Even though R reports
Garbage collection 16808 = 15754+802+252 (level 0) ...
6.1 Mbytes of cons cells used (7%)
1.8 Mbytes of vectors used (1%)
Has anybody encountered this problem and how you guys deal with it? It
seems like a memory leak to me, as tasks are not memory demandind, the
biggest amount of data in a single file is about 40MB.
Thanks
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
