Dear list,
We have a problem which appears to be a bug in either rgdal or raster,
although it could also be a bug in base R or in our understanding of how
to deal with connections.
We have a process which is writing a rather large (~10-20.000) number of
geoTiffs via writeRaster. However, the process has frequently stopped
with an error of the type:
Error in .local(.Object, ...) :
TIFFOpen:/local0/skoiejo/hri/test.tif: Too many open files
The issue seems to be the creation of temp-files in the temp directory
which is given by tempdir(), not by raster:::.tmpdir(). These temp-files
seem to be created by the call
transient <- new("GDALTransientDataset", driver=driver, rows=r@nrows,
cols=r@ncols, bands=nbands, type=dataformat, fname=filename,
options=options, handle=NULL)
from raster:::.getGDALtransient
The temp-files are deleted after writing the geoTiff, but are not
removed from the list of open files in Linux, which on our system was
limited to 1024 files (ulimit -n) per process. Below is a script which
can replicate the issue (takes a few minutes to reach 1024) and
sessionInfo().
Currently we are trying to solve the issue by increasing the limit of
file connections, but we would prefer a solution where the connections
are properly deleted, either before writeRaster finishes, or a command
which we can include in our script, either R-code or a call to System().
The connections are not visible via showConnections(), and
closeAllConnections() does not help.
Thanks,
Jon
r <- raster(system.file("external/test.grd", package="raster"))
for (ifile in 1:2000) {
writeRaster(r, "test.tif", format = "GTiff", overwrite = TRUE)
print(ifile)
}
After the script stops, I checked the open files from the process, and
got the following:
lsof -aPn -p 596 | more
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
R 596 skoiejo cwd DIR 8,17 139264 4213416
/local0/skoiejo/Raster_temp
R 596 skoiejo rtd DIR 253,0 4096 2 /
R 596 skoiejo txt REG 253,0 14192 2500779
/usr/lib64/R/bin/exec/R
R 596 skoiejo mem REG 253,0 2679280 2501177
/usr/lib64/R/lib/libR.so
......................
R 596 skoiejo 1015u REG 253,0 238 4983213
/tmp/RtmpNyOKCk/toptest.tif (deleted)
R 596 skoiejo 1016u REG 253,0 238 4983214
/tmp/RtmpNyOKCk/qxdtest.tif (deleted)
R 596 skoiejo 1017u REG 253,0 238 4983215
/tmp/RtmpNyOKCk/zwotest.tif (deleted)
R 596 skoiejo 1018u REG 253,0 238 4983216
/tmp/RtmpNyOKCk/cnqtest.tif (deleted)
R 596 skoiejo 1019u REG 253,0 238 4983217
/tmp/RtmpNyOKCk/lottest.tif (deleted)
R 596 skoiejo 1020u REG 253,0 238 4983218
/tmp/RtmpNyOKCk/fartest.tif (deleted)
R 596 skoiejo 1021u REG 253,0 238 4983219
/tmp/RtmpNyOKCk/vsqtest.tif (deleted)
R 596 skoiejo 1022u REG 253,0 238 4983220
/tmp/RtmpNyOKCk/czptest.tif (deleted)
Even if tested by someone with a limit higher than 2000, it should still
be possible to see the long list of open connections, as above.
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-redhat-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rgdal_0.8-10 raster_2.1-49 sp_1.0-11
loaded via a namespace (and not attached):
[1] grid_3.0.1 lattice_0.20-15
--
Jon Olav Skøien
Joint Research Centre - European Commission
Institute for Environment and Sustainability (IES)
Land Resource Management Unit
Via Fermi 2749, TP 440, I-21027 Ispra (VA), ITALY
jon.sko...@jrc.ec.europa.eu
Tel: +39 0332 789206
Disclaimer: Views expressed in this email are those of the individual and do
not necessarily represent official views of the European Commission.
[[alternative HTML version deleted]]
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
