Also take a look at using raster:: clusterR() in combination with calc()
to use multiple cores. This works well if your your calculation function
does not require neighboring cells.
--Mel.
On 04/07/2017 09:50 AM, Benjamin Leutner wrote:
You could stick to the native raster format (grd), in which case
calc() writes to your final file directly.
Of course that doesn't change the file size issue, but saves you the
translation step to geotiff.
For the file size you could consider restricting the datatype to
integer (see ?dataType).
If I have reflectance data [0,1] for example; I scale them by a factor
10000 and then save as "INT4U" or "INT2U" (depending on your maximally
expected value range), or "INT4S" or "INT2S" if you have negative
values. That can bring down the filesize quite a bit, while retaining
most of the relevant precission (beware: remaining decimal places will
be cut-off,)
e.g. calc(..., function(x) { yourcalculations(x) * 10000 }, datatype =
"INT4S")
pro tip: the argument in calc() (or more precisely in writeRaster())
is called datatype, not to be confused with the stand-alone function
dataType() with a capital T. That one has bitten me many times,
because due to the "..." argument there will be no warning if you
mistype it ;-)
On 06.04.2017 19:27, Gregovich, Dave P (DFG) wrote:
Hi,
I am performing a math operation on a stack of large rasters. The
code below uses smaller files for illustration and reproducibility.
Any alternative way of performing this task that does not create huge
temporary files, and perhaps cuts down on processing time, would be
greatly appreciated. The 'calc' process creates a couple of
temporary raster files that are huge. The first one is 142 GB, and I
don't have hard drive space for that one and the second one that
begins writing during the process.
Thanks kindly for any advice!
Dave.
#create raster stack and coefficients...
library(raster)
mod.coefs <- rnorm(10)
s <- stack()
r <- raster(nrow = 100, ncol = 100)
#actual rasters I am working with are about 40000, pixels square,
with each GeoTiff raster in the stack taking about 2.5 GB on disk
for(i in 1:10){
r[] <- rnorm(10000)
s <- addLayer(s, r)
}
#attempt to perform raster math...
out.file <- 'C:/ out.rast.tif'
out.rast <- calc(s, function(x){exp(sum(mod.coefs * x))}, filename =
out.file)
#at this point, the temporary files in the
\AppData\Local\Temp\RtmpqQYzfS\raster folder eventually become quite
large,
#with one .GRI file reaching 142 GB, and another now growing to 8 GB
before I ended the process
#the file 'out.file' has not been created at that point.
#_____________end____________________________________________________________________
[[alternative HTML version deleted]]
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
