Re: [R-sig-Geo] stack many files without loading into memory

2015-02-25 Thread dschneiderch
Just wanted to update this thread in case anyone else comes looking. Some of these things were not immediately clear to me. I ended up doing: library(raster) library(ncdf4) fn=list.files('serverpath') fnstack=stack(fn) layerdates=names(fnstack) #instead of writeRaster, use ncdf4 directly to get aro

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-06 Thread dschneiderch
Ok - Looks like it worked this time for 112 files from 2012. The netcdf is 2.25 GB while the compressed multiband geotiff is 510MB. Does the netcdf have so much overhead- the 112 file at 10MB each are only 1.12 GB individually? I like the tidiness of 1 file per year so I'll have to play with how e

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-06 Thread dschneiderch
Hi Michael - Yes saving as GTiff with the compression options reduced the file size from ~10MB to ~2.5M for a single file but I am having a lot of trouble getting it to save the whole stack. I'm definitely running out of memory on my computer so maybe R is being slow and timing out? I've left it ov

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-04 Thread Michael Sumner
On Thu Feb 05 2015 at 7:41:33 AM Dominik Schneider < dominik.schnei...@colorado.edu> wrote: > I think you are correct. > s=stack(fn,quick=T) > writeRaster(s,'localpath/2012data') > > Ugh, sorry yes that's me reading too fast. I should have suggested the next step to writeRaster, I'm not sure why

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-04 Thread Dominik Schneider
I think you are correct. s=stack(fn,quick=T) writeRaster(s,'localpath/2012data') would get the data local. I guess the trade off is that the file size is an order of magnitude bigger than if I saved them in an .RData file but much quicker to access. ds On Wed, Feb 4, 2015 at 12:51 PM, Dominik Sch

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-04 Thread Michael Sumner
Why not stack(fn) ? On Thu, 5 Feb 2015 06:41 Dominik Schneider wrote: > Hi - > I have some data on a server but would like to bring them local in a > somewhat compressed format that is still easy to access. > > /Volumes/hD/2012 -> 100 geotiffs > ~/project/data/ -> store those geotiffs here with

Re: [R-sig-Geo] stack many files without loading into memory

2015-02-04 Thread Dominik Schneider
Wouldn't that keep the link to the server on which they are stored now? Dominik Schneider o 303.735.6296 | c 518.956.3978 On Wed, Feb 4, 2015 at 12:50 PM, Michael Sumner wrote: > Why not stack(fn) > > ? > > On Thu, 5 Feb 2015 06:41 Dominik Schneider > wrote: > >> Hi - >> I have some data on a

[R-sig-Geo] stack many files without loading into memory

2015-02-04 Thread Dominik Schneider
Hi - I have some data on a server but would like to bring them local in a somewhat compressed format that is still easy to access. /Volumes/hD/2012 -> 100 geotiffs ~/project/data/ -> store those geotiffs here without needing server access. untested, I think I could do something like: s=stack() wr