On Sun, 12 Jul 2020, Thomas Adams wrote:
Hi Roger,
I believe I may have been correct about the memory issue... it's possible,
in GRASS, that I did not restrict my REGION sufficiently, using g.region.
Additionally, I applied a MASK for my country of interest, Mozambique,
before starting R, which yielded:
srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
Creating BIL support files...
Exporting raster as integer values (bytes=4)
100%
Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0 +lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded datum unknown in CRS definition
image(srtm)
Error in rasterImage(m, bb[1, 1], bb[2, 1], bb[1, 2], bb[2, 2], interpolate
= interpolate, :
cannot allocate memory block of size 16777216 Tb
I'd got to an errant g.region before r.out.bin too.
graphics::rasterImage() is reporting the error. The metric is wrong, but
the problem is real. The object is about 650M x 4 bytes, which image()
needs to manipulate too, creating several copies (in the imported object
it is a vector, but is cast to a matrix and transposed).
Still, a huge amount of memory needed (is 16777216 Tb correct??)
My thought was to resample the SRTM data in GRASS to 1km resolution, then
start R, and proceed, which quite honestly, for my purposes, really is
sufficient. By doing this, everything worked perfectly! I was able to read
the srtm data quickly and generate an image. I am puzzled by the "cannot
allocate memory block of size 16777216 Tb" message with my GRASS Region
settings in place.
Please try r.out.gdal in 90m, then stars::read_stars(..., proxy=TRUE)
(outside rgrass7). We need to try to get stars to work in rgrass7, but we
had not gone into that yet. In stars and gdalcubes, the idea is to do work
on the back end, and only bring data to (say) display in the feasible
resolution. So I think something like:
library(stars)
big <- read_stars("big.tif", proxy=TRUE)
plot(big)
automates the downsampling. The default arguments to readRAST() assume
that uncompressed binary is the optimal transfer strategy, and that all
the data should be in R's workspace.
I can speculate that if GDAL used the GRASS plugin, it might be possibble
to proxy the actual GRASS CELL or DCELL. However, someone else would need
to take responsibility for implementing prototypes for that. The same
applies to support for stars, alternatively for terra in addition (raster
is used by many, but with limited resources, stars and terra should be
preferred).
Roger
Thank you for all your help!
Best,
Tom
On Sat, Jul 11, 2020 at 11:02 PM Thomas Adams <tea...@gmail.com> wrote:
Hi Roger,
r.out.bin from GRASS did succeed... r.out.bin
input=mozambique_srtm_patch@teaiii output=mozambique_srtm_patch null=-999
Exporting raster as integer values (bytes=2)
A very large file!!
-rw-r--r-- 1 teaiii teaiii 1045368225220 Jul 11 13:26
mozambique_srtm_patch
Tom
On Sat, Jul 11, 2020 at 12:12 PM Roger Bivand <roger.biv...@nhh.no> wrote:
Briefly, please try to run r.out.bin manually, to the same output or
another. This is what is failing on theGRASS side.
Roger
-------
Roger Bivand
Norwegian School of Economics, Bergen, Norway
roger.biv...@nhh.no
------------------------------
*Fra:* Thomas Adams <tea...@gmail.com>
*Sendt:* lørdag 11. juli 2020, 17:59
*Til:* Roger Bivand
*Kopi:* r-sig-Geo@r-project.org
*Emne:* Re: [R-sig-Geo] Issue with rgrass7 and readRAST
Hi Roger!
Here is what was written in
/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS:
drwxr-xr-x 3 teaiii teaiii 4096 Sep 23 2019 ..
-rw-r--r-- 1 teaiii teaiii 131072 Jul 9 10:02 801.0.gpkg
-rw-r--r-- 1 teaiii teaiii 131072 Jul 9 10:04 466.0.gpkg
-rw-r--r-- 1 teaiii teaiii 0 Jul 9 20:32 801.0
-rw-r--r-- 1 teaiii teaiii 0 Jul 9 22:40 188.0
-rw-r--r-- 1 teaiii teaiii 0 Jul 9 22:51 222.0
And there is plenty of space on the disk drive...
Tom
On Sat, Jul 11, 2020 at 9:26 AM Roger Bivand <roger.biv...@nhh.no> wrote:
Tom
On Fri, 10 Jul 2020, Thomas Adams wrote:
Hi,
I'm using R in combination with GRASS GIS 7.8.3, attempting to read a
GRASS
raster file into R with the rgrass7 interface. I have done this kind of
thing countless times previously without problems. It's quite possible
I am
running into memory issues, but my Ubuntu 18.04 Linux computer has 32
GB of
RAM.
I have not succeeded in reproducing this, I'm afraid. I created a
location
with cell counts to match, and ran r.out.bin first to check whether
there
were any issues writing the data to a temporary file. Then I ran the
equivalent of your command (readRAST()), again without problems. I'm on
a
16 GB PC, so the problem is puzzling. I'm curious about the free space
on
your
/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/
teaiii-Leopard-WS
directory. Did you check whether any output arrived there, since the
error
message is coming from r.out.bin?
The error is on line 53 in raster/r.out.bin/main.c, when the system
function fwrite() returns an error.
Thanks for your patience.
Roger
My GRASS raster file:
Type of Map: raster Number of Categories: 0
Data Type: CELL
Rows: 22663
Columns: 28794
Total Cells: 652558422
~90m resolution
Using...
srtm <- readRAST("mozambique_srtm_patch",cat=FALSE)
After quite some time, I get...
Error in execGRASS("r.out.bin", flags = rOutBinFlags, input =
vname[i], :
The command:
r.out.bin -b -i input=mozambique_srtm_patch
output=/media/teaiii/development/grass/greensboro_mercator/teaiii/.tmp/teaiii-Leopard-WS/mozambique_srtm_patch
bytes=4 null=-81
produced an error (1) during execution:
Creating BIL support files...
Exporting raster as integer values (bytes=4)
ERROR: Error writing data
In addition: Warning messages:
1: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded ellps unknown in CRS definition: +proj=merc +lat_ts=0
+lon_0=0
+x_0=0 +y_0=0 +R=6378137 +units=m +no_defs
2: In showSRID(uprojargs, format = "PROJ", multiline = "NO") :
Discarded datum unknown in CRS definition
Libraries loaded include: sp, rgrass7, rgdal, raster...
Any thoughts?
Best,
Tom
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: roger.biv...@nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
--
Thomas E Adams, III
1724 Sage Lane
Blacksburg, VA 24060
tea...@gmail.com (personal)
t...@terrapredictions.org (work)
1 (513) 739-9512 (cell)
--
Thomas E Adams, III
1724 Sage Lane
Blacksburg, VA 24060
tea...@gmail.com (personal)
t...@terrapredictions.org (work)
1 (513) 739-9512 (cell)
--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: roger.biv...@nhh.no
https://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo
