Re: [GRASS-dev] R.terraflow on massive grids
doug_newc...@fws.gov wrote:
> Should the limit to memory be indicated in the gui?
> Would there be any benefit to other parts of GRASS in bumping up this
> memory limit? I can see the reason on 32 bit windows XP, but that is
> starting to go away.
Arbitrary 32-bit limits should ideally be fixed, but it's a game of
Whac-a-Mole. The problem usually arises as a result of using "int"
instead of long, size_t, off_t, etc.
I've fixed the original issue and one other with r47665:
--- raster/r.terraflow/grass2str.h (revision 47663)
+++ raster/r.terraflow/grass2str.h (working copy)
@@ -140,9 +140,9 @@
/* close map files */
Rast_close (infd);
- G_debug(3, "nrows=%d ncols=%dstream_len()=%d", nrows, ncols,
+ G_debug(3, "nrows=%d ncols=%dstream_len()=%"PRI_OFF_T, nrows, ncols,
str->stream_len());
- assert(nrows * ncols == str->stream_len());
+ assert((off_t) nrows * ncols == str->stream_len());
rt_stop(rt);
stats->recordTime("reading raster map", rt);
This might cause problems on non-Linux platforms, but both r.terraflow
and LFS have always had portability issues.
--
Glynn Clements
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
Hamish wrote: > > The iostream library uses size_t. However, r.terraflow > > itself had a bug (scale then cast instead of cast then scale), > > which should be fixed by r47641. > > in the same class, but sort of unrelated- any thoughts on #1107? > https://trac.osgeo.org/grass/ticket/1107 I'm fairly certain that it's supposed to be float and atoi() is a typo. Fixed in r47664. -- Glynn Clements ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
Glynn: > The iostream library uses size_t. However, r.terraflow > itself had a bug (scale then cast instead of cast then scale), > which should be fixed by r47641. in the same class, but sort of unrelated- any thoughts on #1107? https://trac.osgeo.org/grass/ticket/1107 thanks, Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
Markus Metz 08/15/2011 06:28 AM To doug_newc...@fws.gov cc grass-dev@lists.osgeo.org Subject Re: [GRASS-dev] R.terraflow on massive grids > wrote: >> >> Hi Folks, >> I have an fcell grid of elevations for the state of North Carolina (51000 >> rows 133000 columns 678300 cells) . I tried to run r.terraflow in >> GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I >> patched them according to Glynn's email , >> http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and >> tried again ( Would it be better to change the dimension variable to int >> instead of short int?) . >> >> This time my Streams file builds to about 26 GB and then r.terraflow bombs >> with : >> >> MFD flow direction >> D8CUT=99986991104.00 >> Memory size: 808.00M (847249408) bytes >> Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. >> r.terraflow: grass2str.h:145: AMI_STREAM* >> cell2stream(char*, elevation_type, long int*) [with T = >> float, elevation_type = float]: Assertion `nrows * ncols == >> str->stream_len()' failed. >> >> The memory size is interesting, because I'm giving it 8GB of RAM out of 16 >> GB in the command. >Note that the memory manager of the iostream lib used by r.terraflow >can only handle up to 2047 MB of RAM because the memory option in MB >is internally converted to Byte and stored as int, thus the max >recognized value is 2^31 - 1 Byte. >Markus M This appears to be kind of like pulling on a corner of a spiderweb. :-) Should the limit to memory be indicated in the gui? Would there be any benefit to other parts of GRASS in bumping up this memory limit? I can see the reason on 32 bit windows XP, but that is starting to go away. Doug Doug Newcomb USFWS Raleigh, NC 919-856-4520 ext. 14 doug_newc...@fws.gov - The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
Markus Metz wrote: > Note that the memory manager of the iostream lib used by r.terraflow > can only handle up to 2047 MB of RAM because the memory option in MB > is internally converted to Byte and stored as int, thus the max > recognized value is 2^31 - 1 Byte. The iostream library uses size_t. However, r.terraflow itself had a bug (scale then cast instead of cast then scale), which should be fixed by r47641. -- Glynn Clements ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
wrote: > > Hi Folks, > I have an fcell grid of elevations for the state of North Carolina (51000 > rows 133000 columns 678300 cells) . I tried to run r.terraflow in > GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I > patched them according to Glynn's email , > http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and > tried again ( Would it be better to change the dimension variable to int > instead of short int?) . > > This time my Streams file builds to about 26 GB and then r.terraflow bombs > with : > > MFD flow direction > D8CUT=99986991104.00 > Memory size: 808.00M (847249408) bytes > Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. > r.terraflow: grass2str.h:145: AMI_STREAM* > cell2stream(char*, elevation_type, long int*) [with T = > float, elevation_type = float]: Assertion `nrows * ncols == > str->stream_len()' failed. > > The memory size is interesting, because I'm giving it 8GB of RAM out of 16 > GB in the command. Note that the memory manager of the iostream lib used by r.terraflow can only handle up to 2047 MB of RAM because the memory option in MB is internally converted to Byte and stored as int, thus the max recognized value is 2^31 - 1 Byte. Markus M ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] R.terraflow on massive grids
doug_newc...@fws.gov wrote: > I have an fcell grid of elevations for the state of North Carolina (51000 > rows 133000 columns 678300 cells) . I tried to run r.terraflow in > GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I > patched them according to Glynn's email , > http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and > tried again ( Would it be better to change the dimension variable to int > instead of short int?) . I think that dimension_type has to be used consistently. There is a class, "ijBaseType" which contains a pair of dimension_type members (i.e. it's a coordinate pair), and there are also several classes which derive from ijBaseType. If r.terraflow creates large arrays of instances of ijBaseType or its subclasses, changing dimension_type to 32-bit could significantly increase memory consumption. > This time my Streams file builds to about 26 GB and then r.terraflow bombs > with : > > MFD flow direction > D8CUT=99986991104.00 > Memory size: 808.00M (847249408) bytes > Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. > r.terraflow: grass2str.h:145: AMI_STREAM* > cell2stream(char*, elevation_type, long int*) [with T = > float, elevation_type = float]: Assertion `nrows * ncols == > str->stream_len()' failed. > > The memory size is interesting, because I'm giving it 8GB of RAM out of 16 > GB in the command. The temp directory has about 900GB of space, so it has > plenty of room . > > The box is 64 bit Ubuntu 11.04 > > related to ? > > http://trac.osgeo.org/grass/ticket/1006 Yes. The total number of cells (nrows * ncols) is too large to fit into an "int". In general, rasters with more than 2^31-1 cells are problematic. Anything which counts cells risks running into similar problems. In the case of r.terraflow, you may be able to fix this particular issue by casting to "long" or "off_t" in the offending line, e.g.: assert((off_t) nrows * ncols == str->stream_len()); But there may be similar issues elsewhere in r.terraflow. -- Glynn Clements ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] R.terraflow on massive grids
Hi Folks, I have an fcell grid of elevations for the state of North Carolina (51000 rows 133000 columns 678300 cells) . I tried to run r.terraflow in GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I patched them according to Glynn's email , http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and tried again ( Would it be better to change the dimension variable to int instead of short int?) . This time my Streams file builds to about 26 GB and then r.terraflow bombs with : MFD flow direction D8CUT=99986991104.00 Memory size: 808.00M (847249408) bytes Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. r.terraflow: grass2str.h:145: AMI_STREAM* cell2stream(char*, elevation_type, long int*) [with T = float, elevation_type = float]: Assertion `nrows * ncols == str->stream_len()' failed. The memory size is interesting, because I'm giving it 8GB of RAM out of 16 GB in the command. The temp directory has about 900GB of space, so it has plenty of room . The box is 64 bit Ubuntu 11.04 related to ? http://trac.osgeo.org/grass/ticket/1006 I can run it again in verbose mode when I get home tonight. Doug Doug Newcomb USFWS Raleigh, NC 919-856-4520 ext. 14 doug_newc...@fws.gov - The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
