Re: [GRASS-user] Working with a large number of maps
Hi, 2007/12/18, Craig Leat <[EMAIL PROTECTED]>: [snip] > Another question on the usage of r.patch: > I have passed the option --verbose to r.patch and I am expecting the program > to generate VERBOSE output, but all I'm seeing is the following: > > GRASS 6.3.0RC3 (msunduzi_lo31):~ > r.patch input=$MAPS > output="aerial_mosaic.red" --overwrite --verbose > r.patch: percent complete: > 15% In SVN-trunk: $ export GRASS_VERBOSE=0;r.patch in=elevation.10m,elevation.dem out=x [prints nothing] $ export GRASS_VERBOSE=1;r.patch in=elevation.10m,elevation.dem out=x 100% $ export GRASS_VERBOSE=2;r.patch in=elevation.10m,elevation.dem out=x 100% Creating support files for raster map $ export GRASS_VERBOSE=3;r.patch in=elevation.10m,elevation.dem out=x Percent complete... 100% Creating support files for raster map Note that GRASS_VERBOSE=0 is the same as --q, GRASS_VERBOSE=1 -> --v. What kind of verbose output are you expecting? Martin -- Martin Landa <[EMAIL PROTECTED]> * http://gama.fsv.cvut.cz/~landa * ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Hi, 2007/12/18, Martin Landa <[EMAIL PROTECTED]>: > Note that GRASS_VERBOSE=0 is the same as --q, GRASS_VERBOSE=1 -> --v. Ops, GRASS_VERBOSE=3 -> --v See http://grass.gdf-hannover.de/wiki/Development_Specs#Current_abstract_concept Martin -- Martin Landa <[EMAIL PROTECTED]> * http://gama.fsv.cvut.cz/~landa * ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Martin Landa wrote: Hi, 2007/12/18, Craig Leat <[EMAIL PROTECTED]>: [snip] What kind of verbose output are you expecting? Martin Hi I don't know what to expect, but was hoping for something like: r.patch progress: Currently processing map: abc.red, progress: 80% Total progress: 15% This may be something to be added to the wishlist? Craig ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Martin Landa wrote:
> > What kind of verbose output are you expecting?
Craig:
> I don't know what to expect, but was hoping for something like:
> r.patch progress:
> Currently processing map: abc.red, progress: 80%
> Total progress: 15%
>
> This may be something to be added to the wishlist?
--verbose is a no-op for most modules. The module needs to contain
G_verbose_message("some message here") calls in the code for --v to do
anything. That library function is very new (new for GRASS 6.3) so not
much uses it yet. There's an effort to move non-surprising status
messages to only show up in verbose mode, but it is early days for
that.
r.series works by row, then column, then by map. ie it works on all
maps at the same time (sorta) not sequentially. So G_percent() per row
is the best that can be done.
Your region is really really huge, I might suggest making 4 smaller
tiles out of it instead of one master map. I hope you compiled with LFS
(large file support) and your OS is 64bit.
By using 4 or more smaller region tiles to do the patch of non-
overlapping maps it may spend less time looking through empty space so
will be a lot faster. ?
You can use 'g.region -g' + 'r.info -g' to exclude maps which do not at
all fall in the output region. (this optimization might be built
directly into r.series?)
Alternate: let r.series run for x days.
Hamish
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Hi, 2007/12/18, Craig Leat <[EMAIL PROTECTED]>: [snip] > I don't know what to expect, but was hoping for something like: > r.patch progress: > Currently processing map: abc.red, progress: 80% > Total progress: 15% Since r.patch works by row (reads x-row of the first, second, ... given raster map), G_percent() per row seems to be the best solution. Martin -- Martin Landa <[EMAIL PROTECTED]> * http://gama.fsv.cvut.cz/~landa * ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Craig wrote: I can think of three workarounds. 1. Enter Rainer Krug and MapServer. I don't think my time frames allow for learning another software suite right now. 2. Use a script to build a .grc file with a group for gis.m. Before attempting this I would like to know if there are any limits to the number of layers that can be loaded. 3. Use r.patch to build one mosaic for each band. Hamish wrote: Your region is really really huge, I might suggest making 4 smaller tiles out of it instead of one master map. I hope you compiled with LFS (large file support) and your OS is 64bit. I wasn't aware that 32 vs 64 bit architecture affected the maximum file size (RAM size - Yes). I read the following article and since I compiled with LFS on an ext3 filesystem I thought I was good for 2TB. https://help.ubuntu.com/community/LinuxFilesystemsExplained? By using 4 or more smaller region tiles to do the patch of non- overlapping maps it may spend less time looking through empty space so will be a lot faster. ? You can use 'g.region -g' + 'r.info -g' to exclude maps which do not at all fall in the output region. (this optimization might be built directly into r.series?) Alternate: let r.series run for x days. Thanks for the valuable suggestions. I'm still hoping to get some comment on workaround number two mentioned above. Does anyone know of any pitfalls with trying to load 139x3 layers into a group for gis.m? I'm wondering if there might be a limit on the number of layers that can be loaded or if loading so many layers might seriously affect the performance of the map display? Craig ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Craig Leat wrote: > I have 139 high resolution images, which I need to use as a background map > for some digitizing work. Loading 139x3 layers into gis.m is a real pain and > I can think of three workarounds. > 3. Use r.patch to build one mosaic for each band. > I currently have r.patch running. It's been going for about 10.5hrs and has > reached 15%. Does this sound reasonable? I am starting to wonder if I'm > expecting too much. Here are my region settings: > > GRASS 6.3.0RC3 (msunduzi_lo31):~ > g.region -p > projection: 99 (Transverse Mercator) > zone: 0 > datum: wgs84 > ellipsoid: wgs84 > north: -3264858.18753122 > south: -3295545.14772988 > west: -92005.31394386 > east: -43582.78768323 > nsres: 0.3512 > ewres: 0.3519 > rows: 87677 > cols: 138350 > cells: 12130112950 How do the individual maps relate to the overall region? If they are (approximately) tiles, I would suggest using a hierarchical approach, patching groups of adjacent tiles into larger tiles, then patching the larger tiles together, so that each r.patch operation has relatively few maps, and the earlier stages have relatively small regions. The total time taken by r.patch is roughly proportional to the total number of cells in the region multiplied by the number of input maps, so the combination of a large region and a large number of maps will be slow. -- Glynn Clements <[EMAIL PROTECTED]> ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
Glynn Clements wrote: Craig Leat wrote: I have 139 high resolution images, which I need to use as a background map for some digitizing work. How do the individual maps relate to the overall region? If they are (approximately) tiles, I would suggest using a hierarchical approach, patching groups of adjacent tiles into larger tiles, then patching the larger tiles together, so that each r.patch operation has relatively few maps, and the earlier stages have relatively small regions. The total time taken by r.patch is roughly proportional to the total number of cells in the region multiplied by the number of input maps, so the combination of a large region and a large number of maps will be slow. Thanks for the advice Glynn. The images are tiles and the filenames follow a regular pattern, so I am able to patch blocks together quite easily with a simple script. I included lots of echos in the script and now I even have verbose messages ;-) The script is running right now and making good progress... Craig ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
When you get this all working, please post a description of the final outcome in the grass wiki. Thanks, Jerry Original message >Date: Wed, 19 Dec 2007 15:16:10 +0200 >From: Craig Leat <[EMAIL PROTECTED]> >Subject: Re: [GRASS-user] Working with a large number of maps >To: Glynn Clements <[EMAIL PROTECTED]> >Cc: grass-user@lists.osgeo.org > >Glynn Clements wrote: >> Craig Leat wrote: >> >>> I have 139 high resolution images, which I need to use as a background map >>> for some digitizing work. >>> >> How do the individual maps relate to the overall region? >> >> If they are (approximately) tiles, I would suggest using a >> hierarchical approach, patching groups of adjacent tiles into larger >> tiles, then patching the larger tiles together, so that each r.patch >> operation has relatively few maps, and the earlier stages have >> relatively small regions. >> >> The total time taken by r.patch is roughly proportional to the total >> number of cells in the region multiplied by the number of input maps, >> so the combination of a large region and a large number of maps will >> be slow. >> >> >Thanks for the advice Glynn. The images are tiles and the filenames >follow a regular pattern, so I am able to patch blocks together quite >easily with a simple script. I included lots of echos in the script and >now I even have verbose messages ;-) > >The script is running right now and making good progress... > >Craig >___ >grass-user mailing list >grass-user@lists.osgeo.org >http://lists.osgeo.org/mailman/listinfo/grass-user Gerald Nelson Professor, Dept. of Agricultural and Consumer Economics University of Illinois, Urbana-Champaign office: 217-333-6465 cell: 217-390-7888 315 Mumford Hall 1301 W. Gregory Urbana, IL 61801 ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Working with a large number of maps
JerryNelson wrote: > > When you get this all working, please post a description of the final > outcome in the grass wiki. > Thanks, > Jerry > Hi I have added a new entry to the FAQ to the best of my understanding of the problem and its solutions. Please visit http://grass.gdf-hannover.de/wiki/Working_with_many_rasters and modify/improve as necessary. Regards Craig. -- View this message in context: http://www.nabble.com/Working-with-a-large-number-of-maps-tp14384527p14715218.html Sent from the Grass - Users mailing list archive at Nabble.com. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
