Re: comparing r.cost and r.terracost [was: [GRASS-dev] Re: grass-dev Digest, Vol 43, Issue 8]
Hi Markus, Your conclusions are based on the hypothesis that you can model the performance of r.cost in the presence of low memory by tweaking the memory limit in the code and using a machine with a large physical memory. I don't think that this hypothesis is true, and here is the evidence so far: r.cost on a machine with 8GB physical memory:1h r.cost on a machine with 2.5GB physical memory: 4h If you reboot the machine with 1GB RAM, you will see the running time go up (by a lot). Afterwards, try rebooting with 512GB RAM. I have run similar tests in the past, and r.cost did not finish in 40 hours. It may be better now, and you are the best person to re-try these tests as you know how to tweak it. I'll get back about what terracost is doing and why it has such large files after we see these new numbers. -Laura On Nov 17, 2009, at 3:51 PM, Markus Metz wrote: Hi Laura, Laura Toma wrote: Hi Markus, Processing a grid of 312 M cells takes about 8 x 312M = 2GB of RAM, That is only true for r.terracost with numtiles=1, because r.terracost stores costs as float. Is it possible that there is a bug in r.terracost when using numtiles 1, because creating 64GB of temporary files seems a bit inordinate for 2GB of data? And if r.terracost would use double for costs, it would be about 130GB of temporary files? OK, disk space is nearly for free nowadays. r.cost stores costs as double, so the size of temporary files is about 4GB. Additionally, 2GB where used for processing, i.e. at least 6GB of system RAM are required to also keep cached files in RAM. so on a machine with 8GB of RAM it will not use virtual memory at all, irrespective of how you tweak it. Right, but it still uses the disk IO algorithm and reads from/writes to disk. With 8GB of RAM, the correct comparison is between r.cost and r.terracost with numtiles=1 I don't think so because r.cost still uses its disk IO algorithm while r.terracost doesn't. That's like comparing r.watershed in ram mode with r.terraflow. A module not using a disk IO algorithm should always be faster than a corresponding module using a disk IO algorithm, as long as intermediate data fit into RAM. In other words, if you tweak r.cost, you also need to tweak r.terracost, which means you run with numtiles=1 for as long as data fits in real memory. I tweaked the disk IO algorithm to be faster, not to use less disk space. I can also do serious tweaking and write a true all-in-memory version of r.cost and compare that to r.terracost numtiles=1, but I'm interested in the performance of r.cost with the disk IO algorithm and thus compare it to r.terracost with its disk IO algorithm (requires numtiles 1). If you want any real numbers on how r.cost behaves with low memory you need to reboot the machine with 1GB or better 512MB of RAM. There is no way around it. Just try it, it is easy to do. I run experiments like this all the time. OK, would you mind running experiments with r.cost in grass7 and r.terracost numtiles1 so you can see for yourself? I rebooted with 2500MB of RAM in order to run the same test command as before on the 312 million cells region, giving about 2000MB of RAM to r.cost, same like before. I used the same region and start points as before because I think these settings are challenging for r.cost. My test system went into swap space, all memory was used up (system file cache was in swap anyway, OS needs some RAM too), and r.cost took, as expected, longer, namely 4 hours 10 min. Still much less than the 24 hours 22 min of r.terracost with memory=2000 and 8GB of system RAM... The latest version of r.cost (r39749) needs 2 hours 30 min with 2500MB of RAM and 2000MB of RAM assigned to it, remainder used by OS. From a user's perspective, one reason or side-effect concerning modules with disk IO algorithms is IMO that you do not need to use up all available system memory and can still do other things in parallel, so I would always assign max 75% percent of RAM to these modules and can still do other work, potentially preventing the system from caching files. BTW, there was a typo in my g.region command, must be res=30 in order to get 312 million cells, sorry! Markus M -Laura On Nov 14, 2009, at 6:51 AM, Markus Metz wrote: Hi Laura, Laura Toma wrote: my experience is that , if you want to see how an application would behave with 500 MB of RAM, you have to physically reboot the machine with 500 MB of RAM (it's very easy to do this on a Mac, and relatively easy on Linux. on windows, i don't know). if the machine has more than 500MB RAM, even if you restrict the application to use less, the system gives it all it can. in your setup, it is almost as if r.cost would run fully in memory, because even it it places the segments on disk, the system file cache fits all segments in memory. the same is true for terracost, its streams fit in memory. but using tiles has a big
Re: comparing r.cost and r.terracost [was: [GRASS-dev] Re: grass-dev Digest, Vol 43, Issue 8]
Hi Markus, Processing a grid of 312 M cells takes about 8 x 312M = 2GB of RAM, so on a machine with 8GB of RAM it will not use virtual memory at all, irrespective of how you tweak it. With 8GB of RAM, the correct comparison is between r.cost and r.terracost with numtiles=1 (do you have timings for this case?). In other words, if you tweak r.cost, you also need to tweak r.terracost, which means you run with numtiles=1 for as long as data fits in real memory. If you want any real numbers on how r.cost behaves with low memory you need to reboot the machine with 1GB or better 512MB of RAM. There is no way around it. Just try it, it is easy to do. I run experiments like this all the time. -Laura On Nov 14, 2009, at 6:51 AM, Markus Metz wrote: Hi Laura, Laura Toma wrote: my experience is that , if you want to see how an application would behave with 500 MB of RAM, you have to physically reboot the machine with 500 MB of RAM (it's very easy to do this on a Mac, and relatively easy on Linux. on windows, i don't know). if the machine has more than 500MB RAM, even if you restrict the application to use less, the system gives it all it can. in your setup, it is almost as if r.cost would run fully in memory, because even it it places the segments on disk, the system file cache fits all segments in memory. the same is true for terracost, its streams fit in memory. but using tiles has a big CPU overhead, which is why it is slower. I haven't rebooted my Linux box with less RAM, but I set up a test region with about 312 million cells (details below), I think we can agree that this is for current standards a pretty large region, maybe not in the future. Your argument still holds true that r.cost may have some advantage because its temp files are much smaller than the temp files of r.terracost and therefore a larger proportion can be cached by the system (beyond the control of the module). I could however see a lot of disk IO on both modules. For 312 million cells, r.cost needed 51 min, r.terracost needed 24 h 22 min, both got 2GB memory. Now that sounds like really bad news for r.terracost. But this is not the whole truth. First, I had to tweak r.cost a little bit in order to be so fast, still have to come up with a solution to do that tweaking in the module. Second, r.cost may suffer more from memory reduction, not physical RAM reduction, than r.terracost. Reducing the percent_memory option already slows the module down considerably. But that is also true for r.terracost, there the bottleneck seems to be INTERTILE DIJKSTRA which took well over 12 hours with heavy disk IO and full memory consumption. Third, r.cost performs better with less start points keeping region settings constant. I'm not sure if this applies as well to r.terracost. In summary, I think that on even larger regions, say 1 billion cells, and many small separate start points (100 000), r.terracost should outperform r.cost, but I would not bet on it ;-) For what I guess is current everyday use ( 100 million cells), r.cost in grass7 might most of the time outperform r.terracost with numtiles1, sometimes considerably as in my tests. Speed performance of r.cost is variable and dependent on the combination of region size, number and distribution of start points, and the amount of memory it is allowed to use. There may still be some scope for improvement in r.cost, I just did a quick job there, no in-depth code analysis (yet). The extraordinarily large temp files of r.terracost (total 64GB, largest single file was about 56GB, no typo) could be a handicap when processing such large regions. Finally, the results of the tests I did are valid for my test system only, they will be different on other systems. when i did some preliminary testing, i rebooted the machine with 512MB RAM, and ran r.cost on grids of 50M-100M cells. it was slow, completely IO bound, and took several hours or more. or if you use 1GB of RAM, you may need to go to larger grids. Please test r.cost in grass7 yourself, and maybe share your test commands, then others can run the tests too and compare. Here is my test region: The 312 million cells test region was created in the North Carolina sample dataset with g.region rast=elev_state_5...@permanent res=40 Then I created a cost layer with r.mapcalc cost = 1 You wanted many start points, so I generated 1 start points with v.random output=start_points_1 n=1 and converted this vector to raster with v.to.rast start_points_1 use=val val=1 out=start_points_1 --o The test command for r.cost was time r.cost input=cost start_rast=start_points_1 output=dist_random_1 percent_memory=40 --o This setting was equivalent to 2 GB of memory. time: real 51m18.172s user 34m4.067s sys 0m45.100s For r.terracost, I used as temp dir again a directory on a separate hard drive, faster than the one that r.cost used, so let's say tmpdir=/path
Re: [GRASS-dev] Re: grass-dev Digest, Vol 43, Issue 8
Hi Markus, How much memory was available on the machine? 8 GB If the machine had more than 512MB RAM, it is not fair to run terracost running with mem=400MB, and compare it with an algorithm that can use more memory. I don't understand, both modules can use more than 400MB of memory. I set r.terracost to use 400MB max and r.cost to use 135MB max. If anything, this is not fair for r.cost and gives an advantage to r.terracost. I did not mention that I gave r.terracost another advantage by assigning the temporary directories to a folder on a separate, very fast hard drive that had nothing else to do but manage the temp files of r.terracost. The temp files of r.cost are in the standard grass .tmp/$HOST directory, in my case that (slower) hard drive also had other things to do than just manage r.cost's temp files. I really tried to give r.terracost a head start ;-) my experience is that , if you want to see how an application would behave with 500 MB of RAM, you have to physically reboot the machine with 500 MB of RAM (it's very easy to do this on a Mac, and relatively easy on Linux. on windows, i don't know). if the machine has more than 500MB RAM, even if you restrict the application to use less, the system gives it all it can.in your setup, it is almost as if r.cost would run fully in memory, because even it it places the segments on disk, the system file cache fits all segments in memory. the same is true for terracost, its streams fit in memory. but using tiles has a big CPU overhead, which is why it is slower. when i did some preliminary testing, i rebooted the machine with 512MB RAM, and ran r.cost on grids of 50M-100M cells. it was slow, completely IO bound, and took several hours or more. or if you use 1GB of RAM, you may need to go to larger grids. However, I am surprised that withnumtiles=1, it was slower than r.cost. it looks like i misread your numbers. BTW, I took the liberty to fix r.terracost, it works now with numtiles1. See changelog for r39684 https://trac.osgeo.org/grass/changeset/39684 great, thanks! let's see if terracost is worthwhile :) -Laura ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: grass-dev Digest, Vol 40, Issue 17
Good idea, and this should cause no problem for the rest. -Laura Message: 6 Date: Fri, 7 Aug 2009 16:09:42 -0400 From: Andrew Danner adan...@cs.swarthmore.edu Subject: Re: [GRASS-dev] r.terraflow broken To: GRASS developers list grass-dev@lists.osgeo.org Message-ID: 1f298bfc0908071309m668324d5yd9d2a64264339...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Can we just initialize the MM_manager to ignore memory limits until we have a chance to set and enforce a limit? The change below allowed me to at least compile and run the help --- lib/iostream/mm.cc(revision 38633) +++ lib/iostream/mm.cc(working copy) @@ -458,7 +458,7 @@ MM_register MM_manager; int MM_register::instances = 0; // Number of instances. (init) // TPIE's register memory requests flag -MM_mode MM_register::register_new = MM_ABORT_ON_MEMORY_EXCEEDED; +MM_mode MM_register::register_new = MM_IGNORE_MEMORY_EXCEEDED; ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.terraflow broken
The problem is that it attempts to set the memory manager limit to 0 (see below: limit=0B), and the first line in MM_register::set_memory_limit(size_t new_limit) is assert(new_limit0). Therefore the problem. So the question is, why does it try to set the memory limit to 0? Unless the user specifies a value, the default value is 300MB, see main.cc: mem-answer=300; ... opt-mem = atoi(mem-answer); .. size_t mm_size = opt-mem 20;/* opt-mem is in MB */ MM_manager.set_memory_limit(mm_size); Did you use the default value of memory, or a different one? I wonder what size_t is on that platform. -Laura On Aug 3, 2009, at 2:47 PM, Markus Neteler wrote: Hi Laura, we have new problem in GRASS 7 with r.terraflow: On Mon, Aug 3, 2009 at 8:40 PM, Glynn Clementsgl...@gclements.plus.com wrote: It appears to be related to memory management: $ r.terraflow help MM error: limit =0B. allocating 24B. limit exceeded by 32B. r.terraflow: mm.cc:344: void* operator new(size_t): Assertion `0' failed. Aborted (core dumped) Happens at least on Gentoo (?) and Mandriva. But grass64/lib/iostream/mm.cc and grass70/lib/iostream/mm.cc look the same despite the commented header. Any idea? thanks Markus ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: comparing r.cost and r.terracost [was: Re: [GRASS-user] affecting cat of closest feature as pixel value]
This may be because you don't have rights to /var/tmp, or maybe the directory /var/tmp/mlenert does not exist. To change this location, you do not need to change main.cc. Instead use option VTMPDIR Location of intermediate STREAMs default: /var/tmp/username I tried r.terracost ...VTMPDIR=. and it put the intermediate streams in the current folder. The other potential problem is the location of the temporary streams. The default is /var/tmp/, but the user can set it with option STREAM_DIR Location of temporary STREAMs default: /var/tmp Even though main.cc explicitly executes a setenv, it did not work for me. I had to explicitly run from the shell: export STREAM_DIR=/var/tmp/ -Laura On Jul 15, 2009, at 12:49 PM, Moritz Lennert wrote: On 15/07/09 15:40, Laura Toma wrote: Hi Moritz, Terracost runs in two modes: in memory (when numtiles=1), and in external memory, using streams (when numtiles1). When run without a value for numtiles, it sets numtiles to the optimal value. In all your tests numtiles was set to 1. The algorithms in the 2 cases are different, so you also need to test with numtiles1. Trying to run with numtiles1, I get: TileFactory: Sorting internalstr... file=/var/tmp/mlennert/terracost.tileStr:cannot read!: Bad address r.terracost: /home/mlennert/SRC/GRASS/grass64_release/dist.i486-pc- linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAMT*, T*, unsigned int, Compare*) [with T = ijCostTypecostSourceType, Compare = ijTileCostCompareType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed. Aborted And the same, if I change all occurrences of /var/tmp in main.cc to / tmp Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: comparing r.cost and r.terracost [was: Re: [GRASS-user] affecting cat of closest feature as pixel value]
Hi Moritz, Terracost runs in two modes: in memory (when numtiles=1), and in external memory, using streams (when numtiles1). When run without a value for numtiles, it sets numtiles to the optimal value. In all your tests numtiles was set to 1. The algorithms in the 2 cases are different, so you also need to test with numtiles1. The difference in speed should increase a lot with the size of the raster. In the idea of replacing r.cost by r.terracost, I would think that the latter would first have to implement the options in r.cost, e.g. - equivalents of the start_points and coordinates options of r.cost - stop_points/coordinates - max_cost - the knight's move - ... What does the max_cost do ? I can easily add the Knight's move; is it generally agreed that it is a useful option? Without the Knight's move, each point can get to its 8 neighbors in one step. With the Knight's move, it can get to some additional points, which it was going to reach in two steps, at a slightly higher cost. The other options are in principle easy to add, but time is the problem.. -Laura On Jul 15, 2009, at 6:22 AM, Moritz Lennert wrote: Hello Laura, On 15/07/09 00:01, Laura Toma wrote: Hi Markus, Moritz, I could reproduce the problem. It was happening because the cost of the source points were incorrectly assigned to be zero, instead of the actual cost of the points. Because of this, the immediate neighbors of the source points had half costs. I fixed one line in initialize.cc and checked in the change in grass-addons/raster/ r.terracost (hopefully it is the right place). yes, perfect. Thanks for fixing this so quickly. Now results are almost identicial to r.cost without the knight's move. Differences are very small, and (my guess) probably due to differences in rounding: g.region n=10 s=0 w=0 e=10 res=1 -a r.mapcalc one=1.0 echo 5|5 | v.in.ascii out=start --o v.to.rast start use=val val=1 out=start --o time r.cost one start_rast=start out=r_cost --o time r.terracost one start_rast=start out=terra_cost --o r.mapcalc diff3=r_cost-terra_cost r.univar diff n: 100 minimum: -1.90012e-07 maximum: 1.91819e-07 range: 3.81831e-07 mean: -4.1721e-09 And image of diff: http://geog-pc40.ulb.ac.be/grass/misc/ comparison_rcost_rterracost_4.png This seems to indicate that r.cost overestimates diagonals and underestimates the straighter lines when compared to r.terracost. The same, but with a 100x100 region: r.univar diff: n: 1 minimum: -3.04995e-05 maximum: 9.16214e-06 range: 3.96616e-05 mean: 6.50277e-07 And image of diff: http://geog-pc40.ulb.ac.be/grass/misc/ comparison_rcost_rterracost_3.png This seems to confirm the above, but not for farther distances... Second test: g.region rast=elev_state_500m v.to.rast nc_state use=val val=500.0 out=cost --o v.to.rast urbanarea use=val val=1 out=urbanarea --o r.cost in=cost start_rast=urbanarea out=dist_urban --o r.terracost in=cost start_rast=urbanarea out=dist_urban_terracost --o r.mapcalc diff=dist_urban-dist_urban_terracost r.univar diff n: 513065 minimum: -0.0125342 maximum: 0.0684331 range: 0.0809673 mean: 0.00173129 Third test: g.region rast=landclass96 r.reclass landclass96 out=cost_landcover --o EOF 1 = 200 2 = 20 3 = 5 4 = 50 5 = 30 6 = 1000 7 = 5 EOF echo 629980|228527.25 | v.in.ascii out=point --o v.to.rast point use=val val=1 out=point --o r.cost in=cost_landcover out=dist_rcost start_rast=point --o r.terracost in=cost_landcover out=dist_terracost start_rast=point --o r.mapcalc diff2=dist_rcost-dist_terracost r.univar diff2 n: 249323 minimum: -0.554688 maximum: 0.523438 range: 1.07812 mean: -0.0104599 In all cases r.terracost takes about 1/4-1/3 the time of r.cost. In the idea of replacing r.cost by r.terracost, I would think that the latter would first have to implement the options in r.cost, e.g. - equivalents of the start_points and coordinates options of r.cost - stop_points/coordinates - max_cost - the knight's move - ... Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.in.gdal problems
On Jun 10, 2009, at 11:14 PM, Hamish wrote: GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america does it really extend from 180 lon to 180 lon? (ie include asia) It is SRTM3 data grouped by continents loaded from frome0srp01u.ecs.nasa.gov. The continents have different number of rows but the same number of columns and (EW resolution) of 360+ degrees, so i think they contain a whole slice from east to west, like below. Most of it is nodata, so changing the ewresolution is no problem. inline: austr.gif Thanks for suggesting all the things that I could try. -Laura ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] r.in.gdal problems
Hi, I imported a large SRTM grid with r.in.gdal; when starting grass64 with this location there is an error. Some part of Grass seems to work despite this error, except d.rast shows a blank screen. The east and west (see below) have the same value, except E/W. Could this cause troubles? GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america Error in startup script: divide by zero while executing expr round($parts(w)/$parts(ewres))*$parts(ewres) (procedure MapCanvas::zoom_gregion line 17) invoked from within MapCanvas::zoom_gregion $mon (procedure MapCanvas::create line 40) invoked from within MapCanvas::create (procedure Gm::startmon line 11) invoked from within Gm::startmon (procedure Gm::create line 79) invoked from within Gm::create (procedure main line 30) invoked from within main $argc $argv (file /usr/local/grass/grass-6.4.0RC4/etc/gm/gm.tcl line 566) GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america g.region -p projection: 3 (Latitude-Longitude) zone: 0 datum: wgs84 ellipsoid: wgs84 north: 61:00:01.5N south: 9:59:58.5N west: 179:59:58.5E east: 179:59:58.5W nsres: 0:00:03 ewres: 0:00:00.07 rows: 61201 cols: 432001 cells: 26438893201 GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america r.info na_mosaic + + | Layer:na_mosaic Date: Tue Jun 9 21:54:39 2009| | Mapset: PERMANENT Login of Creator: ltoma | | Location: northamericaSRTM3| | DataBase: /mnt/research/gis/DATA/ grassdb/ | | Title: ( na_mosaic ) | | Timestamp: none| | | | | | Type of Map: raster Number of Categories: 5573| | Data Type: CELL | | Rows: 61201 | | Columns: 432001 | | Total Cells: 26438893201| |Projection: Latitude- Longitude | |N: 61:00:01.5NS: 9:59:58.5N Res: 0:00:03 | |E: 179:59:58.5WW: 179:59:58.5E Res: 0:00:00.07| | Range of data:min = -291 max = 5573 | | | | Data Description:| |generated by r.in.gdal | | | | Comments :| |r.in.gdal input=na_mosaic.vrt output=na_mosaic | | | + + -Laura ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.in.gdal problems
Markus, Here is what the cellhd contains (it seems correct). proj: 3 zone: 0 north: 61:00:01.5N south: 9:59:58.5N east: 179:59:58.5W west: 179:59:58.5E cols: 432001 rows: 61201 e-w resol: 0:00:03 n-s resol: 0:00:03 format: 3 compressed: 1 but r.info gives: Projection: Latitude- Longitude | |N: 61:00:01.5NS: 9:59:58.5N Res: 0:00:03 | |E: 179:59:58.5WW: 179:59:58.5E Res: 0:00:00.07| so for some reason the ewres is messed up. Here is the tif file that I used as input in r.in.gdal: gdalinfo na_mosaic.tif Driver: GTiff/GeoTIFF Files: na_mosaic.tif Size is 432001, 61201 Coordinate System is: GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.2572235630016, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]] Origin = (-180.0004166,61.0004166) Pixel Size = (0.0008333,-0.0008333) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left (-180.0004167, 61.0004167) (180d 0'1.50W, 61d 0'1.50N) Lower Left (-180.0004167, 9.9995833) (180d 0'1.50W, 9d59'58.50N) Upper Right ( 180.0004167, 61.0004167) (180d 0'1.50E, 61d 0'1.50N) Lower Right ( 180.0004167, 9.9995833) (180d 0'1.50E, 9d59'58.50N) Center ( -0.000, 35.500) ( 0d 0'0.00W, 35d30'0.00N) Band 1 Block=432001x1 Type=Int16, ColorInterp=Gray NoData Value=-32768 [lt...@dover:~/gis/DATA/SRTM/SRTM-V2-frome0srp01u.ecs.nasa.gov/SRTM3/ north_america]$ On Jun 10, 2009, at 4:10 PM, Markus GRASS wrote: Laura Toma wrote: Hi, I imported a large SRTM grid with r.in.gdal; when starting grass64 with this location there is an error. Some part of Grass seems to work despite this error, except d.rast shows a blank screen. The east and west (see below) have the same value, except E/W. Could this cause troubles? Both values are smaller than 180, so they should not give problems in a ll projection. What could give problems is the ew resolution of 0:00:00.07 which is an amazing resolution for a DEM covering the whole world, especially SRTM (native resolution is 0:00:03 for the world coverage dataset). Should that not be the same like the ns resolution, 0:00:03? Maybe something went wrong with r.in.gdal? Does gdalinfo report the same ew resolution? The resolution of 0:00:00.07 can cause the divide by zero error when trying to set the current region and/or display parameters. A workaround may be to start grass in text mode and set the region resolution to 0:00:03, both ns and ew, then start the gui. Or manually edit the WIND and DEFAULT_WIND files before launching grass. Markus M GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america Error in startup script: divide by zero while executing expr round($parts(w)/$parts(ewres))*$parts(ewres) (procedure MapCanvas::zoom_gregion line 17) invoked from within MapCanvas::zoom_gregion $mon (procedure MapCanvas::create line 40) invoked from within MapCanvas::create (procedure Gm::startmon line 11) invoked from within Gm::startmon (procedure Gm::create line 79) invoked from within Gm::create (procedure main line 30) invoked from within main $argc $argv (file /usr/local/grass/grass-6.4.0RC4/etc/gm/gm.tcl line 566) GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america g.region -p projection: 3 (Latitude-Longitude) zone: 0 datum: wgs84 ellipsoid: wgs84 north: 61:00:01.5N south: 9:59:58.5N west: 179:59:58.5E east: 179:59:58.5W nsres: 0:00:03 ewres: 0:00:00.07 rows: 61201 cols: 432001 cells: 26438893201 GRASS 6.4.0RC4 (northamericaSRTM3):~/gis/DATA/SRTM/SRTM-V2- frome0srp01u.ecs.nasa.gov/SRTM3/north_america r.info na_mosaic + + | Layer:na_mosaic Date: Tue Jun 9 21:54:39 2009| | Mapset: PERMANENT Login of Creator: ltoma | | Location: northamericaSRTM3| | DataBase: /mnt/research/gis/DATA/grassdb/ | | Title: ( na_mosaic ) | | Timestamp: none| | | | | | Type of Map: raster Number of Categories: 5573| | Data Type: CELL | | Rows: 61201 | | Columns: 432001
[GRASS-dev] r.terracost uploaded to grass-addons
Hi, If you need a faster module for computing shortest paths , I have uploaded a module called r.terracost in grass-addons. It runs Dijkstra algorithm, which is the standard algorithm for SP (all costs need to be non-negative). The user specifies the input and output grid, the grid of start points, and the number of tiles.If you give it numtiles=1, it runs the entire computation in memory. If numtiles1, it switches to an external version, where it computes on one tile at a time. This is efficient in the case the grid is large and does not fit in memory. To find the suggested number of tiles, runt it with flag -i. hope this is useful -Laura ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] uploaded r.terracost to GRASS Addons
Andy,
Thanks for looking at this. I am downloading the grass-trunk right
now, and I'll be able to compile shortly.
-Laura
On Apr 8, 2009, at 2:16 PM, Andrew Danner wrote:
Markus Neteler wrote:
On Thu, Apr 2, 2009 at 7:08 PM, Laura Toma lt...@bowdoin.edu wrote:
If you're looking for a fast version of r.cost, try
r.terracost.I think
I put it at some point in the add-ons repository. Otherwise it's
available
from my website. Depending on the amount of memory you specify
on the
command line, it runs in memory (an optimized CPU algorithm) or
an I/O
algorithm.
I have (upon request from Laura) now uploaded r.terracost to Addons:
svn checkout https://svn.osgeo.org/grass/grass-addons/raster/r.terracost
r.terracost
It is r.terracost-1.1 but with various include file fixes, a new
description.html,
a new Makefile. Also I didn't not include the IOSTREAM lib as it is
already
present in GRASS 6 in the lib/ directory.
Compilation:
make MODULE_TOPDIR=/path/to/grass6svn/
Currently, I get these two errors:
iterator.cc: In member function ‘int
Iterator::getNext(ijCostSource*)’:
iterator.cc:42: error: no match for ‘operator=’ in ‘* out =
((Iterator*)this)-Iterator::tile-
Tile
::
anonymous
.mappedTilecostSourceType::anonymous.genericTileT::get
[with T = costSourceType](((int)((short
int)((Iterator*)this)-Iterator::iMarker)), ((int)((short
int)((Iterator*)this)-Iterator::jMarker)))’
input.h:160: note: candidates are: ijCostTypecostSourceType
ijCostTypecostSourceType::operator=(const
ijCostTypecostSourceType)
make: *** [OBJ.x86_64-unknown-linux-gnu/iterator.o] Error 1
get() returns a costSourceType. You want an
typedef ijCostTypecostSourceType ijCostSource returned by
getComplex (I think)
Index: iterator.cc
===
--- iterator.cc (revision 36633)
+++ iterator.cc (working copy)
@@ -39,7 +39,7 @@
if (iMarker tileSizeRows - 1)
return 0;
- *out = tile-get(iMarker, jMarker);
+ *out = tile-getComplex(iMarker, jMarker);
Makefile:15: warning: overriding commands for target
`/home/neteler/grass64/dist.x86_64-unknown-linux-gnu/bin/r.terracost'
/home/neteler/grass64//include/Make/Module.make:14: warning: ignoring
old commands for target
`/home/neteler/grass64/dist.x86_64-unknown-linux-gnu/bin/r.terracost'
c++ -I/home/neteler/grass64/dist.x86_64-unknown-linux-gnu/include -g
-O2 -DPACKAGE=\grassmods\
-I/home/neteler/grass64/dist.x86_64-unknown-linux-gnu/include -o
OBJ.x86_64-unknown-linux-gnu/main.o -c main.cc
main.cc: In function ‘void printGrid(AMI_STREAMijCostTypefloat
*)’:
main.cc:432: warning: deprecated conversion from string constant to
‘char*’
main.cc: In function ‘int main(int, char**)’:
main.cc:976: error: no matching function for call to
‘AMI_sort(AMI_STREAMdistanceType*, AMI_STREAMdistanceType**,
distanceIJCompareType*, int, char*)’
make: *** [OBJ.x86_64-unknown-linux-gnu/main.o] Error 1
Fixes appreciated.
There is no matching signature for that kind of AMI_sort. The last
two parameters are new and missing in the iostream library. Laura,
do you know what these parameters are for and is there an iostream
update?
For the const warnings
Index: stats.h
===
--- stats.h (revision 36633)
+++ stats.h (working copy)
@@ -28,14 +28,14 @@
#define HAVE_STATS
-int noclobberFile(char *);
+int noclobberFile(const char *);
class statsRecorder : public ofstream {
private:
Rtimer tm;
void *bss;
public:
- statsRecorder(char *fname);
+ statsRecorder(const char* fname);
Index: stats.cc
===
--- stats.cc(revision 36633)
+++ stats.cc(working copy)
@@ -113,7 +113,7 @@
int
-noclobberFile(char *fname) {
+noclobberFile(char * const fname) {
int fd=-1;
while(fd0) {
@@ -162,7 +162,7 @@
/*
** */
-statsRecorder::statsRecorder(char *fname) {
+statsRecorder::statsRecorder(char * const fname) {
for suspiciousness (missing template arguments):
Index: input.h
===
--- input.h (revision 36633)
+++ input.h (working copy)
@@ -194,15 +194,14 @@
cout *this;
};
- friend ostream operator (ostream s, const ijCostType ct) {
+ friend ostream operator (ostream s, const ijCostTypeT ct) {
s ( ct.getI() , ct.getJ() : ct.cs );
return s;
};
- friend int operator == (const ijCostType a, const ijCostType b) {
+ friend int operator == (const ijCostTypeT a, const
ijCostTypeT b) {
return (basicIJType(a) == basicIJType(b) a.cs == b.cs);
};
Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http
[GRASS-dev] Re: grass-dev Digest, Vol 36, Issue 4
If you're looking for a fast version of r.cost, try r.terracost.I think I put it at some point in the add-ons repository. Otherwise it's available from my website. Depending on the amount of memory you specify on the command line, it runs in memory (an optimized CPU algorithm) or an I/O algorithm. -Laura Today's Topics: 1. Re: r.cost (Markus Metz) -- Message: 1 Date: Thu, 02 Apr 2009 09:04:19 +0200 From: Markus Metz markus.metz.gisw...@googlemail.com Subject: Re: [GRASS-dev] r.cost To: GRASS developers list grass-dev@lists.osgeo.org Cc: cavall...@faunalia.it Message-ID: 49d46373.4000...@googlemail.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Paolo Cavallini wrote: Paolo Cavallini ha scritto: Markus Metz ha scritto: Could it be that the binary tree implementation in r.cost is not balanced? If yes, the search tree may degenerate on smooth surfaces towards a linked list, search time going from O(log n) to O(n). BTW, there are now three different generic balanced binary search tree implementations in the vector libs, plus sorted heaps. Just an idea. Tested on two different machines, (ubuntu+debian) 3 diff surfaces, always the same result. It seems something more structural. :( My comment was a suggestion to Colin Nielson to use another binary tree in r.cost if the current tree is not a balanced tree, and that there are generic balanced binary tree implementations in the vector libs that can be used by other developers in their modules. AFAIKT r.cost uses the same binary tree implementation in all versions, i.e. the results and the speed should be similar on different machines and with different grass versions. ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [BULK] grass-dev Digest, Vol 30, Issue 9
I remember we replaced the fseek() with an fseeko() because we ran into a bug where we were trying to seek to a large offset (in r.terracost). No objection to removing the 2 includes, given that it still compiles w/o them on other platforms. -Laura On Oct 6, 2008, at 9:18 PM, [EMAIL PROTECTED] wrote: Message: 4 Date: Tue, 7 Oct 2008 00:12:10 +0100 (BST) From: Paul Kelly [EMAIL PROTECTED] Subject: Re: [GRASS-dev] fseeko() in libiostream To: Andrew Danner [EMAIL PROTECTED] Cc: grass-dev@lists.osgeo.org Message-ID: [EMAIL PROTECTED] Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Mon, 6 Oct 2008, Andrew Danner wrote: fseek is not the same as fseeko. int fseek(FILE *stream, long offset, int whence); int fseeko(FILE *stream, off_t offset, int whence); If large file support is enabled, fseeko will convert off_t to a 64 bit type even on a 32-bit OS. I'm not sure if long will be 64 bit on a 64- bit OS for fseek. r.in.xyz may work with large files using plain old fseek as long as r.in.xyz doesn't make long seeks. ami_stream.h could potentially make long seeks (e.g., skip the first 8GB of a 30GB file), though in practice, I'm not sure that code that uses ami_stream (r.terraflow) actually uses this kind of fseek. Still the interface is designed to potentially use long seeks and I would be reluctant to simply replace fseeko with fseek in ami_stream.h. you may to add a #define to check the compiler type and use fseek only if fseeko does not exist. Right - I was vaguely aware of some configure checks for fseeko() but as ami_stream.h was the only file in GRASS using fseeko() I wasn't sure if it had been made redundant by something else. But looking closely at include/config.h, I see that we can conditionalise fseeko() on #ifdef HAVE_LARGEFILES. After doing that, on Windows fseeko() is redefined to fseeko64() and all then works well. A couple of further Windows compile errors in lib/iostream though: sh-2.04$ make c++ -I/c/grass/grass7/dist.i686-pc-mingw32/include -I/c/grass/extra/ include -O2 -s -I/c/grass/extra/include -D_FILE_OFFSET_BITS=64 - DPACKAGE=\grasslibs\ -I/c/grass/grass7/dist.i686-pc-mingw32/ include -o OBJ.i686-pc-mingw32/mm_utils.o -c mm_utils.cc mm_utils.cc:38:22: sys/mman.h: No such file or directory make: *** [OBJ.i686-pc-mingw32/mm_utils.o] Error 1 sh-2.04$ make c++ -I/c/grass/grass7/dist.i686-pc-mingw32/include -I/c/grass/extra/ include -O2 -s -I/c/grass/extra/include -D_FILE_OFFSET_BITS=64 - DPACKAGE=\grasslibs\ -I/c/grass/grass7/dist.i686-pc-mingw32/ include -o OBJ.i686-pc-mingw32/mm_utils.o -c mm_utils.cc c++ -I/c/grass/grass7/dist.i686-pc-mingw32/include -I/c/grass/extra/ include -O2 -s -I/c/grass/extra/include -D_FILE_OFFSET_BITS=64 - DPACKAGE=\grasslibs\ -I/c/grass/grass7/dist.i686-pc-mingw32/ include -o OBJ.i686-pc-mingw32/rtimer.o -c rtimer.cc rtimer.cc:38:26: sys/resource.h: No such file or directory make: *** [OBJ.i686-pc-mingw32/rtimer.o] Error 1 In the two respective cases I commented out the #includes for sys/ mman.h and sys/resource.h and everything compiled fine. r.terraflow compiles fine too. So I'll simply remove those two #includes unless there's any objection? Thanks for the feedback, Paul ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] iostream lib and r.terraflow/r.viewshed
Unfortunately I cannot reproduce the compile errors, so I cannot be of much help. I added r.viewshed in grass_trunk/raster/. It compiles and works for me. Looking forward to comments. -Laura On Aug 10, 2008, at 3:29 PM, Martin Landa wrote: Hi, 2008/8/10 Paul Kelly [EMAIL PROTECTED]: [...] Can you explain for my benefit why you made the change to lib/iostream/ami_stream.cc, and not to include/iostream/ ami_stream.h which is where the problem calls to strcpy seemed to be? Is it something to do with namespaces? well, I should leave fixing iostream lib and r.terraflow module to someone who knows this code. It was just dirty quick fix, iostream and r.terraflow is broken at least one week. Martin -- Martin Landa landa.martin gmail.com * http://gama.fsv.cvut.cz/ ~landa * ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] iostream lib and r.terraflow/r.viewshed
I added r.viewshed in grass_trunk/raster/. It compiles and works for me. Looking forward to comments. Should it maybe go in grass-addons rather than trunk for now? I could not get it to compile in the add-ons (probably i need to set up some grass parameters, etc).My understanding is that not everything that's in the trunk goes into the release brunch, that's why I thought it is safe and simpler to put it there. We can revert that and move it in the addons; just let me know. -Laura Paul ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] iostream lib and r.terraflow/r.viewshed
I added r.viewshed to grass-addons, and deleted it from the trunk. -Laura On Aug 13, 2008, at 4:47 PM, Paul Kelly wrote: On Wed, 13 Aug 2008, Laura Toma wrote: Unfortunately I cannot reproduce the compile errors, so I cannot be of much help. I wondered if you might be able to help with advice on the C++ headers. From the compile errors it seems to me that the problem is a lack of a definition for the strcpy() function in the ami_stream.h header. I suspect that adding #include cstring (thanks to Martin for the hint about the C++ header names) would solve the compilation errors, but I'm not sure if anybody with access to a problem system has tried it out yet - but I'm thinking of maybe just going ahead and making the change as it seems so appropriate. I added r.viewshed in grass_trunk/raster/. It compiles and works for me. Looking forward to comments. Should it maybe go in grass-addons rather than trunk for now? Paul ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: viewshed and iostream issues
Hi Paul, There should be no rtimer in viewshed, only in the library. I was planning to upload the most recent version as soon as I get access to the svn. -Laura On Aug 7, 2008, at 7:00 PM, Paul Kelly wrote: Hi Laura, On Tue, 5 Aug 2008, Laura Toma wrote: I have merged these changes into my most recent version of iostream, which I am attaching; it is split into headers (include_iostream.tar), and cc files (lib_iostream.tar). If you place them into grass_trunk/include/iostream/ and grass_trunk/lib/ iostream/ respectively, the library, terraflow and viewshed all compile and run (macosx 10.4). That all seems fine. I've committed the updates to SVN trunk and also merged them back into develbranch_6 (i.e. 6.4-svn). This is a start to get r.viewshed to compile and run and get some feedback. Will, do you want to commit it to grass-addons now? I've created a directory under raster/ for it to go. If you haven't got your OSGeo id and grass-addons access sorted out yet I can commit it for you if you want and then you could make further updates directly in the SVN. Let me know. One issue I noticed was with viewshed.cc including rtimer.h - I'm not sure if this is supposed to be the rtimer.h in the iostream lib or the rtimer.h in the r.viewshed source (maybe it could be renamed to reduce confusion?). If it's meant to be including the iostream version, you can change the include line to #include grass/iostream/rtimer.h to pick it up. Or if it's meant to be the one in the r.viewshed directory, #include rtimer.h should get it (the quotes rather than angle brackets indicate to look in the current directory for include files before searching the default include directories). Paul ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: iostream issues (formerly r.viewshed ready for testing)
Hi Paul, PEARL is off (not defined) when compiling, so it is not significant. Two libraries is definitely ugly. We'll try to get r.viewshed to work with the iostream version in GRASS. Until then, thanks for all the help. -Laura On Aug 5, 2008, at 12:45 PM, Paul Kelly wrote: On Tue, 5 Aug 2008, Will wrote: Hi Paul, Yes, I am using the updated tarball that you sent me, and I'm getting the same errors, regardless of if ami.h (the header for iostream) is included in distribute.cc or not. That seems to just be a coincidence. If I copy the ami_stream.h from the version of iostream that you included with r.viewshed into include/iostream (and run make in the top level directory so that this gets copied into the correct location for compilation), the compilation of r.viewshed gets a lot further, albeit with a lot of warnings. Basically the issue seems to be that the version of iostream that you have been working with is quite different from the version that r.terraflow uses. In particular, the licence statement is different (includes Duke University advertising clause) and there is a comment * PEARL upgrades: Rajiv Wickremesinghe 2004, 2005 I don't know what PEARL is, but other than that do you know (or can find out) are the updates by Rajiv Wickremesinghe significant to the functionality, and should they be included in GRASS to work with r.terraflow as well? I notice some comments around the code initialled RW which suggest there are a lot of little changes all over the place. I guess what we need to find out is how important these are, and should they be merged into GRASS. Or, a simpler solution would be can you make r.viewshed work with the version of iostream in GRASS. The alternative is having separate versions of the iostream library for the two modules which is really ugly IMHO. Good luck, Paul ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: Line of Sight Update
On Jul 9, 2008, at 12:40 PM, Paul Kelly wrote: Regarding the output: my suggestion was to output, for every cell that is visible, its height (or slope) *above* the line of sight; this will give an indication of how visible the point is. Similarly, for the cells that are *not* visible, you could compute their height (or slope) below the line of sight. High positive values will mean the point is very visible; high negative values will mean the point is very un-visible. I don't quite understand what the line of sight is here. What I think of as the line of sight is a straight line from the observer to the visible cell, passing straight through both. Some kind of output relating to how visible the cell is definitely sounds like a very good idea though. Sorry, I meant the height/slope above or below the horizon. In other words, at a point p, in addition to boolean visibility, we can compute the height difference between p and the lowest point at that location that would be visible by the observer. -Laura ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
