On Mon, Nov 27, 2017 at 3:56 AM, Giuseppe Amatulli < [email protected]> wrote: > > Just finished the test with 100G and the computation finish without error. > > So my conclusion is: > even if setting the max value of RAM in the r.stream.basins to memory=25000 the effective requirements is a bit less the then 100G.
For r.stream.basins you need to use the -m flag, otherwise everything is done in RAM. A nice enhancement for all affected r.stream.* modules would be to switch automatically to disk swap mode if the amount of memory needed for all-in-RAM processing is larger than what is allocated with the memory=MB option. > A similar situation happens with r.stream.extract where the requested used RAM is a bit less then 50G. This is strange. In my tests, r.stream.extract always uses a bit less than what is given with the memory option. I could process a raster with 11,612,160,000 cells and memory=25000 and actual peak memory consumption was about 24000 MB. Markus M > > Any thoughts? > > Thanks you > Best > Giuseppe > > > > > > > On 26 November 2017 at 20:33, Giuseppe Amatulli < [email protected]> wrote: >> >> Hi Markus M, >> I re-tested (size 80490 x 142690 = 11,485,118,100 ) r.stream.extract & r.stream.basins setting 25G RAM for the 2 commands with a slurm upper limit of 60G RAM. >> >> The r.stream.extract use this RAM >> ############################################################ >> JobID MaxVMSize >> ------------ ---------- >> 6731334.bat+ 47459272K >> ############################################################ >> and finish without problem >> >> rather r.stream.basins use >> ############################################################ >> JobID MaxVMSize >> ------------ ---------- >> 6731334.bat+ 90241708K >> ############################################################ >> and get kill with this error. >> >> Reading raster map <stream>... >> 0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88: 15041 Bus error /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l stream_rast=stream direction=dir basins=lbasin memory=25000 --o --verbose >> >> I think something is implemented different in r.stream.basins compare to r.stream.extract. >> I will try to ask for more RAM (100G) but I'm afraid that is going to fail again. >> >> Any thoughts? >> >> Thanks you >> Best >> Giuseppe >> >> >> >> >> >> >> >> >> >> >> >> >> On 17 November 2017 at 03:33, Markus Metz <[email protected]> wrote: >>> >>> >>> >>> On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli < [email protected]> wrote: >>> > >>> > >>> > Hi Markus M. >>> > >>> > I was testing the r.stream.extract for 2 tiff >>> > >>> > 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error >>> >>> Great, that means large maps with more than 2 billion cells are supported. >>> > >>> > 2) 80490 x 142690 = 11,485,118,100 i got the following error >>> > >>> > A* Search... >>> > 0..2../var/spool/slurmd/job6514787/slurm_script: line 80: 28925 Bus error r.stream.extract elevation=elv accumulation=upa threshold=0.5 depression=dep direction=dir stream_raster=stream memory=45000 --o --verbose >>> >>> From wikipedia: >>> "a bus error is a fault raised by hardware, notifying an operating system (OS) that a process is trying to access memory that the CPU cannot physically address: an invalid address for the address bus, hence the name." >>> >>> I guess you either need to raise the RAM limit in slurm or slightly reduce the memory option for r.stream.extract. >>> >>> Markus M >>> >>> > >>> > Do you think that is something with the slurm ram limitation or is something with the r.extract.stream? >>> > If the the stream output have a number of stream segments larger than 2,147,483,647 what is happen? >>> > Do I get an error or all the value larger than 2,147,483,647 are just rounded to 2,147,483,647 ? >>> > >>> > >>> > Moreover, if use the stream obtain from option 1) and I use the stream as input for the r.stream.basins >>> > I got the following error >>> > >>> > reading raster map <stream>... >>> > 0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..45../var/spool/slurmd/job6514788/slurm_script: line 82: 17687 Bus error /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l stream_rast=stream direction=dir >>> > >>> > is this something that need to be fixed in r.stream.basins, or should i think that is due to other problems >>> > >>> > Thank you >>> > Giuseppe >>> > >>> > >>> > >>> > >>> > On 10 November 2017 at 11:21, Markus Neteler <[email protected]> wrote: >>> >> >>> >> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz >>> >> <[email protected]> wrote: >>> >> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli >>> >> > <[email protected]> wrote: >>> >> >> >>> >> >> Thanks Markus!! >>> >> >> I will test and I will let you know how it works. >>> >> > >>> >> > Your feedback is very helpful! >>> >> >> >>> >> >> I have few more questions >>> >> >> 1) now how much is the upper limit matrix cell number that >>> >> >> r.stream.extract can handle? >>> >> > >>> >> > About 1.15e+18 cells. >>> >> > >>> >> > Another limitation is the number of detected stream segments. This must not >>> >> > be larger than 2,147,483,647 streams, >>> >> ... >>> >> >>> >> (Added as a note to >>> >> https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some_benchmarks >>> >> ) >>> >> >>> >> best, >>> >> markusN >>> > >>> > >>> > >>> > >>> > -- >>> > Giuseppe Amatulli, Ph.D. >>> > >>> > Research scientist at >>> > Yale School of Forestry & Environmental Studies >>> > Yale Center for Research Computing >>> > Center for Science and Social Science Information >>> > New Haven, 06511 >>> > Teaching: http://spatial-ecology.net >>> > Work: https://environment.yale.edu/profile/giuseppe-amatulli/ >>> >> >> >> >> -- >> Giuseppe Amatulli, Ph.D. >> >> Research scientist at >> Yale School of Forestry & Environmental Studies >> Yale Center for Research Computing >> Center for Science and Social Science Information >> New Haven, 06511 >> Teaching: http://spatial-ecology.net >> Work: https://environment.yale.edu/profile/giuseppe-amatulli/ > > > > > -- > Giuseppe Amatulli, Ph.D. > > Research scientist at > Yale School of Forestry & Environmental Studies > Yale Center for Research Computing > Center for Science and Social Science Information > New Haven, 06511 > Teaching: http://spatial-ecology.net > Work: https://environment.yale.edu/profile/giuseppe-amatulli/
_______________________________________________ grass-user mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/grass-user
