Just finished the test with 100G and the computation finish without error. So my conclusion is: even if setting the max value of RAM in the r.stream.basins to memory=25000 the effective requirements is a bit less the then 100G. A similar situation happens with r.stream.extract where the requested used RAM is a bit less then 50G.
Any thoughts? Thanks you Best Giuseppe On 26 November 2017 at 20:33, Giuseppe Amatulli <[email protected] > wrote: > Hi Markus M, > I re-tested (size 80490 x 142690 = 11,485,118,100 ) r.stream.extract > & r.stream.basins setting 25G RAM for the 2 commands with a slurm upper > limit of 60G RAM. > > The r.stream.extract use this RAM > ############################################################ > JobID MaxVMSize > ------------ ---------- > 6731334.bat+ 47459272K > ############################################################ > and finish without problem > > rather r.stream.basins use > ############################################################ > JobID MaxVMSize > ------------ ---------- > 6731334.bat+ 90241708K > ############################################################ > and get kill with this error. > > Reading raster map <stream>... > 0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88: > 15041 Bus error /gpfs/home/fas/sbsc/ga254/. > grass7/addons/bin/r.stream.basins -l stream_rast=stream direction=dir > basins=lbasin memory=25000 --o --verbose > > I think something is implemented different in r.stream.basins compare to > r.stream.extract. > I will try to ask for more RAM (100G) but I'm afraid that is going to fail > again. > > Any thoughts? > > Thanks you > Best > Giuseppe > > > > > > > > > > > > > On 17 November 2017 at 03:33, Markus Metz <[email protected]> > wrote: > >> >> >> On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli < >> [email protected]> wrote: >> > >> > >> > Hi Markus M. >> > >> > I was testing the r.stream.extract for 2 tiff >> > >> > 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error >> >> Great, that means large maps with more than 2 billion cells are supported. >> > >> > 2) 80490 x 142690 = 11,485,118,100 i got the following error >> > >> > A* Search... >> > 0..2../var/spool/slurmd/job6514787/slurm_script: line 80: 28925 Bus >> error r.stream.extract elevation=elv accumulation=upa >> threshold=0.5 depression=dep direction=dir stream_raster=stream >> memory=45000 --o --verbose >> >> From wikipedia: >> "a bus error is a fault raised by hardware, notifying an operating system >> (OS) that a process is trying to access memory that the CPU cannot >> physically address: an invalid address for the address bus, hence the name." >> >> I guess you either need to raise the RAM limit in slurm or slightly >> reduce the memory option for r.stream.extract. >> >> Markus M >> >> > >> > Do you think that is something with the slurm ram limitation or is >> something with the r.extract.stream? >> > If the the stream output have a number of stream segments larger than >> 2,147,483,647 what is happen? >> > Do I get an error or all the value larger than 2,147,483,647 are just >> rounded to 2,147,483,647 ? >> > >> > >> > Moreover, if use the stream obtain from option 1) and I use the stream >> as input for the r.stream.basins >> > I got the following error >> > >> > reading raster map <stream>... >> > 0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..45.. >> /var/spool/slurmd/job6514788/slurm_script: line 82: 17687 Bus error >> /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins >> -l stream_rast=stream direction=dir >> > >> > is this something that need to be fixed in r.stream.basins, or should i >> think that is due to other problems >> > >> > Thank you >> > Giuseppe >> > >> > >> > >> > >> > On 10 November 2017 at 11:21, Markus Neteler <[email protected]> wrote: >> >> >> >> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz >> >> <[email protected]> wrote: >> >> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli >> >> > <[email protected]> wrote: >> >> >> >> >> >> Thanks Markus!! >> >> >> I will test and I will let you know how it works. >> >> > >> >> > Your feedback is very helpful! >> >> >> >> >> >> I have few more questions >> >> >> 1) now how much is the upper limit matrix cell number that >> >> >> r.stream.extract can handle? >> >> > >> >> > About 1.15e+18 cells. >> >> > >> >> > Another limitation is the number of detected stream segments. This >> must not >> >> > be larger than 2,147,483,647 streams, >> >> ... >> >> >> >> (Added as a note to >> >> https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some >> _benchmarks >> >> ) >> >> >> >> best, >> >> markusN >> > >> > >> > >> > >> > -- >> > Giuseppe Amatulli, Ph.D. >> > >> > Research scientist at >> > Yale School of Forestry & Environmental Studies >> > Yale Center for Research Computing >> > Center for Science and Social Science Information >> > New Haven, 06511 >> > Teaching: http://spatial-ecology.net >> > Work: https://environment.yale.edu/profile/giuseppe-amatulli/ >> >> > > > -- > Giuseppe Amatulli, Ph.D. > > Research scientist at > Yale School of Forestry & Environmental Studies > Yale Center for Research Computing > Center for Science and Social Science Information > New Haven, 06511 > Teaching: http://spatial-ecology.net > Work: https://environment.yale.edu/profile/giuseppe-amatulli/ > -- Giuseppe Amatulli, Ph.D. Research scientist at Yale School of Forestry & Environmental Studies Yale Center for Research Computing Center for Science and Social Science Information New Haven, 06511 Teaching: http://spatial-ecology.net Work: https://environment.yale.edu/profile/giuseppe-amatulli/
_______________________________________________ grass-user mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/grass-user
