Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6
Hi, 2011/12/3 Hamish hamis...@yahoo.com: I think that it would be better just to move such dirs to `grass6`. After reviewing this experiment I'm thinking the same thing. done in r49502 r49503. [...] Martin -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6
2011/12/3 Hamish hamis...@yahoo.com: And that we should do it before releasing 6.4.2 with its (hopefully) working-for-everyone release of g.extension.[*] should be done in r49504 (devbr6) and r49505 (relbr64). Martin -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS 6: GISRC on Windows
Hamish wrote: also fwiw, run commands from the era when ~/.files were `source`d at program startup is pretty meaningless in this context. if it were going to be renamed (ie in trunk :-) perhaps it is better to change it to something more understandable than a semi-obsolete two letter acronym? On UNIX there is enough entrenched tradition to perhaps justify it, Except that the grassrc file isn't actually an rc file. It may have been at one time (given the G_getenv() name, I suspect that GRASS variables may once have been actual environment variables), but that's ancient history now. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6
2011/12/3 Hamish hamis...@yahoo.com: It would require to update URL in g.extension and GRASS Addons Wiki page. most of the broken links should be fixed [1]. Martin [1] http://grass.osgeo.org/grass-wiki/index.php?title=GRASS_AddOnsaction=historysubmitdiff=14457oldid=14413 -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] wxGUI: new packages layout
Hi Markus, 2011/12/2 Markus Neteler nete...@osgeo.org: [...] I think that this period is perfect for GUI modules reorganization also in devbr6/relbr64. I would suggest the roadmap below: 1) within next days reorganize wxGUI modules in devbr6 based on the layout from trunk You mean: shuffle files around but not new code from GRASS 7? that could be fine. Exactly. 2) stabilize wxGUI in devb6 within one-two weeks I guess it will be a bit more effectively... 3) *after* releasing 6.4.2 backport new layout also to relbr64 Question: how much to 6.4 and 6.5 differ *now*? Differs a bit. Anyway there is a plan to backport wxGUI after releasing 6.4.2 from `devbr6` to `relbr64`. This model is working since 6.4.0. The wxGUI has been backported immediately after release, first two months kept in sync with `devbr6` and next four months backports limited only to bug-fixes - assuming half of year period for new release. It keeps wxGUI up-to-date and stable for release. Martin -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements
#1444: g.parser does not filter for duplicate output map statements
--+-
Reporter: neteler | Owner: grass-dev@…
Type: defect | Status: reopened
Priority: minor| Milestone: 6.4.2
Component: LibGIS | Version: 6.4.1
Resolution: |Keywords: parser
Platform: All | Cpu: All
--+-
Changes (by neteler):
* cc: lutra, pcav (added)
Comment:
I have locally backported it and it helps:
{{{
GRASS 6.4.2svn (nc_spm_08):~ v.what.vect myhospitals qvect=urbanarea
column=urb_name qcolumn=NAME =silly ==d layer=1
Sorry =silly is not a valid option
Sorry ==d is not a valid option
}}}
any risk to submit this backport to SVN?
--
Ticket URL: http://trac.osgeo.org/grass/ticket/1444#comment:11
GRASS GIS http://grass.osgeo.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements
#1444: g.parser does not filter for duplicate output map statements --+- Reporter: neteler | Owner: grass-dev@… Type: defect | Status: reopened Priority: minor| Milestone: 6.4.2 Component: LibGIS | Version: 6.4.1 Resolution: |Keywords: parser Platform: All | Cpu: All --+- Comment(by neteler): Backported to 6.5 in r49509. I suggest to also backport to 6.4 (for 6.4.2) if no risk is involved. -- Ticket URL: http://trac.osgeo.org/grass/ticket/1444#comment:12 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] parallelizing GRASS modules
I lost the previous thread but wanted to respond to your question about which modules might benefit from speedup. In our recursive landscape evolution module (r.landscape.evol.py), the two GRASS modules that take the most time are r.watershed, r.stats, and r.walk, especially r.watershed and r.stats since we need to run these every model cycle. The speedup of r.watershed of a few years back made an enormous difference in our model run times. But it is still time consuming on landscapes with large numbers of cells. If parallelization could speed this up, it would be great. I'm not sure that r.stats can be parallelized or not, but speedup would be helpful. Michael C. Michael Barton Director, Center for Social Dynamics Complexity Professor of Anthropology, School of Human Evolution Social Change Arizona State University voice: 480-965-6262 (SHESC), 480-727-9746 (CSDC) fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC) www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] parallelizing GRASS modules
If you decide to use opencl, keep in contact and I might be able to help. I'm traveling for at least the next few days, though. ~Seth via iPhone On Dec 3, 2011, at 3:07 PM, Barton Michael c.michael.bar...@gmail.com wrote: I lost the previous thread but wanted to respond to your question about which modules might benefit from speedup. In our recursive landscape evolution module (r.landscape.evol.py), the two GRASS modules that take the most time are r.watershed, r.stats, and r.walk, especially r.watershed and r.stats since we need to run these every model cycle. The speedup of r.watershed of a few years back made an enormous difference in our model run times. But it is still time consuming on landscapes with large numbers of cells. If parallelization could speed this up, it would be great. I'm not sure that r.stats can be parallelized or not, but speedup would be helpful. Michael C. Michael Barton Director, Center for Social Dynamics Complexity Professor of Anthropology, School of Human Evolution Social Change Arizona State University voice:480-965-6262 (SHESC), 480-727-9746 (CSDC) fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC) www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: parallelizing GRASS modules
Michael wrote: I lost the previous thread but wanted to respond to your question about which modules might benefit from speedup. In our recursive landscape evolution module (r.landscape.evol.py), the two GRASS modules that take the most time are r.watershed, r.stats, and r.walk, especially r.watershed and r.stats since we need to run these every model cycle. The speedup of r.watershed of a few years back made an enormous difference in our model run times. But it is still time consuming on landscapes with large numbers of cells. If parallelization could speed this up, it would be great. I'm not sure that r.stats can be parallelized or not, but speedup would be helpful. wrt r.walk: I would think to start with parallelizing r.cost, then porting the method over to r.walk. Both modules use the segment library, so coding it to process each segment in its own thread seems like the way to do that. (and more generally formulate + document a method to parallelize things that use the segment library. perhaps a simple proof-of-method module to do that should come first) wrt r.watershed: I guess we'd want a segment mode like the -m flag, but keeping things in RAM instead of using disk swap? Segment mode does make use of the segment library.. MarkusM? wrt r.stats: I suspect it is mostly I/O bound so I don't know how much faster it can be made, but I ran it through the valgrind profiler anyway just to see. (see the wiki Bugs page for recipe) 22% of the time is spent in G_quant_get_cell_value() 13% of the time is spent in lib/gis/get_row.c's cell_values_double() 10% of the time is spent in r.stats/stats.c's update_cell_stats() updating a hash table. may be possible to parallelize that. if multiple input maps are used perhaps each could be processed in their own thread, but I don't think you are doing that with LandDyn. perhaps the way that r.stats is called/used by LandDyn could be refined? ie is there a lot of unnecessary calculations going on just to get a simple stat out which could more efficiently be answered in another way? (no idea, but worth exploring) Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS 6: GISRC on Windows
Hi, 2011/12/3 Hamish hamis...@yahoo.com: currently GIRC file is stored on Windows in $HOME/.grassrc6 hmmm, I thought that already had happened. I guess not. This location seems to be probably quite weird for normal Windows user. I would suggest to store GISRC in `$APPDATA\GRASS6\rc`. It seems like a good natural idea to move it into $APPDATA\GRASS6 on Windows -- but do keep the 'grassrc6' name. done in r49506, build for testing available at [1]. Martin [1] http://wingrass.fsv.cvut.cz/grass65/WinGRASS-6.5.SVN-r49506-1-Setup.exe -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: parallelizing GRASS modules
On Dec 3, 2011, at 3:38 PM, Hamish wrote: Michael wrote: I lost the previous thread but wanted to respond to your question about which modules might benefit from speedup. In our recursive landscape evolution module (r.landscape.evol.py), the two GRASS modules that take the most time are r.watershed, r.stats, and r.walk, especially r.watershed and r.stats since we need to run these every model cycle. The speedup of r.watershed of a few years back made an enormous difference in our model run times. But it is still time consuming on landscapes with large numbers of cells. If parallelization could speed this up, it would be great. I'm not sure that r.stats can be parallelized or not, but speedup would be helpful. wrt r.walk: I would think to start with parallelizing r.cost, then porting the method over to r.walk. Both modules use the segment library, so coding it to process each segment in its own thread seems like the way to do that. (and more generally formulate + document a method to parallelize things that use the segment library. perhaps a simple proof-of-method module to do that should come first) I was hoping that this would be the answer. So improvements to the segment library should help both (and also v.surf.bspline and v.surf.rst??) wrt r.watershed: I guess we'd want a segment mode like the -m flag, but keeping things in RAM instead of using disk swap? Segment mode does make use of the segment library.. MarkusM? Sounds promising. wrt r.stats: I suspect it is mostly I/O bound so I don't know how much faster it can be made, but I ran it through the valgrind profiler anyway just to see. (see the wiki Bugs page for recipe) 22% of the time is spent in G_quant_get_cell_value() 13% of the time is spent in lib/gis/get_row.c's cell_values_double() 10% of the time is spent in r.stats/stats.c's update_cell_stats() updating a hash table. may be possible to parallelize that. But not much gain it seems. if multiple input maps are used perhaps each could be processed in their own thread, but I don't think you are doing that with LandDyn. You're right perhaps the way that r.stats is called/used by LandDyn could be refined? ie is there a lot of unnecessary calculations going on just to get a simple stat out which could more efficiently be answered in another way? (no idea, but worth exploring) No help in that regard. There are other things we're doing like working through long lists that might be speeded up, but these are in Python and Java, not GRASS. So they can't be helped by parallelization. This overall sounds encouraging, however. Mihael Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] parallelizing GRASS modules
Seth wrote: If you decide to use opencl, keep in contact and I might be able to help. I'm traveling for at least the next few days, though. I plan to work on adding OpenCL support to the build system soon after the OpenMP support is done. the last missing piece for the OpenMP stuff is to add a test compile to configure.in which does $(CC) $(OMPCFLAGS) to check if the named compiler really has OpenMP support built in. (e.g. for gcc test if `gcc -fopenmp` works, as that only exists for gcc versions = 4.2.1) it should be trivial but I'm no autoconf expert. I see that Autoconf has a AC_OPENMP macro, but don't know if that applies to version 2.13. I'm guessing that it doesn't. http://www.gnu.org/software/autoconf/manual/html_node/Generic-Compiler-Characteristics.html#Generic-Compiler-Characteristics Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] winGRASS: PATHEXT question
Hi all, when working with CLI (sh.exe) in winGRASS7, I am able to launch C-modules, eg. r.buffer2 but not a python script r.buffer - command not found The user has to type r.buffer.py I have checked $PATHEXT which contains beside EXE also PY extension. The problem is that sh.exe ignores PATHEXT, right? Martin -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: parallelizing GRASS modules
Michael: So improvements to the segment library should help both [r.cost and r.walk] Perhaps better stated as learning how to run the segment library in parallel. The changes would be in the modules AFAIU, not in the library per se. (although perhaps the could be, or both the lib and module working together for it, ??) (and also v.surf.bspline and v.surf.rst??) v.surf.bspline does use GRASS's segment library, but v.surf.rst quadtree segmentation is something different. note Soeren has recently parallelized the bit of the gmath library which v.surf.bspline uses which for me made it 3.5x faster on 6-cores, but incurred a 40% overhead penalty. Hopefully by parallelizing the segments instead of the inner loops of the gmath library we can get that overhead down near 0% of the overall task. also to consider is that the segment library was created mainly as a way to keep RAM usage down on big jobs. running all segments at once negates that advantage. (but of course you can limit the number of threads launched at run time using environment variables if that is a problem) Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] winGRASS: PATHEXT question
Martin wrote: when working with CLI (sh.exe) in winGRASS7, I am able to launch C-modules, eg. r.buffer2 but not a python script r.buffer - command not found The user has to type r.buffer.py I have checked $PATHEXT which contains beside EXE also PY extension. The problem is that sh.exe ignores PATHEXT, right? I'm pretty sure that is correct, PATHEXT is only recognized by DOS cmd.exe if sh.exe (bash) does too it will only be because the MinGW folks have added custom support for it. but I would guess that they won't have done that. Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: parallelizing GRASS modules
Michael wrote: In our recursive landscape evolution module (r.landscape.evol.py), ... There are other things we're doing like working through long lists that might be speeded up, but these are in Python and Java, not GRASS. So they can't be helped by parallelization. For help in finding the hogs I've just added a section to the wiki about profiling python scripts: http://grass.osgeo.org/wiki/GRASS_Debugging#Profiling_python_scripts which just points you to this page: http://docs.python.org/library/profile.html Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] winGRASS: PATHEXT question
Hamish wrote: I have checked $PATHEXT which contains beside EXE also PY extension. The problem is that sh.exe ignores PATHEXT, right? I'm pretty sure that is correct, PATHEXT is only recognized by DOS cmd.exe Right. But note that cmd.exe doesn't just mean running cmd.exe interactively in a console window, but also C's system() and popen(), Python's subprocess.Popen(), Windows' ShellExecute(), etc. If the .py extension is removed, bash will still recognise it as a script via the shebang, but native Windows APIs won't be able to execute it. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
