date:20111203

Re: [GRASS-dev] winGRASS: PATHEXT question

2011-12-03 Thread Glynn Clements

Hamish wrote:

> > I have checked $PATHEXT which contains beside EXE
> > also PY extension.
> > The problem is that sh.exe ignores PATHEXT, right?
> 
> I'm pretty sure that is correct, PATHEXT is only
> recognized by DOS & cmd.exe

Right. But note that "cmd.exe" doesn't just mean running cmd.exe
interactively in a console window, but also C's system() and popen(),
Python's subprocess.Popen(), Windows' ShellExecute(), etc.

If the .py extension is removed, bash will still recognise it as a
script via the shebang, but native Windows APIs won't be able to
execute it.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: parallelizing GRASS modules

2011-12-03 Thread Hamish

Michael wrote:
> >> In our recursive landscape evolution module
> >> (r.landscape.evol.py),
...
> There are other things we're doing like working
> through long lists that might be speeded up,
> but these are in Python and Java, not GRASS.
> So they can't be helped by parallelization.

For help in finding the hogs I've just added a
section to the wiki about profiling python scripts:
  http://grass.osgeo.org/wiki/GRASS_Debugging#Profiling_python_scripts

which just points you to this page:
  http://docs.python.org/library/profile.html


Hamish
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] winGRASS: PATHEXT question

2011-12-03 Thread Hamish

Martin wrote:
> when working with CLI (sh.exe) in winGRASS7, I am
> able to launch C-modules, eg.
> r.buffer2
>
> but not a python script
> r.buffer -> command not found
> 
> The user has to type
> r.buffer.py
> 
> I have checked $PATHEXT which contains beside EXE
> also PY extension.
> The problem is that sh.exe ignores PATHEXT, right?

I'm pretty sure that is correct, PATHEXT is only
recognized by DOS & cmd.exe

if sh.exe (bash) does too it will only be because
the MinGW folks have added custom support for it.
but I would guess that they won't have done that.


Hamish
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: parallelizing GRASS modules

2011-12-03 Thread Hamish

Michael:
> So improvements to the segment library should
> help both
[r.cost and r.walk]

Perhaps better stated as learning how to run the
segment library in parallel. The changes would be
in the modules AFAIU, not in the library per se.
(although perhaps the could be, or both the lib and
module working together for it, ??)


> (and also v.surf.bspline and v.surf.rst??)

v.surf.bspline does use GRASS's segment library,
but v.surf.rst quadtree segmentation is something
different.

note Soeren has recently parallelized the bit of
the gmath library which v.surf.bspline uses which
for me made it 3.5x faster on 6-cores, but incurred
a 40% overhead penalty. Hopefully by parallelizing
the segments instead of the inner loops of the
gmath library we can get that overhead down near 0%
of the overall task.


also to consider is that the segment library was
created mainly as a way to keep RAM usage down on
big jobs. running all segments at once negates that
advantage. (but of course you can limit the number
of threads launched at run time using environment
variables if that is a problem)


Hamish
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] winGRASS: PATHEXT question

2011-12-03 Thread Martin Landa

Hi all,

when working with CLI (sh.exe) in winGRASS7, I am able to launch C-modules, eg.

r.buffer2

but not a python script

r.buffer -> command not found

The user has to type

r.buffer.py

I have checked $PATHEXT which contains beside EXE also PY extension.
The problem is that sh.exe ignores PATHEXT, right?

Martin

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] parallelizing GRASS modules

2011-12-03 Thread Hamish

Seth wrote:
> If you decide to use opencl, keep in contact and I
> might be able to help. I'm traveling for at least
> the next few days, though.

I plan to work on adding OpenCL support to the build
system soon after the OpenMP support is done.

the last missing piece for the OpenMP stuff is to
add a test compile to configure.in which does
  $(CC) $(OMPCFLAGS)
to check if the named compiler really has OpenMP
support built in.  (e.g. for gcc test if `gcc -fopenmp`
works, as that only exists for gcc versions >=
4.2.1)

it should be trivial but I'm no autoconf expert.

I see that Autoconf has a AC_OPENMP macro, but don't
know if that applies to version 2.13. I'm guessing
that it doesn't.
http://www.gnu.org/software/autoconf/manual/html_node/Generic-Compiler-Characteristics.html#Generic-Compiler-Characteristics

Hamish
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: parallelizing GRASS modules

2011-12-03 Thread Michael Barton



On Dec 3, 2011, at 3:38 PM, Hamish wrote:

> Michael wrote:
>> I lost the previous thread but wanted to respond
>> to your question about which modules might
>> benefit from speedup.
>> In our recursive landscape evolution module
>> (r.landscape.evol.py), the two GRASS modules that
>> take the most time are r.watershed, r.stats, and
>> r.walk, especially r.watershed and r.stats since
>> we need to run these every model cycle.
>> The speedup of r.watershed of a few years back
>> made an enormous difference in our model run
>> times. But it is still time consuming on
>> landscapes with large numbers of cells. If
>> parallelization could speed this up, it would be
>> great. I'm not sure that r.stats can be
>> parallelized or not, but speedup would be helpful.
> 
> 
> wrt r.walk: I would think to start with parallelizing r.cost, then porting
> the method over to r.walk. Both modules use the segment library, so coding
> it to process each segment in its own thread seems like the way to do that.
> (and more generally formulate + document a method to parallelize things that 
> use the segment library.
> perhaps a simple proof-of-method module to do that should come first)
> 

I was hoping that this would be the answer. So improvements to the segment 
library should help both (and also v.surf.bspline and v.surf.rst??)

> 
> wrt r.watershed: I guess we'd want a segment mode like the -m flag, but
> keeping things in RAM instead of using disk swap? Segment mode does make
> use of the segment library..   MarkusM?

Sounds promising.

> 
> 
> wrt r.stats: I suspect it is mostly I/O bound so I don't know how much faster
> it can be made, but I ran it through the valgrind profiler anyway just to see.
> (see the wiki Bugs page for recipe)
> 
> 22% of the time is spent in G_quant_get_cell_value()
> 
> 13% of the time is spent in lib/gis/get_row.c's cell_values_double()
> 
> 10% of the time is spent in r.stats/stats.c's update_cell_stats() updating a
> hash table. may be possible to parallelize that.

But not much gain it seems. 

> 
> if multiple input maps are used perhaps each could be processed in their own
> thread, but I don't think you are doing that with LandDyn.

You're right

> 
> perhaps the way that r.stats is called/used by LandDyn could be refined? ie
> is there a lot of unnecessary calculations going on just to get a simple
> stat out which could more efficiently be answered in another way? (no idea,
> but worth exploring)

No help in that regard. There are other things we're doing like working through 
long lists that might be speeded up, but these are in Python and Java, not 
GRASS. So they can't be helped by parallelization. 

This overall sounds encouraging, however.

Mihael

> 
> 
> Hamish

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GRASS 6: GISRC on Windows

2011-12-03 Thread Martin Landa

Hi,

2011/12/3 Hamish :
>> currently GIRC file is stored on Windows in
>>
>> $HOME/.grassrc6
>
> hmmm, I thought that already had happened. I guess
> not.
>
>> This location seems to be probably quite weird
>> for normal Windows user. I would suggest to store
>> GISRC in `$APPDATA\GRASS6\rc`.
>
> It seems like a good & natural idea to move it into
> $APPDATA\GRASS6 on Windows -- but do keep the
> 'grassrc6' name.

done in r49506, build for testing available at [1].

Martin

[1] http://wingrass.fsv.cvut.cz/grass65/WinGRASS-6.5.SVN-r49506-1-Setup.exe

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: parallelizing GRASS modules

2011-12-03 Thread Hamish

Michael wrote:
> I lost the previous thread but wanted to respond
> to your question about which modules might
> benefit from speedup.
> In our recursive landscape evolution module
> (r.landscape.evol.py), the two GRASS modules that
> take the most time are r.watershed, r.stats, and
> r.walk, especially r.watershed and r.stats since
> we need to run these every model cycle.
> The speedup of r.watershed of a few years back
> made an enormous difference in our model run
> times. But it is still time consuming on
> landscapes with large numbers of cells. If
> parallelization could speed this up, it would be
> great. I'm not sure that r.stats can be
> parallelized or not, but speedup would be helpful.

wrt r.walk: I would think to start with parallelizing r.cost, then porting
the method over to r.walk. Both modules use the segment library, so coding
it to process each segment in its own thread seems like the way to do that.
(and more generally formulate + document a method to parallelize things that 
use the segment library.
perhaps a simple proof-of-method module to do that should come first)

wrt r.watershed: I guess we'd want a segment mode like the -m flag, but
keeping things in RAM instead of using disk swap? Segment mode does make
use of the segment library..   MarkusM?

wrt r.stats: I suspect it is mostly I/O bound so I don't know how much faster
it can be made, but I ran it through the valgrind profiler anyway just to see.
(see the wiki Bugs page for recipe)

22% of the time is spent in G_quant_get_cell_value()

13% of the time is spent in lib/gis/get_row.c's cell_values_double()

10% of the time is spent in r.stats/stats.c's update_cell_stats() updating a
hash table. may be possible to parallelize that.

if multiple input maps are used perhaps each could be processed in their own
thread, but I don't think you are doing that with LandDyn.

perhaps the way that r.stats is called/used by LandDyn could be refined? ie
is there a lot of unnecessary calculations going on just to get a simple
stat out which could more efficiently be answered in another way? (no idea,
but worth exploring)

Hamish
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] parallelizing GRASS modules

2011-12-03 Thread Seth Price

If you decide to use opencl, keep in contact and I might be able to help. I'm 
traveling for at least the next few days, though.

~Seth


via iPhone

On Dec 3, 2011, at 3:07 PM, Barton Michael  wrote:

> I lost the previous thread but wanted to respond to your question about which 
> modules might benefit from speedup.
> 
> In our recursive landscape evolution module (r.landscape.evol.py), the two 
> GRASS modules that take the most time are r.watershed, r.stats, and r.walk, 
> especially r.watershed and r.stats since we need to run these every model 
> cycle. 
> 
> The speedup of r.watershed of a few years back made an enormous difference in 
> our model run times. But it is still time consuming on landscapes with large 
> numbers of cells. If parallelization could speed this up, it would be great. 
> I'm not sure that r.stats can be parallelized or not, but speedup would be 
> helpful.
> 
> Michael
> 
> C. Michael Barton
> Director, Center for Social Dynamics & Complexity 
> Professor of Anthropology, School of Human Evolution & Social Change
> Arizona State University
> 
> voice:480-965-6262 (SHESC), 480-727-9746 (CSDC)
> fax:  480-965-7671 (SHESC),  480-727-0709 (CSDC)
> www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] parallelizing GRASS modules

2011-12-03 Thread Barton Michael

I lost the previous thread but wanted to respond to your question about which 
modules might benefit from speedup.

In our recursive landscape evolution module (r.landscape.evol.py), the two 
GRASS modules that take the most time are r.watershed, r.stats, and r.walk, 
especially r.watershed and r.stats since we need to run these every model 
cycle. 

The speedup of r.watershed of a few years back made an enormous difference in 
our model run times. But it is still time consuming on landscapes with large 
numbers of cells. If parallelization could speed this up, it would be great. 
I'm not sure that r.stats can be parallelized or not, but speedup would be 
helpful.

Michael

C. Michael Barton
Director, Center for Social Dynamics & Complexity 
Professor of Anthropology, School of Human Evolution & Social Change
Arizona State University

voice:  480-965-6262 (SHESC), 480-727-9746 (CSDC)
fax:  480-965-7671 (SHESC),  480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu











___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements

2011-12-03 Thread GRASS GIS

#1444: g.parser does not filter for duplicate output map statements
--+-
  Reporter:  neteler  |   Owner:  grass-dev@…  
  Type:  defect   |  Status:  reopened 
  Priority:  minor|   Milestone:  6.4.2
 Component:  LibGIS   | Version:  6.4.1
Resolution:   |Keywords:  parser   
  Platform:  All  | Cpu:  All  
--+-

Comment(by neteler):

 Backported to 6.5 in r49509. I suggest to also backport to 6.4 (for 6.4.2)
 if no risk is involved.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements

2011-12-03 Thread GRASS GIS

#1444: g.parser does not filter for duplicate output map statements
--+-
  Reporter:  neteler  |   Owner:  grass-dev@…  
  Type:  defect   |  Status:  reopened 
  Priority:  minor|   Milestone:  6.4.2
 Component:  LibGIS   | Version:  6.4.1
Resolution:   |Keywords:  parser   
  Platform:  All  | Cpu:  All  
--+-
Changes (by neteler):

 * cc: lutra, pcav (added)


Comment:

 I have locally backported it and it helps:

 {{{
 GRASS 6.4.2svn (nc_spm_08):~ > v.what.vect myhospitals qvect=urbanarea
 column=urb_name qcolumn=NAME =silly ==d layer=1
 Sorry <=silly> is not a valid option
 Sorry <==d> is not a valid option
 }}}

 any risk to submit this backport to SVN?

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] wxGUI: new packages layout

2011-12-03 Thread Martin Landa

Hi Markus,

2011/12/2 Markus Neteler :

[...]

>> I think
>> that this period is perfect for GUI modules reorganization also in
>> devbr6/relbr64. I would suggest the roadmap below:
>>
>> 1) within next days reorganize wxGUI modules in devbr6 based on the
>> layout from trunk
>
> You mean: shuffle files around but not new code from GRASS 7?
> that could be fine.

Exactly.

>> 2) stabilize wxGUI in devb6 within one-two weeks
>
> I guess it will be a bit more effectively...
>
>> 3) *after* releasing 6.4.2 backport new layout also to relbr64
>
> Question: how much to 6.4 and 6.5 differ *now*?

Differs a bit. Anyway there is a plan to backport wxGUI after
releasing 6.4.2  from `devbr6` to `relbr64`. This model is working
since 6.4.0. The wxGUI has been backported immediately after release,
first two months kept in sync with `devbr6` and next four months
backports limited only to bug-fixes - assuming half of year period for
new release. It keeps wxGUI up-to-date and stable for release.

Martin

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

2011-12-03 Thread Martin Landa

2011/12/3 Hamish :
>> It would require to update URL in g.extension and
>> GRASS Addons Wiki page.

most of the broken links should be fixed [1].

Martin

[1] 
http://grass.osgeo.org/grass-wiki/index.php?title=GRASS_AddOns&action=historysubmit&diff=14457&oldid=14413

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GRASS 6: GISRC on Windows

2011-12-03 Thread Glynn Clements

Hamish wrote:

> also fwiw, "run commands" from the era when ~/.files
> were `source`d at program startup is pretty
> meaningless in this context. if it were going to be
> renamed (ie in trunk :-) perhaps it is better to
> change it to something more understandable than a
> semi-obsolete two letter acronym? On UNIX there is
> enough entrenched tradition to perhaps justify it,

Except that the "grassrc" file isn't actually an "rc" file.

It may have been at one time (given the G_getenv() name, I suspect
that GRASS variables may once have been actual environment variables),
but that's ancient history now.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

2011-12-03 Thread Martin Landa

2011/12/3 Hamish :
> And that we should do it before releasing 6.4.2
> with its (hopefully) working-for-everyone release
> of g.extension.[*]

should be done in r49504 (devbr6) and r49505 (relbr64).

Martin

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

2011-12-03 Thread Martin Landa

Hi,

2011/12/3 Hamish :
>> I think that it would be better just to move such
>> dirs to `grass6`.
>
> After reviewing this experiment I'm thinking the
> same thing.

done in r49502 & r49503.

[...]

Martin

-- 
Martin Landa  * http://geo.fsv.cvut.cz/~landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] winGRASS: PATHEXT question

[GRASS-dev] Re: parallelizing GRASS modules

Re: [GRASS-dev] winGRASS: PATHEXT question

[GRASS-dev] Re: parallelizing GRASS modules

[GRASS-dev] winGRASS: PATHEXT question

Re: [GRASS-dev] parallelizing GRASS modules

[GRASS-dev] Re: parallelizing GRASS modules

Re: [GRASS-dev] GRASS 6: GISRC on Windows

[GRASS-dev] Re: parallelizing GRASS modules

Re: [GRASS-dev] parallelizing GRASS modules

[GRASS-dev] parallelizing GRASS modules

[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements

[GRASS-dev] Re: [GRASS GIS] #1444: g.parser does not filter for duplicate output map statements

Re: [GRASS-dev] wxGUI: new packages layout

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

Re: [GRASS-dev] GRASS 6: GISRC on Windows

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

Re: [GRASS-dev] Re: [GRASS-SVN] r49486 - grass-addons/grass6

18 matches

Site Navigation

Mail list logo

Footer information