Re: [GRASS-dev] [SoC] GSoC 2021 - Final Report [Parallelization of raster modules for GRASS GIS]

[Vaclav Petras]

On Tue, Aug 24, 2021 at 8:48 AM Veronica Andreo 
wrote:

>
> Huge thanks as well to Vaclav, Huidae and Maris for their commitment to
> the project and mentoring! You all did a great team work!
>

...and also to Anna who did a lot of testing!

>
> One question: are these changes planned to be merged before the creation
> of grass 8 branch or will they remain for 8+? Maybe add a milestone to
> clarify?
>

They are planned to be merged for v8.2. Now reflected in the milestone.

For those wondering how soon the merge will happen: Each PR now has a check
list of items which need to be confirmed before merging. Many of the tasks
are pretty accessible, so checking one or two yourself is the way to get
this merged faster.

There is also a GitHub project in the repo for this:

https://github.com/OSGeo/grass/projects/8
https://github.com/OSGeo/grass/pulls?q=is%3Aopen+is%3Apr+project%3AOSGeo%2Fgrass%2F8
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Update list of addons

[Vaclav Petras]

Hi Paulo,

Thanks for reporting these.

On Tue, Aug 24, 2021 at 8:27 AM Paulo van Breugel 
wrote:

>
> The list of addons on https://grass.osgeo.org/grass78/manuals/addons/ is
> not updated. See e.g., my previous email about the r.suitability.regions
> addon. Another example, it still lists v.clip, which is now part of the
> core.
>

I assume that's part of the restructuring of addons repo for v8. Please,
see #528 where we track the progress. Comment and/or edit as needed. Any
help in tracking down the issues and identifying required changes is
appreciated!

https://github.com/OSGeo/grass-addons/issues/528


>
> In addition, the links to the source code on github points to the master
> branch. Should that be updated to the grass7 branch?
>

That is correct. There might be a similar problem in core in case I missed
it. Open an issue or even tracking down the outdated line of code would be
helpful.

Vaclav



> Cheers,
>
> Paulo
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GSoC 2021: Final report: First steps towards a new GRASS GIS Single-Window GUI

[Veronica Andreo]

Dear Linda,

Thanks for such a detailed final report and for all your contributions to
make GRASS GUI easier and straightforward for users, especially first time
users! And thanks much for this cool new experimental single window GUI,
I'm eager to test :)

Once again, huge thanks to your mentors Anna, Vaclav, Stefan and Martin for
their dedication and for their commitment to the project!! Great team!

I'm sure we'll keep seeing your contributions after GSoC!

All the best,
Vero


El dom, 22 ago 2021 a las 20:15, Linda Kladivová ()
escribió:

> Hello everyone,
>
>
> I am sending my Final GSoC report. The more detailed version with
> permanent links on GitHub PRs and with several screenshots can be found at
> the project wiki:
> https://trac.osgeo.org/grass/wiki/GSoC/2021/SingleWindowLayout#FinalReport
> .
>
> *Abstract:*
> The project was focused mainly on the extensive Graphical User Interface
> refactoring necessary to prepare GRASS for Single-Window GUI. In addition
> to a good programming base, the simple working Single-Window GUI prototype
> was built in a parallel environment. This Single-Window frame largely
> copies the design (mockup) which the author had proposed for the GSoC
> application (see ​
> [1]).
> All main functionality will be put into operation after the successful
> merge of PR [2] and PR [3]. However,
> to completely replace the current Multi-Window GUI solution, many other
> improvements are still expected.
>
>
> *The state of the art BEFORE the start of GSoC:*
> Although the GRASS GUI has been enriched with many new features since last
> year, the basic Multi-Window GUI concept used in all historical GRASS
> versions has been preserved. As the name suggests Multi-Window GUI consists
> of several windows (frames) - the control window and additional separate
> map display windows. The control window includes a notebook containing 5
> tabs in a standard 2D map view - Data, Display, Modules, Console, and
> Python.
>
>
> *The state of the art AFTER GSoC:*
> A large part of the project was focused on GUI refactoring. The core work
> had to be done all at once in the PR ​
> [2] and since it overturns the
> core logic of map display widgets, it was decided to merge it after GSoC
> when the GRASS 8 will be released. To sum it up, we are probably not
> entirely done with the refactoring, but the main part was managed.
>
> In addition, a Single-Window GUI prototype has been coded in a parallel
> environment. I with help of my mentors handled to get to the stage where
> most of the functions are working. There is just missing completion of PR
> [3] and possible check/fix of workspaces functioning. It is important to
> emphasize here that I'm talking about basic functionality. To provide a
> really user-friendly environment, many other things will have to be changed
> or reprogrammed. All those steps are summed up in the "Next steps"
> paragraph.
>
>
> *Conclusion:*
> To follow the project idea I wrote for the GSoC application, I succeeded
> (with help of my mentors) to provide the first steps for a new era of
> Single-Window GUI. Those steps have a form of refactoring as well as a new
> parallel simple Single-Window GUI prototype. I believe it's one big step
> forward and I am looking forward to further development of which I would
> like to be a part.
>
>
> *Next Steps:*
> All these following points assume that the main functionality is working.
> In other words, we must have a merged main refactoring PR [2] and a merged
> PR [3]  aiming at the
> implementation of the AuiNotebook closing event. Then several improvements
> are possible (and probably necessary):
>
>- Enable undocking the Map Display notebook tab to a separate window.
>It will allow each user to use the Single-Window GUI as the Multi-Window
>GUI.
>- Fix switching to the Console pane in an automatic notebook if any
>information is written there.
>- Check/test if workspaces are functioning properly.
>- Focus on a newly added Map Display tab in the Display pane (now it
>may not be clear at first glance that it has been added).
>- Modify the appearance for the dark mode. Some parts are ugly and
>illegible (names of AuiNotebook tabs, names of panes, ugly gradients etc.).
>- Add the option for starting GRASS as the Single-Window GUI => we
>want GRASS geeks to try it out. :-)
>
> Other improvements mainly concern the organization of widgets and possible
> ideas for the future:
>
>- It is necessary to change the rendering of the 3D View panel. Now
>the 3D View pane is added as another panel under the Display tab - very
>problematic in terms of space.
>- A part of the Console pane in the startup Single-Window layout is
>not visible. We should completely reorganize the Console tab, su

Re: [GRASS-dev] [SoC] GSoC 2021 - Final Report [Parallelization of raster modules for GRASS GIS]

[Veronica Andreo]

Dear Aaron,

Thanks for your report and all your work to make GRASS raster modules run
faster, we like that :) Huge thanks as well to Vaclav, Huidae and Maris for
their commitment to the project and mentoring! You all did a great team
work!

One question: are these changes planned to be merged before the creation of
grass 8 branch or will they remain for 8+? Maybe add a milestone to
clarify?
One minor observation: all links currently point to the same PR

We hope to keep seeing you around, Aaron!

All the best,
Vero

El mar, 24 ago 2021 a las 3:18, Aaron Saw Min Sern ()
escribió:

> Hello everyone,
>
> Here is my final report for GSoC 2021 project, *Parallelization of Raster
> Modules for GRASS GIS*.
>
> *Abstract*
> The goal of this project is to introduce parallelization to existing
> raster modules in GRASS GIS using OpenMP. This will allow users to take
> advantage of more cores in their hardware to speed up the computation time
> especially for large raster files with large computation cost. The key
> challenge of this project is to separate the parallelizable components from
> the sequential part of the modules without introducing too much overhead in
> terms of memory, disk or computation resources
> *. *
>
> *Milestones*
>
> In total, I have introduced OpenMP support to 8 raster modules in GRASS
> GIS. The pull requests to each module are as follows:
>
>- r.univar - https://github.com/OSGeo/grass/pull/1634
>- r.neighbors - https://github.com/OSGeo/grass/pull/
>1724
>
>- r.mfilter - https://github.com/OSGeo/grass/pull/
>1708
>
>- r.resamp.filter - https://github.com/OSGeo/grass/pull/
>1759
>
>- r.resamp.interp - https://github.com/OSGeo/grass/pull/
>1771
>
>- r.slope.aspect - https://github.com/OSGeo/grass/pull/
>1767
>
>- r.series - https://github.com/OSGeo/grass/pull/
>1776
>
>- r.patch - https://github.com/OSGeo/grass/pull/
>1782
>
>
> Firstly, I have greatly underestimated the complexity of the work. Up to
> 20 modules were initially proposed at first but after the second week.
> However, it became clear that we had to cut down on the number of target
> modules and focus more on designing the algorithms. The modules we targeted
> behave differently as compared to some modules that had received OpenMP
> support in the past such as *r.sun*. In particular, the modules need to
> keep the same of behavior of having low memory footprint even after the
> parallelization, unlike *r.sun* which loads the entire raster map
> in-memory.
>
>
> During the first half of the GSoC, with the mentors’ discussion, we have
> come out with three different approaches to introducing parallel support to
> *r.neighbors*. After benchmarking their performance and taking account of
> their memory/disk usage, we decided to settle with the last approach which
> requires us to add an extra parameter *memory* to allow users to adjust
> their memory consumption. With this approach, we have to allow the modules
> to process the raster map by chunks. Once we settled about the design, we
> started applying the same approach to other similar modules with low memory
> footprints.
>
> For more information regarding the implementation, see
> https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP.
>
> Furthermore, test scripts were included in the modules to ensure the
> consistency of the results. Benchmark scripts were added to allow users to
> easily benchmark the performance of the parallelization to monitor the
> speedup in their own local machine. User documentations were also modified
> to include sections detailing how to make use of the newly added features.
>
> *Future Work*
>
> In the future, more raster modules can be parallelized using similar
> approach. Then, we can consider tackling more complex modules such as
> *r.watershed* and *r.mapcalc*. Also, we could consider exploring 3D
> raster modules as well.
>
>
> Furthermore, when we implement parallelization for *r.univar*, we notice
> that modules that produce statistics involving arithmetic can often have
> floating point discrepancies when dealing with large summation. Because of
> this, computation using different number of threads will now produce
> different results due to having different order of arithmetic. One idea
> would be to introduce *Kahan Summation algorithm* to reduce the
> floa

[GRASS-dev] Update list of addons

[Paulo van Breugel]

 Dear devs,

The list of addons on https://grass.osgeo.org/grass78/manuals/addons/ is
not updated. See e.g., my previous email about the r.suitability.regions
addon. Another example, it still lists v.clip, which is now part of the
core.

In addition, the links to the source code on github points to the master
branch. Should that be updated to the grass7 branch?

Cheers,

Paulo
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] lib/gis/band_reference.c ?

[Maris Nartiss]

Band reference rules were totally relaxed 7 days a go:
https://github.com/OSGeo/grass/commit/abe194dce78adf5f63885a6a09c452fc7ae4f735

New relaxed rule changes haven't propagated to the documentation yet
(PR's welcome).

Band reference editing via r.support was added on 4th of July:
https://github.com/OSGeo/grass/commit/ca1551206eaa0fc462f4849a06ebb035808470da

Still – keep in mind – there are no changes in band metadata as
managed by g.bands. Changes only deal with ability to have a band
reference without extended metadata in json files. Thus limitation of
not being able to define your own band reference is still in place,
now only you can assign a band reference without having a definition
(and thus any extended metadata apart from its name).

Māris.

2021-08-24 11:18 GMT+03:00, Stefan Blumentrath :
> Thanks Maris for the reply.
>
> I could not find that option in r.support.
>
> i.band allows to add e.g. S2_12 as band reference, but nothing custom. And
> the g.bands manual says that user-defined band references are not yet
> supported.
>
> Has that been added recently? Or would I have to edit the bands json file of
> the system first?
>
> My system is:
> g.version -ger
> version=8.0.dev
> date=2021
> revision=dd9d36830
> build_date=2021-07-08
> build_platform=x86_64-pc-linux-gnu
> build_off_t_size=8
> libgis_revision=d2a5e8e8f
> libgis_date=2021-06-16T04:05:21+00:00
> proj=7.0.0
> gdal=3.0.4
> geos=3.8.0
> sqlite=3.22.0
>
> Cheers
> Stefan
>
> -Original Message-
> From: Maris Nartiss 
> Sent: tirsdag 24. august 2021 09:03
> To: Stefan Blumentrath 
> Cc: Martin Landa ; Markus Metz
> ; GRASS developers list
> 
> Subject: Re: [GRASS-dev] lib/gis/band_reference.c ?
>
> GRASS supports arbitrary band reference names. Just make them unique enough
> to not mix together apples with oranges by accident (e.g. "min"
> is a bad idea, "min_t_c" is better; "NDVI" would work; "elevation" – bad,
> "elevation_m" – better).
> You can set band references after import with r.support module.
>
> Have fun,
> Māris.
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] lib/gis/band_reference.c ?

[Stefan Blumentrath]

Thanks Maris for the reply.

I could not find that option in r.support.

i.band allows to add e.g. S2_12 as band reference, but nothing custom. And the 
g.bands manual says that user-defined band references are not yet supported. 

Has that been added recently? Or would I have to edit the bands json file of 
the system first?

My system is:
g.version -ger
version=8.0.dev
date=2021
revision=dd9d36830
build_date=2021-07-08
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=d2a5e8e8f
libgis_date=2021-06-16T04:05:21+00:00
proj=7.0.0
gdal=3.0.4
geos=3.8.0
sqlite=3.22.0

Cheers
Stefan

-Original Message-
From: Maris Nartiss  
Sent: tirsdag 24. august 2021 09:03
To: Stefan Blumentrath 
Cc: Martin Landa ; Markus Metz 
; GRASS developers list 

Subject: Re: [GRASS-dev] lib/gis/band_reference.c ?

GRASS supports arbitrary band reference names. Just make them unique enough to 
not mix together apples with oranges by accident (e.g. "min"
is a bad idea, "min_t_c" is better; "NDVI" would work; "elevation" – bad, 
"elevation_m" – better).
You can set band references after import with r.support module.

Have fun,
Māris.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] lib/gis/band_reference.c ?

[Maris Nartiss]

GRASS supports arbitrary band reference names. Just make them unique
enough to not mix together apples with oranges by accident (e.g. "min"
is a bad idea, "min_t_c" is better; "NDVI" would work; "elevation" –
bad, "elevation_m" – better).
You can set band references after import with r.support module.

Have fun,
Māris.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev