[GRASS-dev] Fwd: Memory consumption using pygrass.utils.get_raster_for_points

[Steven Pawley]

As a follow-up to this, I tried tracking the memory usage using the python
Pympler package.

I ran the following code block:

from grass.pygrass.vector import VectorTopo
from grass.pygrass.raster import RasterRow
from grass.pygrass.modules.shortcuts import raster as r
from grass.pygrass.gis.region import Region
from grass.pygrass.utils import get_raster_for_points

# set region
reg = Region()
reg.from_rast("landclass96")
reg.write()
reg.set_raster_region()

# generate a large point dataset
r.random(input="landclass96", npoints=20, vector="landclass96_roi",
 overwrite=True)

# memory tracking
from pympler.tracker import SummaryTracker
tracker = SummaryTracker()

points = VectorTopo("landclass96_roi")
points.open("r")

# repeat spatial query of raster
for i in range(10):
print(i)
with RasterRow("lsat5_1987_10") as src:
arr = get_raster_for_points(points, src)
points.close()

tracker.print_diff()

The memory tracker results are:

   types |   # objects
|   total size
 | ===
| 
  
Date: Sat, Feb 9, 2019 at 9:48 AM
Subject: Memory consumption using pygrass.utils.get_raster_for_points
To: GRASS developers list 


Hello devs,

When running pygrass.utils.get_raster_for_points repeatedly, it appears
that the python memory allocation continuously increases until all ram is
consumed, even if the extracted values are not being collected (or are
overwriting the same variable).

I noticed this when extracting raster data at point locations, when using a
large point dataset, even though I had pre-allocated a numpy array to
receive the results.

Below is an example on the nc_spm_08_grass7 example data (in the landsat
mapset), repeating the operation say 50 times on the same point vector
dataset. I wouldn't have expected the memory consumption to continuously
increase for this operation, because I'm overwriting the 'arr' variable
each time. However, if you repeat this enough times, you will run out of
system memory and the allocated memory does not appear to be released, i.e.
even if you manually force garbage collection.

Any suggestions?


from grass.pygrass.vector import VectorTopo
from grass.pygrass.raster import RasterRow
from grass.pygrass.modules.shortcuts import raster as r
from grass.pygrass.gis.region import Region
from grass.pygrass.utils import get_raster_for_points

reg = Region()
reg.from_rast("landclass96")
reg.write()
reg.set_raster_region()

r.random(input="landclass96", npoints=20, vector="landclass96_roi",
 overwrite=True)

points = VectorTopo("landclass96_roi")
points.open("r")

# repeat spatial query of raster
for i in range(50):
with RasterRow("lsat5_1987_10") as src:
arr = get_raster_for_points(points, src)
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Survey: GRASS GIS migration to git

[Markus Neteler]

Am Mi., 13. Feb. 2019, 19:40 hat Markus Neteler 
geschrieben:

> On Tue, Feb 5, 2019 at 11:08 AM Markus Neteler  wrote:
> > Dear all,
> >
> > Time has come to also move the GRASS GIS development from SVN to git.
> >
> > In order to understand what you believe is the best solution (which
> > git hosting platform, how and where to handle tickets/issues, etc),
> > please fill in this survey by Feb 12, 2019:
> >
> >
> https://docs.google.com/forms/d/1BoTFyZRNebqVX98A3rh5GpUS2gKFfmuim78gbradDjc
> >
> > Everybody is invited to participate! We want to know your opinion!
>
> While already quite a few answers came in we extended the deadline to
> Feb, 12th 4pm UTC.
>

Sorry, Feb 15th of course.


> Best
> Markus
>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Memory consumption using pygrass.utils.get_raster_for_points

[Markus Neteler]

hi Steven

(yes, the mail came through)

On Sat, Feb 9, 2019 at 5:39 PM Steven Pawley  wrote:
>
> Hello devs,
>
> When running pygrass.utils.get_raster_for_points repeatedly, it appears
> that the python memory allocation continuously increases until all ram
> is consumed, even if the extracted values are not being collected (or
> are overwriting the same variable).
>
> I noticed this when extracting raster data at point locations, when
> using a large point dataset, even though I had pre-allocated a numpy
> array to receive the results.

Do you see a chance to run valgrind on this? (maybe there are more
appropriate tools in the Python world):

https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind

> Below is an example on the nc_spm_08_grass7 example data (in the landsat
> mapset), repeating the operation say 50 times on the same point vector
> dataset. I wouldn't have expected the memory consumption to continuously
> increase for this operation, because I'm overwriting the 'arr' variable
> each time. However, if you repeat this enough times, you will run out of
> system memory and the allocated memory does not appear to be released,
> i.e. even if you manually force garbage collection.
>
> Any suggestions?
>
>
> from grass.pygrass.vector import VectorTopo
> from grass.pygrass.raster import RasterRow
> from grass.pygrass.modules.shortcuts import raster as r
> from grass.pygrass.gis.region import Region
> from grass.pygrass.utils import get_raster_for_points
>
> reg = Region()
> reg.from_rast("landclass96")
> reg.write()
> reg.set_raster_region()
>
> r.random(input="landclass96", npoints=20, vector="landclass96_roi",
>   overwrite=True)
>
> points = VectorTopo("landclass96_roi")
> points.open("r")
>
> # repeat spatial query of raster
> for i in range(50):
>  with RasterRow("lsat5_1987_10") as src:
>  arr = get_raster_for_points(points, src)

Hope someone here can support you!

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Survey: GRASS GIS migration to git

[Markus Neteler]

On Tue, Feb 5, 2019 at 11:08 AM Markus Neteler  wrote:
> Dear all,
>
> Time has come to also move the GRASS GIS development from SVN to git.
>
> In order to understand what you believe is the best solution (which
> git hosting platform, how and where to handle tickets/issues, etc),
> please fill in this survey by Feb 12, 2019:
>
> https://docs.google.com/forms/d/1BoTFyZRNebqVX98A3rh5GpUS2gKFfmuim78gbradDjc
>
> Everybody is invited to participate! We want to know your opinion!

While already quite a few answers came in we extended the deadline to
Feb, 12th 4pm UTC.

Best
Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] i.sentinel improvements

[Luca Delucchi]

On Wed, 13 Feb 2019 at 15:02, Roberta Fagandini  wrote:
>
> I took a look at the interface and I saw a new text form for the output 
> suffix name..if the -c flag is checked, does the module create two different 
> output (13 atmospherically corrected bands and 13 topographically corrected 
> bands)?

right now yes

> maybe only one output (i.e. 13 bands to which both corrections are applied) 
> would be better.
>

could make sense, probably a new flag to keep/remove atmospherically
corrected bands could be useful

>
> ciao
> Roberta
>


-- 
ciao
Luca

www.lucadelu.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3624: Let g.search.modules list all modules

[GRASS GIS]

#3624: Let g.search.modules list all modules
---+-
  Reporter:  Nikos Alexandris  |  Owner:  grass-dev@…
  Type:  enhancement   | Status:  new
  Priority:  minor |  Milestone:
 Component:  Default   |Version:  unspecified
Resolution:|   Keywords:
   CPU:  Unspecified   |   Platform:  Unspecified
---+-

Comment (by Nikos Alexandris):

 See also #3751.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3583: g.search.modules to include addons

[GRASS GIS]

#3583: g.search.modules to include addons
---+--
  Reporter:  Nikos Alexandris  |  Owner:  grass-dev@…
  Type:  enhancement   | Status:  new
  Priority:  normal|  Milestone:  7.6.1
 Component:  Python|Version:  svn-trunk
Resolution:|   Keywords:  g.search.modules
   CPU:  Unspecified   |   Platform:  Unspecified
---+--

Comment (by Nikos Alexandris):

 See also #3751.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] [GRASS GIS] #3751: Logical operators for g.search.modules

[GRASS GIS]

#3751: Logical operators for g.search.modules
--+-
 Reporter:  Nikos Alexandris  |  Owner:  grass-dev@…
 Type:  enhancement   | Status:  new
 Priority:  normal|  Milestone:
Component:  Default   |Version:  unspecified
 Keywords:  g.search.modules  |CPU:  Unspecified
 Platform:  All   |
--+-
 Following are
 [https://trac.osgeo.org/grass/search?q=g.search.modules=1=on
 results searching for `g.search.modules`] tickets:  #3624, #3583, #3405,
 #3341, #3149, #386, #151.

 Considering these and the fact that
 [https://grass.osgeo.org/grass76/manuals/g.search.modules.html
 g.search.modules] features currently three flags (`-a`, `-n` and `-k`) to
 minapulate the retrieval of modules based on (combination) keywords,

 what about using parameters like `subset=` based on pre-defined sets that
 list modules **and/or** `selection=` based on logical operators to combine
 the requested keywords?

 Examples of describing the inerface:

 {{{
 subset Subset of available modules
options: all, core, addons, other(?)
default: core
 }}}

 or even (which might overlap the functionality of `keywords=`?)

 {{{
 subset Subset of available modules
options: all, core, addons, generic, raster, vector, miscellaneous,
 temporal
default: core
 }}}

 (under options also things like the
 [https://grass.osgeo.org/grass76/manuals/topics.html topics index] if
 topis are not covered by the
 [https://grass.osgeo.org/grass76/manuals/keywords.html keywords index] ?)

 and

 {{{
 selection Define selection of keywords
   options: and,or,not,xor,xnor,nand
   default: or
 }}}

 (obviously, excluding operators which would not make sense).

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] i.sentinel improvements

[Roberta Fagandini]

Il giorno mar 12 feb 2019 alle ore 11:35 Luca Delucchi 
ha scritto:

> On Mon, 17 Dec 2018 at 10:38, Luca Delucchi  wrote:
> >
> > Hi all,
> >
>
> Hi devs,
>

Hi Luca,


>
> > - i.sentinel.preproc add capability to compute topographic correction
> > using the following prcedure
> >
>
> done in r74080, please test it
>

Unfortunately I have no time to test it in the next days but I will as soon
as possible.
I took a look at the interface and I saw a new text form for the output
suffix name..if the -c flag is checked, does the module create two
different output (13 atmospherically corrected bands and 13 topographically
corrected bands)?
maybe only one output (i.e. 13 bands to which both corrections are applied)
would be better.


>
> --
> ciao
> Luca
>

ciao
Roberta


>
> www.lucadelu.org
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #3624: Let g.search.modules list all modules

[GRASS GIS]

#3624: Let g.search.modules list all modules
---+-
  Reporter:  Nikos Alexandris  |  Owner:  grass-dev@…
  Type:  enhancement   | Status:  new
  Priority:  minor |  Milestone:
 Component:  Default   |Version:  unspecified
Resolution:|   Keywords:
   CPU:  Unspecified   |   Platform:  Unspecified
---+-

Comment (by neteler):

 Replying to [ticket:3624 Nikos Alexandris]:
 > This would make it easy to get the full list of existing modules for
 further processing with standard *nix tools.

 That would be very good. Especially, since JSON output is available.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev