Re: [GRASS-dev] Memory consumption using pygrass.utils.get_raster_for_points
hi Steven (yes, the mail came through) On Sat, Feb 9, 2019 at 5:39 PM Steven Pawley wrote: > > Hello devs, > > When running pygrass.utils.get_raster_for_points repeatedly, it appears > that the python memory allocation continuously increases until all ram > is consumed, even if the extracted values are not being collected (or > are overwriting the same variable). > > I noticed this when extracting raster data at point locations, when > using a large point dataset, even though I had pre-allocated a numpy > array to receive the results. Do you see a chance to run valgrind on this? (maybe there are more appropriate tools in the Python world): https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind > Below is an example on the nc_spm_08_grass7 example data (in the landsat > mapset), repeating the operation say 50 times on the same point vector > dataset. I wouldn't have expected the memory consumption to continuously > increase for this operation, because I'm overwriting the 'arr' variable > each time. However, if you repeat this enough times, you will run out of > system memory and the allocated memory does not appear to be released, > i.e. even if you manually force garbage collection. > > Any suggestions? > > > from grass.pygrass.vector import VectorTopo > from grass.pygrass.raster import RasterRow > from grass.pygrass.modules.shortcuts import raster as r > from grass.pygrass.gis.region import Region > from grass.pygrass.utils import get_raster_for_points > > reg = Region() > reg.from_rast("landclass96") > reg.write() > reg.set_raster_region() > > r.random(input="landclass96", npoints=20, vector="landclass96_roi", > overwrite=True) > > points = VectorTopo("landclass96_roi") > points.open("r") > > # repeat spatial query of raster > for i in range(50): > with RasterRow("lsat5_1987_10") as src: > arr = get_raster_for_points(points, src) Hope someone here can support you! Markus ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Memory consumption using pygrass.utils.get_raster_for_points
Hello devs, When running pygrass.utils.get_raster_for_points repeatedly, it appears that the python memory allocation continuously increases until all ram is consumed, even if the extracted values are not being collected (or are overwriting the same variable). I noticed this when extracting raster data at point locations, when using a large point dataset, even though I had pre-allocated a numpy array to receive the results. Below is an example on the nc_spm_08_grass7 example data (in the landsat mapset), repeating the operation say 50 times on the same point vector dataset. I wouldn't have expected the memory consumption to continuously increase for this operation, because I'm overwriting the 'arr' variable each time. However, if you repeat this enough times, you will run out of system memory and the allocated memory does not appear to be released, i.e. even if you manually force garbage collection. Any suggestions? from grass.pygrass.vector import VectorTopo from grass.pygrass.raster import RasterRow from grass.pygrass.modules.shortcuts import raster as r from grass.pygrass.gis.region import Region from grass.pygrass.utils import get_raster_for_points reg = Region() reg.from_rast("landclass96") reg.write() reg.set_raster_region() r.random(input="landclass96", npoints=20, vector="landclass96_roi", overwrite=True) points = VectorTopo("landclass96_roi") points.open("r") # repeat spatial query of raster for i in range(50): with RasterRow("lsat5_1987_10") as src: arr = get_raster_for_points(points, src) ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Memory consumption using pygrass.utils.get_raster_for_points
Hello devs, When running pygrass.utils.get_raster_for_points repeatedly, it appears that the python memory allocation continuously increases until all ram is consumed, even if the extracted values are not being collected (or are overwriting the same variable). I noticed this when extracting raster data at point locations, when using a large point dataset, even though I had pre-allocated a numpy array to receive the results. Below is an example on the nc_spm_08_grass7 example data (in the landsat mapset), repeating the operation say 50 times on the same point vector dataset. I wouldn't have expected the memory consumption to continuously increase for this operation, because I'm overwriting the 'arr' variable each time. However, if you repeat this enough times, you will run out of system memory and the allocated memory does not appear to be released, i.e. even if you manually force garbage collection. Any suggestions? from grass.pygrass.vector import VectorTopo from grass.pygrass.raster import RasterRow from grass.pygrass.modules.shortcuts import raster as r from grass.pygrass.gis.region import Region from grass.pygrass.utils import get_raster_for_points reg = Region() reg.from_rast("landclass96") reg.write() reg.set_raster_region() r.random(input="landclass96", npoints=20, vector="landclass96_roi", overwrite=True) points = VectorTopo("landclass96_roi") points.open("r") # repeat spatial query of raster for i in range(50): with RasterRow("lsat5_1987_10") as src: arr = get_raster_for_points(points, src) ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev