Re: [GRASS-user] G_malloc errors in i.segment
Hi Paul,
On 21/06/18 16:54, Paul Shapley wrote:
Hi Marcus,
Thanks again for your patience with this issue. It seems that whatever i
set in the memory of 'i.segment' to it throws up the same 'G_malloc'
error with the same 3 gb shortfall. I had another idea which is to make
a smaller subset image (but if i do this) will the new subset of 'id'
segments be applicable if applied to the entire image. For example if a
value of '5134' represents 'peat' could i use 'r.mapcalc' to query the
entire image for those values of '5134'. Supervised and Unsupervised
Classification would not yield the differences in the landscape i'm
looking for which 'i.segment' succeeds.
i.segment does not classify (i.e. label segments with a semantic
meaning). It only creates segments. Across an image you will have many
(without your size of image probably millions of segments which will
correspond to the same class, but each segment will have its own id. In
addition these ids are somewhat random, so you cannot deduct any meaning
from them.
In order to classify your map you will have to either use classification
rules or go for supervised classification of your segments. Currently
GRASS GIS offers the addons i.segment.stats and v.class.mlR to do this.
Check this article [1] and the related Jupyter notebook [2] for a more
detailled explanation.
Another tip linked to your large image: you can cut the image into small
tiles and segment each tile separately. You might have some border
effects, but by using i.cutlines to define the tiles you can reduce
those (or by using overlaps in a rectangular tiling scheme). You can
parallelize the segmentation process in order to speed it up. We use
i.segment.uspo for that here (at the same time optimizing the threshold
segmentation parameter separately for each tile), but you can also use
i.segment.hierarchical or simply create a little python script using
GridModule (which creates rectangular tiles), i.e. something like this:
from grass.pygrass.modules.grid.grid import GridModule
import grass.script as gscript
# Create dictionary with the parameter arguments for i.segment
kwargs = {'group' : 'YourGroup',
'output' : 'YourOutputMap',
'threshold' : YourDesiredThreshold,
'minsize' : YourDesiredMinsize,
etc...
'quiet' : True}
# Define your desired gridding, overlap between grids and number of
processes
grd = GridModule('i.segment',
width=width,
height=height,
overlap=overlap,
processes=processes,
split=False,
**kwargs)
# Run the module on each of the grid cells
grd.run()
Moritz
[1] http://www.mdpi.com/2072-4292/9/4/358
[2] https://github.com/tgrippa/Opensource_OBIA_processing_chain
Thanks,
Paul Shapley
On 21 June 2018 at 08:52, Markus Metz > wrote:
[please keep it in the list]
On Thu, Jun 21, 2018 at 9:12 AM, Paul Shapley mailto:p.shap...@gmail.com>> wrote:
>
> Thank you Markus...here are the results of g.region -p and r.info
>
> g.region -p
> projection: 99 (OSGB 1936 / British National Grid)
> zone: 0
> datum: osgb36
> ellipsoid: airy
> north: 242815
> south: 200862.25
> west: 263134.25
> east: 298833.75
> nsres: 0.25
> ewres: 0.25
> rows: 167811
> cols: 142798
> cells: 23963075178
> (Thu Jun 21 08:03:51 2018) Command finished (0 sec)
>
> (Thu Jun 21 08:08:02 2018)
> r.info map=APGB_aerial_1.1@PERMANENT
>
++
> | Map: APGB_aerial_1.1@PERMANENT Date: Tue May 29
09:11:31 2018 |
> | Mapset: PERMANENT Login of Creator:
administrator |
> | Location: TempLocation
|
> | DataBase: C:\
|
> | Title: APGB_aerial_1.1
|
> | Timestamp: none
|
>
||
> |
|
> | Type of Map: raster Number of Categories: 0
|
> | Data Type: CELL
|
> | Rows: 167811
|
> | Columns: 142798
|
> | Total Cells: 23963075178
|
> | Projection: OSGB 1936 / British National
Re: [GRASS-user] G_malloc errors in i.segment
Hi Marcus, Thanks again for your patience with this issue. It seems that whatever i set in the memory of 'i.segment' to it throws up the same 'G_malloc' error with the same 3 gb shortfall. I had another idea which is to make a smaller subset image (but if i do this) will the new subset of 'id' segments be applicable if applied to the entire image. For example if a value of '5134' represents 'peat' could i use 'r.mapcalc' to query the entire image for those values of '5134'. Supervised and Unsupervised Classification would not yield the differences in the landscape i'm looking for which 'i.segment' succeeds. Thanks, Paul Shapley On 21 June 2018 at 08:52, Markus Metz wrote: > [please keep it in the list] > > On Thu, Jun 21, 2018 at 9:12 AM, Paul Shapley wrote: > > > > Thank you Markus...here are the results of g.region -p and r.info > > > > g.region -p > > > projection: 99 (OSGB 1936 / British National Grid) > > zone: 0 > > datum: osgb36 > > ellipsoid: airy > > north: 242815 > > south: 200862.25 > > west: 263134.25 > > east: 298833.75 > > nsres: 0.25 > > ewres: 0.25 > > rows: 167811 > > cols: 142798 > > cells: 23963075178 > > (Thu Jun 21 08:03:51 2018) Command finished (0 sec) > > > > (Thu Jun 21 08:08:02 2018) > > > r.info map=APGB_aerial_1.1@PERMANENT > > > +-- > --+ > > | Map: APGB_aerial_1.1@PERMANENT Date: Tue May 29 09:11:31 > 2018| > > | Mapset: PERMANENT Login of Creator: > administrator | > > | Location: TempLocation > | > > | DataBase: C:\ > | > > | Title:APGB_aerial_1.1 > | > > | Timestamp: none > | > > |-- > --| > > | > | > > | Type of Map: raster Number of Categories: 0 > | > > | Data Type:CELL > | > > | Rows: 167811 > | > > | Columns: 142798 > | > > | Total Cells: 23963075178 > | > > |Projection: OSGB 1936 / British National Grid > | > > |N: 242815S: 200862.25 Res: 0.25 > | > > |E: 298833.75W: 263134.25 Res: 0.25 > | > > | Range of data:min = 0 max = 255 > | > > | > | > > | Data Description: > | > > |generated by r.in.gdal > | > > | > | > > | Comments: > | > > |r.in.gdal -o -k input="J:\Spatial Data\Aerial\APGB_received_20180416\ > | > > |\source\RGB_25cm\output\test\JPEG.png" output="APGB_aerial_1" > memory\ | > > |=300 offset=0 num_digits=0 > | > > | > | > > +-- > --+ > > (Thu Jun 21 08:08:02 2018) Command finished (0 sec) > > OK, so you really have a group of rasters with 24 billion cells. Still, do > you really need to run i.segment on the full extents with such a high > resolution? If yes, try i.segment with memory=25000. Even if the system has > more RAM, the operating system and any other programs also need some RAM, > therefore it is safer to not use all free memory. Be aware that i.segment > will take a long time (several days) to process such a large region. > > > > > Should the output image from 'r.in.gdal' have had more memory allocated > than the 300mb? > > The memory option of r.in.gdal applies to the input image, not the output > image. Sometimes more memory can help, sometimes it does not have any > effect. However, it does not affect subsequent processing steps. > > Markus M > > > > > Thanks again Markus...I appreciate your help. > > > > Paul Shapley > > > > > > > > > > On 20 June 2018 at 16:15, Markus Metz > wrote: > >> > >> > >> > >> On Wed, Jun 20, 2018 at 1:30 PM, Paul Shapley > wrote: > >> > > >> > Hi Users, > >> > > >> > Apologies if i'm posting this error again but i need to get pass the > memory allocation issue of trying to allocate 3gb memory. The Grass version > is (using file r.contour). Processor has 32 gb ram and i've allocated 3 > mb under memory option of 'i.segment'. > >> > > >> > System Info > > >> > GRASS version: 7.4.1 > > >> > GRASS SVN revision: r72807 > > >> > Build date: 2018-06-13 > > >> > Build platform: i386-w64-mingw32 > > >> > GDAL: 2.2.4 > > >> > PROJ.4: 4.9.3 > > >> > GEOS: 3.5.0 > > >> > SQLite: 3.17.0 > > >> > Python: 2.7.4 > > >> > wxPython: 2.8.12.1 > > >> > Platform: Windows-2008ServerR2-6.1.7601-SP1 (OSGeo4W) > > >> > > > >> > > >> > ERROR: G_malloc: unable to allocate 2995426350 bytes of memory at > imagery/i.segment/flag.c:25 > >> > >> this error appears because all other memory has already been allocated, > there is less than 3 GB free RAM left. > >> > >> It seems that the current region has more than 20 billion grid cells. > Does this match the input to i.segment? You might want to check the current > region settings and compare them with extents
Re: [GRASS-user] G_malloc errors in i.segment
[please keep it in the list] On Thu, Jun 21, 2018 at 9:12 AM, Paul Shapley wrote: > > Thank you Markus...here are the results of g.region -p and r.info > > g.region -p > projection: 99 (OSGB 1936 / British National Grid) > zone: 0 > datum: osgb36 > ellipsoid: airy > north: 242815 > south: 200862.25 > west: 263134.25 > east: 298833.75 > nsres: 0.25 > ewres: 0.25 > rows: 167811 > cols: 142798 > cells: 23963075178 > (Thu Jun 21 08:03:51 2018) Command finished (0 sec) > > (Thu Jun 21 08:08:02 2018) > r.info map=APGB_aerial_1.1@PERMANENT > ++ > | Map: APGB_aerial_1.1@PERMANENT Date: Tue May 29 09:11:31 2018| > | Mapset: PERMANENT Login of Creator: administrator | > | Location: TempLocation | > | DataBase: C:\ | > | Title:APGB_aerial_1.1 | > | Timestamp: none | > || > | | > | Type of Map: raster Number of Categories: 0 | > | Data Type:CELL | > | Rows: 167811 | > | Columns: 142798 | > | Total Cells: 23963075178 | > |Projection: OSGB 1936 / British National Grid | > |N: 242815S: 200862.25 Res: 0.25 | > |E: 298833.75W: 263134.25 Res: 0.25 | > | Range of data:min = 0 max = 255 | > | | > | Data Description: | > |generated by r.in.gdal | > | | > | Comments: | > |r.in.gdal -o -k input="J:\Spatial Data\Aerial\APGB_received_20180416\ | > |\source\RGB_25cm\output\test\JPEG.png" output="APGB_aerial_1" memory\ | > |=300 offset=0 num_digits=0 | > | | > ++ > (Thu Jun 21 08:08:02 2018) Command finished (0 sec) OK, so you really have a group of rasters with 24 billion cells. Still, do you really need to run i.segment on the full extents with such a high resolution? If yes, try i.segment with memory=25000. Even if the system has more RAM, the operating system and any other programs also need some RAM, therefore it is safer to not use all free memory. Be aware that i.segment will take a long time (several days) to process such a large region. > > Should the output image from 'r.in.gdal' have had more memory allocated than the 300mb? The memory option of r.in.gdal applies to the input image, not the output image. Sometimes more memory can help, sometimes it does not have any effect. However, it does not affect subsequent processing steps. Markus M > > Thanks again Markus...I appreciate your help. > > Paul Shapley > > > > > On 20 June 2018 at 16:15, Markus Metz wrote: >> >> >> >> On Wed, Jun 20, 2018 at 1:30 PM, Paul Shapley wrote: >> > >> > Hi Users, >> > >> > Apologies if i'm posting this error again but i need to get pass the memory allocation issue of trying to allocate 3gb memory. The Grass version is (using file r.contour). Processor has 32 gb ram and i've allocated 3 mb under memory option of 'i.segment'. >> > >> > System Info >> > GRASS version: 7.4.1 >> > GRASS SVN revision: r72807 >> > Build date: 2018-06-13 >> > Build platform: i386-w64-mingw32 >> > GDAL: 2.2.4 >> > PROJ.4: 4.9.3 >> > GEOS: 3.5.0 >> > SQLite: 3.17.0 >> > Python: 2.7.4 >> > wxPython: 2.8.12.1 >> > Platform: Windows-2008ServerR2-6.1.7601-SP1 (OSGeo4W) >> > >> > >> > ERROR: G_malloc: unable to allocate 2995426350 bytes of memory at imagery/i.segment/flag.c:25 >> >> this error appears because all other memory has already been allocated, there is less than 3 GB free RAM left. >> >> It seems that the current region has more than 20 billion grid cells. Does this match the input to i.segment? You might want to check the current region settings and compare them with extents and resolution as reported by r.info on the input to i.segment. >> >> Please provide the output of g.region -p and r.info if you want more detailed help. >> >> Markus M >> >> > >> > -- >> > Paul J. Shapley MSc CGeog (GIS) FRGS >> > >> > >> > ___ >> > grass-user mailing list >> > grass-user@lists.osgeo.org >> > https://lists.osgeo.org/mailman/listinfo/grass-user >> > > > > -- > Paul J. Shapley MSc CGeog (GIS) FRGS > ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] G_malloc errors in i.segment
On Wed, Jun 20, 2018 at 1:30 PM, Paul Shapley wrote: > > Hi Users, > > Apologies if i'm posting this error again but i need to get pass the memory allocation issue of trying to allocate 3gb memory. The Grass version is (using file r.contour). Processor has 32 gb ram and i've allocated 3 mb under memory option of 'i.segment'. > > System Info > GRASS version: 7.4.1 > GRASS SVN revision: r72807 > Build date: 2018-06-13 > Build platform: i386-w64-mingw32 > GDAL: 2.2.4 > PROJ.4: 4.9.3 > GEOS: 3.5.0 > SQLite: 3.17.0 > Python: 2.7.4 > wxPython: 2.8.12.1 > Platform: Windows-2008ServerR2-6.1.7601-SP1 (OSGeo4W) > > > ERROR: G_malloc: unable to allocate 2995426350 bytes of memory at imagery/i.segment/flag.c:25 this error appears because all other memory has already been allocated, there is less than 3 GB free RAM left. It seems that the current region has more than 20 billion grid cells. Does this match the input to i.segment? You might want to check the current region settings and compare them with extents and resolution as reported by r.info on the input to i.segment. Please provide the output of g.region -p and r.info if you want more detailed help. Markus M > > -- > Paul J. Shapley MSc CGeog (GIS) FRGS > > > ___ > grass-user mailing list > grass-user@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/grass-user ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
