Re: [GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
Hi! Very interested in this discussion. I am working with some QuickBird and WorldView images myself, for classification as well (in the coming days). However, I only want to deal with small areas. Let me know if I can load my new system to run any tests (although it has more RAM and disk-space than laptops or low-end PCs). I.e., I might try to replicate what Moritz is doing (meaning the region extent). Best, N [all deleted] ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
On Mon, Jul 8, 2013 at 2:18 PM, Moritz Lennert wrote: > After some improvements to the module by MarkusM, I tried again during the > night, but it got stuck again with heavy swapping going on. I'm definitively > giving up to segment the panchromatic image in its entirety on my machine. > > Question: Does the memory= parameter limit total memory usage of the module, > or does it only limit the memory usage for certain parts of it ? The memory= parameter should limit the total memory usage. In my tests, it used always a bit less memory than specified. It would be a bug if the module uses more memory than specified. The only limitations are 1) disk space, 2) how long you want to wait. Memory should not be a limit because of the memory option. > > > On 07/07/13 11:52, Yann Chemin wrote: >> > >> >> 2 - Cut in pieces (See Markus Comment about supercomputing) and run 4 >> quads on 4 CPUs instead of one large image in one CPU. > > > I can do that. Will this have any influence on segmentation results ? Of course. There will be edge effects, objects will not cross region borders. Therefore you would need one final run with the pieces patched together, thus using the former output as input seeds for the last run. This last run should be faster and use less memory because it uses seeds. On a standard HPC system, you are usually allowed to compile your own software in your $HOME. You could compile trunk there, then set up the tiles and tile management, and launch jobs with the HPC's job manager (typically gridengine or derivates, torque and/or MPI). >> >> 3 - Recode i.segment to work natively on heterogeneous computers... >> (would be fun!) > > > Would parallization help ? Should i.segment be included into the list of > OpenMP candidates: > http://grasswiki.osgeo.org/wiki/OpenMP#Candidates ? Hmm, because the final global run is needed to remove edge effects, I would expect the results to be different (not wrong, but different). But definitively an interesting idea! Markus M > > Moritz > > >> Good luck! >> >> On 4 July 2013 14:10, Moritz Lennert wrote: >>> >>> Hello, >>> >>> In parallel to the discussion going on in another thread, I have a >>> question >>> concering the segmentation of another Worldview 2 scene: >>> >>> I first used all 8 multispectral bands and managed to get a series of >>> results with increasing thresholds in very reasonable running times. The >>> region was as follows: >>> g.region -p >>> >>> projection: 1 (UTM) >>> zone: -36 >>> datum: wgs84 >>> ellipsoid: wgs84 >>> north: 7251172 >>> south: 7234772 >>> west: 333792 >>> east: 350192 >>> nsres: 2 >>> ewres: 2 >>> rows: 8200 >>> cols: 8200 >>> cells: 6724 >>> >>> and the command line: >>> >>> i.segment group=xs out=seg_xs minsize=2 memory=3072 threshold=0.05 (and >>> thresh=0.1 and 0.2 in successive runs using the results of the previous >>> run >>> as seeds). >>> >>> Now, I would like to test segmentation of just the panchromatic band. >>> This >>> means the following region settings: >>> >>> projection: 1 (UTM) >>> zone: -36 >>> datum: wgs84 >>> ellipsoid: wgs84 >>> north: 7251172 >>> south: 7234772 >>> west: 333792 >>> east: 350192 >>> nsres: 0.5 >>> ewres: 0.5 >>> rows: 32800 >>> cols: 32800 >>> cells: 107584 >>> >>> Trying to run with the following command line on my i3, 8GB RAM machine: >>> >>> i.segment group=pan out=seg_pan_005 threshold=0.05 memory=3072 >>> >>> had the process running for almost 13 hours with it then becoming >>> apparently >>> stuck in the fourth pass at 10%. At that point the percent didn't change >>> for >>> over an hour, so I decided to kill the process. Can I assume that I'm >>> here >>> above the capacities of my machine ? Is there anything (besides working >>> on a >>> smaller subsample of the image) that I can do to make it work ? What kind >>> of >>> resources would I need to be able to run such a segmentation? >>> >>> I guess I'll have to move these kinds of treatments to our university >>> supercomputer, but I first have to get them to install GRASS... >>> >>> Moritz >>> ___ >>> grass-dev mailing list >>> grass-dev@lists.osgeo.org >>> http://lists.osgeo.org/mailman/listinfo/grass-dev >> >> >> >> > > ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
After some improvements to the module by MarkusM, I tried again during the night, but it got stuck again with heavy swapping going on. I'm definitively giving up to segment the panchromatic image in its entirety on my machine. Question: Does the memory= parameter limit total memory usage of the module, or does it only limit the memory usage for certain parts of it ? On 07/07/13 11:52, Yann Chemin wrote: 1 - Find a faster CPU machine with 16Gb RAM and a SSD for a start, it will be better, Also if you have the opportunity check that the RAM is a 1800MHz one, details like this may actually add your computation performance. I'll try on a machine with more RAM and better CPU (i7 instead of i3). 2 - Cut in pieces (See Markus Comment about supercomputing) and run 4 quads on 4 CPUs instead of one large image in one CPU. I can do that. Will this have any influence on segmentation results ? 3 - Recode i.segment to work natively on heterogeneous computers... (would be fun!) Would parallization help ? Should i.segment be included into the list of OpenMP candidates: http://grasswiki.osgeo.org/wiki/OpenMP#Candidates ? Moritz Good luck! On 4 July 2013 14:10, Moritz Lennert wrote: Hello, In parallel to the discussion going on in another thread, I have a question concering the segmentation of another Worldview 2 scene: I first used all 8 multispectral bands and managed to get a series of results with increasing thresholds in very reasonable running times. The region was as follows: g.region -p projection: 1 (UTM) zone: -36 datum: wgs84 ellipsoid: wgs84 north: 7251172 south: 7234772 west: 333792 east: 350192 nsres: 2 ewres: 2 rows: 8200 cols: 8200 cells: 6724 and the command line: i.segment group=xs out=seg_xs minsize=2 memory=3072 threshold=0.05 (and thresh=0.1 and 0.2 in successive runs using the results of the previous run as seeds). Now, I would like to test segmentation of just the panchromatic band. This means the following region settings: projection: 1 (UTM) zone: -36 datum: wgs84 ellipsoid: wgs84 north: 7251172 south: 7234772 west: 333792 east: 350192 nsres: 0.5 ewres: 0.5 rows: 32800 cols: 32800 cells: 107584 Trying to run with the following command line on my i3, 8GB RAM machine: i.segment group=pan out=seg_pan_005 threshold=0.05 memory=3072 had the process running for almost 13 hours with it then becoming apparently stuck in the fourth pass at 10%. At that point the percent didn't change for over an hour, so I decided to kill the process. Can I assume that I'm here above the capacities of my machine ? Is there anything (besides working on a smaller subsample of the image) that I can do to make it work ? What kind of resources would I need to be able to run such a segmentation? I guess I'll have to move these kinds of treatments to our university supercomputer, but I first have to get them to install GRASS... Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
1 - Find a faster CPU machine with 16Gb RAM and a SSD for a start, it will be better, Also if you have the opportunity check that the RAM is a 1800MHz one, details like this may actually add your computation performance. 2 - Cut in pieces (See Markus Comment about supercomputing) and run 4 quads on 4 CPUs instead of one large image in one CPU. 3 - Recode i.segment to work natively on heterogeneous computers... (would be fun!) Good luck! On 4 July 2013 14:10, Moritz Lennert wrote: > Hello, > > In parallel to the discussion going on in another thread, I have a question > concering the segmentation of another Worldview 2 scene: > > I first used all 8 multispectral bands and managed to get a series of > results with increasing thresholds in very reasonable running times. The > region was as follows: > >> g.region -p > projection: 1 (UTM) > zone: -36 > datum: wgs84 > ellipsoid: wgs84 > north: 7251172 > south: 7234772 > west: 333792 > east: 350192 > nsres: 2 > ewres: 2 > rows: 8200 > cols: 8200 > cells: 6724 > > and the command line: > > i.segment group=xs out=seg_xs minsize=2 memory=3072 threshold=0.05 (and > thresh=0.1 and 0.2 in successive runs using the results of the previous run > as seeds). > > Now, I would like to test segmentation of just the panchromatic band. This > means the following region settings: > > projection: 1 (UTM) > zone: -36 > datum: wgs84 > ellipsoid: wgs84 > north: 7251172 > south: 7234772 > west: 333792 > east: 350192 > nsres: 0.5 > ewres: 0.5 > rows: 32800 > cols: 32800 > cells: 107584 > > Trying to run with the following command line on my i3, 8GB RAM machine: > > i.segment group=pan out=seg_pan_005 threshold=0.05 memory=3072 > > had the process running for almost 13 hours with it then becoming apparently > stuck in the fourth pass at 10%. At that point the percent didn't change for > over an hour, so I decided to kill the process. Can I assume that I'm here > above the capacities of my machine ? Is there anything (besides working on a > smaller subsample of the image) that I can do to make it work ? What kind of > resources would I need to be able to run such a segmentation? > > I guess I'll have to move these kinds of treatments to our university > supercomputer, but I first have to get them to install GRASS... > > Moritz > ___ > grass-dev mailing list > grass-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/grass-dev -- ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
On Thu, Jul 4, 2013 at 10:40 AM, Moritz Lennert wrote: ... > I guess I'll have to move these kinds of treatments to our university > supercomputer, but I first have to get them to install GRASS... If you/they need assistance with that, we have accumulated some experience in the past years with GRASS GIS 7 on various supercomputer operating systems. Markus ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] i.segment on panchromatic band of Worldview 2 scene: what resources are necessary to complete segmentation ?
Hello, In parallel to the discussion going on in another thread, I have a question concering the segmentation of another Worldview 2 scene: I first used all 8 multispectral bands and managed to get a series of results with increasing thresholds in very reasonable running times. The region was as follows: > g.region -p projection: 1 (UTM) zone: -36 datum: wgs84 ellipsoid: wgs84 north: 7251172 south: 7234772 west: 333792 east: 350192 nsres: 2 ewres: 2 rows: 8200 cols: 8200 cells: 6724 and the command line: i.segment group=xs out=seg_xs minsize=2 memory=3072 threshold=0.05 (and thresh=0.1 and 0.2 in successive runs using the results of the previous run as seeds). Now, I would like to test segmentation of just the panchromatic band. This means the following region settings: projection: 1 (UTM) zone: -36 datum: wgs84 ellipsoid: wgs84 north: 7251172 south: 7234772 west: 333792 east: 350192 nsres: 0.5 ewres: 0.5 rows: 32800 cols: 32800 cells: 107584 Trying to run with the following command line on my i3, 8GB RAM machine: i.segment group=pan out=seg_pan_005 threshold=0.05 memory=3072 had the process running for almost 13 hours with it then becoming apparently stuck in the fourth pass at 10%. At that point the percent didn't change for over an hour, so I decided to kill the process. Can I assume that I'm here above the capacities of my machine ? Is there anything (besides working on a smaller subsample of the image) that I can do to make it work ? What kind of resources would I need to be able to run such a segmentation? I guess I'll have to move these kinds of treatments to our university supercomputer, but I first have to get them to install GRASS... Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
