Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote:
I don't know how to debug this...
Can you identify a repeatable test case?
If I could make it happen, I could debug it.
You can get a location names TEST here:
http://tomahawk.ulb.ac.be/moritz/mask_bug_testlocation.tgz
This contains only a PERMANENT mapset.
In that mapset, launch the following command:
r.mask vect=hull; for map in $(g.list rast pat=firm_rate*); do echo
$map ; r.mapcalc temp_prob = float($map) / sum_rates --o --q; done;
r.mask -r
I get the error arbitrarily for different firm_rate_* maps, sometimes
only for one, sometimes for many, but at each run its for different
maps.
So it's non-deterministic (I'm getting one error for every 10-20
passes over the data, i.e. every 1200-2500 commands), and only applies
to r.mapcalc.
My first guess was a race condition related to pthreads. I tried
export WORKERS=0
before running the test, and it hasn't happened since.
And actually I'm now fairly certain as to the specific cause.
When compiled with pthread support, r.mapcalc has a mutex for each map
to prevent concurrent access to a single map from multiple threads.
Concurrent access to different maps (and to core lib/gis and and
lib/raster functionality) from different threads is supposed to be
safe (see r34485 and the interval surrounding it), but the MASK was
overlooked.
If a MASK is in use, reading a row from any raster map will read the
corresponding row from the MASK, and there's nothing to prevent
different threads from concurrently accessing two different maps and
thus accessing the MASK.
So, in read_data_{compressed,uncompressed,read_data_fp_compressed} in
lib/raster/get_row.c we have code like:
if (lseek(fcb-data_fd, (off_t) row * bufsize, SEEK_SET) == -1)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
if (read(fcb-data_fd, data_buf, bufsize) != bufsize)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
If multiple threads execute this code concurrently, you can end up
with the calls being interleaved like so:
Thread 1Thread 2
lseek
lseek
read
read
meaning that the file offset has changed betwee the lseek() and the
read() (this is why X/Open and POSIX added pread(), but that's still
relatively new).
This only results in an error at the end of the file (the first read()
will leave the file offset at EOF, so the second read() fails), but in
other situations it's likely causing the wrong row of the MASK to be
read.
A possible quick fix:
if (R__.auto_mask 0)
putenv(WORKERS=0);
A slightly better fix would be to check for masking and if it's
enabled, have a single mutex which guards *all* raster reads so that
even concurrent access to different maps is blocked. Unlike the above
hack, this still allows computations to be executed in parallel.
Better still would be to guard access to the MASK so that the other
aspects of raster input can be parallelised (raster I/O is still a
major bottleneck, and mostly because of processing rather than actual
disc access).
But that would involve either adding pthread code directly into the
base raster input code in lib/raster/get_row.c (undesirable) or at
least adding a mechanism to allow r.mapcalc to hook into it to provide
the mutex.
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
On 14/07/15 09:46, Glynn Clements wrote:
Moritz Lennert wrote:
I don't know how to debug this...
Can you identify a repeatable test case?
If I could make it happen, I could debug it.
You can get a location names TEST here:
http://tomahawk.ulb.ac.be/moritz/mask_bug_testlocation.tgz
This contains only a PERMANENT mapset.
In that mapset, launch the following command:
r.mask vect=hull; for map in $(g.list rast pat=firm_rate*); do echo
$map ; r.mapcalc temp_prob = float($map) / sum_rates --o --q; done;
r.mask -r
I get the error arbitrarily for different firm_rate_* maps, sometimes
only for one, sometimes for many, but at each run its for different
maps.
So it's non-deterministic (I'm getting one error for every 10-20
passes over the data, i.e. every 1200-2500 commands), and only applies
to r.mapcalc.
My first guess was a race condition related to pthreads. I tried
export WORKERS=0
before running the test, and it hasn't happened since.
And actually I'm now fairly certain as to the specific cause.
When compiled with pthread support, r.mapcalc has a mutex for each map
to prevent concurrent access to a single map from multiple threads.
Concurrent access to different maps (and to core lib/gis and and
lib/raster functionality) from different threads is supposed to be
safe (see r34485 and the interval surrounding it), but the MASK was
overlooked.
If a MASK is in use, reading a row from any raster map will read the
corresponding row from the MASK, and there's nothing to prevent
different threads from concurrently accessing two different maps and
thus accessing the MASK.
So, in read_data_{compressed,uncompressed,read_data_fp_compressed} in
lib/raster/get_row.c we have code like:
if (lseek(fcb-data_fd, (off_t) row * bufsize, SEEK_SET) == -1)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
if (read(fcb-data_fd, data_buf, bufsize) != bufsize)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
If multiple threads execute this code concurrently, you can end up
with the calls being interleaved like so:
Thread 1Thread 2
lseek
lseek
read
read
meaning that the file offset has changed betwee the lseek() and the
read() (this is why X/Open and POSIX added pread(), but that's still
relatively new).
This only results in an error at the end of the file (the first read()
will leave the file offset at EOF, so the second read() fails), but in
other situations it's likely causing the wrong row of the MASK to be
read.
A possible quick fix:
if (R__.auto_mask 0)
putenv(WORKERS=0);
A slightly better fix would be to check for masking and if it's
enabled, have a single mutex which guards *all* raster reads so that
even concurrent access to different maps is blocked. Unlike the above
hack, this still allows computations to be executed in parallel.
Better still would be to guard access to the MASK so that the other
aspects of raster input can be parallelised (raster I/O is still a
major bottleneck, and mostly because of processing rather than actual
disc access).
But that would involve either adding pthread code directly into the
base raster input code in lib/raster/get_row.c (undesirable) or at
least adding a mechanism to allow r.mapcalc to hook into it to provide
the mutex.
Thanks for the detailed analysis and explanation !
So, for me, the best solution at this stage is to just set WORKERS to 0 ?
The rest of your proposed solutions is above my head, so I couldn't help
with implementation.
Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Implement new airborne hyperspectral sensor in i.atcorr
Hello everybody, I would like to atmospheric correct some AISA Eagle data. It's a hyperspectral, airborne sensor. There are 60 bands in the range from 405 to 951 nm (about a 9,2 nm - step). So my question(s)... Is it possible to implement a new hyperspectral, airborne sensors in i.atcorr? I know there is already a hyperspectral satellite sensor included. If it is possible, how much time will be roughly consumed to undertake this task? Days, weeks or months? Furthermore, I've already read the i.atcorr README to add new band filters (http://trac.osgeo.org/grass/browser/grass/trunk/imagery/i.atcorr/README), but I'm an absolute beginner, so unfortunately I wasn't able to figure out how to create this needed filter function. Can somebody point me in the right direction, please of course only if it is possible. Thanks for your effords! Alexander K. ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r65307 - grass/trunk/lib/init
Hi, 2015-07-08 18:31 GMT+02:00 Huidae Cho gras...@gmail.com: I think the following change has to be reverted because it breaks aliases and custom prompts defined in ~/.grass7/bashrc. Currently, only NAME=VALUE lines are parsed from this file in load_env(). sorry for delay, I took liberty to partly re-introduce r65307 in r65585. It should work with aliases now. Martin -- Martin Landa http://geo.fsv.cvut.cz/gwiki/Landa http://gismentors.cz/mentors/landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS 7.0.1
Martin Landa wrote
Hi,
2015-07-03 14:46 GMT+02:00 Helmut Kudrnovsky lt;
hellik@
gt;:
just tested in trunk and nc data set, I get this error:
g.copy vector=firestations@PERMANENT,myfire
I am feeling lost in this thread, it's also related to grass70 branch?
I wonder if we really need RC2 or we could just relase relbr70 as
7.0.1 final as it is now?
Thanks a lot for clarification. Martin
---
GRASS Version: 7.0.1RC1
GRASS SVN Revision: 65472
Erstellungsdatum: 2015-06-15
Build Platform: i686-pc-mingw32
GDAL/OGR: 1.11.2
PROJ.4: 4.8.0
GEOS: 3.4.2
SQLite: 3.7.17
Python: 2.7.4
wxPython: 2.8.12.1
Platform: Windows-Vista-6.0.6002-SP2 (OSGeo4W)
RC1 has this bug:
https://trac.osgeo.org/grass/ticket/2634
---
System Info
GRASS Version: 7.0.1svn
GRASS SVN Revision: 65576
Erstellungsdatum: 2015-07-13
Build Platform: i686-pc-mingw32
GDAL/OGR: 1.11.2
PROJ.4: 4.8.0
GEOS: 3.4.2
SQLite: 3.7.17
Python: 2.7.4
wxPython: 2.8.12.1
Platform: Windows-Vista-6.0.6002-SP2
open v.db.addcolumn, choose vector layer, following error:
Traceback (most recent call last):
File C:\Program Files\GRASS GIS
7.0.1svn\gui\wxpython\gui_core\forms.py, line 2188, in
OnSetValue
self.OnUpdateValues(event)
File C:\Program Files\GRASS GIS
7.0.1svn\gui\wxpython\gui_core\forms.py, line 628, in
updateValuesHook
self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xf6 in position 12: ordinal
not in range(128)
or
open v.to,db, choose vector layer, following error:
Traceback (most recent call last):
File C:\Program Files\GRASS GIS
7.0.1svn\gui\wxpython\gui_core\forms.py, line 2188, in
OnSetValue
self.OnUpdateValues(event)
File C:\Program Files\GRASS GIS
7.0.1svn\gui\wxpython\gui_core\forms.py, line 628, in
updateValuesHook
self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xf6 in position 11: ordinal
not in range(128)
---
System Info
GRASS Version: 7.1.svn
GRASS SVN revision: 65575
Build date: 2015-07-13
Build platform: i686-pc-mingw32
GDAL: 1.11.2
PROJ.4: 4.8.0
GEOS: 3.4.2
SQLite: 3.7.17
Python: 2.7.4
wxPython: 2.8.12.1
Platform: Windows-Vista-6.0.6002-SP2 (OSGeo4W)
open v.db.addcolumn, choose vector layer, following error:
Traceback (most recent call last):
File C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\forms.py, line 2221, in OnSetValue
self.OnUpdateValues(event)
File C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\forms.py, line 628, in updateValuesHook
self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xf6 in position 11: ordinal
not in range(128)
or
open v.to,db, choose vector layer, following error:
Traceback (most recent call last):
[GRASS-dev] r.in.proj v.in.proj to trunk
Hi, I would like to express my wish to copy r|v.in.proj from addons to trunk as a core module. The modules will be still available via addons for GRASS 7.0 users. It has one disadvange the code duplication in trunk and addons. What do you think? Martin -- Martin Landa http://geo.fsv.cvut.cz/gwiki/Landa http://gismentors.cz/mentors/landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS 7.0.1
Hi, 2015-07-03 14:46 GMT+02:00 Helmut Kudrnovsky hel...@web.de: just tested in trunk and nc data set, I get this error: g.copy vector=firestations@PERMANENT,myfire I am feeling lost in this thread, it's also related to grass70 branch? I wonder if we really need RC2 or we could just relase relbr70 as 7.0.1 final as it is now? Thanks a lot for clarification. Martin -- Martin Landa http://geo.fsv.cvut.cz/gwiki/Landa http://gismentors.cz/mentors/landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r65307 - grass/trunk/lib/init
Thanks, Martin. Is there any good reason why grass.py imports environment variables in load_env() and writes non-/^export/ lines (r65585 does not strip out whitespaces) into MAPSET/.bashrc in bash_startup() instead of simply sourcing or fully writing .grass7/bashrc? I see a couple of problems with this two step approach. 1. /^export/ lines in .grass.bashrc don't work anymore with r65585. This file used to be fully written into MAPSET/.bashrc. 2. Conditional constructs like the following are not supported in .grass7/bashrc: case $TERM in xterm*) export PS1=xterm # no space before export ;; screen) export PS1=screen ;; esac because load_env() would overwrite PS1 and bash_startup() won't write out the /^export PS1/ lines. It will work if there are whitespaces before export, but I would say that was not intended by you... Regards, Huidae On Tue, Jul 14, 2015 at 8:51 AM, Martin Landa landa.mar...@gmail.com wrote: Hi, 2015-07-08 18:31 GMT+02:00 Huidae Cho gras...@gmail.com: I think the following change has to be reverted because it breaks aliases and custom prompts defined in ~/.grass7/bashrc. Currently, only NAME=VALUE lines are parsed from this file in load_env(). sorry for delay, I took liberty to partly re-introduce r65307 in r65585. It should work with aliases now. Martin -- Martin Landa http://geo.fsv.cvut.cz/gwiki/Landa http://gismentors.cz/mentors/landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.in.proj v.in.proj to trunk
+1 ! On Tue, Jul 14, 2015 at 11:33 PM Markus Neteler nete...@osgeo.org wrote: On Tue, Jul 14, 2015 at 6:02 PM, Martin Landa landa.mar...@gmail.com wrote: Hi, I would like to express my wish to copy r|v.in.proj from addons to trunk as a core module. +1! The modules will be still available via addons for GRASS 7.0 users. It has one disadvange the code duplication in trunk and addons. What do you think? Since it is not subject to heavy changes I don't see this as a real problem. Markus ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: So, for me, the best solution at this stage is to just set WORKERS to 0 ? That should work. Also, the issue should be fixed by r65591. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.in.proj v.in.proj to trunk
On Tue, Jul 14, 2015 at 6:02 PM, Martin Landa landa.mar...@gmail.com wrote: Hi, I would like to express my wish to copy r|v.in.proj from addons to trunk as a core module. +1! The modules will be still available via addons for GRASS 7.0 users. It has one disadvange the code duplication in trunk and addons. What do you think? Since it is not subject to heavy changes I don't see this as a real problem. Markus ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
