#2764: corrupt data written to FCELL and DCELL rasters, hard to re-produce ---------------------+------------------------- Reporter: dylan | Owner: grass-dev@… Type: defect | Status: new Priority: normal | Milestone: 7.2.3 Component: Raster | Version: unspecified Resolution: | Keywords: CPU: x86-64 | Platform: Linux ---------------------+-------------------------
Comment (by dylan): Replying to [comment:32 mmetz]: > Markus Neteler in particular spent a lot of time to fix various systems for parallel execution of GRASS commands. GRASS itself was never the problem, instead the main problem was that the multiple outputs to be written to a single storage device were too much for that storage device. OK. Good to know. Are there any other diagnostics for these kind of problems, other than looking through the output from `dmesg` or kernel messages? I typically run `dstat` while developing parallel processing scripts, but I haven't noticed any disk-thrashing in this case. I see that the SSD is at about 5% of its "write bandwidth" as reported by `dstat`. Each instance of `r.sun` is writing out ~5000 rows of data over a period of about 8 minutes, so that is 5000 rows * 8 processes * 1/8 proc per minute * 1/60 minutes per second = ~80 write operations per second (assuming rows are written as processed). Are rows written 1-by-1, or in batches? > > I know this is a lot to ask, but did you try testing using ZLIB compression and running it multiple times? It took a couple of tiles before I noticed the error. > > I did use ZLIB compression when running the test with the data and scripts provided. Do you mean I should run the test several times with the same data? Yes (please). It wasn't until I had ran a couple of tiles that I encountered errors. -- Ticket URL: <https://trac.osgeo.org/grass/ticket/2764#comment:33> GRASS GIS <https://grass.osgeo.org>
_______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev