Re: [GRASS-user] r.patch: number of rasters
On Tue, Nov 2, 2010 at 2:03 PM, razmjoo...@faunalia.co.uk wrote: Dear all Is there a limit on the number of rasters for r.patch? How funny, we had the same question at the same time... :) From GRASS 6.4+ there is no more limit despite the limit of open files per process (which depends on the operating system; for Linux it is usually 1024 files). To increase that: The limit of open files per process was reached (64bit Linux: 1024 files; ulimit -n) To overcome this limit, the nofile parameter needs to be increased in the /etc/security/limits.conf file. Here an increase to 1500 files (use * instead of neteler to aply it to all users): #domain type item value neteler hardnofile 1500 But myself, I got troubles when having registered the maps with r.external: GRASS 6.4.1svn (patUTM32):~ r.patch in=`g.mlist type=rast pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif: Too many open files WARNING: Unable to open '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format' WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing WARNING: Unable to open raster map dtm000...@dtm_1m_external ERROR: One or more input raster maps not found So far I could not find out the reason. Here I am using 500 files registered with r.external. Help welcome, Markus ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] r.patch: number of rasters
Hi Markus Many thanks for your response. In the past versions, the patched raster used to have an odd values. I use Grass 6.4 from Ubuntu 10.10 64 bit repo. Cheers Sab On Tue, Nov 2, 2010 at 2:03 PM, razmjoo...@faunalia.co.uk wrote: Dear all Is there a limit on the number of rasters for r.patch? How funny, we had the same question at the same time... :) From GRASS 6.4+ there is no more limit despite the limit of open files per process (which depends on the operating system; for Linux it is usually 1024 files). To increase that: The limit of open files per process was reached (64bit Linux: 1024 files; ulimit -n) To overcome this limit, the nofile parameter needs to be increased in the /etc/security/limits.conf file. Here an increase to 1500 files (use * instead of neteler to aply it to all users): #domain type item value neteler hardnofile 1500 But myself, I got troubles when having registered the maps with r.external: GRASS 6.4.1svn (patUTM32):~ r.patch in=`g.mlist type=rast pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif: Too many open files WARNING: Unable to open '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format' WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing WARNING: Unable to open raster map dtm000...@dtm_1m_external ERROR: One or more input raster maps not found So far I could not find out the reason. Here I am using 500 files registered with r.external. Help welcome, Markus ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] r.patch: number of rasters
On Tue, Nov 2, 2010 at 4:41 PM, razmjoo...@faunalia.co.uk wrote: Hi Markus Many thanks for your response. In the past versions, the patched raster used to have an odd values. No, this is rather impossible (perhaps you refer to the odd number of pixels in a moving raster window). I use Grass 6.4 from Ubuntu 10.10 64 bit repo. Master question: how many files do you plan to patch? Markus Cheers Sab On Tue, Nov 2, 2010 at 2:03 PM, razmjoo...@faunalia.co.uk wrote: Dear all Is there a limit on the number of rasters for r.patch? How funny, we had the same question at the same time... :) From GRASS 6.4+ there is no more limit despite the limit of open files per process (which depends on the operating system; for Linux it is usually 1024 files). To increase that: The limit of open files per process was reached (64bit Linux: 1024 files; ulimit -n) To overcome this limit, the nofile parameter needs to be increased in the /etc/security/limits.conf file. Here an increase to 1500 files (use * instead of neteler to aply it to all users): #domain type item value neteler hard nofile 1500 But myself, I got troubles when having registered the maps with r.external: GRASS 6.4.1svn (patUTM32):~ r.patch in=`g.mlist type=rast pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif: Too many open files WARNING: Unable to open '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format' WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing WARNING: Unable to open raster map dtm000...@dtm_1m_external ERROR: One or more input raster maps not found So far I could not find out the reason. Here I am using 500 files registered with r.external. Help welcome, Markus ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] r.patch: number of rasters
Markus Neteler wrote: Here an increase to 1500 files (use * instead of neteler to aply it to all users): #domain type item value neteler hardnofile 1500 But myself, I got troubles when having registered the maps with r.external: GRASS 6.4.1svn (patUTM32):~ r.patch in=`g.mlist type=rast pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif: Too many open files This message matches strerror(EMFILE): #define EMFILE 24 /* Too many open files */ It's possible that some library (i.e. GDAL or TIFF) has its own limit, and returns EMFILE if the limit is reached. But I'd start by assuming that you're hitting an OS limit. WARNING: Unable to open '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format' WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing WARNING: Unable to open raster map dtm000...@dtm_1m_external ERROR: One or more input raster maps not found So far I could not find out the reason. Here I am using 500 files registered with r.external. I assume that it's holding multiple open descriptors per map. Note that 7.0 will typically have at least 2 open descriptors per map: one for the raster data, the other for the null data. But not for r.external linked files, which don't have a separate null file. 6.x doesn't keep the null file open, which reduces the number of open descriptors at the expense of performance (open() can be fairly expensive). Run r.patch under gdb, set a breakpoint on G_fatal_error(). When the breakpoint is hit, ls -l /proc/pid/fd (where pid is the PID of the r.patch process) to see *which* files it has open. Or run r.patch under strace, and manually examine the output to see which files are open()ed and which are close()d. -- Glynn Clements gl...@gclements.plus.com ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] r.patch: number of rasters
I am having about 700 LiDAR tiles (~4 GB). Did it a couple of hours ago and all worked well...the DTM looks as I expected! The problem I had last year when I did it last, there were 1000+, values for one tile were changed. Never investigated the problem. Simply changed the region and things worked smoothly. Many thanks Sab On Tue, 2010-11-02 at 16:54 +0100, Markus Neteler wrote: On Tue, Nov 2, 2010 at 4:41 PM, razmjoo...@faunalia.co.uk wrote: Hi Markus Many thanks for your response. In the past versions, the patched raster used to have an odd values. No, this is rather impossible (perhaps you refer to the odd number of pixels in a moving raster window). I use Grass 6.4 from Ubuntu 10.10 64 bit repo. Master question: how many files do you plan to patch? Markus Cheers Sab On Tue, Nov 2, 2010 at 2:03 PM, razmjoo...@faunalia.co.uk wrote: Dear all Is there a limit on the number of rasters for r.patch? How funny, we had the same question at the same time... :) From GRASS 6.4+ there is no more limit despite the limit of open files per process (which depends on the operating system; for Linux it is usually 1024 files). To increase that: The limit of open files per process was reached (64bit Linux: 1024 files; ulimit -n) To overcome this limit, the nofile parameter needs to be increased in the /etc/security/limits.conf file. Here an increase to 1500 files (use * instead of neteler to aply it to all users): #domain type item value neteler hardnofile 1500 But myself, I got troubles when having registered the maps with r.external: GRASS 6.4.1svn (patUTM32):~ r.patch in=`g.mlist type=rast pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif: Too many open files WARNING: Unable to open '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format' WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing WARNING: Unable to open raster map dtm000...@dtm_1m_external ERROR: One or more input raster maps not found So far I could not find out the reason. Here I am using 500 files registered with r.external. Help welcome, Markus ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
