Re: [gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Le mercredi 17 avril 2013 00:01:41, Eli Adam a écrit : On Tue, Apr 16, 2013 at 2:11 PM, Even Rouault even.roua...@mines-paris.org wrote: I've investigated a bit and I've discovered that while computing overviews, libtiff constantly switches from TIFF directories (main image, first overview,second overview, etc...). While doing this, it calls each time client code to install its own TIFF tags (libgeotiff for the geotiff tags, GDAL for the GDAL specific TIFF tags), but doesn't reset its array of existing tags. So each time the tags are added again and again, hence the huge memory allocation. Could you test the attached patch (assuming you build GDAL with internal libtiff) ? I can apply and test this on linux but don't know how to do it on windows. I usually use the nightly windows builds from Tamas. Should I test this on linux? It would have been good to test this on the environmenet on which you got the problem, but while it isn't committed, I can see it is not really doable for you. It would be good if you could also open a ticket in Trac about that issue. I've opened a ticket, http://trac.osgeo.org/gdal/ticket/5054 Thanks. I've attached my patch and I'm waiting for the review of a libtiff committer (a.k.a. Frank) before committing it in GDAL. Eli ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Le lundi 15 avril 2013 23:47:33, Eli Adam a écrit : Hi all, I'm adding internal overviews to a ~20G tiff file and I get this error (1000+ times): gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) gdaladdo still completes and the file seems to work fine. What does that error mean? Does it impact the output file or overviews? Is there a different way I should approach this? Are there any general rules for which compression to select? I've found JPEG-In-TIFF, using PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure of the best compression for the type of raster (in this case single band, byte, colormap). That's interesting. I've tried to reproduce your setup with an artificial blank input file of the same dimensions and here's what I get (after several hours): $ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS 0...10...20...30...40...50...60...70...80...90...100 - done. Current VSI memory usage: 0 bytes Maximum VSI memory usage: 2361761086 bytes Number of calls to VSIMalloc() : 43981823 Number of calls to VSICalloc() : 1259 Number of calls to VSIRealloc() : 17378556 Number of calls to VSIFree(): 43983082 VSIMalloc + VSICalloc - VSIFree : 0 The interesting part is the memory usage report. It is based on an instrumented version of GDAL that instruments each memory allocation routine. This is with the internal version of libtiff that redirects the TIFF memory allocation routines to the GDAL ones. So apparently there wouldn't be a memory leak (some valgrind'ing should be done on a smaller file to confirm), but a maximum memory usage of 2.3 GB ! No wonder why it fails on a Windows 32 bit (my test was done a Linux 64bit). I'm not sure why such a high memory use occurs. Some investigation should be done. As far as the error message you've seen, I wouldn't swear that there's not some pixel corruption in the result you get... As a potential track for a workaround (completely untested), perhaps you could try to do : $ gdaladdo topo_mosaic_4326.tif 2 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 4 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 8 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 16 --config COMPRESS_OVERVIEW PACKBITS etc.. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Even, Thanks for checking into this and the ideas. Eli On Tue, Apr 16, 2013 at 10:18 AM, Even Rouault even.roua...@mines-paris.org wrote: Le lundi 15 avril 2013 23:47:33, Eli Adam a écrit : Hi all, I'm adding internal overviews to a ~20G tiff file and I get this error (1000+ times): gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) gdaladdo still completes and the file seems to work fine. What does that error mean? Does it impact the output file or overviews? Is there a different way I should approach this? Are there any general rules for which compression to select? I've found JPEG-In-TIFF, using PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure of the best compression for the type of raster (in this case single band, byte, colormap). That's interesting. I've tried to reproduce your setup with an artificial blank input file of the same dimensions and here's what I get (after several hours): $ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS 0...10...20...30...40...50...60...70...80...90...100 - done. Current VSI memory usage: 0 bytes Maximum VSI memory usage: 2361761086 bytes Number of calls to VSIMalloc() : 43981823 Number of calls to VSICalloc() : 1259 Number of calls to VSIRealloc() : 17378556 Number of calls to VSIFree(): 43983082 VSIMalloc + VSICalloc - VSIFree : 0 The interesting part is the memory usage report. It is based on an instrumented version of GDAL that instruments each memory allocation routine. This is with the internal version of libtiff that redirects the TIFF memory allocation routines to the GDAL ones. So apparently there wouldn't be a memory leak (some valgrind'ing should be done on a smaller file to confirm), but a maximum memory usage of 2.3 GB ! No wonder why it fails on a Windows 32 bit (my test was done a Linux 64bit). I'm not sure why such a high memory use occurs. Some investigation should be done. As far as the error message you've seen, I wouldn't swear that there's not some pixel corruption in the result you get... As a potential track for a workaround (completely untested), perhaps you could try to do : $ gdaladdo topo_mosaic_4326.tif 2 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 4 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 8 --config COMPRESS_OVERVIEW PACKBITS $ gdaladdo topo_mosaic_4326.tif 16 --config COMPRESS_OVERVIEW PACKBITS etc.. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Le mardi 16 avril 2013 22:07:52, Eli Adam a écrit :
Even,
Thanks for checking into this and the ideas.
I've investigated a bit and I've discovered that while computing overviews,
libtiff constantly switches from TIFF directories (main image, first
overview,second overview, etc...). While doing this, it calls each time client
code to install its own TIFF tags (libgeotiff for the geotiff tags, GDAL for
the
GDAL specific TIFF tags), but doesn't reset its array of existing tags. So each
time the tags are added again and again, hence the huge memory allocation.
Could you test the attached patch (assuming you build GDAL with internal
libtiff) ?
It would be good if you could also open a ticket in Trac about that issue.
Even
Eli
Index: frmts/gtiff/libtiff/tif_dir.c
===
--- frmts/gtiff/libtiff/tif_dir.c (révision 25916)
+++ frmts/gtiff/libtiff/tif_dir.c (copie de travail)
@@ -1302,8 +1302,20 @@
tif-tif_tagmethods.printdir = NULL;
/*
* Give client code a chance to install their own
- * tag extensions methods, prior to compression overloads.
+ * tag extensions methods, prior to compression overloads,
+ * but do some prior cleanup first.
*/
+if (tif-tif_nfieldscompat 0) {
+uint32 i;
+
+for (i = 0; i tif-tif_nfieldscompat; i++) {
+if (tif-tif_fieldscompat[i].allocated_size)
+_TIFFfree(tif-tif_fieldscompat[i].fields);
+}
+_TIFFfree(tif-tif_fieldscompat);
+tif-tif_nfieldscompat = 0;
+tif-tif_fieldscompat = NULL;
+}
if (_TIFFextender)
(*_TIFFextender)(tif);
(void) TIFFSetField(tif, TIFFTAG_COMPRESSION, COMPRESSION_NONE);
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
On Tue, Apr 16, 2013 at 2:11 PM, Even Rouault even.roua...@mines-paris.org wrote: I've investigated a bit and I've discovered that while computing overviews, libtiff constantly switches from TIFF directories (main image, first overview,second overview, etc...). While doing this, it calls each time client code to install its own TIFF tags (libgeotiff for the geotiff tags, GDAL for the GDAL specific TIFF tags), but doesn't reset its array of existing tags. So each time the tags are added again and again, hence the huge memory allocation. Could you test the attached patch (assuming you build GDAL with internal libtiff) ? I can apply and test this on linux but don't know how to do it on windows. I usually use the nightly windows builds from Tamas. Should I test this on linux? It would be good if you could also open a ticket in Trac about that issue. I've opened a ticket, http://trac.osgeo.org/gdal/ticket/5054 Eli ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Hi all, I'm adding internal overviews to a ~20G tiff file and I get this error (1000+ times): gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) gdaladdo still completes and the file seems to work fine. What does that error mean? Does it impact the output file or overviews? Is there a different way I should approach this? Are there any general rules for which compression to select? I've found JPEG-In-TIFF, using PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure of the best compression for the type of raster (in this case single band, byte, colormap). I just checked my version and I am running 1.9.0 on a older modest XP laptop (I figured that I would have been running something more recent). I can try again with a nightly or compile on Ubuntu or use a computer with more resources (the process takes a long time to run, so reporting back on those will take some time). Here is information (I've omitted some previous mosaicing and reprojecting) of how I got to this point and some gdalinfo reports: gdal_translate mosaic_4326.vrt topo_mosaic_4326.tif -co COMPRESS=PACKBITS -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 400 gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) original input files: gdalinfo ID_Cuddy_Mountain_20110715_TM_geo.tif Driver: GTiff/GeoTIFF Files: ID_Cuddy_Mountain_20110715_TM_geo.tif Size is 4880, 6845 Coordinate System is: PROJCS[NAD83 / UTM zone 11N, GEOGCS[NAD83, DATUM[North_American_Datum_1983, SPHEROID[GRS 1980,6378137,298.2572221010002, AUTHORITY[EPSG,7019]], AUTHORITY[EPSG,6269]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4269]], PROJECTION[Transverse_Mercator], PARAMETER[latitude_of_origin,0], PARAMETER[central_meridian,-117], PARAMETER[scale_factor,0.9996], PARAMETER[false_easting,50], PARAMETER[false_northing,0], UNIT[metre,1, AUTHORITY[EPSG,9001]], AUTHORITY[EPSG,26911]] Origin = (509873.30982669181,4969095.03341848590) Pixel Size = (2.032010116947724,-2.032010116947745) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 509873.310, 4969095.033) (116d52'30.00W, 44d52'30.74N) Lower Left ( 509873.310, 4955185.924) (116d52'30.97W, 44d44'59.97N) Upper Right ( 519789.519, 4969095.033) (116d44'58.04W, 44d52'30.00N) Lower Right ( 519789.519, 4955185.924) (116d44'59.99W, 44d44'59.23N) Center ( 514831.415, 4962140.479) (116d48'44.75W, 44d48'45.05N) Band 1 Block=4880x1 Type=Byte, ColorInterp=Palette NoData Value=195 Color Table (RGB with 256 entries) 0: 0,0,0,255 1: 128,0,0,255 2: 0,128,0,255 3: 128,128,0,255 ... 254: 0,255,255,255 255: 255,255,255,255 final output file with overviews: gdalinfo -checksum topo_mosaic_4326.tif Driver: GTiff/GeoTIFF Files: topo_mosaic_4326.tif Size is 398449, 210532 Coordinate System is: GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]] Origin = (-124.732439702451200,46.380533997595272) Pixel Size = (0.20993062915,-0.20993062915) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=PACKBITS INTERLEAVE=BAND Corner Coordinates: Upper Left (-124.7324397, 46.3805340) (124d43'56.78W, 46d22'49.92N) Lower Left (-124.7324397, 41.9608225) (124d43'56.78W, 41d57'38.96N) Upper Right (-116.3677748, 46.3805340) (116d22' 3.99W, 46d22'49.92N) Lower Right (-116.3677748, 41.9608225) (116d22' 3.99W, 41d57'38.96N) Center (-120.5501072, 44.1706782) (120d33' 0.39W, 44d10'14.44N) Band 1 Block=256x256 Type=Byte, ColorInterp=Palette Checksum=45365 NoData Value=195 Overviews: 199225x105266, 99613x52633, 49807x26317, 24904x13159, 12452x6580, 6 226x3290, 3113x1645, 1557x823, 779x412, 390x206 Overviews checksum: 27605, 16665, 58099, 63342, 4876, 64021, 44152, 14528, 574 63, 15006 Color Table (RGB with 256 entries) 0: 0,0,0,255 1: 128,0,0,255 2: 0,128,0,255 3: 128,128,0,255 ... 253: 255,0,255,255 254: 0,255,255,255 255: 255,255,255,255 Best Regards, Eli ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
