[gdal-dev] gdaladdo creates very large files in 1.8 but not trunk
This is an informational report only, there are not problems in trunk. Using 1.8.0 from OSGeo4W on XP, I made a mosaic of about 1200 uncompressed tifs totally 80G. gdalbuildvrt mosaic.vrt *.tif Then gdal_translated that to a compressed tif of about 8G gdal_translate mosaic.vrt mosaic_ycbcr_big.tif -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 800 Last step was to add some overviews: gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL --config BIGTIFF_OVERVIEW YES --config GDAL_CACHEMAX 800 -r average -ro mosaic_ycbcr_big.tif 2 4 8 16 32 64 128 It works (at least in OpenEV it displays quickly like it is using overviews), except the file sizes for the overviews seem odd and too large. The tiles are about 80G, the result of gdal_translate is 8G, the result of gdaladdo, mosaic_ycbcr_big.tif.ovr, is 375G! That was using 1.8.0. I tested on trunk on ubuntu and XP from here, http://vbkto.dyndns.org/sdk/ which produce reasonable results. The ovr is about 3G. A dif of gdalinfo on the .tif and .ovr are the same. (I hope that is not the result of a typo when I initially typed the command. I unfortunately don't have the shell around any more but took the above listed commands from some notes I took.) I have tried making the overviews internal and external with similar results. I also tried making the overviews on the uncompressed files (the vrt) with the same results. Then I made the overviews 1 level at a time and those results seemed reasonable. I didn't find any tickets about this in trac, but don't experience this in trunk so apparently it is fixed. Just reporting in case it is useful. Do you want any other information? Thanks, Eli ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdaladdo creates very large files in 1.8 but not trunk
Le jeudi 14 avril 2011 17:02:36, Eli Adam a écrit : I'm not aware of any recent change in the GTiff driver that could have "solved" the issue you've seen. This is a bit surprising. Could you also try with the -stable builds (based on 1.8 branch) from http://vbkto.dyndns.org/sdk/ to compare ? > This is an informational report only, there are not problems in trunk. > > Using 1.8.0 from OSGeo4W on XP, I made a mosaic of about 1200 uncompressed > tifs totally 80G. > > gdalbuildvrt mosaic.vrt *.tif > > Then gdal_translated that to a compressed tif of about 8G > > gdal_translate mosaic.vrt mosaic_ycbcr_big.tif -co COMPRESS=JPEG -co > PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 800 > > Last step was to add some overviews: > > gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW > YCBCR --config INTERLEAVE_OVERVIEW PIXEL --config BIGTIFF_OVERVIEW YES > --config GDAL_CACHEMAX 800 -r average -ro mosaic_ycbcr_big.tif 2 4 8 16 32 > 64 128 > > It works (at least in OpenEV it displays quickly like it is using > overviews), except the file sizes for the overviews seem odd and too > large. > > The tiles are about 80G, the result of gdal_translate is 8G, the result of > gdaladdo, mosaic_ycbcr_big.tif.ovr, is 375G! That was using 1.8.0. > > I tested on trunk on ubuntu and XP from here, http://vbkto.dyndns.org/sdk/ > which produce reasonable results. The ovr is about 3G. > > A dif of gdalinfo on the .tif and .ovr are the same. > > (I hope that is not the result of a typo when I initially typed the > command. I unfortunately don't have the shell around any more but took > the above listed commands from some notes I took.) > > I have tried making the overviews internal and external with similar > results. I also tried making the overviews on the uncompressed files (the > vrt) with the same results. Then I made the overviews 1 level at a time > and those results seemed reasonable. > > I didn't find any tickets about this in trac, but don't experience this in > trunk so apparently it is fixed. Just reporting in case it is useful. > > Do you want any other information? > > Thanks, Eli > > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdaladdo creates very large files in 1.8 but not trunk
> I'm not aware of any recent change in the GTiff driver that could have > "solved" > the issue you've seen. This is a bit surprising. > > Could you also try with the -stable builds (based on 1.8 branch) from > http://vbkto.dyndns.org/sdk/ to compare ? I am trying it with, http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1500-gdal-1-8-mapserver-5-6.zip Although not finished, based on progress so far, it looks to be working properly and on track to create a 3G ovr. That is consistent with trunk. I'll provide more information when it finishes. Eli > >> This is an informational report only, there are not problems in trunk. >> >> Using 1.8.0 from OSGeo4W on XP, I made a mosaic of about 1200 uncompressed >> tifs totally 80G. >> >> gdalbuildvrt mosaic.vrt *.tif >> >> Then gdal_translated that to a compressed tif of about 8G >> >> gdal_translate mosaic.vrt mosaic_ycbcr_big.tif -co COMPRESS=JPEG -co >> PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 800 >> >> Last step was to add some overviews: >> >> gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW >> YCBCR --config INTERLEAVE_OVERVIEW PIXEL --config BIGTIFF_OVERVIEW YES >> --config GDAL_CACHEMAX 800 -r average -ro mosaic_ycbcr_big.tif 2 4 8 16 32 >> 64 128 >> >> It works (at least in OpenEV it displays quickly like it is using >> overviews), except the file sizes for the overviews seem odd and too >> large. >> >> The tiles are about 80G, the result of gdal_translate is 8G, the result of >> gdaladdo, mosaic_ycbcr_big.tif.ovr, is 375G! That was using 1.8.0. >> >> I tested on trunk on ubuntu and XP from here, http://vbkto.dyndns.org/sdk/ >> which produce reasonable results. The ovr is about 3G. >> >> A dif of gdalinfo on the .tif and .ovr are the same. >> >> (I hope that is not the result of a typo when I initially typed the >> command. I unfortunately don't have the shell around any more but took >> the above listed commands from some notes I took.) >> >> I have tried making the overviews internal and external with similar >> results. I also tried making the overviews on the uncompressed files (the >> vrt) with the same results. Then I made the overviews 1 level at a time >> and those results seemed reasonable. >> >> I didn't find any tickets about this in trac, but don't experience this in >> trunk so apparently it is fixed. Just reporting in case it is useful. >> >> Do you want any other information? >> >> Thanks, Eli >> >> >> >> >> ___ >> gdal-dev mailing list >> gdal-dev@lists.osgeo.org >> http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdaladdo creates very large files in 1.8 but not trunk
>> I'm not aware of any recent change in the GTiff driver that could have >> "solved" >> the issue you've seen. This is a bit surprising. >> >> Could you also try with the -stable builds (based on 1.8 branch) from >> http://vbkto.dyndns.org/sdk/ to compare ? > > I am trying it with, > http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1500-gdal-1-8-mapserver-5- > > 6.zip > > Although not finished, based on progress so far, it looks to be working > properly and on track to create a 3G ovr. That is consistent with trunk. > I'll provide more information when it finishes. Yes, this produced the expected results. Here is some information on that, (comments of creation included too) E:\Orthos>dir mosaic_ycbcr_big* 04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big.tif 04/11/2011 08:43 PM 3,262,697,247 mosaic_ycbcr_big.tif.ovr (created from svn a few days ago on ubuntu) 04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_1_8.tif 04/14/2011 09:47 PM 3,260,246,941 mosaic_ycbcr_big_1_8.tif.ovr (created from http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1500-gdal-1-8-mapserver-5-6.zip downloaded today or yesterday) 04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_addo_v2.tif 04/07/2011 03:55 AM 385,580,179,214 mosaic_ycbcr_big_addo_v2.tif.ovr (Created from OSGEO4W 1.8.0) 04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_win.tif 04/12/2011 08:17 PM 3,260,246,941 mosaic_ycbcr_big_win.tif.ovr (created from a trunk version download from Tamas's site from a few days ago) gdalinfo for mosaic_ycbcr_big.tif and mosaic_ycbcr_big_addo_v2.tif produces identical results (diff'ed). Here is one for reference. Driver: GTiff/GeoTIFF Files: mosaic_ycbcr_big_addo_v2.tif mosaic_ycbcr_big_addo_v2.tif.ovr Size is 141400, 280500 Coordinate System is: PROJCS["NAD83(HARN) / Oregon North (ft)", GEOGCS["NAD83(HARN)", DATUM["NAD83_High_Accuracy_Regional_Network", SPHEROID["GRS 1980",6378137,298.2572221010002, AUTHORITY["EPSG","7019"]], AUTHORITY["EPSG","6152"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4152"]], PROJECTION["Lambert_Conformal_Conic_2SP"], PARAMETER["standard_parallel_1",46], PARAMETER["standard_parallel_2",44.34], PARAMETER["latitude_of_origin",43.66], PARAMETER["central_meridian",-120.5], PARAMETER["false_easting",8202099.738], PARAMETER["false_northing",0], UNIT["foot",0.3048, AUTHORITY["EPSG","9002"]], AUTHORITY["EPSG","2913"]] Origin = (7255000.00046718680,522299.9997485) Pixel Size = (1.018,-1.018) Metadata: AREA_OR_POINT=Area Image Structure Metadata: SOURCE_COLOR_SPACE=YCbCr COMPRESSION=YCbCr JPEG INTERLEAVE=PIXEL Corner Coordinates: Upper Left ( 7255000.000, 522300.000) (124d 9'55.56"W, 45d 2'25.33"N) Lower Left ( 7255000.000, 241800.000) (124d 7' 0.72"W, 44d16'18.24"N) Upper Right ( 7396400.000, 522300.000) (123d37' 7.62"W, 45d 3'23.93"N) Lower Right ( 7396400.000, 241800.000) (123d34'38.80"W, 44d17'16.07"N) Center ( 7325700.000, 382050.000) (123d52'10.27"W, 44d39'52.08"N) Band 1 Block=256x256 Type=Byte, ColorInterp=Red Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210 x4383, 1105x2192 Band 2 Block=256x256 Type=Byte, ColorInterp=Green Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210 x4383, 1105x2192 Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210 x4383, 1105x2192 gdalinfo for mosaic_ycbcr_big.tif.ovr and mosaic_ycbcr_big_addo_v2.tif.ovr produces identical results. Here for reference too. Driver: GTiff/GeoTIFF Files: mosaic_ycbcr_big_addo_v2.tif.ovr Size is 70700, 140250 Coordinate System is `' Image Structure Metadata: SOURCE_COLOR_SPACE=YCbCr COMPRESSION=YCbCr JPEG INTERLEAVE=PIXEL Corner Coordinates: Upper Left (0.0,0.0) Lower Left (0.0,140250.0) Upper Right (70700.0,0.0) Lower Right (70700.0,140250.0) Center (35350.0,70125.0) Band 1 Block=128x128 Type=Byte, ColorInterp=Red Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21 92 Band 2 Block=128x128 Type=Byte, ColorInterp=Green Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21 92 Band 3 Block=128x128 Type=Byte, ColorInterp=Blue Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21 92 I can run more tests, although for results you will have to wait since it takes a long time to run these. Wait, this is easier, it is reproducible with OSGEO4W 1.8.0 and http://svn.osgeo.org/gdal/trunk/autotest/warp/data/test3658.tif (I don't know what this tif is designed to test, so hopefully it isn't causing problems.) C:\Documents and Settings\eadam\Desktop>gdaladdo -ro test3658.tif 2 0...10
