Re: [gdal-dev] GDAL/OGR 1.10.0 Release Candidate 2 available for testing
Hi Even, RC2 passes my Windows tests. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ On 2013-04-14 4:25 PM, Even Rouault wrote: > Hi, > > After the issue found by Ari about Perl module versionning in RC1, I've > prepared a second release candidate for GDAL/OGR 1.10.0. > Please test and report any problems promptly. Soon I will initiate a motion > to > declare this release candidate the final release and PSC members can vote on > that after doing some checks themselves. > > Source Code: > http://download.osgeo.org/gdal/gdal-1.10.0RC2.tar.gz > http://download.osgeo.org/gdal/gdal-1.10.0RC2.tar.gz.md5 > http://download.osgeo.org/gdal/gdal1100RC2.zip > http://download.osgeo.org/gdal/gdal1100RC2.zip.md5 > > Autotest Suite: > http://download.osgeo.org/gdal/gdalautotest-1.10.0.tar.gz > > Documentation: > http://download.osgeo.org/gdal/gdal1100doc.zip > > NEWS: > http://svn.osgeo.org/gdal/tags/1.10.0/gdal/NEWS > > Best regards, > > Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff
Hi all, I'm adding internal overviews to a ~20G tiff file and I get this error (1000+ times): gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) gdaladdo still completes and the file seems to work fine. What does that error mean? Does it impact the output file or overviews? Is there a different way I should approach this? Are there any general rules for which compression to select? I've found JPEG-In-TIFF, using PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure of the best compression for the type of raster (in this case single band, byte, colormap). I just checked my version and I am running 1.9.0 on a older modest XP laptop (I figured that I would have been running something more recent). I can try again with a nightly or compile on Ubuntu or use a computer with more resources (the process takes a long time to run, so reporting back on those will take some time). Here is information (I've omitted some previous mosaicing and reprojecting) of how I got to this point and some gdalinfo reports: gdal_translate mosaic_4326.vrt topo_mosaic_4326.tif -co COMPRESS=PACKBITS -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 400 gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config COMPRESS_OVERVIEW PACKBITS ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields array (1410046 elements of 16 bytes each) original input files: >gdalinfo ID_Cuddy_Mountain_20110715_TM_geo.tif Driver: GTiff/GeoTIFF Files: ID_Cuddy_Mountain_20110715_TM_geo.tif Size is 4880, 6845 Coordinate System is: PROJCS["NAD83 / UTM zone 11N", GEOGCS["NAD83", DATUM["North_American_Datum_1983", SPHEROID["GRS 1980",6378137,298.2572221010002, AUTHORITY["EPSG","7019"]], AUTHORITY["EPSG","6269"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4269"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",-117], PARAMETER["scale_factor",0.9996], PARAMETER["false_easting",50], PARAMETER["false_northing",0], UNIT["metre",1, AUTHORITY["EPSG","9001"]], AUTHORITY["EPSG","26911"]] Origin = (509873.30982669181,4969095.03341848590) Pixel Size = (2.032010116947724,-2.032010116947745) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left ( 509873.310, 4969095.033) (116d52'30.00"W, 44d52'30.74"N) Lower Left ( 509873.310, 4955185.924) (116d52'30.97"W, 44d44'59.97"N) Upper Right ( 519789.519, 4969095.033) (116d44'58.04"W, 44d52'30.00"N) Lower Right ( 519789.519, 4955185.924) (116d44'59.99"W, 44d44'59.23"N) Center ( 514831.415, 4962140.479) (116d48'44.75"W, 44d48'45.05"N) Band 1 Block=4880x1 Type=Byte, ColorInterp=Palette NoData Value=195 Color Table (RGB with 256 entries) 0: 0,0,0,255 1: 128,0,0,255 2: 0,128,0,255 3: 128,128,0,255 ... 254: 0,255,255,255 255: 255,255,255,255 final output file with overviews: >gdalinfo -checksum topo_mosaic_4326.tif Driver: GTiff/GeoTIFF Files: topo_mosaic_4326.tif Size is 398449, 210532 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]] Origin = (-124.732439702451200,46.380533997595272) Pixel Size = (0.20993062915,-0.20993062915) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=PACKBITS INTERLEAVE=BAND Corner Coordinates: Upper Left (-124.7324397, 46.3805340) (124d43'56.78"W, 46d22'49.92"N) Lower Left (-124.7324397, 41.9608225) (124d43'56.78"W, 41d57'38.96"N) Upper Right (-116.3677748, 46.3805340) (116d22' 3.99"W, 46d22'49.92"N) Lower Right (-116.3677748, 41.9608225) (116d22' 3.99"W, 41d57'38.96"N) Center (-120.5501072, 44.1706782) (120d33' 0.39"W, 44d10'14.44"N) Band 1 Block=256x256 Type=Byte, ColorInterp=Palette Checksum=45365 NoData Value=195 Overviews: 199225x105266, 99613x52633, 49807x26317, 24904x13159, 12452x6580, 6 226x3290, 3113x1645, 1557x823, 779x412, 390x206 Overviews checksum: 27605, 16665, 58099, 63342, 4876, 64021, 44152, 14528, 574 63, 15006 Color Table (RGB with 256 entries) 0: 0,0,0,255 1: 128,0,0,255 2: 0,128,0,255 3: 128,128,0,255 ... 253: 255,0,255,255 254: 0,255,255,255 255: 255,255,255,255 Best Regards, Eli ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Feature request: gauss and other interpolations in gdalwarp
Hi Etienne and Jan,
For what it's worth, I would also use a gdalwarp gauss interpolation method.
I've found that the gdaladdo gauss interpolation provides the best anti-aliased
downsampling, especially for rasters that contain many small nodata holes
surrounded by valid data. Cubic and lanczos both propagate the nodata holes.
However, in order to obtain optimal results with gdaladdo gauss, I have to
iteratively build overviews using the 3x3 gaussian kernel (gdaladdo 2 4 8). As
far as I understand, this is providing gaussian pyramid downsampling with
simple edge-inward inpainting of the nodata holes. A single downsampling step
with the larger gaussian kernels (gdaladdo 8) doesn't provide the same output
quality. So I suppose I would be most interested in an implementation of a
more generalized iterative gaussian resampling for gdalwarp, allowing for
arbitrary output resolution (not limited to integer steps).
I understand that this is likely a special use case, and my
BuildOverviews/FillNodata hack for downsampling and filling remaining holes
works for now. Regardless, I appreciate the new interpolation options. Thanks.
-David
On Apr 24, 2013, at 12:51 AM, Jan Hartmann wrote:
> Hi folks, just a general question: I opened this thread asking to implement a
> few gdaladdo filters in gdalwarp, and am happy to see that happen now. Would
> it be possible to add a few new filters to gdalwarp/gdaladdo? I'm thinking
> about the gauss filter for gdalwarp, and the unsharp mask filter for gdaladdo
> and gdalwarp. Funding would be no problem, but the question remains: does
> this make sense to you, and is there someone willing and able to implement it?
>
> Jan
>
> On 04/09/2013 11:57 PM, Etienne Tourigny wrote:
>>
>>
>>
>> On Tue, Apr 9, 2013 at 6:50 PM, Even Rouault
>> wrote:
>> Le mardi 09 avril 2013 20:34:40, Even Rouault a écrit :
>> > Le mardi 09 avril 2013 19:06:28, Etienne Tourigny a écrit :
>> > > I have committed new warping methods average and mode to trunk, this will
>> > > be part of gdal-1.10
>> >
>> > Hi Etienne,
>> >
>> > It would be good if you could extend the autotest suite to add tests for
>> > those new warping methods. For that, you can likely take inspiration from
>> > the first tests of autotest/warp/warp.py. "Reference" images based on
>> > utmsmall.tif reference image is a bit big, but you can likely start from a
>> > smaller source image like byte.tif instead that will produce reference
>> > images of reasonable size to be put in svn.
>> >
>> > Regarding nAlgo == 2 (mode with foating point data), the allocations of
>> > pafVals and panSums have the potential to fail if warping is done on a
>> > large image whose floating point values are rarely identical. So I think
>> > that VSIRealloc shoud be used instead with a test on the result to fail
>> > properly. I'm also a bit doubtfull about the practical usefulness of this
>> > case on real data. There might also be a performance issue due to the loop
>> > "//Check array for existing entry" that is at the most inner level of the
>> > algorithm.
>>
>> I stand corrected on the above comment about big memory consumption. The size
>> of the array is limited to the number of source pixels needed to compute a
>> target pixel, so unless you do extreme subsampling, that should be OK.
>> yes
>>
>> I've just noticed however that pafVals and panSums aren't free'd, so there's
>> a
>> memory leak currently. And the CPLRealloc() are a bit weird as currently
>> coded
>> :
>>
>>
>> int nMaxNumPx = 0;
>> float* pafVals = NULL;
>> int* panSums = NULL;
>>
>> if (nNumPx > nMaxNumPx)
>> {
>> pafVals = (float*) CPLRealloc(pafVals, nNumPx *
>> sizeof(float));
>> panSums = (int*) CPLRealloc(panSums, nNumPx *
>> sizeof(int));
>> nMaxNumPx = nNumPx;
>> }
>>
>> The test is always true, so CPLMalloc() would be clearer. But I think there
>> was a will to move nMaxNumPx, pafVals, panSums before the top loops. So that
>> should likely be done.
>>
>> I thought it is weird also, but again copied over from overview code.
>>
>> I thought about running valgrind, but then forgot. I will take your
>> suggestions into consideration, thanks!
>>
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
