Re: [gdal-dev] Check of sibling files when opening an image file
Just in case someone is interested in a solution for this issue: The most efficient setting was to set the gdal option GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR This can reduce the file stat/open calls noticeably, depending on the raster format. The worst format I found was JP2, which is now unfortunately more often used like e.g. in Sentinel-2 data. For a JP2 image, a gdal Open() request for a single JP2 file makes 46 file stat/open calls... With the above setting this is reduced to 6. In order to set this as default, one needs to change this in the source code in file gcore/gdalopeninfo.cpp. If in addition the option GDAL_PAM_ENABLED is set to NO, then another 5 stat/open calls are eliminated. Here this setting can be put as default at compile time by adding to the "configure" script the option --without-pam Cheers Armin On 21/03/2019 19:54, Armin Burger wrote: On 21/03/2019 19:09, Even Rouault wrote: On jeudi 21 mars 2019 18:34:11 CET Armin Burger wrote: Hi all When gdal opens a file, it checks for various sibling files, like *.MTL, *.aux.xml, *.RPC, etc (this is the case when the full directory scanning is deactivated). This can be ~30 stat or open requests when opening a single image file. Is there a way to define the file types that gdal checks for their existence, thus allowing to reduce this to the pure minimum, like *.ovr files only? Is this check driver dependent? If using /vsicurl/ and derived file systems, you can use CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.ovr" for example Even Hi Even Thanks, but we're using a POSIX file system (mounted via FUSE client), so I assume this will not work then. It would also be fine for us to reduce the number of extensions at compile time by modifying configuration or source code files. But so far I haven't found any appearances in the source code files for file extensions that are obviously by default checked for their existence, like e.g. "RPC.TXT". So any hints if/how this is possible would be great. Cheers Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Check of sibling files when opening an image file
On 21/03/2019 19:09, Even Rouault wrote: On jeudi 21 mars 2019 18:34:11 CET Armin Burger wrote: Hi all When gdal opens a file, it checks for various sibling files, like *.MTL, *.aux.xml, *.RPC, etc (this is the case when the full directory scanning is deactivated). This can be ~30 stat or open requests when opening a single image file. Is there a way to define the file types that gdal checks for their existence, thus allowing to reduce this to the pure minimum, like *.ovr files only? Is this check driver dependent? If using /vsicurl/ and derived file systems, you can use CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.ovr" for example Even Hi Even Thanks, but we're using a POSIX file system (mounted via FUSE client), so I assume this will not work then. It would also be fine for us to reduce the number of extensions at compile time by modifying configuration or source code files. But so far I haven't found any appearances in the source code files for file extensions that are obviously by default checked for their existence, like e.g. "RPC.TXT". So any hints if/how this is possible would be great. Cheers Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Check of sibling files when opening an image file
Hi all When gdal opens a file, it checks for various sibling files, like *.MTL, *.aux.xml, *.RPC, etc (this is the case when the full directory scanning is deactivated). This can be ~30 stat or open requests when opening a single image file. Is there a way to define the file types that gdal checks for their existence, thus allowing to reduce this to the pure minimum, like *.ovr files only? Is this check driver dependent? Thanks in advance for any hints Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Set gdal option at compile time
On 20/03/2019 22:01, Even Rouault wrote: On mercredi 20 mars 2019 21:06:21 CET Armin Burger wrote: Dear all is it possible to change the default value of a GDAL option at compile time? No Must be done at runtime. Can be done by setting it for example as environment variable in your batch script. Even Thanks for the quick reply. When opening a file, gdal checks for various sibling files, like *.AUX, *.aux.xml, etc. Is there a sort of configuration file that defines for which files and file extensions gdal is searching for any file open? Best Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Set gdal option at compile time
Dear all is it possible to change the default value of a GDAL option at compile time? I would in particular like to set the option "GDAL_DISABLE_READDIR_ON_OPEN" to TRUE as a default since in a batch processing environment this can cause some overload of storage systems. Thanks in advance for any hint Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Read error on large VRT files
I meant the GDAL_MAX_DATASET_POOL_SIZE option. Don't know if it really has any influence, but these read errors happened multiple times in the past when trying to translate VRT's with lots of input files, for tiles that were actually readable. It could be of course also a problem of the network file system, short network time-outs etc. Armin On 21/12/16 10:49, Even Rouault wrote: On mercredi 21 décembre 2016 10:29:27 CET Armin Burger wrote: Margherita thanks, now that you say this, I remeber a special GDAL option "GDAL_MAX_DATASET_POOL_SIZE" which by default is 100. The operating system limit itself is high enough on our server (65k). When running the processing again with less memory cache (and hence likely less concurrent opened files) it worked without errors. I will set this GDAL option and see if I ever run into this errors again. On which option did you play exactly ? The TIFF error you get is a rather low level one (really trying to read a range of bytes from a file), and I can't explain why GDAL options would play a role on it. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Read error on large VRT files
Margherita thanks, now that you say this, I remeber a special GDAL option "GDAL_MAX_DATASET_POOL_SIZE" which by default is 100. The operating system limit itself is high enough on our server (65k). When running the processing again with less memory cache (and hence likely less concurrent opened files) it worked without errors. I will set this GDAL option and see if I ever run into this errors again. armin Gesendet: Mittwoch, 21. Dezember 2016 um 07:29 Uhr Von: "Margherita Di Leo" An: armin.bur...@gmx.net Betreff: Re: [gdal-dev] Read error on large VRT files Hey Armin, Could it be a limitation in number of open files at same time? See https://grasswiki.osgeo.org/wiki/Large_raster_data_processing#Troubleshooting Il giorno mar 20 dic 2016 alle 22:21 Armin Burger <armin.bur...@gmx.net> ha scritto: Hi all I sometimes get read errors when trying to convert large input VRT files into a single BigTIFF file, typically applying an outsize of e.g. 50% or 25%. Errors are like ERROR 1: TIFFFillTile:Read error at row 11264, col 11264; got 46 by tes, expected 84 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: 2352/2592_BUCNFD.tif, band 1: IReadBlock failed at X offset 45, Y offset 47 The tile that returns the error can be converted alone without problems. All together the number of tiles of the VRT can be towards 6000, having 100 GB in total, tiles are LZW or Deflate compressed TIFF's. Is there a possible source of errors when trying to read and convert VRT's of compressed TIFF of that size? Thanks for any hint Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Read error on large VRT files
Hi all I sometimes get read errors when trying to convert large input VRT files into a single BigTIFF file, typically applying an outsize of e.g. 50% or 25%. Errors are like ERROR 1: TIFFFillTile:Read error at row 11264, col 11264; got 46 by tes, expected 84 ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: 2352/2592_BUCNFD.tif, band 1: IReadBlock failed at X offset 45, Y offset 47 The tile that returns the error can be converted alone without problems. All together the number of tiles of the VRT can be towards 6000, having 100 GB in total, tiles are LZW or Deflate compressed TIFF's. Is there a possible source of errors when trying to read and convert VRT's of compressed TIFF of that size? Thanks for any hint Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Problem when creating overviews for large input files with gdaladdo
Hi Even thanks a lot for the fast reply and the explanation! I will try your workaround. The patience needed to wait for the completion of the "normal" method seems to be a bit too high, I stopped the last test after 4 days... Best Armin On 19/12/16 22:26, Even Rouault wrote: Armin, This is a "well known" performance issue with the geotiff driver / libtiff when switching back and forth between IFD (image file directories). At ~ 75%, you start generating overview factor 4 and that implies switching regularly between the one previously computed (read operations) and the new one (write operations). Each time you switch, you need to read and write the TileOffsets and TileByteCounts tags (those can be 10's of megabytes large on huge file, and are the root cause for the performance problem). With infinite patience, this would eventually complete. A trick you can use is : gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE 2 gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr 2 gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr 2 gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr.ovr 2 Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Problem when creating overviews for large input files with gdaladdo
Dear all I ran into a problem when trying to create overviews for large LZW-compressed GeoTIFF images using gdaladdo. The TIFF's are internally tiled, the file type is Byte. GDAL versions tested are 2.1.1 and 1.11, running on 64bit Linux. The first try was using an input file of size 1038336 x 565248 pixels. Setting CPL_DEBUG=ON the processing goes up to the message ...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1 of tileindex_nodsm_otsu_2 and then it remains at this "70.." % forever, with no further log output. This happened for all files of similar size, on multiple machines, on both GDAL versions. Always hangs at this "70.." state, the process remains alive but nothing is written any more to the output file. I used the command like gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE 2 4 8 16 32 64 Trying to specify BIGTIFF or not for the overview did not make any difference. Using a smaller file or converting the large input file to 50% size and running the gdaladdo on this smaller file finished successfully, but also here I got some warnings of a similar kind, and the process got stuck for a while after the ".70..GDAL: Potential thrashing..." warning, but then continued to the end. The full log is attached. An additional observation I made was that the overview file stopping at 70% was already bigger than the input file. I was thinking to create single lower resolution files and merging them to an overview file, but there's no built-in functionality for this, so did not follow it up further. Could there be a sort of limitation of maximum pixels x*y, or with the LZW compression? Any ideas how this could be solved? Thanks in advance for any hint Armin GDAL: GDALOpen(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif, this=0xc36800) succeeds as GTiff. GTiff: File open for read-only accessing, creating overviews externally. GDAL: GDALDefaultOverviews::OverviewScan() GDAL: GDALOpen(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0xc485f0) succeeds as GTiff. GTiff: ScanDirectories() GTiff: Opened 55296x38400 overview. GTiff: Opened 27648x19200 overview. GTiff: Opened 13824x9600 overview. GTiff: Opened 6912x4800 overview. GTiff: Opened 3456x2400 overview. 0GDAL: GDAL_CACHEMAX = 116073 MB ...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1 of /scratch/mosaic_20161219/tileindex_esm_otsu_2 .tif.ovr. .80...90.GDAL: Potential thrashing on band 1 of . ..GDAL: Potential thrashing on band 1 of . GDAL: Potential thrashing on band 1 of . GDAL: Potential thrashing on band 1 of . GDAL: 259200 block reads on 129600 block band 1 of . GDAL: 64800 block reads on 32400 block band 1 of . GDAL: 16200 block reads on 8100 block band 1 of . GDAL: 4104 block reads on 2052 block band 1 of . GDAL: GDALClose(/scratch/mosaic_20161219/tileindex_esm_otsu_2.tif.ovr, this=0x
Re: [gdal-dev] Fwd: Fwd: OpenJPEG: The Slumbering Giant Awakens
Aaron, thanks a lot! I will try it the next days. Armin On 12/01/16 22:10, Aaron Boxer wrote: -- Forwarded message -- From: *Aaron Boxer* mailto:boxe...@gmail.com>> Date: Tue, Jan 12, 2016 at 4:06 PM Subject: Re: [gdal-dev] Fwd: OpenJPEG: The Slumbering Giant Awakens To: armin.bur...@gmx.net <mailto:armin.bur...@gmx.net> Hi Armin, Yes, indeed. You can find it here: https://github.com/CodecCentral/openjpeg/tree/omnibus and run cmake with openmp flag turned on. This is a complete version of openjpeg, with my patch applied on top. Cheers, Aaron On Tue, Jan 12, 2016 at 3:43 PM, Armin Burger mailto:armin.bur...@gmx.net>> wrote: Hi Aaron would this patch (or a patched version of complete Openjpeg source) already be available for testing to assess the difference to current release? Thanks Armin On 12/01/16 15:13, Aaron Boxer wrote: Yes, it will be a lot faster. By the way, this patch may be delayed a little bit while we test some more. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Fwd: OpenJPEG: The Slumbering Giant Awakens
Hi Aaron would this patch (or a patched version of complete Openjpeg source) already be available for testing to assess the difference to current release? Thanks Armin On 12/01/16 15:13, Aaron Boxer wrote: Yes, it will be a lot faster. By the way, this patch may be delayed a little bit while we test some more. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] OpenFileGDB, feature classes in feature dataset
Hi Even Thanks a lot again! I added the fix to the current 1.11.2 version and afterwards everything was fine, all layers in the FGDB v9 were found. Cheers Armin On 03/15/2015 04:08 AM, Even Rouault wrote: Armin, I've committed a fix in http://trac.osgeo.org/gdal/ticket/5875 that should hopefully fix your issue. There was a problem with the presence of non spatial tables that could cause later tables not to be listed. This wasn't specifically related to feature classes. Even Hi Even thanks for the reply! The FGDB dataset where I discovered that has unfortunately some commercial IPR restrictions, so I cannot share it (and it in addition has some 800 MB...). I will try to create a test FGDB with a similar internal structure, I have to check if I can create an FGDB v9 format with recent ArcGIS versions. Armin On 03/10/2015 06:48 AM, Even Rouault wrote: Selon Armin Burger : Dear all I have a question regarding the OpenFileGDB driver in 1.11.2 release. It now lists also non-spatial tables as layers of an FGDB (as mentioned in the changelog). But it seems to not list any more layers that are "feature classes" under a parent "feature dataset" (that's how the relation is displayed in ArcGIS). This is at least the case for FGDB's in older v9 format. In GDAL version 1.11.0 these feature classes were still listed as layers. Is this a bug or an intended behavior? Looks like an unfortunate side effect. could you open a ticket and attach such a GDB or provide a link to it ? Best regards Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] OpenFileGDB, feature classes in feature dataset
Hi Even thanks for the reply! The FGDB dataset where I discovered that has unfortunately some commercial IPR restrictions, so I cannot share it (and it in addition has some 800 MB...). I will try to create a test FGDB with a similar internal structure, I have to check if I can create an FGDB v9 format with recent ArcGIS versions. Armin On 03/10/2015 06:48 AM, Even Rouault wrote: Selon Armin Burger : Dear all I have a question regarding the OpenFileGDB driver in 1.11.2 release. It now lists also non-spatial tables as layers of an FGDB (as mentioned in the changelog). But it seems to not list any more layers that are "feature classes" under a parent "feature dataset" (that's how the relation is displayed in ArcGIS). This is at least the case for FGDB's in older v9 format. In GDAL version 1.11.0 these feature classes were still listed as layers. Is this a bug or an intended behavior? Looks like an unfortunate side effect. could you open a ticket and attach such a GDB or provide a link to it ? Best regards Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] OpenFileGDB, feature classes in feature dataset
Dear all I have a question regarding the OpenFileGDB driver in 1.11.2 release. It now lists also non-spatial tables as layers of an FGDB (as mentioned in the changelog). But it seems to not list any more layers that are "feature classes" under a parent "feature dataset" (that's how the relation is displayed in ArcGIS). This is at least the case for FGDB's in older v9 format. In GDAL version 1.11.0 these feature classes were still listed as layers. Is this a bug or an intended behavior? Best regards Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] LZW Compression on geotiffs
I would also always recommend to use -co PREDICTOR=2 when compressing with LZW (or DEFLATE). So far I made quite good experiences with LZW and PREDICTOR=2, with typical compression rates close to gzip compression (meaning when putting the tif file into a tgz archive file). The only time that I discovered where LZW does not work well was when using gdalwarp and directly writing the output as (untiled) LZW TIFF. This could create LZW files up to 2-3 times bigger than the uncompressed input TIFF. When using gdalwarp and one wants to save disk space one should either create tiled LZW TIFF's, or warp to uncompressed TIFF and then use gdal_translate with LZW to create the final LZW image, PREDICTOR=2 is recommended for both methods. armin On 10/02/2014 01:13 AM, Even Rouault wrote: Le jeudi 02 octobre 2014 01:06:57, David Strip a écrit : On 10/1/2014 12:02 PM, Jukka Rahkonen wrote: For comparison: Tiff as zipped347 MB Tiff into png 263 MB If I have understood right both zip and png are using deflate algorithm so there might be some place for improving deflate compression in GDAL. I was curious how png could achieve such better compression if it is using the same deflate algorithm. I wouldn't think different implementations would account for so much improvement. It turns out the png compression uses a "filtering" step ahead of compression. This is explained here. The filter is similar to a differential pulse code modulation, in which the pixel is represented as the difference from the pixels to the left, left upper diagonal, and above. This typically reduces the magnitude of the value to something close to zero, making the encoding more efficient. True, a way to improve things might be to specify -co PREDICTOR=2. Should apply to both LZW and DEFLATE. This is one of the filter that might be used by PNG, except that PNG has different filters, so it will eventually beat TIFF deflate. David ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Re: TiffAppendToStrip errors with uncompressed output
Jukka I downloaded and converted your test VRT file to a single Geotiff without any problems - on Linux... Conversion took a few minutes on a fast server. Just a guess, but in my opinion this problem is related to the limited memory handling capabilities of Windows. It does not seem to be a shortage of total memory but more a sort of memory "fragmentation". Windows is not able to provide a contiguous block of memory that is required by some software in some cases. Some time ago a colleague had this type of problem when working with IDL scripts, it worked fine on Linux or Mac OS, never on Windows with larger datasets. On Windows, working with large datasets and gdal (especially compressed and VRT) or allowing gdal_translate to use more memory via GDAL_CACHEMAX config, caused always problems for me. So I stopped trying this... On Linux I can easily work with VRT files that reference several 100's of GB. There on Linux however, higher settings for GDAL_CACHEMAX were speeding up conversions noticeably when working with lots of tiles, especially when they were compressed Tiffs. So I can just recommend, if feasible, to try out Linux or any type of Unix for your processing and see if the problem disappears. Greetings Armin On 08/05/2012 14:32, Jukka Rahkonen wrote: Jukka Rahkonen mmmtike.fi> writes: Hi, I have some LZW compressed paletted 8-bit tiff files which I look through a .vrt file. I have been taking some timings with different gdal_translate parameters but for some reason the most common command fails always for me. The result is gdal_translate -of GTiff test.vrt uncompressed.tif Input file size is 96000, 48000 0...10...20...30ERROR 1: TIFFAppendToStrip:Write error at scanline 14650 ERROR 1: TIFFAppendToStrip:Write error at scanline 14650 ERROR 1: An error occured while writing a dirty block ERROR 1: TIFFAppendToStrip:Write error at scanline 14664 ERROR 1: TIFFAppendToStrip:Write error at scanline 14664 I can repeat the error with another computer (Vista 32-bit). This time the error occurs at scanline 44731 when progress bar had advanced to value of 90. Obviously there is a lack of some resource but I cannot imagine what it is and how I could give more of it for gdal_translate. Both computers have 2 GB of memory but resource manager does not show signs about running out of memory. There is now a 460 MB zip file at http://latuviitta.org/documents/TiffAppendToStrip_error.zip It contains the source tiffs and .vrt file "UM52.vrt" and after unzipping it should be ready for a test with gdal_translate -of GTiff UM542.vrt output.tif SRS is EPSG:3067 if it matters. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] VRT derived band, pixel functions and Python bindings
On 22/03/2012 21:00, Even Rouault wrote:
Le jeudi 22 mars 2012 19:26:56, Armin Burger a écrit :
Hi all
I set up 2 pixel functions to be used with VRT images and derived bands.
This works fine with the GDAL utilities and also with MapServer and
Python MapScript.
If I try however to load the pixel functions in GDAL-Python as drivers,
I run into problems. The following code
from osgeo import gdal
import os
os.putenv("GDAL_DRIVER_PATH", "/usr/local/share/gdal/gdalplugins")
gdal.AllRegister()
produces the error:
python: symbol lookup error: /usr/local/share/gdal/gdalplugins
/gdal_SpotBlue.so: undefined symbol: GDALAddDerivedBandPixelFunc
So is this sort of a bug or is this functionality not available via the
GDAL Python bindings? Environment is Linux, GDAL 1.9, Python 2.6.
No fundamental reason it should not work. I'm pretty sure to have done that in
the past. Did you make sure that your gdal_SpotBlue.so links to libgdal.so
(check with ldd) ? But I'm not sure why loading throug python would make a
difference w.r.t loading with GDAL utilities or MapServer.
Even
compilation was with
gcc -fPIC -c SpotBlue.c
gcc -shared SpotBlue.o -o SpotBlue.so
So no spacial linking against libgdal.so. ldd therefore does not report
anything related to the libgdal.so. But since it directly worked from
the normal GDAL utilities and MapScript I did not check anything further.
What would be your suggestion for additional flags to add for the
compilation?
Armin
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] VRT derived band, pixel functions and Python bindings
Hi all
I set up 2 pixel functions to be used with VRT images and derived bands.
This works fine with the GDAL utilities and also with MapServer and
Python MapScript.
If I try however to load the pixel functions in GDAL-Python as drivers,
I run into problems. The following code
from osgeo import gdal
import os
os.putenv("GDAL_DRIVER_PATH", "/usr/local/share/gdal/gdalplugins")
gdal.AllRegister()
produces the error:
python: symbol lookup error: /usr/local/share/gdal/gdalplugins
/gdal_SpotBlue.so: undefined symbol: GDALAddDerivedBandPixelFunc
So is this sort of a bug or is this functionality not available via the
GDAL Python bindings? Environment is Linux, GDAL 1.9, Python 2.6.
Thanks for any help
Armin
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdaladdo, overviews and NODATA
On 21/02/2012 22:36, Frank Warmerdam wrote: On Tue, Feb 21, 2012 at 1:17 PM, Armin Burger wrote: Then I tried to set the NODATA value to 0 using Gdal-Python, and afterwards they had Band 1 Block=3000x1 Type=Byte, ColorInterp=Red NoData Value=0 Band 2 Block=3000x1 Type=Byte, ColorInterp=Green NoData Value=0 Band 3 Block=3000x1 Type=Byte, ColorInterp=Blue NoData Value=0 But the result was the same, when using "-r average" a 1-2 pixel area of the original NODATA values along the border to the DATA areas seem to have got pixels with higher values than 0 (at least in Mapserver they cannot any more be "hidden" with OFFSITE). Is there a simple way to write out the overviews to separate Tiff files so that I can check them directly? Armin, You can use the gdal/apps/dumpoverviews app to write out overviews. It may not be built and installed by default. If you confirm the overviews are built wrong then please file a ticket, and provide with it a small sample file with which you encounter the problem. Best regards, Frank I checked the overviews after dumping them into Tiff files. I could not see anything strange with them along the border DATA-NODATA. The supposed NODATA pixels all had value 0 in all bands. So the problem seems a bit related how Mapserver handles the files. Strangely enough that the problem does not appear when the OV are created with nearest resampling. Being somehow more related to Mapserver I filed a ticket there http://trac.osgeo.org/mapserver/ticket/4210 But the ticket owner are already you anyway ;-) I could not upload a small test image due to file size restrictions (the zip has 3 MB). Should I send the file to you or someone else via mail? I discovered an additional strange effect when using the layer processing parameter PROCESSING "RESAMPLE=BILINEAR" which will create partially colored effects along the DATA-NODATA line. All of this happens just in the case that this line is not exactly horizontal or vertical but oblique due to image warping. I will create another ticket for this. So far it seems that I can use no "smoothing" via resampling, neither with overviews nor with the layer definition without having some "border effects". Thanks Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdaladdo, overviews and NODATA
Frank the original images showed ... Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=PIXEL ... Band 1 Block=3000x1 Type=Byte, ColorInterp=Red Band 2 Block=3000x1 Type=Byte, ColorInterp=Green Band 3 Block=3000x1 Type=Byte, ColorInterp=Blue Then I tried to set the NODATA value to 0 using Gdal-Python, and afterwards they had Band 1 Block=3000x1 Type=Byte, ColorInterp=Red NoData Value=0 Band 2 Block=3000x1 Type=Byte, ColorInterp=Green NoData Value=0 Band 3 Block=3000x1 Type=Byte, ColorInterp=Blue NoData Value=0 But the result was the same, when using "-r average" a 1-2 pixel area of the original NODATA values along the border to the DATA areas seem to have got pixels with higher values than 0 (at least in Mapserver they cannot any more be "hidden" with OFFSITE). Is there a simple way to write out the overviews to separate Tiff files so that I can check them directly? Armin On 21/02/2012 21:48, Frank Warmerdam wrote: Armin, Reviewing the overview building code in gdal/gcore/overview.cpp it clearly tries to exclude nodata values from the averaging calculation. What does the gdalinfo report look like for the file before you run gdaladdo? Best regards, On Tue, Feb 21, 2012 at 12:36 PM, Armin Burger wrote: Hi all I have a question regarding the gdaladdo tool and NODATA pixels: When using resampling like "average" it seems that along the border of DATA and NODATA pixels there happens an averaging also of some of the original NODATA pixels (looks like 1-2 pixel width). This way in the overviews they get values higher than 0 (0 is defined as NODATA). Using this type of data in mapserver leads to small black borders when the overviews are taken since the tag OFFSITE 0 0 0 does not work anymore for the small border of now only "nearly-black" pixels. I tried to set the metadata NODATA of the images to 0 for each band but it had no effect. Only when I use resampling "nearest" then every NODATA pixel of the overviews retains the full black 0 0 0. Is there a way to force to exclude NODATA pixels during the "average" resampling so that they retain their NODATA values? Thanks Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] gdaladdo, overviews and NODATA
Hi all I have a question regarding the gdaladdo tool and NODATA pixels: When using resampling like "average" it seems that along the border of DATA and NODATA pixels there happens an averaging also of some of the original NODATA pixels (looks like 1-2 pixel width). This way in the overviews they get values higher than 0 (0 is defined as NODATA). Using this type of data in mapserver leads to small black borders when the overviews are taken since the tag OFFSITE 0 0 0 does not work anymore for the small border of now only "nearly-black" pixels. I tried to set the metadata NODATA of the images to 0 for each band but it had no effect. Only when I use resampling "nearest" then every NODATA pixel of the overviews retains the full black 0 0 0. Is there a way to force to exclude NODATA pixels during the "average" resampling so that they retain their NODATA values? Thanks Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using VRT images with derived bands (pixel function) with MapServer
Hello Yves I succeeded to get it running with Mapserver, after some testing and additional settings also with PHP MapScript. For others who might be interested in the various steps, I added a short how-to to the Mapserver Wiki, see http://trac.osgeo.org/mapserver/wiki/RasterVrtDerivedBands Thanks again for the help! Armin On 16/02/2012 18:30, Yves Jacolin (free) wrote: Hello, I used the Makefile file in an archive I found in a GDAL ticket. So I guess I am using a .so file. Y. Le jeudi 16 février 2012 10:15:58, Armin Burger a écrit : Yves thanks a lot for the help. So it is sufficient to add SetEnv GDAL_DRIVER_PATH ..., place the file with the pixel function there, and Mapserver can find pixel function referenced in the VRT file? What kind of compilation did you use? Create a shared library *.so or is the *.o object file sufficient? Armin Original-Nachricht Datum: Thu, 16 Feb 2012 09:31:11 +0100 Von: Yves Jacolin An: gdal-dev@lists.osgeo.org, armin.bur...@gmx.net Betreff: Re: [gdal-dev] Using VRT images with derived bands (pixel function) with MapServer Armin, I am using such application stack: netcdf> vrt and pixel functions> GDAL MapServer We have a tiled layer inside PostGIS and we juste configure the layer like all other tiled datasource: LAYER STATUS ON NAME "layer_idx" TYPE POLYGON CONNECTION "dbname=mydb host=localhost user=myUser password=mypassword" CONNECTIONTYPE postgis DATA "the_geom FROM myLayer USING srid=4326" PROJECTION "init=epsg:4326" END METADATA "wms_srs" "EPSG:4326" "wms_timeextent" "2010-12-20/2010-12-26" "wms_timeitem" "datetime" #column in postgis table of type timestamp "wms_timedefault" "2010-12-24T00:00:00Z" "wms_enable_request" "*" END END If you want to use the vrt file directly it is also possible as easily as: DATA "vrt/myLayer.vrt" About register the plugin, in Apache config you have to use SetEnv this way: SetEnv GDAL_DRIVER_PATH directory/mapserver/misc SetEnv MS_MAPFILE directory/mapserver/mymapfile.map SetEnv MS_MAPFILE_PATTERN "^directory/mapserver/mymapfile.map" Hth, Y. Le Mercredi 15 Février 2012 19:52:21 Armin Burger a écrit : Dear all I would be interested in using VRT images with "derived bands" as described on the VRT image documentation page. This requires the definition of a pixel function and the registration of this function in the used application via "GDALAddDerivedBandPixelFunc". The provided sample function is already towards what I'm looking for. I would just need to adapt it accordingly (calculating band 1 as combination of source band 1 and 3). Since my main interest is to use such image definition in the context of Mapserver, my question is if there could be a way to use this also from Mapserver as application? If yes, where would I need to register the function and how would I best make the function available to Mapserver (maybe via an .o file that is created when compiling)? Maybe somewhere inside mapgdal.c ? Would such a pixel function be used also in case when not the full resolution image (i.e. TIFF file referenced in the VRT file) is read but one of the overviews from the .ovr file associated to the Tiff file? Thanks in advance for any hints Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using VRT images with derived bands (pixel function) with MapServer
Yves thanks a lot for the help. So it is sufficient to add SetEnv GDAL_DRIVER_PATH ..., place the file with the pixel function there, and Mapserver can find pixel function referenced in the VRT file? What kind of compilation did you use? Create a shared library *.so or is the *.o object file sufficient? Armin Original-Nachricht > Datum: Thu, 16 Feb 2012 09:31:11 +0100 > Von: Yves Jacolin > An: gdal-dev@lists.osgeo.org, armin.bur...@gmx.net > Betreff: Re: [gdal-dev] Using VRT images with derived bands (pixel function) > with MapServer > Armin, > > I am using such application stack: netcdf > vrt and pixel functions > GDAL > > > MapServer > > We have a tiled layer inside PostGIS and we juste configure the layer like > all > other tiled datasource: > > LAYER > STATUS ON > NAME "layer_idx" > TYPE POLYGON > CONNECTION "dbname=mydb host=localhost user=myUser password=mypassword" > CONNECTIONTYPE postgis > DATA "the_geom FROM myLayer USING srid=4326" > PROJECTION > "init=epsg:4326" > END > METADATA >"wms_srs" "EPSG:4326" >"wms_timeextent" "2010-12-20/2010-12-26" >"wms_timeitem" "datetime" #column in postgis table of type timestamp >"wms_timedefault" "2010-12-24T00:00:00Z" >"wms_enable_request" "*" > END > END > > If you want to use the vrt file directly it is also possible as easily as: > DATA "vrt/myLayer.vrt" > > About register the plugin, in Apache config you have to use SetEnv this > way: >SetEnv GDAL_DRIVER_PATH directory/mapserver/misc >SetEnv MS_MAPFILE directory/mapserver/mymapfile.map >SetEnv MS_MAPFILE_PATTERN "^directory/mapserver/mymapfile.map" > > Hth, > > Y. > > Le Mercredi 15 Février 2012 19:52:21 Armin Burger a écrit : > > Dear all > > > > I would be interested in using VRT images with "derived bands" as > > described on the VRT image documentation page. This requires the > > definition of a pixel function and the registration of this function in > > the used application via "GDALAddDerivedBandPixelFunc". > > > > The provided sample function is already towards what I'm looking for. I > > would just need to adapt it accordingly (calculating band 1 as > > combination of source band 1 and 3). > > > > Since my main interest is to use such image definition in the context of > > Mapserver, my question is if there could be a way to use this also from > > Mapserver as application? If yes, where would I need to register the > > function and how would I best make the function available to Mapserver > > (maybe via an .o file that is created when compiling)? Maybe somewhere > > inside mapgdal.c ? > > > > Would such a pixel function be used also in case when not the full > > resolution image (i.e. TIFF file referenced in the VRT file) is read but > > one of the overviews from the .ovr file associated to the Tiff file? > > > > Thanks in advance for any hints > > > > Armin > > ___ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > -- > Yves Jacolin > > http://yjacolin.gloobe.org -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Using VRT images with derived bands (pixel function) with MapServer
Dear all I would be interested in using VRT images with "derived bands" as described on the VRT image documentation page. This requires the definition of a pixel function and the registration of this function in the used application via "GDALAddDerivedBandPixelFunc". The provided sample function is already towards what I'm looking for. I would just need to adapt it accordingly (calculating band 1 as combination of source band 1 and 3). Since my main interest is to use such image definition in the context of Mapserver, my question is if there could be a way to use this also from Mapserver as application? If yes, where would I need to register the function and how would I best make the function available to Mapserver (maybe via an .o file that is created when compiling)? Maybe somewhere inside mapgdal.c ? Would such a pixel function be used also in case when not the full resolution image (i.e. TIFF file referenced in the VRT file) is read but one of the overviews from the .ovr file associated to the Tiff file? Thanks in advance for any hints Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Optimisation of ECW files
On 08/11/2011 22:27, Yves Jacolin (free) wrote: It seems there was a performance issue with this files. I though using a lot of small ECW files was not a good practice. Am I wrong? Yves my experience is that making Mapserver (via GDAL) reading lots of small ECW files becomes quite inefficient. I would say having more than maybe 50 or 100 files to be read for one Mapserver request starts to slow down the response time of Mapserver. Geotiff with overview files I found a bit better in this respect, but still suboptimal. ECW works well for very large single files, having them even with several 10's of GB will not cause any problems. If you have frequent updates of the single files then merging the files into few large ones might not be a feasible option. I would then try to use Geotiff with overviews, it should be at least a bit better than the ECW. Automatically converting the ECW files to Geotiff is done quickley. Regarding the required space, Geotiff with overviews need ~5 to 15 times more disc space, depending on the ECW compression rate. If you need to display the image layer already at very low scales (hence requiring many files to read) then a single merged low-resolution file with ~10-20% resolution (and with overviews) as active layer for low scales would help quite much. Then only when the request is at higher scales, the small images are read, but the number of files will be much lower. OK, this low-resolution files needs to be updated every time you replace some of the ECW files, but it should be much faster to create a 10% resolution file than a full-resolution one. Using high values for "--config GDAL_CACHEMAX" with "gdal_translate" can speed up quite much merging the small files. armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Help on GDAL 1.8.1 .dll files for C#
Zhenyu, have you tried the compiled versions from http://www.gisinternals.com/sdk/ they come in various flavours of compilers, in 32/64 bit, contain java, python and c# bindings. So I guess one of them should fit your environment. Best regards armin On 02/10/2011 23:01, Zhenyu Lu wrote: Hi, Is there any possibilities that any kind-hearted guys can send me the compiled .dll files of GDAL 1.8.1 for C#. I am trying to compile these .dll files this whole afternoon, but failed all the time. I used to create tools using GDAL 1.5.2 and C#. These tools run well under the OS of XP, but these days I would like to run the tools under the Windows 7. It always remind me that "initialize the OSGeo.GDAL.Gdal throw an exception". I am guessing is that because the 1.5.2 is too old to work on Windows 7. Then I decided to compile the most current version of GDAL 1.8.1 to see whether it will be useful for solving the problem. I have already obtained three .dll files for GDAL, which are gdal18, gdal_wrap, gdal_csharp. But I noticed that two files named as gdalconst_csharp.dll and gdalconst_wrap.dll are missing. Does the GDAL 1.8.1 require same .dll files as the GDAL 1.5.2? Anyone send me the compiled .dll files or let me know where the possible problem would be might save my frustrated life ^-^. Thanks in advance for whoever can help! -- Zhenyu Lu SUNY-ESF ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GTiff optimisation
On 01/09/2011 14:23, Even Rouault wrote: As I did not use DATUM and PROJ creation option GDAL don't add it! Yes, the mapping between WKT representation and ECW DATUM/PROJ is not perfect, especially in the WKT -> DATUM/PROJ direction (the other direction is simpler). You could perhaps try the updated ecw_cs.wkt file attached to ticket http://trac.osgeo.org/gdal/ticket/4127 Finally I saw that gdal create only an aux.xml file to store his own proj information and don't use the projection information from the header file. If I create this aux.xml file for all of my raster data it could work. Yes, .aux.xml is a solution that works for most GDAL drivers. But of course, it will only work in the GDAL-based software world. I made the experience that ECW files do not work too well with the *.aux.xml file and less common projections. Often gdalinfo cannot convert the projected coordinates into lat/lon in it's output. And especially in combination with Mapserver reprojection does often not work correctly with ECW files that should read the projection info from the *.aux.xml file. But as soon as you apply the ECW specific DATUM and PROJ definitions and add the corresponding projections/datums to the ecw_cs.wkt file (if missing there) everything works like a charm. Also desktop clients like ArcGIS 10 and QGIS work well with those ECW's if you add the missing projections and datums to their own copy of the ecw_cs.wkt file. Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL/OGR 1.8.1 Advance Warning
On 29/06/2011 19:53, Even Rouault wrote: Le mardi 28 juin 2011 21:32:01, Armin Burger a écrit : Frank a functionality that I would find useful in combination with clients like ArcGIS or Erdas are the Erdas overviews in *.rrd files. When creating overviews with gdaladdo using the "--config USE_RRD YES" option, aux files instead of rrd files are created. They are not fully recognized by ArcGis/Erdas, depending on the image format. I'm afraid I know very little of the subtleties of Erdas overviews, but I see that the HFA_USE_RRD config option can also be set to YES and produces a .rrd file. Perhaps this is was you are looking for. See http://gdal.org/frmt_hfa.html . I've found also a past thread where this is somehow discussed : http://www.mail-archive.com/gdal- d...@lists.osgeo.org/msg00129.html Even thanks a lot for pointing me to the thread. I was not aware of this double option --config HFA_USE_RRD YES --config USE_RRD YES since the documentation and the --help of gdaladdo just mention the USE_RRD option, which puts the overview inside a single big .aux file. It looks a bit like the small .aux file created with the 2 config options contains also image statistics, but I will check this tomorrow at work. Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL/OGR 1.8.1 Advance Warning
Frank a functionality that I would find useful in combination with clients like ArcGIS or Erdas are the Erdas overviews in *.rrd files. When creating overviews with gdaladdo using the "--config USE_RRD YES" option, aux files instead of rrd files are created. They are not fully recognized by ArcGis/Erdas, depending on the image format. For me an ideal solution would therefore be being able to create overviews in standard rrd files (I guess the format is not much different from the one used for the overviews in aux files) and the possibility to create standard Erdas aux files that contain only information like the histogram, but no overviews. Looks to me like the creation of HFA formats supports that already. Clients like Erdas and ArcGis use and create the aux file for any type of image, also for Tiff or ECW files when applying image stretching. Having the possibility to create such aux files with a command line tool for various image formats could be useful in an image archive where clients do not have write access and cannot create neither rrd nor aux files. And the creation could then be automated and performed eg. on a server, and it could also be much faster than clients like ArcGis calculating the histograms. Just my ideas. Greetings Armin On 27/06/2011 14:31, Frank Warmerdam wrote: Folks, I would like to prepare a 1.8.1 release candidate in the next couple days. Developers are encouraged to address any bug fixes they would like promptly. Is there anything folks feel strongly needs to be addressed in 1.8.1 that has not yet made it into the 1.8 branch? Best regards, ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] datum projection info in ecw
Nicholas I was interested about similar things a few weeks ago and got quick help in this forum. As jean-Claude already mentioned there's a header editor for Windows that allows you to see/edit the datum and projection info. Jean-Claude also created a tool "ecwhed" that allows you to get and set these parameters via the command line, at least under Linux it compiles easily, it requires the ECW SDK. If you create ECW's via GDAL you should use the options -co "PROJ=..." and -co "DATUM=..." to be sure what GDAL writes to the ECW header. For reading ECW files GDAL uses the definitions in the file "ecw_cs.wkt" (on Unix under /usr/local/share/gdal/). There you can add your own projections and datums or modify existing ones, eg. adding datum shift parameters. Programs that use GDAL for raster rendering like QGIS or ArcGis 10 can also work with these new projection definitions if you add them to their own "ecw_cs.wkt" file, Mapserver takes the file of the GDAL installation. Regards Armin On 28/06/2011 03:09, nicholas.g.lawre...@tmr.qld.gov.au wrote: Hello all, Does anyone know where the datum and projection information is stored inside an ecw? Kind regards, * Nicholas Lawrence* Senior Spatial Science Officer | Darling Downs Region / Toowoomba Office * Assets & Operations* Division | Department of Transport and Main Roads Floor 1 | Toowoomba - Phillip Street | 1-5 Phillip Street (cnr Clopton Street) | Toowoomba Qld 4350 PO Box 645 | Toowoomba Qld 4350 P: (07) 46390764 | F: (07) 46390750 E: nicholas.g.lawre...@tmr.qld.gov.au W: www.tmr.qld.gov.au * Tomorrow's Queensland: strong, green, smart, healthy and fair – www.towardQ2.qld.gov.au* P | Please consider the environment before printing this email *** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority. If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake. It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system). Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, Maritime Safety Queensland or endorsed organisations utilising the same infrastructure. *** ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdal_translate - ECW format and non-UTM projections
On 08/06/2011 17:44, Frank Warmerdam wrote: Armin, GDAL skips with a file data/ecw_cs.wkt that lists many ECW projections and datums. Perhaps it will be helpful. There are definately problems translating coordinate system into the ECW dictionary. A ticket for a particular case *might* lead to some improvement but it is an iffy process. Part of the problem in the thread mentioned by Jean-Claude is that when the coordinate system translation fails GDAL also gives up on read the georeferencing transform - or at least I think it used to work this way. I think there may have been some recent fixes in this regard. If not, a ticket might be appropriate. Best regards, Frank, Jean-Claude I made some tests and the tool from Jean-Claude was very useful also for this. I defined new projections by adding them to the "ecw_cs.wkt" file, taking the definitions GDAL returns using Python bindings for the OSR and the combination of ImportFromEPSG/ExportToWkt methods. If with gdalinfo a projection was not recognized for an ECW file I first tried to add it via the -co "PROJ=..." flag. This alone had no effect and gdalinfo still did not display any projection. Only when adding as well the correct datum via -co "DATUM=..." gdalinfo returned the full and correct projection definition. I tested this then not only with gdal tools directly but also with programs that use GDAL for image rendering, like QGIS and ArcGis 10. Also there, when adding the new projections to the "ecw_cs.wkt", the programs recognized the projection information correctly. So for me a first solution would be to define all missing projections and adding them to the "ecw_cs.wkt" file. In principal using the EPSG code as projection name would be sufficient (and easy to define), but this seems not to follow standard naming conventions so I will define short names for them. The projections I'm interested in are mainly national European projections and some pan-European ones (the ETRS types). I would be glad if these definitions could be added later on to the GDAL sources and go into future releases. Maybe sooner or later they might be included as well in the programs that use GDAL for image rendering. Then I need to define at least for my own use a CSV table with EPSG, projection, datum in order to identify the projection names and datums corresponding to a certain EPSG code. I will use it when creating ECW files or updating the ECW headers where I have the projection available as EPSG code. For me a great solution would be if GDAL could make use of such a table eg. in gdal_translate when it can identify the EPSG code or when it is passed via the "-a_srs" option, but I don't know if this is somehow feasible. Another helpful addition would be the possibility to set projection and datum in an ECW header via update functions of GDAL SWIG bindings, like it is currently possible eg. for GeoTiff files. Frank, if you think some of this might be feasible I will create ticket(s) for this. Cheers Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdal_translate - ECW format and non-UTM projections
Frank, Jean-Claude, thanks for pointing me to the ECW definitions. My main problem are currently the missing ETRS89/GRS80 definitions for several pan-European projections that are missing. Some national projections I need I could already identify and I will try them. If a projection is missing can I add it and GDAL will try to use it when reading and creating ECW files? Regarding the datums I guess I will be limited to the ones available in this list an understood by the ECW library, right? Is there a way to update an ECW file adding this type of information via GDAL API functions? I know there was a Windows GUI tool from ER Mapper some years a go that could modify some kind of ECW header, but I would need to run this on at least several 100's of files, so a manual approach is not feasible. Or do I need to re-create all ECW's and add the projection options? I will try to put together a good description for a ticket and then I will create one. One funny observation I made for the mentioned case where using the ECW file via MapServer WMS was not applying a datum shift: I made a quick test defining the ECW as only source of a virtual image using the standard projection definition for the VRT image. Using this VRT as layer data source in Mapserver the datum shift was applied... Best regards Armin On 08/06/2011 17:44, Frank Warmerdam wrote: On 11-06-08 11:30 AM, Armin Burger wrote: Jean-Claude I guess it's not only a problem of gdalinfo but of some parts of GDAL in general dealing with the ECW. I made some more tests and when I add the options -co DATUM=WGS84 -co PROJ=NUTM32 to the gdal_translate call then gdalinfo identifies the coordinates correctly afterwards, but then of course the whole projection is set to UTM... It seems that the default value entered for DATUM is "RAW" and that seems to cause some of the problems afterwards for GDAL. The problem for me is that I need to insert the correct datum and projection using the syntax of ECW, and I don't know if all available datums/projections used e.g. in Europe are supported, and which syntax to use for the available ones. I found on my Windows installation a file "PcskeyProjDatum.dat" with a few definitions of projection and datum used but this was very limited and covered mainly UTM and NAD. Most European projections and datums were missing in this file. Armin, GDAL skips with a file data/ecw_cs.wkt that lists many ECW projections and datums. Perhaps it will be helpful. There are definately problems translating coordinate system into the ECW dictionary. A ticket for a particular case *might* lead to some improvement but it is an iffy process. Part of the problem in the thread mentioned by Jean-Claude is that when the coordinate system translation fails GDAL also gives up on read the georeferencing transform - or at least I think it used to work this way. I think there may have been some recent fixes in this regard. If not, a ticket might be appropriate. Best regards, Original-Nachricht Datum: Wed, 08 Jun 2011 16:27:22 +0200 Von: Jean-Claude Repetto An: Armin Burger CC: gdal-dev@lists.osgeo.org Betreff: Re: [gdal-dev] gdal_translate - ECW format and non-UTM projections Hi, I think there is a bug in gdalinfo, I already noticed this kind of problem, for example <http://osgeo-org.1803224.n2.nabble.com/gdal-dev-corner-coords-in-tif-vs-ecw-td5608144.html> You should file a bug at<http://trac.osgeo.org/gdal/> . Jean-Claude ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdal_translate - ECW format and non-UTM projections
Jean-Claude I guess it's not only a problem of gdalinfo but of some parts of GDAL in general dealing with the ECW. I made some more tests and when I add the options -co DATUM=WGS84 -co PROJ=NUTM32 to the gdal_translate call then gdalinfo identifies the coordinates correctly afterwards, but then of course the whole projection is set to UTM... It seems that the default value entered for DATUM is "RAW" and that seems to cause some of the problems afterwards for GDAL. The problem for me is that I need to insert the correct datum and projection using the syntax of ECW, and I don't know if all available datums/projections used e.g. in Europe are supported, and which syntax to use for the available ones. I found on my Windows installation a file "PcskeyProjDatum.dat" with a few definitions of projection and datum used but this was very limited and covered mainly UTM and NAD. Most European projections and datums were missing in this file. Armin Original-Nachricht > Datum: Wed, 08 Jun 2011 16:27:22 +0200 > Von: Jean-Claude Repetto > An: Armin Burger > CC: gdal-dev@lists.osgeo.org > Betreff: Re: [gdal-dev] gdal_translate - ECW format and non-UTM projections > Hi, > > I think there is a bug in gdalinfo, I already noticed this kind of > problem, for example > <http://osgeo-org.1803224.n2.nabble.com/gdal-dev-corner-coords-in-tif-vs-ecw-td5608144.html> > > You should file a bug at <http://trac.osgeo.org/gdal/> . > > Jean-Claude -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] gdal_translate - ECW format and non-UTM projections
Hi all I am using "gdal_translate" to convert GeoTiff or Erdas images to ECW. This works fine when using standard projections like UTM-WGS84. For UTM I guess GDAL even writes the correct projection information into the ECW header. Several problems appear however if I convert images that are in one of the various European national projections. The way I'm trying to assure GDAL uses the projection information it understands is to call gdal_translate with a flag like -a_srs "EPSG:3912" This usually creates an XML file (*.ecw.aux.xml) that contains the projection information in an XML tag . But then some quite strange issues arise that I don't understand: - 1. "gdalinfo" displays the projection correctly, but the values for the corner coordinates are just the pixel values, starting with 0/0 for upper-left. This for the above listed projection 3912 gives eg. for an ECW file Size is 40258, 20016 Coordinate System is: PROJCS["MGI 1901 / Slovene National Grid", GEOGCS["MGI 1901", DATUM["MGI_1901", SPHEROID["Bessel 1841",6377397.155,299.1528128, AUTHORITY["EPSG","7004"]], AUTHORITY["EPSG","1031"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","3906"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",15], PARAMETER["scale_factor",0.], PARAMETER["false_easting",50], PARAMETER["false_northing",-500], AUTHORITY["EPSG","3912"], AXIS["Y",EAST], AXIS["X",NORTH]] Corner Coordinates: Upper Left (0.0,0.0) Lower Left (0.0,20016.0) Upper Right (40258.0,0.0) Lower Right (40258.0,20016.0) Center (20129.0,10008.0) In programs like ArcGIS 10 the coordinates are displayed correctly and the projection information is also right. But in QGIS I have just like with gdalinfo the pixel coordinates even though the projection information is correct (displayed in proj4 notation). Using such an image as layer in MapServer via WMS, defining the projection like "init=EPSG:3912" the image has more or less the correct projected coordinates, but datum shift parameters defined in the epsg file are not taken into account. If I instead of the ECW file use the GeoTIFF version instead, the returned imagettes from the WMS are correct with datum shift parameters applied. 2. If I remove the *.ecw.aux.xml file then gdalinfo does not find any more any projection information but at least displays correctly the corner coordinates. The same for QGIS. Also MapServer WMS then returns the images correctly with datum shift parameters applied. 3. An even more exotic problem appears eg. with an image in European ETM projection (the EU-specific UTM projection using ETRS89 datum, but in practical terms +/- identical to UTM WGS84). If I convert a GeoTiff with such a projection (EPSG 25832) PROJCS["UTM Zone 32, Northern Hemisphere", GEOGCS["ETRF 1989", DATUM["ETRF 1989", SPHEROID["WGS 84",6378137,298.2572235630016], TOWGS84[0,0,0,0,0,0,0]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",9], PARAMETER["scale_factor",0.9996], PARAMETER["false_easting",50], PARAMETER["false_northing",0], UNIT["Meter",1]] the resulting ECW shows the same problems mentioned above, just pixel coordinates displayed. But in addition, it does not display any pixels in some programs like ERDAS, ER Viewer or gives errors in ERDAS IWS It doesn't matter if I keep the aux.xml file or if I remove it. If I only re-write the projection information in the GeoTiff header to the equivalent UTM using GDAL Python bindings, convert the image again to ECW with gdal_translate, then the resulting ECW file is perfectly correct and displays fine in all programs I tested. --- The GDAL version was 1.8.0, using the ECW library 3.3, running on Linux. The same problems appear though on Windows using GDAL 1.7.3 from FWTools or a newer 1.8.0 compilation from www.gisinternals.com. Did others stumble over similar types of problem as well? Is there any possibility this could be sorted out somehow? If needed I can provide some test images that show the problems via FTP download. Best regards Armin -- NEU: FreePhone - kostenlos mobil telefonieren! Jetzt informieren: http://www.gmx.net/de/go/freephone ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Datum Shifts with OSR.CoordinateTransform
Steve In newer versions of the WKT definition there is sometimes a TOWGS84[...] tag included which contains the 7 shifting parameters, but I don't know if this is already a standard notation and if GDAL automatically takes this into account when found. What should normally work is to use the Proj4 notation which includes the tag +towgs84=. and then define the projection via the method ImportFromProj4() armin Original-Nachricht > Datum: Mon, 9 May 2011 20:07:23 -0700 (PDT) > Von: srweal > An: gdal-dev@lists.osgeo.org > Betreff: [gdal-dev] Datum Shifts with OSR.CoordinateTransform > Hi all, > > I'm using the C# wrapper and GDAL/OGR binaries from Tamas' repository > (http://vbkto.dyndns.org:1280/sdk/) and am trying to do coordinate > transformation that includes a datum shift on a geometry. > > I'm wanting to understand how I can control what shift parameters are used > when calling the MyGeometry.Transform(MyTransform) method. > > I have been creating the CoordinateTransform by using the basic > constructor > (i.e. new CoordinateTransform(SourceSR, TargetSR)) but this doesn't give > me > any control over the shift. Do I need to define the shift parameters on > the > actual SourceSR and TargetSR? If so, how is this best done. > > Currently I'm creating the SourceSR via the ImportFromWkt() method and > obtaining the WKT from spatialreference.org. > > Thanks for any advice, examples or links you can provide with more > information. > > Steve > > -- > View this message in context: > http://osgeo-org.1803224.n2.nabble.com/Datum-Shifts-with-OSR-CoordinateTransform-tp6346391p6346391.html > Sent from the GDAL - Dev mailing list archive at Nabble.com. > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL Java bindings, using gdal plugins under Windows
Even thanks for the reply. The path to the standard GDAL dll's was included in the system path definition when running the command from the Windows console. I guess also that otherwise the gdalinfo.java running on a GeoTiff file would have produced a similar error. armin On 02/05/2011 18:53, Even Rouault wrote: Le lundi 02 mai 2011 14:14:26, Armin Burger a écrit : Hello all I tried to use GDAL plugins via the Java bindings under Windows. The compiled GDAL libraries are the ones taken from http://www.gisinternals.com/sdk/, I used the "release-1400-gdal-1-8-0-mapserver-5-6-6" built. Compilation of the sample gdalinfo.java worked fine and I can apply it on a Tiff file. It does not work however for formats supplied via the plugins dll's. Following the description from the site I set the environment variable GDAL_DRIVER_PATH pointing to the directory with the plugin dll's. But when running the gdalinfo eg. on an ECW file I get the error ERROR 1: Can't load requested DLL: X:\gdal\gdalplugins\gdal_ECW_JP2ECW.dll 127: The specified procedure could not be found. You also need to make sure that your PATH points to the X:\gdal\bin directory (where libecwj2.dll should be) So it looks for the correct dll in the right place, but cannot use it. Is the usage of the plugins not supported via the Java bindings? It works fine when I use the gdalinfo.exe file. Thanks in advance for any hints Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDAL Java bindings, using gdal plugins under Windows
Hello all I tried to use GDAL plugins via the Java bindings under Windows. The compiled GDAL libraries are the ones taken from http://www.gisinternals.com/sdk/, I used the "release-1400-gdal-1-8-0-mapserver-5-6-6" built. Compilation of the sample gdalinfo.java worked fine and I can apply it on a Tiff file. It does not work however for formats supplied via the plugins dll's. Following the description from the site I set the environment variable GDAL_DRIVER_PATH pointing to the directory with the plugin dll's. But when running the gdalinfo eg. on an ECW file I get the error ERROR 1: Can't load requested DLL: X:\gdal\gdalplugins\gdal_ECW_JP2ECW.dll 127: The specified procedure could not be found. So it looks for the correct dll in the right place, but cannot use it. Is the usage of the plugins not supported via the Java bindings? It works fine when I use the gdalinfo.exe file. Thanks in advance for any hints Armin -- NEU: FreePhone - kostenlos mobil telefonieren und surfen! Jetzt informieren: http://www.gmx.net/de/go/freephone ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Calculate footprints of shapefiles
Chaitanya I can understand how the UnionCascade works in PostDIS since it's an aggregate function. I just don't understand how I should use it in OGR. The documentation has a single line UnionCascaded(self) -> Geometry So it returns a geometry from another geometry, but I would need a returned geometry from a *list* of input geometries, not from a single one. So I don't have a clue if this is possible in OGR. Another option could be the "Collect" function, but this seems to be only implemented in the C libraries, not in the Python bindings. Looks like geometry collections are not available in OGR for Python at all. Armin Original-Nachricht > Datum: Thu, 17 Mar 2011 10:16:06 +0530 > Von: Chaitanya kumar CH > An: Armin Burger > CC: gdal-dev@lists.osgeo.org, mariusjigm...@hotmail.com > Betreff: Re: [gdal-dev] Calculate footprints of shapefiles > Armin, > > Cascaded Union works pretty much like Union except that it is optimized to > work on more than two geometries. > Both OGR and PostGIS uses the GEOS library to perform this. > > Read this blog entry by Paul Ramsey: > http://blog.cleverelephant.ca/2009/01/must-faster-unions-in-postgis-14.html > > On Wed, Mar 16, 2011 at 3:41 PM, Armin Burger > wrote: > > > Chaitanya > > > > thanks for the hints. So a sort of Union operation seems to be > necessary. I > > need to check how fast that is with OGR on larger shapefiles (I want to > > store in PostGIS just the footprint, not the full geometry collection). > The > > ConvexHull could be a good alternative to a simplified boundary. > > > > I understand how the normal Union() works, it merges 2 geometries into a > > new one. But I don't get it what UnionCascade() is doing. It applies a > Union > > on a single geometry and returns the new geometry, what is it actually > > unioning/merging then? > > > > Regards, > > Armin -- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Calculate footprints of shapefiles
Chaitanya thanks for the hints. So a sort of Union operation seems to be necessary. I need to check how fast that is with OGR on larger shapefiles (I want to store in PostGIS just the footprint, not the full geometry collection). The ConvexHull could be a good alternative to a simplified boundary. I understand how the normal Union() works, it merges 2 geometries into a new one. But I don't get it what UnionCascade() is doing. It applies a Union on a single geometry and returns the new geometry, what is it actually unioning/merging then? Regards, Armin Original-Nachricht > Datum: Wed, 16 Mar 2011 10:06:57 +0530 > Von: Chaitanya kumar CH > An: armin.bur...@gmx.net > CC: Marius Jigmond , gdal-dev@lists.osgeo.org > Betreff: Re: [gdal-dev] Calculate footprints of shapefiles > Armin, > > This can be done in either OGR or PostGIS. > > For OGR, refer to the OGRGeometry class reference, especially the > ConvexHull(), Union() and UnionCascaded() functions. > For PostGIS, ST_Boundary() and ST_Union(). > -- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Calculate footprints of shapefiles
Marius, thanks for the suggestion. I don't know if shapely supports re-projection of geometries or has a simplify algorithm, which I will both need. There could be some other functions of GDAL/OGR that I might need in the future as well. So in principal I'd prefer to stick to GDAL. armin On 16/03/2011 01:35, Marius Jigmond wrote: Armin, Not sure if this would be straightforward in GDAL but you might want to consider a combination of GDAL + Shapely (http://gispython.org/shapely/docs/1.2/manual.html#cascading-unions). What you're trying to do is spatial analysis not directly implemented in GDAL. Shapely requires that you're working with Cartesian coordinates. The upside is that it has WKT/WKB support for direct loading into PostGIS. -marius On Wed, 2011-03-16 at 01:08 +0100, Armin Burger wrote: Hi all I would like to catalogue shapefiles scattered over lots of directories of the file system and store retrievable information of the shapefiles in a PostGIS layer. Extracting parameters like extent, projection, fields, etc works very fine with GDAL's Python bindings. But I would also like to store a sort of "footprint" of the whole shapefile as a geometry object since the extent is a bit coarse geographic representation of the shapefile. So far I have no better idea than eg. for polygon shapefiles looping through all features, applying a Union function on them. And at the end trying to use the Simplify method on the resulting polygon that will be used as the footprint. This is for sure not very efficient for larger shapefiles with lots of records. And for line and point shapefiles I still don't have a clue how their records could be represented by an enclosing polygon (maybe the Boundary functions does something like this...). Any ideas how this footprint generation could be achieved in a feasible way using GDAL/OGR Python? Cheers, Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Calculate footprints of shapefiles
Hi all I would like to catalogue shapefiles scattered over lots of directories of the file system and store retrievable information of the shapefiles in a PostGIS layer. Extracting parameters like extent, projection, fields, etc works very fine with GDAL's Python bindings. But I would also like to store a sort of "footprint" of the whole shapefile as a geometry object since the extent is a bit coarse geographic representation of the shapefile. So far I have no better idea than eg. for polygon shapefiles looping through all features, applying a Union function on them. And at the end trying to use the Simplify method on the resulting polygon that will be used as the footprint. This is for sure not very efficient for larger shapefiles with lots of records. And for line and point shapefiles I still don't have a clue how their records could be represented by an enclosing polygon (maybe the Boundary functions does something like this...). Any ideas how this footprint generation could be achieved in a feasible way using GDAL/OGR Python? Cheers, Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Windows builds and wildcard usage for gdaltindex & gdalbuildvrt
Tamas, Frank thanks a lot for the quick response to this issue (I just noticed today...). I will download the new Windows binaries. Armin On 24/02/2011 22:40, Tamas Szekeres wrote: Hi Armin, I've just added wildcard support to the builds at http://vbkto.dyndns.org/sdk/ according to Frank's suggestion. The corresponding binaries should be available with the next daily builds. Best regards, Tamas 2011/2/24 Armin Burger mailto:armin.bur...@gmx.net>> Hi all I wanted to use a more recent Windows build than the latest FW Tools which are a bit old now. I used the binaries from http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1310-gdal-1-8-0-mapserver-5-6-6.zip All works fine, just the usage of wildcards in tools like gdaltindex or gdalbuildvrt does not work any more. If I run eg gdaltindex tileindex.shp *.tif I just get the error that the file "*.tif" could not be found. Same happens for gdalbuildvrt.exe. So the wildcard resolution seems not to work any more (it still did work in the last FWTools 2.4.7). For gdaltindex I can use a "for" loop command because it adds every tif to the same shape, but I don't think this workaround works for the gdalbuildvrt. On Linux builds the wildcards usage still works fine with GDAL 1.8. Any ideas if this will be fixed for Windows? Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org> http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Windows builds and wildcard usage for gdaltindex & gdalbuildvrt
Hi all I wanted to use a more recent Windows build than the latest FW Tools which are a bit old now. I used the binaries from http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1310-gdal-1-8-0-mapserver-5-6-6.zip All works fine, just the usage of wildcards in tools like gdaltindex or gdalbuildvrt does not work any more. If I run eg gdaltindex tileindex.shp *.tif I just get the error that the file "*.tif" could not be found. Same happens for gdalbuildvrt.exe. So the wildcard resolution seems not to work any more (it still did work in the last FWTools 2.4.7). For gdaltindex I can use a "for" loop command because it adds every tif to the same shape, but I don't think this workaround works for the gdalbuildvrt. On Linux builds the wildcards usage still works fine with GDAL 1.8. Any ideas if this will be fixed for Windows? Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] ERDAS RRD format for overviews
Hi all I would like to create ERDAS Imagine compliant overview files for a larger satellite archive (mainly GeoTIFF files) in an automated way. The "gdaladdo" tool has the possibility of defining this with "--config USE_RRD YES". But this writes out not the typical ERDAS *.rrd files but adds the overviews into an *.aux file instead. This is not recognized properly by Erdas or ArcGIS. Just renaming the *.aux file to *.rrd does not fully work since Erdas complains and spits out some error messages (it opens the files after that, though). Are there any plans to implement the plain *.rrd output and leave the *.aux file for the typical usage of Erdas/ArcGIS for storing other parameters like histograms etc? I guess the format change between the plain *.rrd and the *.aux is very low, proably the aux file adds some additional header lines or something like that. Best regards Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] SPHEROID["unretrievable - using WGS84"...
Edi this is a typical issue with GDAL using less "common" projections specified with their EPSG code (it always works fine with UTM projections for example). The transformation is correct, but the GeoTiff header is a bit weird. This happens for me eg. when I use a pan-European projection like ETRS LAEA (EPSG 3035). The best way to avoid this is to specify the "-t_srs" not with the EPSG:2199 but with a file containing the WKT description of your projection. This seems to always write the complete WKT from your prj file into the GeoTiff header, so something like ... -t_srs path/to/epsg2199.prj ... where epsg2199.prj is a text file containing the WKT representation of your projection. WKT for 2199 should be: PROJCS["Albanian 1987 / Gauss Kruger zone 4 (deprecated)", GEOGCS["Albanian 1987", DATUM["Albanian_1987", SPHEROID["Krassowsky 1940",6378245,298.3, AUTHORITY["EPSG","7024"]], AUTHORITY["EPSG","6191"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4191"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",21], PARAMETER["scale_factor",1], PARAMETER["false_easting",450], PARAMETER["false_northing",0], AUTHORITY["EPSG","2199"]] An alternative is to use the full Proj4 definition string, like for your case ... -t_srs "+proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=450 +y_0=0 +ellps=krass +units=m +no_defs" ... You can also specify your datum by adding in addition a clause to this string like +datum=your-datum but this depends if GDAL recognizes the datum definition (I have not found a list of supported datums that one can use this way). An advantage using the latter approach is that you can also specify datum shift parameters with the additional clause like +towgs84=570.8,85.7,462.8,4.998,1.587,5.261,3.56 (values depend of course of your projection). I hope this helps Greetings Armin Original-Nachricht > Datum: Thu, 5 Aug 2010 05:42:40 -0700 (PDT) > Von: "Edi.Karadumi" > An: gdal-dev@lists.osgeo.org > Betreff: [gdal-dev] SPHEROID["unretrievable - using WGS84"... > > when i use gdalwarp -t_srs EPSG:2199 source.tif dest.tif i get an ok but > when > i use gdalinfo i get > the following result: > > Driver: GTiff/GeoTIFF > Files: dest.tif > dest.tfw > Size is 9375, 6250 > Coordinate System is: > PROJCS["unnamed", >GEOGCS[, >DATUM["unknown", > SPHEROIK["unretrievable - using > WGS84",6375137,298.257223563]], >PRIMEM["Greenwich",0], >UNIT["metre",1, >AUTHORITY["EPSG","9001"]], >AUTHORITY["EPSG","2199"]] > Origin = (375000.,4581000.00) > Pixel Size = (0.08,-0.080) > Medatata: > > .. > > is this normal? > what is the meaning of all those unknown, unretrievable? > > > > > -- > View this message in context: > http://osgeo-org.1803224.n2.nabble.com/SPHEROID-unretrievable-using-WGS84-tp5376320p5376320.html > Sent from the GDAL - Dev mailing list archive at Nabble.com. > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev -- GMX DSL SOMMER-SPECIAL: Surf & Phone Flat 16.000 für nur 19,99 ¿/mtl.!* http://portal.gmx.net/de/go/dsl ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] gdaladdo and cubic resampling
Hi all I have a question regarding the resampling modes for the gdaladdo utility. Since the overviews are used by Mapserver afterwards, I ran some tests with a map file referencing the Tiff image and the shp2img tool to output the results. I found the gauss resampling smoothening the image a bit too much. Therefore I wanted to try the "cubic" resample method using the GDAL version included in FWTools 2.4.7 (1.7.0b2), but to me it looks like that this does not have any effect compared to "nearest" resampling. Does anybody have a clue why? Command used was gdaladdo -ro -r cubic image.tif 2 4 8 16 32 64 Thanks in advance for any help Armin -- GMX DSL SOMMER-SPECIAL: Surf & Phone Flat 16.000 für nur 19,99 ¿/mtl.!* http://portal.gmx.net/de/go/dsl ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] VRT images and overview types
Hi everybody I noticed some strange behaviour of virtual images with the Erdas RRD overview type. I am using VRT images referencing Geotiff files and applying image stretching parameters (offset and scale) to improve the display in Mapserver applications. The VRT images are created with Python and GDAL bindings. This works fine if the overview is created e.g. with the -ro flag, the stretching parameters are then applied at all overview levels. If I create the overviews using the Erdas format (--config USE_RRD YES) then the vrt image is applying the stretching parameters only in case the full Tiff image is read. At a scale level where just the overview file is read the stretching parameters are not applied. This causes a confusing display effect by enabling the stretching at a certain scale. I'd prefer to continue with the RRD overviews since the images are also read via Desktop programs like Erdas or ArcMap that make use of the RRD .aux files. Is there a way to have RRD overviews and stretching parameters applied on them as well? Or would this be possible in future GDAL versions? Thanks for any clarification Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Writing KML with GDAL Python bindings
Even
thanks a lot for the help. Works perfectly.
armin
On 15/06/2009 22:07, Even Rouault wrote:
Yes, try something like :
ogr.GetDriverByName('KML').CreateDataSource('out.kml', options =
['NameField=foo'] )
Le Monday 15 June 2009 21:52:45 Armin Burger, vous avez écrit :
Hi everybody
I would like to write KML output with GDAL Python bindings based on a
database query. Everything works fine, I just have not found a way to
define the NameField that is used in Google Earth as name displayed in
the TOC. The ogr2ogr tool lets me define this using the flag "-dsco
NameField=...". Is there anything similar for GDAL Python?
Thanks in advance for any hints.
Armin
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Writing KML with GDAL Python bindings
Hi everybody I would like to write KML output with GDAL Python bindings based on a database query. Everything works fine, I just have not found a way to define the NameField that is used in Google Earth as name displayed in the TOC. The ogr2ogr tool lets me define this using the flag "-dsco NameField=...". Is there anything similar for GDAL Python? Thanks in advance for any hints. Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Status of Spatialite support
Even thanks again. Now it works. armin On 23/05/2009 11:32, Even Rouault wrote: Armin, The path behind the --with-spatialite flag must be the "prefix" of installation of spatialite, not the path to the include files. In your case, I'd try --with-spatialite=/usr/local. It will search a libspatialite.so in /usr/local/lib, and the following include files : /usr/local/include/spatialite.h and /usr/local/include/spatialite/sqlite3.h Note that I've done my tests with the libspatialite-linux-x86-2.3.0.tar.gz archive. FYI, the 'spatialite_init' is a function that is looked in the libspatialite.so. It is not a separate file. Best regards, Even Le Saturday 23 May 2009 11:03:19 Armin Burger, vous avez écrit : Even thanks for the reply. I downloaded the 1.7 dev version and used the flag --with-spatialite=/usr/local/include/spatialite But the configure output was checking for SpatiaLite... checking for spatialite_init in -lspatialite... yes disabled and then from the summary at the end: SpatiaLite support:no The installation of the spatialite libraries did no add a 'spatialite_init' file. All files installed were under /usr/local/include/ and /usr/local/lib. Is there anything to change for running configure? Best regards Armin On 22/05/2009 12:13, Even Rouault wrote: Armin, There is already some spatialite support in GDAL 1.6 that enables to read simple SpatiaLite geometries (point, linestring, polygon) without needing to link against libspatialite. In GDAL 1.7dev, write support has also been added, as well as creating SpatiaLite compatible databases. Yesterday, I've finally added the (optionnal) capability to link against libspatialite instead of libsqlite3, to benefit from the spatial enabled SQL functions that it provides (spatial queries + spatial indexing). So spatialite support should be mostly complete by now. You should be able to take the ogr/ogrsf_frmts/sqlite directory from trunk in GDAL 1.6, but if you want to benefit from linking against libspatialite, you'll also have to patch configure and GDALmakeopt.in. See http://trac.osgeo.org/gdal/changeset/17083 For the sake of simplicity, if you don't have to deal with sticking to GDAL 1.6 ABI, I'd advise you to grab a snapshot from GDAL 1.7dev. Best regards, Even Le Friday 22 May 2009 11:13:46, vous avez écrit : Hi everybody I would like to test the spatialite format that should be supported by gdal starting with v1.7. Is it already working in current dev version (I would mainly need the read functionality with spatial index support in interaction with Mapserver)? Would I need to take the full export of current trunk or could I just use gdal 1.6 and add the sqlite directory from trunk? Thanks for any help Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Status of Spatialite support
Even thanks for the reply. I downloaded the 1.7 dev version and used the flag --with-spatialite=/usr/local/include/spatialite But the configure output was checking for SpatiaLite... checking for spatialite_init in -lspatialite... yes disabled and then from the summary at the end: SpatiaLite support:no The installation of the spatialite libraries did no add a 'spatialite_init' file. All files installed were under /usr/local/include/ and /usr/local/lib. Is there anything to change for running configure? Best regards Armin On 22/05/2009 12:13, Even Rouault wrote: Armin, There is already some spatialite support in GDAL 1.6 that enables to read simple SpatiaLite geometries (point, linestring, polygon) without needing to link against libspatialite. In GDAL 1.7dev, write support has also been added, as well as creating SpatiaLite compatible databases. Yesterday, I've finally added the (optionnal) capability to link against libspatialite instead of libsqlite3, to benefit from the spatial enabled SQL functions that it provides (spatial queries + spatial indexing). So spatialite support should be mostly complete by now. You should be able to take the ogr/ogrsf_frmts/sqlite directory from trunk in GDAL 1.6, but if you want to benefit from linking against libspatialite, you'll also have to patch configure and GDALmakeopt.in. See http://trac.osgeo.org/gdal/changeset/17083 For the sake of simplicity, if you don't have to deal with sticking to GDAL 1.6 ABI, I'd advise you to grab a snapshot from GDAL 1.7dev. Best regards, Even Le Friday 22 May 2009 11:13:46, vous avez écrit : Hi everybody I would like to test the spatialite format that should be supported by gdal starting with v1.7. Is it already working in current dev version (I would mainly need the read functionality with spatial index support in interaction with Mapserver)? Would I need to take the full export of current trunk or could I just use gdal 1.6 and add the sqlite directory from trunk? Thanks for any help Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Status of Spatialite support
Hi everybody I would like to test the spatialite format that should be supported by gdal starting with v1.7. Is it already working in current dev version (I would mainly need the read functionality with spatial index support in interaction with Mapserver)? Would I need to take the full export of current trunk or could I just use gdal 1.6 and add the sqlite directory from trunk? Thanks for any help Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] directory for ECW tmp files
Hi everybody does anybody know if the tools like gdal_translate have a config parameter that allows to specify the directory used for the tmp files when creating ECW files? I made some tests on Linux and no environment vars like TMP, TEMP or TMPDIR were taken into account when creating the ECW files. The temporary files were always written to the same dir as the final ECW file. Greetings Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question on image formats vs performance and quality
Steve ECW needs Byte data *per band*, so you could convert 3 Band RGB (every band in byte) data without problems into ECW. Performance is another issue. As long as you do not have problems with disk space I would recommend continuing to use uncompressed Tiff with overviews. In combination with Mapserver/Gdal the best solution regarding performance. In our tests most of the time ~ 1.5 to 2 times faster than ECW, and becoming worse for ECW for more concurrent users. And I think the current ECW license conditions also have issues for serving this format. Other formats like Erdas Imagine are not faster than Tiff and with the BigTiff library also Tiff files can become > 4GB. Reagrds, Armin On 22/03/2009 05:49, Stephen Woodbridge wrote: Hi all, I would be interest in you thoughts on comparing MrSID, ECW, and Tiff images from the perspective of image quality and performance. The scenario is I have imagery in MrSID. I typically decompress that into Tiled Tiff uncompressed imagery with overviews. I server it via mapserver and it has reasonable performance. I have not considered serving it from the MrSID files because I don't have a license to compress the files and I would need to reproject them. Also I'm guessing it in not a good idea to uncompress and compress them again as it is lossy and the image degrades. I have never worked with ECW, but I'm concerned about: 1) ECW is 8-bit data only and the source data is 24 bit RGB data 2) how does it compare performance wise to say the tiff images when rendering? Are there other formats that would be good to consider? What are their characteristics? Thanks, -Steve ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL and numpy, masking arrays
I guess I found it, looks like the function filled(c, 255) does this. Armin On 12/11/2008 21:54, Armin Burger wrote: Chris thanks a lot for the quick and very helpful explanations. Is there a possibility to reset the masked values '--' after the calculation to a special value? Best regards Armin On 12/11/2008 21:27, Christopher Barker wrote: Armin Burger wrote: maybe someone on this list has some experience using numpy in combination with GDAL/Python and could give me some advice. For further questions about numpy, the numpy list is very helpful. difference can be easily calculated after reading both images into an array and substract one from the other. The problem is that both images can contain clouds or snow that have predefined pixel values (252,253). Does anybody know if this is possible and how to perform it? yep. Looking through the numpy docs I was not able to identify required methods or functions for this. There is something like 'masked arrays' but I have not understood if this could be used for my purpose. This is exactly the kind of thing masked arrays are for. Yu can create a masked array out of your data with something like: >>> a array([ 1, 2, 3, 4, 5, 6, 3, 4, 67, 4, 3, 5, 6, 7]) #a regular array >>> import numpy.ma as ma # create a masked array with the mask set at all elements with a value of 3: >>> a = ma.masked_values(a, 3) >>> a masked_array(data = [1 2 -- 4 5 6 -- 4 67 4 -- 5 6 7], mask = [False False True False False False True False False False True False False False], fill_value=3) #another one: >>> b = np.array((1,3,4,2,4,7,4,5,23,5,7,3,8,5)) >>> b = ma.masked_values(b, 3) >>> b masked_array(data = [1 -- 4 2 4 7 4 5 23 5 7 -- 8 5], mask = [False True False False False False False False False False False True False False], fill_value=3) add them together: >>> c = a+b >>> c masked_array(data = [2 -- -- 6 9 13 -- 9 90 9 -- -- 14 12], mask = [False True True False False False True False False False True True False False], fill_value=3) If your values are Floating Point, then Another option would be to replace all the "cloud" values with NaN: >>> a = np.array((1,2,3,4,5,6,3,4,67,4,3,5,6,7), dtype=np.float) >>> a[a==3] = np.nan >>> a array([ 1., 2., NaN, 4., 5., 6., NaN, 4., 67., 4., NaN, 5., 6., 7.]) >>> b = np.array((1,3,4,2,4,7,4,5,23,5,7,3,8,5), dtype=np.float) >>> b[b==3] = np.nan >>> b array([ 1., NaN, 4., 2., 4., 7., 4., 5., 23., 5., 7., NaN, 8., 5.]) >>> a+b array([ 2., NaN, NaN, 6., 9., 13., NaN, 9., 90., 9., NaN, NaN, 14., 12.]) -Chris ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDAL and numpy, masking arrays
Chris thanks a lot for the quick and very helpful explanations. Is there a possibility to reset the masked values '--' after the calculation to a special value? Best regards Armin On 12/11/2008 21:27, Christopher Barker wrote: Armin Burger wrote: maybe someone on this list has some experience using numpy in combination with GDAL/Python and could give me some advice. For further questions about numpy, the numpy list is very helpful. difference can be easily calculated after reading both images into an array and substract one from the other. The problem is that both images can contain clouds or snow that have predefined pixel values (252,253). Does anybody know if this is possible and how to perform it? yep. Looking through the numpy docs I was not able to identify required methods or functions for this. There is something like 'masked arrays' but I have not understood if this could be used for my purpose. This is exactly the kind of thing masked arrays are for. Yu can create a masked array out of your data with something like: >>> a array([ 1, 2, 3, 4, 5, 6, 3, 4, 67, 4, 3, 5, 6, 7]) #a regular array >>> import numpy.ma as ma # create a masked array with the mask set at all elements with a value of 3: >>> a = ma.masked_values(a, 3) >>> a masked_array(data = [1 2 -- 4 5 6 -- 4 67 4 -- 5 6 7], mask = [False False True False False False True False False False True False False False], fill_value=3) #another one: >>> b = np.array((1,3,4,2,4,7,4,5,23,5,7,3,8,5)) >>> b = ma.masked_values(b, 3) >>> b masked_array(data = [1 -- 4 2 4 7 4 5 23 5 7 -- 8 5], mask = [False True False False False False False False False False False True False False], fill_value=3) add them together: >>> c = a+b >>> c masked_array(data = [2 -- -- 6 9 13 -- 9 90 9 -- -- 14 12], mask = [False True True False False False True False False False True True False False], fill_value=3) If your values are Floating Point, then Another option would be to replace all the "cloud" values with NaN: >>> a = np.array((1,2,3,4,5,6,3,4,67,4,3,5,6,7), dtype=np.float) >>> a[a==3] = np.nan >>> a array([ 1., 2., NaN, 4., 5., 6., NaN, 4., 67., 4., NaN, 5., 6., 7.]) >>> b = np.array((1,3,4,2,4,7,4,5,23,5,7,3,8,5), dtype=np.float) >>> b[b==3] = np.nan >>> b array([ 1., NaN, 4., 2., 4., 7., 4., 5., 23., 5., 7., NaN, 8., 5.]) >>> a+b array([ 2., NaN, NaN, 6., 9., 13., NaN, 9., 90., 9., NaN, NaN, 14., 12.]) -Chris ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDAL and numpy, masking arrays
Hi everybody maybe someone on this list has some experience using numpy in combination with GDAL/Python and could give me some advice. I wanted to try to use GDAL/Python/numpy to caculate the difference between 2 images. The images are vegetation indexes for different dates. The direct difference can be easily calculated after reading both images into an array and substract one from the other. The problem is that both images can contain clouds or snow that have predefined pixel values (252,253). And I of course would like to exclude these parts in both images from the calculation because they would cause wrong results. So I wanted to mask the non wanted areas. Does anybody know if this is possible and how to perform it? Looking through the numpy docs I was not able to identify required methods or functions for this. There is something like 'masked arrays' but I have not understood if this could be used for my purpose. Thanks in advance for any help. Best regards, Armin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
