Re: [gdal-dev] Question: is there a way to set the TileMatrix value (aka Tile Zoom Level) for GDAL_WMTS?
The TileLevel option is not available in the WMTS driver (as not mentionned in http://gdal.org/frmt_wmts.html) Looking at, $ gdalinfo test.xml Driver: WMTS/OGC Web Mab Tile Service Files: test.xml Size is 51220, 32712 Coordinate System is: PROJCS["WGS 84 / Pseudo-Mercator", GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]], PROJECTION["Mercator_1SP"], PARAMETER["central_meridian",0], PARAMETER["scale_factor",1], PARAMETER["false_easting",0], PARAMETER["false_northing",0], UNIT["metre",1, AUTHORITY["EPSG","9001"]], AXIS["X",EAST], AXIS["Y",NORTH], EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"], AUTHORITY["EPSG","3857"]] Origin = (-9662904.575207639485598,4329581.393627353012562) Pixel Size = (0.298582141647579,-0.298582141647579) Metadata: TITLE=USGSShadedReliefOnly Image Structure Metadata: INTERLEAVE=PIXEL Corner Coordinates: Upper Left (-9662904.575, 4329581.394) ( 86d48'12.06"W, 36d12'36.67"N) Lower Left (-9662904.575, 4319814.175) ( 86d48'12.06"W, 36d 8'21.70"N) Upper Right (-9647611.198, 4329581.394) ( 86d39'57.48"W, 36d12'36.67"N) Lower Right (-9647611.198, 4319814.175) ( 86d39'57.48"W, 36d 8'21.70"N) Center (-9655257.887, 4324697.784) ( 86d44' 4.77"W, 36d10'29.21"N) Band 1 Block=128x128 Type=Byte, ColorInterp=Red Overviews: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Band 2 Block=128x128 Type=Byte, ColorInterp=Green Overviews: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Band 3 Block=128x128 Type=Byte, ColorInterp=Blue Overviews: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 Band 4 Block=128x128 Type=Byte, ColorInterp=Alpha Overviews: 25610x16356, 12805x8178, 6402x4089, 3201x2044, 1601x1022, 800x511, 400x256, 200x128 One can see that the resolution of the full resolution dataset is 0.298582141647579 m/pixel. This indeed corresponds to zoom level = >>> math.log((2 * 20037508.3427892 / 256) / 0.298582141647579)/math.log(2) 19.044158 So if you want to query at zoom level 15, you need to select the 5th overview (the one of size 1601x1022) : $ gdal_translate -outsize 1601 1022 --debug HTTP test.xml out.tif [...] HTTP: Requesting [1/7] https://basemap.nationalmap.gov/arcgis/rest/services/ USGSShadedReliefOnly/MapServer/WMTS/tile/1.0.0/USGSShadedReliefOnly/default/ default028mm/14/6424/4241.jpg HTTP: Requesting [2/7] https://basemap.nationalmap.gov/arcgis/rest/services/ USGSShadedReliefOnly/MapServer/WMTS/tile/1.0.0/USGSShadedReliefOnly/default/ default028mm/14/6424/4242.jpg [...] Or equivalently use the OVERVIEW_LEVEL open option ( see http://www.gdal.org/classGDALDataset.html#a9cb8585d0b3c16726b08e25bcc94274a ) $ gdal_translate -oo OVERVIEW_LEVEL=4 --debug HTTP test.xml out.tif Or equivalently, download http://basemap.nationalmap.gov/arcgis/rest/services/ USGSShadedReliefOnly/MapServer/WMTS/1.0.0/WMTSCapabilities.xml locally. Edit it to remove all TileMatrix > 14. And refer to this file as the . $ gdalinfo test.xml Driver: WMTS/OGC Web Mab Tile Service Files: test.xml Size is 1601, 1023 Coordinate System is: PROJCS["WGS 84 / Pseudo-Mercator", GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]], PROJECTION["Mercator_1SP"], PARAMETER["central_meridian",0], PARAMETER["scale_factor",1], PARAMETER["false_easting",0], PARAMETER["false_northing",0], UNIT["metre",1, AUTHORITY["EPSG","9001"]], AXIS["X",EAST], AXIS["Y",NORTH], EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"], AUTHORITY["EPSG","3857"]] Origin = (-9662911.140854565426707,4329584.374663999304175) Pixel Size = (9.554628535632952,-9.554628535632952) Metadata:
[gdal-dev] Question: is there a way to set the TileMatrix value (aka Tile Zoom Level) for GDAL_WMTS?
Question: is there a way to set the TileMatrix value (aka Tile Zoom Level) for GDAL_WMTS? WMTS Tile Retrieval issue: WMTS Is retrieving at the wrong TileMatrix value. WMTS retrieves tiles at the maximum tile matrix (tile level or zoom level). No data is being returned. Server: http://basemap.nationalmap.gov/arcgis/rest/services/USGSShadedReliefOnly/MapServer/WMTS? Capabilities: http://basemap.nationalmap.gov/arcgis/rest/services/USGSShadedReliefOnly/MapServer/WMTS?SERVICE=WMTS=GetCapabilities 1. Tiles are always requested at the maximum tile zoom level. Using the max tile zoom level can result in many tiles returned. When zoomed out, a smaller tile zoom level will give enough data and fewer tiles. It really needs to respond to the TileLevel like Google Maps running the TMS driver. -- Possible Problem in WMTS Code: -- The TileMatrix hard coded maximum tile matrix problem may be here: GDAL 2.1.2 File: wmtsdataset.cpp, Function: WMTSDataset::Open, Line 1587 Loop starts at Maximum matrix id value. The loop breaks at end of processing the maximum value. At that point the tile matrix (tlie zoom level) is 19 (maximum). -- Here's the first tile request built by GDAL 2.1.2: -- AskMiniDriverForBlock - First Tile Request: url = "http://basemap.nationalmap.gov/arcgis/rest/services/USGSShadedReliefOnly/MapServer/WMTS/tile/1.0.0/USGSShadedReliefOnly/default/default028mm/19/205501/135727.jpg; For TileMatrix 19 the tile coordinates look right. But the TileMatrix value (tile zoom level) is set to the maximum for the TileMatrixSet. TileMatrix was requested at 14 based on the standard ESRI map scale computation. Using the WMS/TMS driver, Google Maps honors the TileLevel request. I added the TileMatrix and/or TileLevel parameters in the GDAL_WMTS request below. It did not make any difference. -- Here's the WMTS XML Request file: -- http://basemap.nationalmap.gov/arcgis/rest/services/USGSShadedReliefOnly/MapServer/WMTS/1.0.0/WMTSCapabilities.xml USGSShadedReliefOnly default028mm image/jpg -9662904.298813 4329581.328124 -9647611.198187 4319814.305876 14 // DOES NOT WORK, IGNORED 14 // DOES NOT WORK, IGNORED 4 true 404 true -- View this message in context: http://osgeo-org.1560.x6.nabble.com/Question-is-there-a-way-to-set-the-TileMatrix-value-aka-Tile-Zoom-Level-for-GDAL-WMTS-tp5297011.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Nodata values in RGB orthoimages (GeoTiff)
On mardi 22 novembre 2016 09:08:02 CET Neumann, Andreas wrote: > Hi, > > Is there a way to store nodata values in GeoTiffs that don't match an > RGB value? I used either white or black as nodata values, but in some > rare cases these black or white pixel values do appear as valid data in > the image. > > How can I properly store nodata values in RGB Orthoimages in GeoTiffs > where the borders aren't purely rectangular? > > Would I have to use a hard mask for regions where there is no data, or > is there a trick with an alpha band? > There are several potential issues: * by default, nodata is dealt band-per-band. So if you have nodata at 255, but a RGB triplet is (255,0,0), then it will be considered as nodata for the red band, but valid for the green and blue bands. To avoid that you can set the "NODATA_VALUES=255 255 55" metadata item to indicate that the 3 components must match at the same time. For example with: gdal_edit.py foo.tif -mo "NODATA_VALUES=255 255 55" -unsetnodata Then gdaladdo should properly take that into account * If you indeed have valid pixels at (255,255,255) in the middle of the raster, then the above is not sufficient, and you need to create a dedicated alpha channel or mask band that gdaladdo will honour as well. You can do that with nearblack -white -setalpha/-setmask Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] slow netCDF read times
Thank you Julien, Your comment is very helpful and I think the issue that you're pointing out is very much related to my problem. I did play with setting different chunking patterns and I saw that chunking per line solved pretty much the performance issue, but I didn't understand why. Also, ticket 5291 talks about BottomUp NetCDF causing grief to GDAL. I've checked and my file is BottomUp. I've flipped it by doing: gdal_translate -of netcdf -co "WRITE_BOTTOMUP=NO" chirps-v2.0.1981.dekads.classic.nc out.nc And transformed it again into NetCDF4 with its original chunking pattern and deflate level: nccopy -7 -c time/4,lat/250,lon/900 out.nc out2.nc In this case GDAL reads the file almost as fast as the native netcdf library does. From now on I'll make sure I don't produce netCDF files which are bottom up. Does anyone know where this bottom up convention comes from in netcdf files or why is there? Cheers, Pablo On Tue, Nov 22, 2016 at 8:07 PM, Julien Demariawrote: > Hi, > > > > Maybe this is a problem with your NetCDF internal chunks cache too small > and related to this ticket: https://trac.osgeo.org/gdal/ticket/5291 > > You can change this per-variable cache using this C function: > http://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html# > ga2788cbfc6880ec70c304292af2bc7546 > > Else a workaround may be to rechunk your data using nccopy to have chunks > of the same size than your reading window. > > Another solution is to recompile your NetCDF library to set more chunks > cache. > > > > Regards, > > > > Julien > > > > *De :* gdal-dev [mailto:gdal-dev-boun...@lists.osgeo.org] *De la part de* > Pablo Rozas Larraondo > *Envoyé :* mardi 22 novembre 2016 08:53 > *À :* gdal-dev@lists.osgeo.org > *Objet :* [gdal-dev] slow netCDF read times > > > > Hello, > > > I've come across some NetCDF4 files where GDAL is taking a surprisingly > long time to read data from them. For example this is an example public > file containing precipitation data: > > > > ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0/ > global_dekad/netcdf/chirps-v2.0.2015.dekads.nc > > > > If I use GDAL to read a small top left block (500x500) from one of its > time bands, it takes approximately 1 minute on my computer. Source code is > available here: > > > > https://gist.github.com/monkeybutter/769a24bcf87682171eb87ac05c9347c5 > > > > The equivalent operation is completed in less than a second using the > NetCDF library and even reading the whole file takes around 6 seconds with > the same library. > > > > I've tried to profile the GDAL program to get more insight and understand > what's causing the overhead with not much success. All I know is that the > deflate function is using 96% of the resources. I also guess that the way > this file is chunked has something to do with its performance. Can anyone > suggest any idea for better understanding what's happening here? > > > > Thank you for your help, > > Pablo > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Antialiasing problem with gdaladdo at edges/nodata sections
Andreas, I assume you run a recent enough version of GDAL ? As the image uses JPEG compression, one explanation would be that the transition betweeen data and nodata is not "clean" (which is a feature of JPEG compresssion). Therefore you would need to first translate your image to a uncompressed one (or one with a non -JPEG compression), and run nearblack - white on it so as to clean those edges. Even > Hi, > > I have Orthoimages where the boundaries aren't rectangular, but they > have corners and sometimes even diagonal edges. > > I noticed, that after creating pyramids/overview with gdaladdo, that I > get antialiasing/rendering artefacts that aren't present in the original > image at full resolution. See attached image that shows the problem. I > used a pink background to emphasize the problem. > > My nodata values are set to white (rgb 255,255,255) in the original > image. > > Here is the gdaladdo command that I used: > > gdaladdo -r cubic --config BIGTIFF_OVERVIEW YES --config > COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY 85 --config > PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL > dop2002_lv95.tif 2 4 8 16 32 64 128 256 512 1024 > > This is the output of gdalinfo on the original image: > > --- > > Driver: GTiff/GeoTIFF > Files: dop2002_lv95 - Kopie.tif > Size is 24, 19 > Coordinate System is: > PROJCS["CH1903+ / LV95", > GEOGCS["CH1903+", > DATUM["CH1903+", > SPHEROID["Bessel 1841",6377397.155,299.1528128, > AUTHORITY["EPSG","7004"]], > TOWGS84[674.374,15.056,405.346,0,0,0,0], > AUTHORITY["EPSG","6150"]], > PRIMEM["Greenwich",0, > AUTHORITY["EPSG","8901"]], > UNIT["degree",0.0174532925199433, > AUTHORITY["EPSG","9122"]], > AUTHORITY["EPSG","4150"]], > PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"], > PARAMETER["latitude_of_center",46.952406], > PARAMETER["longitude_of_center",7.4395833], > PARAMETER["azimuth",90], > PARAMETER["rectified_grid_angle",90], > PARAMETER["scale_factor",1], > PARAMETER["false_easting",260], > PARAMETER["false_northing",120], > UNIT["metre",1, > AUTHORITY["EPSG","9001"]], > AXIS["Easting",EAST], > AXIS["Northing",NORTH], > AUTHORITY["EPSG","2056"]] > Origin = (672000.000,234000.000) > Pixel Size = (0.100,-0.100) > Metadata: > AREA_OR_POINT=Area > TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch) > TIFFTAG_XRESOLUTION=1 > TIFFTAG_YRESOLUTION=1 > Image Structure Metadata: > COMPRESSION=YCbCr JPEG > INTERLEAVE=PIXEL > SOURCE_COLOR_SPACE=YCbCr > Corner Coordinates: > Upper Left ( 672000.000, 234000.000) ( 13d51' 5.78"W, 35d55'48.10"N) > Lower Left ( 672000.000, 215000.000) ( 13d47'45.10"W, 35d46' 1.03"N) > Upper Right ( 696000.000, 234000.000) ( 13d35'53.15"W, 35d59'13.56"N) > Lower Right ( 696000.000, 215000.000) ( 13d32'34.61"W, 35d49'25.99"N) > Center ( 684000.000, 224500.000) ( 13d41'49.73"W, 35d52'37.44"N) > Band 1 Block=256x256 Type=Byte, ColorInterp=Red > NoData Value=255 > Band 2 Block=256x256 Type=Byte, ColorInterp=Green > NoData Value=255 > Band 3 Block=256x256 Type=Byte, ColorInterp=Blue > NoData Value=255 > > > --- > > > Is there a way to get rid of these artefacts at the nodata borders? > > Thank you and greetings, > > Andreas -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] CAD (DWG) Driver
Hi Nigel, Thank you for information. I think enhancing OpenCAD library with support of additional DWG versions is right way. I'll study your code. Best regards, Dmitry 21.11.16 13:10, Nigel Westbury пишет: The DWG driver contribution is much appreciated. It currently supports older versions of files up to R1015 (AutoCAD 2000). The DWG format was relatively straight forward up to that version. In 2004 the format became rather more complex, including simple xor encryption and LZ77 compression of most of the records. In 2007 the format became considerably more complex again, including Reed-Solomon encoding. In 2010 the format became simpler again, going back to a format more similar to the 2004 format. Our experience here at 1Spatial is that customers are on versions 2010 or 2013. I have not received a DWG file from an earlier version from a customer. Therefore we have implemented a library for reading versions 2010 and later (https://github.com/westbury/dwg-lib). This has been implemented in Java and could readily be ported to C or otherwise compiled to a DLL, though as it is currently under active development this would need some thought to avoid maintaining two copies. Alternatively we could leave it in Java and call it much as the GDAL GeoMedia driver does for Jackcess. We have put the code under the MIT licence so that it would be available to GDAL. The work was implemented by reference to the Open Design Specification for .dwg files Version 5.3. We do plan on limited writing too, but probably updates to certain elements only, no support for outputting a complete DWG file from scratch. Nigel Westbury -Original Message- From: gdal-dev [mailto:gdal-dev-boun...@lists.osgeo.org] On Behalf Of gdal-dev-requ...@lists.osgeo.org Sent: 19 November 2016 19:49 To: gdal-dev@lists.osgeo.org Subject: gdal-dev Digest, Vol 150, Issue 33 Send gdal-dev mailing list submissions to gdal-dev@lists.osgeo.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.osgeo.org/mailman/listinfo/gdal-dev or, via email, send a message with subject or body 'help' to gdal-dev-requ...@lists.osgeo.org You can reach the person managing the list at gdal-dev-ow...@lists.osgeo.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gdal-dev digest..." Today's Topics: 1. Re: CAD (DWG) Driver (Even Rouault) 2. Re: CAD (DWG) Driver (Kurt Schwehr) 3. Re: CAD (DWG) Driver (Even Rouault) 4. Re: CAD (DWG) Driver (Dmitry Baryshnikov) 5. Re: identifying the fields from the .osm file (Djordje Spasic) -- Message: 1 Date: Sat, 19 Nov 2016 17:49:25 +0100 From: Even RouaultTo: Dmitry Baryshnikov , schw...@gmail.com, gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] CAD (DWG) Driver Message-ID: <7873061.FYxrsrM0Qa@even-i700> Content-Type: text/plain; charset="us-ascii" On samedi 19 novembre 2016 19:34:02 CET Dmitry Baryshnikov wrote: Hi Even, The big_endian test now is ok. But osx crash at the exiting of python after all test finished successfully. This is error message: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument ./gdal/ci/travis/osx/script.sh: line 13: 56363 Abort trap: 6 python run_all.py I find out that this may be the result of resources free order: https://github.com/dmlc/mxnet/issues/309 https://github.com/google/benchmark/issues/52 No idea why this occurred then C++11 used. I'll try to use osx CI test disabling the CAD driver to exclude this possible issue. Hum, I think this might be indeed an issue with resource free order between the GDALDestroy() destructor function (in gcore/gdaldllmain.cpp) and the static C++11 mutex Kurt introduced in https://github.com/OSGeo/gdal/commit/a9b947d6850d496f09e668f4cd148826d45d9fa9 If this mutex gets destroyed before GDALDestroy() is called, then things might indeed crash. I think we should remove this static C++11 mutex. Kurt ? (Another option would be to remove the library destructor function. I've the feeling they cause more harm than good.) -- Spatialys - Geospatial professional services http://www.spatialys.com -- Message: 2 Date: Sat, 19 Nov 2016 10:06:26 -0800 From: Kurt Schwehr To: Even Rouault Cc: Dmitry Baryshnikov , gdal dev Subject: Re: [gdal-dev] CAD (DWG) Driver Message-ID:
Re: [gdal-dev] slow netCDF read times
Hi, Maybe this is a problem with your NetCDF internal chunks cache too small and related to this ticket: https://trac.osgeo.org/gdal/ticket/5291 You can change this per-variable cache using this C function: http://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#ga2788cbfc6880ec70c304292af2bc7546 Else a workaround may be to rechunk your data using nccopy to have chunks of the same size than your reading window. Another solution is to recompile your NetCDF library to set more chunks cache. Regards, Julien De : gdal-dev [mailto:gdal-dev-boun...@lists.osgeo.org] De la part de Pablo Rozas Larraondo Envoyé : mardi 22 novembre 2016 08:53 À : gdal-dev@lists.osgeo.org Objet : [gdal-dev] slow netCDF read times Hello, I've come across some NetCDF4 files where GDAL is taking a surprisingly long time to read data from them. For example this is an example public file containing precipitation data: ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0/global_dekad/netcdf/chirps-v2.0.2015.dekads.nc If I use GDAL to read a small top left block (500x500) from one of its time bands, it takes approximately 1 minute on my computer. Source code is available here: https://gist.github.com/monkeybutter/769a24bcf87682171eb87ac05c9347c5 The equivalent operation is completed in less than a second using the NetCDF library and even reading the whole file takes around 6 seconds with the same library. I've tried to profile the GDAL program to get more insight and understand what's causing the overhead with not much success. All I know is that the deflate function is using 96% of the resources. I also guess that the way this file is chunked has something to do with its performance. Can anyone suggest any idea for better understanding what's happening here? Thank you for your help, Pablo ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Nodata values in RGB orthoimages (GeoTiff)
Hi, Is there a way to store nodata values in GeoTiffs that don't match an RGB value? I used either white or black as nodata values, but in some rare cases these black or white pixel values do appear as valid data in the image. How can I properly store nodata values in RGB Orthoimages in GeoTiffs where the borders aren't purely rectangular? Would I have to use a hard mask for regions where there is no data, or is there a trick with an alpha band? Thank you for any hints and greetings, Andreas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev