Re: [gdal-dev] Passing open options along dataset name in a string ?
Even, On Mon, Nov 2, 2020 at 1:16 PM Even Rouault wrote: > Sean, > > > We already have a way of passing "open" options for vsicurl: > > > https://gdal.org/user/virtual_file_systems.html#vsicurl-http-https-ftp-files > > -random-access. What about reusing that conceptual framework and syntax? > > > > For example: > > > > "foo.csv?AUTODETECT_TYPE=YES&KEEP_GEOM_COLUMNS=NO" > > I actually considered that, but realized that things would get messy if > you want > to use that vsicurl syntax and open options... > > You would then have strings like > > /vsicurl?max_retry=5&url= > http://example.com/foo.csv&AUTODETECT_TYPE=YES&KEEP_GEOM_COLUMNS=NO > > and the GDALOpen() logic would have to figure out whas is the /vsicurl > part and the open option part. > > Or we would have to URL-escape the "/vsicurl?max_retry=5&url= > http://example.com/foo.csv"; part > to avoid using '?' and '&', like: > > /vsicurl%3Fmax_retry=5%26url= > http://example.com/foo.csv?AUTODETECT_TYPE=YES&KEEP_GEOM_COLUMNS=NO > > > Another issue is we have connection strings like "WFS: > http://example.com/wfs?SERVICE=WFS&VERSION=2.0.0"; (or actually > just the "/vsicurl?max_retry=5&url=http://example.com/foo.csv"; string > mentioned above). > GDALOpen() would then mis-interpret this as dataset name = "WFS: > http://example.com/wfs"; > with open options SERVICE=WFS and VERSION=2.0.0 > I see. I wish our data formats were more standard and less slippery and didn't need these open options. But it's true that some files are very different without the proper combination of opening options and there's a benefit to helping applications use the right combination. I'm not a fan of the mix of JSON and not-JSON elements in the syntax you proposed. I think a good solution for naming datasets and including all the driver options and vsi options looks more like a URN [1] and I think we should write a GDAL RFC to standardize it. I also think that we should get some people outside of GDAL involved. Like folks from the Dask community, who might share some lessons learned from writing fsspec [2]. The URN or GDN version might look something like the thing below, using ?+ and ?= [3] to identify vsi and driver option sections gdn:curl:csv: example.com/foo.csv?a=1&b=2?+max_retry=5?=autodetect_type=yes&keep_geom_columns=no Bringing a little more order to how we name and address datasets was on my todo list at the start of the year, but then 2020 went into a spiral. I don't think rasterio's "zip+s3" etc approach is the best. We should start from scratch and come up with something excellent and expressive and broadly supported in GDAL, QGIS, rasterio, GeoPandas, GeoTrellis etc. [1] https://en.wikipedia.org/wiki/Uniform_Resource_Name [2] https://filesystem-spec.readthedocs.io/en/latest/index.html [3] https://tools.ietf.org/html/rfc8141#section-2.3 -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Reading a VRT file & overview usage
Even,
Thank you for your response.
Since you can't reproduce this with your test it makes me think there is
something about the input data. I put together some sample data in a public
S3 bucket and some commands. Would you be able to try them out to see if
that reproduces the behaviour? I tried GDAL 2.2.3, GDAL 2.4.2, and GDAL
3.0.1 and saw the same thing. No overlay use with any resampling method
other than nearest.
The COGS are single band float32 with zero indicating NODATA and the VRT is
built with using gdal.BuildVRT('mosaic.vrt', files,
options=gdal.BuildVRTOptions(srcNodata=0, VRTNodata=0)) in Python. The VRT
also has pre-built statistics, if that is a factor.
# v3.0.1
gdalinfo --version
# Output looks normal to me.
gdalinfo /vsicurl/
https://dev-myheat-ca.s3-us-west-2.amazonaws.com/data/gdal/mosaic.vrt
--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR
# This is actually your tool I think! Reports the COG as valid.
python validate_cloud_optimized_geotiff.py /vsicurl/
https://dev-myheat-ca.s3-us-west-2.amazonaws.com/data/gdal/20.cog.tiff
# ~4 seconds (--debug ON will show GDAL loading overviews)
gdal_translate /vsicurl/
https://dev-myheat-ca.s3-us-west-2.amazonaws.com/data/gdal/mosaic.vrt
/tmp/tile.tiff -projwin -9177335.364031402 5305341.259217514
-9174889.379126277 5302895.274312388 -outsize 256 256 -r nearest --config
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR
# ~60 seconds (--debug ON will show GDAL loading full resolution data)
gdal_translate /vsicurl/
https://dev-myheat-ca.s3-us-west-2.amazonaws.com/data/gdal/mosaic.vrt
/tmp/tile.tiff -projwin -9177335.364031402 5305341.259217514
-9174889.379126277 5302895.274312388 -outsize 256 256 -r bilinear --config
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR
Could it have to do with the data being float32 or the way the overviews
are built? GDAL seems to be making the choice to skip the overviews for
some reason.
Thank again,
Angus
On Fri, Oct 30, 2020 at 8:53 AM Even Rouault
wrote:
> Angus,
>
> > But the second command ignores the overviews and downloads the full
> > resolution imagery leading to very long download times. Is it expected
> > behaviour that all re-sampling methods except 'nearest neighbour' need to
> > request the full resolution data? Can't they apply the selected
> re-sampling
> > method to the overviews? Is it something to do with the way I created my
> > VRT file?
>
> I can't confirm with the following little test
>
> gdal_translate byte.tif test.tif -outsize 2048 2048
> gdal_translate test.tif left.tif -srcwin 0 0 1024 2048 -b 1 -b 1 -b 1
> gdal_translate test.tif right.tif -srcwin 1024 0 1024 2048 -b 1 -b 1 -b 1
> gdalbuildvrt vrt.vrt left.tif right.tif
> gdaladdo left.tif
> gdaladdo right.tif
> python -c "from osgeo import gdal; ds = gdal.Open('left.tif',
> gdal.GA_Update);
> ds.GetRasterBand(1).GetOverview(2).Fill(0)"
>
> copy left.tif, right.tif and vrt.vrt on S3
>
> gdal_translate /vsis3/mybucket/vrt.vrt out.tif -r bilinear -outsize 128 128
>
> Works fine with GDAL 2.4.4 and master
>
> If you can come up with a minimum reproducing scenario (input files +
> script/
> command to reproduce), please file a ticket on github
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Build GDAL with static PROJ6
On Tue, Nov 3, 2020 at 2:44 PM Lars Sunde wrote: > > Hello, > > Does GDAL 3.0.4 support being statically linked with Proj 6.2.1? > > Environment is RedHat 7.2 (64 bit) and GCC 9.3. > > We have successfully built Proj 6.2.1 as static library. The proj build > folder is located at /tmp/proj and contains /tmp/proj/lib/libproj.a. > > We have tried below command to configured gdal but it failed with error > message "configure error: PROJ 6 symbols not found" > ./configure ---with-proj=/tmp/proj Please check towards the end of the file "config.log" which is generated by "configure". It may contain the error message. Best, Markus ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Build GDAL with static PROJ6
Hello, Does GDAL 3.0.4 support being statically linked with Proj 6.2.1? Environment is RedHat 7.2 (64 bit) and GCC 9.3. We have successfully built Proj 6.2.1 as static library. The proj build folder is located at /tmp/proj and contains /tmp/proj/lib/libproj.a. We have tried below command to configured gdal but it failed with error message "configure error: PROJ 6 symbols not found" ./configure ---with-proj=/tmp/proj What are we doing wrong? kind regards, Lars ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
