Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Hint: use EPSG:4326 for the SRS of the .gti.fgb, because this is the SRS of the tiles. The GTI driver isn't smart enough to figure out that OGC:CRS84 ~= EPSG:4326, and thus a rather dummy on-the-fly warping is done. Le 31/01/2024 à 01:37, Michael Sumner a écrit : just to follow up, I got it all working in latest GDAL: script at https://github.com/mdsumner/cog-example/blob/main/gti/cop90.py creates dsn /vsicurl/https://github.com/mdsumner/cog-example/raw/main/gti/cop90.gti.fgb which works nicely, thanks! Cheers, Mike On Tue, Jan 30, 2024 at 10:57 PM Even Rouault wrote: as you could also attach the XRES, YRES, BANDCOUNT, etc. metadata items that would help the GTI driver avoid probing one of the tiles. PS: OGRLayer::SetMetadataItem(key, value) only works on FlatGeoBuf and GPKG drivers with latest master as well -- http://www.spatialys.com My software is free, but my time generally not. -- Michael Sumner Software and Database Engineer Australian Antarctic Division Hobart, Australia e-mail: mdsum...@gmail.com -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
just to follow up, I got it all working in latest GDAL: script at https://github.com/mdsumner/cog-example/blob/main/gti/cop90.py creates dsn /vsicurl/https://github.com/mdsumner/cog-example/raw/main/gti/cop90.gti.fgb which works nicely, thanks! Cheers, Mike On Tue, Jan 30, 2024 at 10:57 PM Even Rouault wrote: > > as you could also attach the XRES, YRES, BANDCOUNT, etc. metadata items > that would help the GTI driver avoid probing one of the tiles. > > PS: OGRLayer::SetMetadataItem(key, value) only works on FlatGeoBuf and > GPKG drivers with latest master as well > > -- http://www.spatialys.com > My software is free, but my time generally not. > > -- Michael Sumner Software and Database Engineer Australian Antarctic Division Hobart, Australia e-mail: mdsum...@gmail.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
as you could also attach the XRES, YRES, BANDCOUNT, etc. metadata items that would help the GTI driver avoid probing one of the tiles. PS: OGRLayer::SetMetadataItem(key, value) only works on FlatGeoBuf and GPKG drivers with latest master as well -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
ah thanks, all very helpful - no it's not done with master (for various
reasons), I'll follow up with details if relevant, mostly I was just
excited to get a useable workflow for the entire process.
Cheers, Mike
On Tue, 30 Jan 2024, 22:46 Even Rouault, wrote:
> Michael,
>
> You need to attach the CRS at layer creation time with:
>
> layer = ds.CreateLayer(layer_name, geom_type=ogr.wkbPolygon, srs=sr)
>
> Setting it with featureDefn.GetGeomFieldDefn(0).SetSpatialRef() is
> illegal. Now that
> https://gdal.org/development/rfc/rfc97_feature_and_fielddefn_sealing.html
> has been implemented, you should have got an error... (assuming you run
> your script with recent GDAL master)
>
> from osgeo import ogr
> ogr.UseExceptions()
> ds = ogr.GetDriverByName('ESRI Shapefile').CreateDataSource('/tmp/foo.shp')
> lyr = ds.CreateLayer('foo')
> from osgeo import osr
> sr = osr.SpatialReference()
> sr.SetFromUserInput("OGC:CRS84")
> featureDefn = lyr.GetLayerDefn()
> featureDefn.GetGeomFieldDefn(0).SetSpatialRef(sr)
> Traceback (most recent call last):
> File "", line 1, in
> File "/home/even/gdal/gdal/build_cmake/swig/python/osgeo/ogr.py", line
> 6017, in SetSpatialRef
> return _ogr.GeomFieldDefn_SetSpatialRef(self, *args)
> RuntimeError: OGRGeomFieldDefn::SetSpatialRef() not allowed on a sealed
> object
>
> I can't think of a reason why your script wouldn't work with a FlatGeoBuf
> output (did you get an error message?), which would be much better, as you
> could also attach the XRES, YRES, BANDCOUNT, etc. metadata items that would
> help the GTI driver avoid probing one of the tiles.
>
> Even
> Le 30/01/2024 à 12:31, Michael Sumner a écrit :
>
> awesome, thanks Even I'm having fun with this one.
>
> For anyone interested I created Python to parse the OpenTopography COP90
> VRT (I have to wget it locally as I don't know how to hit the URL for the
> xml yet).
>
>
> https://github.com/mdsumner/cog-example/blob/1ca74f3baffe69830180031ddabcbbc569816150/gti/cop90.py
>
> (I'm using the DstRect to get efficient extent polygon of each tif without
> opening them).
>
> This writes a cop90.shp which can be GTI'd via
>
> gdalinfo GTI:/vsicurl/
> https://github.com/mdsumner/cog-example/raw/main/gti/cop90.shp
>
> (seems I didn't get the SRS attached correctly).
>
> I also have an R script in there for COP30 that I did previously to
> FlatGeoBuf, I couldn't get FlatGeoBuf to work from Python and don't know
> why.
>
> If there's any pythonic-idiomatic advice on my code I'm very interested,
> slowly I'll try to use this to do the same in C++. If there's a faster way
> to get these extents with GDAL I'm also keen to hear, thanks!
>
> Cheers, Mike
>
> On Fri, Jan 26, 2024 at 10:19 PM Even Rouault via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Hi,
>>
>> the driver has recently landed into GDAL master, renamed as GTI:
>> https://gdal.org/drivers/raster/gti.html
>>
>> Can be tested using
>> https://gdal.org/download.html#gdal-master-conda-builds , the refreshed
>> ghcr.io/osgeo/gdal Docker image, or other nightly builds of GDAL master
>>
>> Even
>>
>> Le 20/12/2023 à 19:33, Even Rouault via gdal-dev a écrit :
>> > Hi,
>> >
>> > For those not actively following github tickets & PR, I just want to
>> > point to a new pending major functionality to improve management of
>> > virtual mosaics with a very large number of tiles/sources (> tens of
>> > thousands of tiles), by referencing them as features of a vector layer
>> > (typically created by gdaltindex), instead of a XML file as in
>> > traditional VRT, augmented with additional metadata.
>> >
>> > More details in https://github.com/OSGeo/gdal/pull/8983 (and in
>> > initial ticket in https://github.com/OSGeo/gdal/issues/8861)
>> >
>> > Even
>> >
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> --
> Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> Hobart, Australia
> e-mail: mdsum...@gmail.com
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Michael,
You need to attach the CRS at layer creation time with:
layer = ds.CreateLayer(layer_name, geom_type=ogr.wkbPolygon, srs=sr)
Setting it with featureDefn.GetGeomFieldDefn(0).SetSpatialRef() is
illegal. Now that
https://gdal.org/development/rfc/rfc97_feature_and_fielddefn_sealing.html
has been implemented, you should have got an error... (assuming you run
your script with recent GDAL master)
from osgeo import ogr
ogr.UseExceptions()
ds = ogr.GetDriverByName('ESRI Shapefile').CreateDataSource('/tmp/foo.shp')
lyr = ds.CreateLayer('foo')
from osgeo import osr
sr = osr.SpatialReference()
sr.SetFromUserInput("OGC:CRS84")
featureDefn = lyr.GetLayerDefn()
featureDefn.GetGeomFieldDefn(0).SetSpatialRef(sr)
Traceback (most recent call last):
File "", line 1, in
File "/home/even/gdal/gdal/build_cmake/swig/python/osgeo/ogr.py",
line 6017, in SetSpatialRef
return _ogr.GeomFieldDefn_SetSpatialRef(self, *args)
RuntimeError: OGRGeomFieldDefn::SetSpatialRef() not allowed on a sealed
object
I can't think of a reason why your script wouldn't work with a
FlatGeoBuf output (did you get an error message?), which would be much
better, as you could also attach the XRES, YRES, BANDCOUNT, etc.
metadata items that would help the GTI driver avoid probing one of the
tiles.
Even
Le 30/01/2024 à 12:31, Michael Sumner a écrit :
awesome, thanks Even I'm having fun with this one.
For anyone interested I created Python to parse the OpenTopography
COP90 VRT (I have to wget it locally as I don't know how to hit the
URL for the xml yet).
https://github.com/mdsumner/cog-example/blob/1ca74f3baffe69830180031ddabcbbc569816150/gti/cop90.py
(I'm using the DstRect to get efficient extent polygon of each tif
without opening them).
This writes a cop90.shp which can be GTI'd via
gdalinfo
GTI:/vsicurl/https://github.com/mdsumner/cog-example/raw/main/gti/cop90.shp
(seems I didn't get the SRS attached correctly).
I also have an R script in there for COP30 that I did previously to
FlatGeoBuf, I couldn't get FlatGeoBuf to work from Python and don't
know why.
If there's any pythonic-idiomatic advice on my code I'm very
interested, slowly I'll try to use this to do the same in C++. If
there's a faster way to get these extents with GDAL I'm also keen to
hear, thanks!
Cheers, Mike
On Fri, Jan 26, 2024 at 10:19 PM Even Rouault via gdal-dev
wrote:
Hi,
the driver has recently landed into GDAL master, renamed as GTI:
https://gdal.org/drivers/raster/gti.html
Can be tested using
https://gdal.org/download.html#gdal-master-conda-builds , the
refreshed
ghcr.io/osgeo/gdal Docker image, or
other nightly builds of GDAL master
Even
Le 20/12/2023 à 19:33, Even Rouault via gdal-dev a écrit :
> Hi,
>
> For those not actively following github tickets & PR, I just
want to
> point to a new pending major functionality to improve management of
> virtual mosaics with a very large number of tiles/sources (>
tens of
> thousands of tiles), by referencing them as features of a vector
layer
> (typically created by gdaltindex), instead of a XML file as in
> traditional VRT, augmented with additional metadata.
>
> More details in https://github.com/OSGeo/gdal/pull/8983 (and in
> initial ticket in https://github.com/OSGeo/gdal/issues/8861)
>
> Even
>
--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com
--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
awesome, thanks Even I'm having fun with this one. For anyone interested I created Python to parse the OpenTopography COP90 VRT (I have to wget it locally as I don't know how to hit the URL for the xml yet). https://github.com/mdsumner/cog-example/blob/1ca74f3baffe69830180031ddabcbbc569816150/gti/cop90.py (I'm using the DstRect to get efficient extent polygon of each tif without opening them). This writes a cop90.shp which can be GTI'd via gdalinfo GTI:/vsicurl/ https://github.com/mdsumner/cog-example/raw/main/gti/cop90.shp (seems I didn't get the SRS attached correctly). I also have an R script in there for COP30 that I did previously to FlatGeoBuf, I couldn't get FlatGeoBuf to work from Python and don't know why. If there's any pythonic-idiomatic advice on my code I'm very interested, slowly I'll try to use this to do the same in C++. If there's a faster way to get these extents with GDAL I'm also keen to hear, thanks! Cheers, Mike On Fri, Jan 26, 2024 at 10:19 PM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > the driver has recently landed into GDAL master, renamed as GTI: > https://gdal.org/drivers/raster/gti.html > > Can be tested using > https://gdal.org/download.html#gdal-master-conda-builds , the refreshed > ghcr.io/osgeo/gdal Docker image, or other nightly builds of GDAL master > > Even > > Le 20/12/2023 à 19:33, Even Rouault via gdal-dev a écrit : > > Hi, > > > > For those not actively following github tickets & PR, I just want to > > point to a new pending major functionality to improve management of > > virtual mosaics with a very large number of tiles/sources (> tens of > > thousands of tiles), by referencing them as features of a vector layer > > (typically created by gdaltindex), instead of a XML file as in > > traditional VRT, augmented with additional metadata. > > > > More details in https://github.com/OSGeo/gdal/pull/8983 (and in > > initial ticket in https://github.com/OSGeo/gdal/issues/8861) > > > > Even > > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -- Michael Sumner Software and Database Engineer Australian Antarctic Division Hobart, Australia e-mail: mdsum...@gmail.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Hi, the driver has recently landed into GDAL master, renamed as GTI: https://gdal.org/drivers/raster/gti.html Can be tested using https://gdal.org/download.html#gdal-master-conda-builds , the refreshed ghcr.io/osgeo/gdal Docker image, or other nightly builds of GDAL master Even Le 20/12/2023 à 19:33, Even Rouault via gdal-dev a écrit : Hi, For those not actively following github tickets & PR, I just want to point to a new pending major functionality to improve management of virtual mosaics with a very large number of tiles/sources (> tens of thousands of tiles), by referencing them as features of a vector layer (typically created by gdaltindex), instead of a XML file as in traditional VRT, augmented with additional metadata. More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial ticket in https://github.com/OSGeo/gdal/issues/8861) Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Sean, I'm wondering how this relates to STAC. Do you imagine that GDAL users should and will use this to publish large collections of data? When should they use this and when should they use STAC collections instead? I don't imagine this for publishing (that is not publishing the VRRTI tile index itself), but more for internal use in the software stack of users with large image collections. Similarly to good-ol' XML VRT. Back in the day our tile indexes were shapefiles. I don't think this would be a good practice now. Would all vector formats be supported? yes, in theory. But as documented, it is strongly recommended to use vector formats with efficient spatial filtering like GeoPackage, FlatGeoBuf, PostGIS. Those 3 ones in particular have been enhanced to support setting the metadata items used by VRTTI driver. So you don't necessarily need to create a XML VRTTI wrapper file for those. How about column-oriented formats? that should work with them too. The fact that they are column-oriented is just an implementation detail. That said, until GeoParquet gets efficient spatial filtering (in the works at the spec level: https://github.com/opengeospatial/geoparquet/pull/191), this isn't yet a good choice (as a provision the VRTTI driver actually already supports .vrt.parquet) Can I just do a gdalinfo ":example.shp" on a MapServer tile index? Yes. However, given that the metadata items suggested by VRTTI will not be available, this will require the driver to GDALOpen() one of the tile to figure out the number of bands, data type, etc. at opening time. Not a big deal however if the VRTTI dataset is opened for a long time to satisfy many requests. This feature has been latent and available for a while, yes? It's obviously useful. My main misgivings are about the overlap with existing GDAL features and protocols (VRT, WMS, WMTS, STAC, MBTiles). I think too many choices can create a not great experience for users. Yes there is clearly overlap (like most GDAL supported formats have strong functional overlap since they can be used with a common API :-)) Here are the main differences I see: - WMS: requires a client&server architecture - WMTS: generally requires a client&server architecture too (local use possible if using file:// URIs), but requires regular tiling - STAC(ITems, with STAC projection extensions): functionally a bit similar to VRTTI, but using JSON format to catalog the sources. If you want to manage a catalog with a large set of sources, static STAC will not work well and you'll need a STAC server. Although with STAC, you want each "asset" to have user-facing metadata. In a VRTTI use case, you are more interested by the resulting mosaic than each of its source. - STAC(Temporal Asset): functionally similar to WMTS (regular tiling) - MBTiles/GeoPackage/COG/etc.: single-file tiled formats Even On Wed, Dec 20, 2023, 11:33 AM Even Rouault via gdal-dev wrote: Hi, For those not actively following github tickets & PR, I just want to point to a new pending major functionality to improve management of virtual mosaics with a very large number of tiles/sources (> tens of thousands of tiles), by referencing them as features of a vector layer (typically created by gdaltindex), instead of a XML file as in traditional VRT, augmented with additional metadata. More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial ticket in https://github.com/OSGeo/gdal/issues/8861) Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Hi Even, I'm wondering how this relates to STAC. Do you imagine that GDAL users should and will use this to publish large collections of data? When should they use this and when should they use STAC collections instead? Back in the day our tile indexes were shapefiles. I don't think this would be a good practice now. Would all vector formats be supported? How about column-oriented formats? How backwards compatible will this be with existing tile index files? Can I just do a gdalinfo ":example.shp" on a MapServer tile index? This feature has been latent and available for a while, yes? It's obviously useful. My main misgivings are about the overlap with existing GDAL features and protocols (VRT, WMS, WMTS, STAC, MBTiles). I think too many choices can create a not great experience for users. On Wed, Dec 20, 2023, 11:33 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > For those not actively following github tickets & PR, I just want to > point to a new pending major functionality to improve management of > virtual mosaics with a very large number of tiles/sources (> tens of > thousands of tiles), by referencing them as features of a vector layer > (typically created by gdaltindex), instead of a XML file as in > traditional VRT, augmented with additional metadata. > > More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial > ticket in https://github.com/OSGeo/gdal/issues/8861) > > Even > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
Andrew, Le 21/12/2023 à 13:55, Andrew C Aitchison via gdal-dev a écrit : On Wed, 20 Dec 2023, Even Rouault via gdal-dev wrote: Hi, For those not actively following github tickets & PR, I just want to point to a new pending major functionality to improve management of virtual mosaics with a very large number of tiles/sources (> tens of thousands of tiles), by referencing them as features of a vector layer (typically created by gdaltindex), instead of a XML file as in traditional VRT, augmented with additional metadata. More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial ticket in https://github.com/OSGeo/gdal/issues/8861) Am I right in thinking that all the tiles must have the same bit-depth and colour interpretation Yes. If different bit-depth and rescaling is wished, you'll have to manually use for example "vrt://the.tif?scale=src_min,src_max,dst_min,dst_max" as the source name. Cf https://gdal.org/drivers/raster/vrt.html#vrt-connection-string If bands are ordered differently, similarly use things like "vrt://the.tif?bands=3,2,1" to reorder them in a uniform way. and for paletted tiles the same colour table ? I've just added support for automatically expanding color tables to RGB(A) if the BAND_COUNT metadata item is set to 3 or 4. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements
On Wed, 20 Dec 2023, Even Rouault via gdal-dev wrote: Hi, For those not actively following github tickets & PR, I just want to point to a new pending major functionality to improve management of virtual mosaics with a very large number of tiles/sources (> tens of thousands of tiles), by referencing them as features of a vector layer (typically created by gdaltindex), instead of a XML file as in traditional VRT, augmented with additional metadata. More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial ticket in https://github.com/OSGeo/gdal/issues/8861) Am I right in thinking that all the tiles must have the same bit-depth and colour interpretation and for paletted tiles the same colour table ? I am aware of a vendor whose products had different colour tables, and even different bit-depths, for each tile - a sort of adaptive compression. -- Andrew C. Aitchison Kendal, UK and...@aitchison.me.uk ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
