Hi Stefan, "If there would be OGR driver for vector tiles it would make a similar resource extremely useful also for other purposes in GIS tools - not only for tileservers and direct visualisation.“
I was thinking you want to do more than just rendering. Then at least, as Blake also mentioned, the features need to be merged. And I think some degenerate geometries are to be expected for a few cases. You will also have multiple polygons cut along tile bounds for what should be a single polygon. And the introduced points to close a polygon’s path at the tile bounds may add another inaccuracy when you later reproject the data. It depends what your goal is. Serious map editing isn’t possible. ~Ben Am 02.02.16, 00:27 schrieb "Stefan Keller" unter <sfkel...@gmail.com>: >Hi Ben > >Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles. >I like projections (and also GeoPackage :-) ). >But reprojection is solved here, since OGR/QGIS can transform on the fly. >This thread is about Petr's question about a reader "for mapbox-like >vector tiles stored in MBTiles or downloaded from a url...". > >So I'd like to come back to the issue: >1. of a generic MVT decoder/enoder library (C++ or Python) => Blake? >2. and of a "feature provider" in an interactive environment (like >QGIS) or a file/stream converter (like OGR) => anybody? > >:Stefan > > > >2016-02-02 0:04 GMT+01:00 Stadin, Benjamin ><benjamin.sta...@heidelberg-mobil.com>: >> Hi Stefan, >> >> A vector tile may contain data for several zoom levels. But what data is >> shown at which zoom level is defined by the styling, not by the data >> itself. It is handled on client side, by the web (or native) tile >> renderer. >> >> The path is the same as with mercator tiles (zoomLevel/x/y): >> https://example.com/17/65535/43602.mvt >> >> >> The tile for zoom level 17 may contain features of class „building“, and >> the style definition may contain a rule that filters type „building“ for >> zoomLevels <= 18. >> >> ~Ben >> >> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter >> <gdal-dev-boun...@lists.osgeo.org on behalf of sfkel...@gmail.com>: >> >>>That's good news! >>>We'll be happy to add Python bindings to such a generic C++ library >>>for Mapbox vector tiles. >>>We have still some time (say few weeks?) left here to decide to >>>contribute either to these bindings or to Mapzen's pure Python lib. >>> >>>One reason why I'm insisting on Python (bindings) - besides easier >>>testing with QGIS - is the following: >>>AFAIK with vector tiles we have to deal with quite a new kind of >>>feature provider: >>>One which contains "zoom levels" and one which "suddenly" changes >>>feature class at different zoom levels. >>>If a client like QGIS consumes vector tiles from OGR in a one-shot >>>call, how should the client deal with such "zoom level" dependent >>>issues? >>>So my thinking is, to consume only the most appropriate zoom level, >>>probably only within current map canvas... >>> >>>:Stefan >>> >>>2016-02-01 18:27 GMT+01:00 Flippmoke <flippm...@gmail.com>: >>>> >>>> Stefan, >>>> >>>> I am not aware of any libraries besides mapnik vector tile that have >>>>been attempting to implement v2 specification. I think mapzen's >>>>implementation might be best? We have been wanting to add vector tiles >>>>to the python mapnik bindings for quite some time but we don't have the >>>>time to do it right now. >>>> >>>> I will try to think this week about how we could make a generic C++ >>>>library quickly for Mapbox vector tiles. >>>> >>>> Thanks, >>>> >>>> Blake Thompson >>>> >>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <sfkel...@gmail.com> wrote: >>>>> >>>>> Hi Blake >>>>> >>>>> It's clear to me that a good generic c++ library for encoding and >>>>> decoding vector tiles is needed. >>>>> On the other hand, I'd like to experiment with vector tiles and >>>>> related QGIS parallel loading issues for which Python is more >>>>> convenient. >>>>> >>>>> It seems that Mapzen's [1] Python implementation is further along. >>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's >>>>>variant [2] ? >>>>> >>>>> :Stefan >>>>> >>>>> [1] https://github.com/mapzen/mapbox-vector-tile >>>>> [2] https://github.com/mapbox/vector-tile-py >>>>> >>>>> >>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippm...@gmail.com>: >>>>>> All, >>>>>> >>>>>> The first thing that is going to be required likely is a good >>>>>>generic >>>>>>c++ library for encoding and decoding vector tiles IMO. >>>>>> >>>>>> I have spent a lot of time working on developing the mapnik vector >>>>>>tile library to be a solid implementation, especially around all the >>>>>>work done for 2.0 of Mapbox vector tile specification. I would love >>>>>>to >>>>>>eventually help make this a generalized implementation so that it is >>>>>>portable to a library like GDAL. >>>>>> >>>>>> We have talked about doing something like this at Mapbox but will >>>>>>require a good deal of time, so it hasn't been pushed forward yet. >>>>>>The >>>>>>other issues that come to mind to me is that some of the libraries >>>>>>that mapnik vector tile depends upon most likely need good custom >>>>>>implementations if we wanted to make it more generic. >>>>>> >>>>>> The reason for this is that I would prefer for it to be a header >>>>>>only >>>>>>library with no dependencies. Adding more dependencies for GDAL seems >>>>>>like it could be a mess. >>>>>> >>>>>> Blake Thompson >>>>>> >>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkel...@gmail.com> >>>>>>>wrote: >>>>>>> >>>>>>> Hi Petr, Blake, Even and everybody >>>>>>> >>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles. >>>>>>> I'd already asked Even a similar question. >>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles >>>>>>>reader >>>>>>> client in Python if possible as part of a QGIS plugin. >>>>>>> And I'm interested not to do redundant implementations. >>>>>>> Are there any news on this matter? >>>>>>> >>>>>>> :Stefan >>>>>>> Prof. at HSR and leader of Geometa Lab >>>>>>> >>>>>>> @Even: Regarding OGR MVT development can you send to me (directly) >>>>>>>a >>>>>>> contact name of the group which won the bidding? >>>>>>> >>>>>>> >>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal >>>>>>><petr.pri...@klokantech.com>: >>>>>>>> Hi everybody, >>>>>>>> >>>>>>>> thanks for the comments. Regarding the technical details: >>>>>>>> >>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part >>>>>>>>>would be >>>>>>>>> merging of features into the original features across tiles. You >>>>>>>>>could use >>>>>>>>> some complicated merging technique, but that could be very >>>>>>>>>expensive. >>>>>>>> >>>>>>>> >>>>>>>> The driver could require "id" parameter, which specifies name of >>>>>>>>an >>>>>>>> attribute with unique identifier on the features shared across the >>>>>>>>tiles. >>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique >>>>>>>>non-zero >>>>>>>> "osm_id" in place. The first version of driver could read only >>>>>>>>features with >>>>>>>> such ID. The identifier does not have to be always filled, but >>>>>>>>often it is, >>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap >>>>>>>>- but >>>>>>>> yeh, a completely general stitching would be much harder. >>>>>>>> >>>>>>>> The clipping + merging of shapes itself can be implemented with >>>>>>>>underlaying >>>>>>>> GEOS library functions. >>>>>>>> >>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript >>>>>>>>X-Ray >>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3: >>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html >>>>>>>> >>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL >>>>>>>>JS >>>>>>>> powered X-Ray: >>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html >>>>>>>> >>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure >>>>>>>> JavaScript: >>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html >>>>>>>> >>>>>>>> The OGR driver could be also practical for the unsimplified / >>>>>>>>ungeneralised >>>>>>>> MBTiles with OSM data available at: >>>>>>>>http://osmlab.github.io/osm-qa-tiles/ >>>>>>>> >>>>>>>>> >>>>>>>>> If you wanted to simply view vector tile data it could get >>>>>>>>>confusing as >>>>>>>>> well since most vector tiles need a buffer when they are used for >>>>>>>>> visualization programs. This is important for things like >>>>>>>>>labeling >>>>>>>>>on maps >>>>>>>>> etc. Therefore, you could get a large amount of overlapping data >>>>>>>>>on tiles if >>>>>>>>> you loaded them all in at once. >>>>>>>> >>>>>>>> >>>>>>>> Clipping on border of each tile must be applied before stitching >>>>>>>>the vector >>>>>>>> features together - so the buffer is removed. Buffer is there >>>>>>>>only >>>>>>>>for >>>>>>>> visualisation and labeling. >>>>>>>> >>>>>>>>> >>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile >>>>>>>>>decoder isn't >>>>>>>>> currently able to decode vector tiles into a form not supported >>>>>>>>>by >>>>>>>>>mapnik. >>>>>>>>> Therefore, OGR would not be able to easily use this. >>>>>>>> >>>>>>>> >>>>>>>> Correct. It can't be used directly (GDAL can't link against), but >>>>>>>>the >>>>>>>> relevant code can be extracted and reused directly. For basic >>>>>>>>decoding >>>>>>>> critical is the Protobuf library (which is already in GDAL) and >>>>>>>> >>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vect >>>>>>>>or >>>>>>>>_tile.proto >>>>>>>> + the code for converting pixel coordinates in features to >>>>>>>>relevant >>>>>>>>geo >>>>>>>> coordinates and GEOS-based merging of features. >>>>>>>> >>>>>>>>> We plan to write a generalized encoder and decoder eventually so >>>>>>>>>that any >>>>>>>>> other library can plugin with out the mapnik requirement. >>>>>>>> >>>>>>>> >>>>>>>> This would be amazing. >>>>>>>> >>>>>>>>> If you have any questions specifically about Mapnik Vector Tile >>>>>>>>>or >>>>>>>>>the >>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would >>>>>>>>>note >>>>>>>>>that you >>>>>>>>> guys will want to update your vector tiles once the V2 >>>>>>>>>specification changes >>>>>>>>> are into Mapnik-Vector-Tile. >>>>>>>> >>>>>>>> >>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers >>>>>>>>on a >>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so >>>>>>>>upgrades on >>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering >>>>>>>>software. >>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right? >>>>>>>> >>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I >>>>>>>>>don't >>>>>>>>> see >>>>>>>>> mention of that neither in the MVT spec ( >>>>>>>>> https://github.com/mapbox/vector- >>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one ( >>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), >>>>>>>>>although >>>>>>>>> I >>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data >>>>>>>>>column. >>>>>>>> >>>>>>>> >>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector >>>>>>>>Tiles >>>>>>>>inside - >>>>>>>> and Mapnik reads them directly when rasterising. Storing the >>>>>>>>vector >>>>>>>>tiles in >>>>>>>> SQLite make sense - and it simplifies deployment, and you can then >>>>>>>>easily >>>>>>>> have complete world on each node of a cluster of tileservers. If >>>>>>>>you don't >>>>>>>> want to upgrade your base map too often, this approach is pretty >>>>>>>>efficient >>>>>>>> and removes the centralised database as typical bottleneck. >>>>>>>> >>>>>>>> I did some tests on vector tiles vs raster in MBTiles and >>>>>>>>documented these >>>>>>>> in README.md at and >>>>>>>>https://github.com/klokantech/vector-tiles-sample before >>>>>>>> we started to work on OSM2VectorTiles project. >>>>>>>> There are also sample data and offline MapBox GL JS viewer in that >>>>>>>>repo. >>>>>>>> Vector tiles can be hosted also "unpacked", just as files in >>>>>>>>folders on a >>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does >>>>>>>>by >>>>>>>> default for raster tiles. >>>>>>>> >>>>>>>> If OGR driver is implemented, the primary source should be >>>>>>>>probably >>>>>>>>reading >>>>>>>> from an URL via curl. >>>>>>>> >>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most >>>>>>>>practical >>>>>>>> portable alternative to it. >>>>>>>> >>>>>>>> Technically people may store the tiles in different ways - for >>>>>>>>example >>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and >>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, >>>>>>>>I >>>>>>>>guess >>>>>>>> MapBox internally is also on a different storage for production >>>>>>>>servers ... >>>>>>>> for transfers and streaming tilesets there is the - but tiles are >>>>>>>>ALWAYS >>>>>>>> exposed via HTTP. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Petr >>>>>>>> -- >>>>>>>> Petr Pridal, Ph.D. >>>>>>>> CEO >>>>>>>> >>>>>>>> Klokan Technologies GmbH >>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland >>>>>>>> Tel: +41 (0)41 511 26 12 >>>>>>>> Email: i...@klokantech.com >>>>>>>> Web: http://www.klokantech.com/ >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> gdal-dev mailing list >>>>>>>> gdal-dev@lists.osgeo.org >>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev >>>_______________________________________________ >>>gdal-dev mailing list >>>gdal-dev@lists.osgeo.org >>>http://lists.osgeo.org/mailman/listinfo/gdal-dev >> _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
