Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Stadin, Benjamin]

I know I¹m a bit late...

Am 01.11.15, 04:04 schrieb "Paul Norman" unter :

>>#2 SQL -> Vectors
>>This is very similar to the set of SQL statements that osm-carto uses
>>to generate data. Mapbox is now on version 6 of vtile structure, so it
>>is obviously a hard problem to nail down. [4] We could make it
>>compatible, or come up with a different structure.
>>TBD: vtile structure
>
>A style designed for showing off OSM data and mapper feedback will put
>different data into vector tile different ways than one designed for
>Wikimedia's purposes. If osm-carto was vector tile based, we'd probably
>be changing the vector tile definitions every 6-12 weeks, and these
>would be backwards-incompatible changes.

Maybe one option would be to "borrow" some parts of the GeoPackage spec
[1] to handle the data aspect.
There is no single answer for how to add tile data to GPKG, but how about
starting with defining a default "OSM presentation data export":
- A standard GPKG container
- including a (probably large) set of gpkg_data_columns definitions
- pure vector features (no tile data at this point)
- versioning (could be done in a quite simple way then probably, since
several concerns are defined by GPKG itself)

Other "consumers" could then add own data and definitions to this
container in a standardized way
(e.g. append some data field and expect the existence of a certain
gpkg_data_columns definition).

There could be a several such OSM containers (though I think the feedback
data could be made part of the presentation data container).

In a next step, this data container can then be further optimized for
rendering by creating tile sets or else - while retaining all data as
defined within the GPKG.
Maybe there can be a tile data BLOB and a user data BLOB (both protocol
buffers) as part of the tile data table.
The user data proto buf must then by definition be an exact match to the
gpkg_data_columns.
So this custom data could be a simple addition to the GPKG tile data spec,
and it should work for both raster and vector tile data.
I've recently made a similar suggestion to a related topic for MapBox on
Github [2]. John Firebaugh's suggestion for vector tiling was to create a
WebP like extension for MapBox vector tiles within GPKG for their next
offline db.

In the end there could (ideally) be a concise set of standardized OSM
exports for different purposes,
a standardized way to deal with and extend this data by external tools and
data, and some way to deal
with different render data formats (e.g. vector tiles or else) with
embedded data. The latter can¹t be standardized easily in my opinion, we
have for example pretty different demands about the rendering data for our
native GL map renderer.

[1] http://www.geopackage.org/spec/
[2] 
https://github.com/mapbox/mapbox-gl-native/issues/3373#issuecomment-1671620
61


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Paul Norman]

On 10/31/2015 6:32 PM, Yuri Astrakhan wrote:

#1 OSM download -> SQL tables
We have used osm2pgsql to produce default table structure. Yet, it is 
not the most efficient way to parse data afterwards, and Paul has been 
working on the new ClearTables structure [3].
TBD: We would need to agree on a prefered table structure for this 
project.


Database schema is not going to be the same for every project. I'll 
probably be announcing something soon, but I expect my work to live 
alongside osm2pgsql C transforms, osm2pgsql pgsql Lua transforms, and 
imposm3, as well as vector tile generation platforms that don't involve 
a database.



#2 SQL -> Vectors
This is very similar to the set of SQL statements that osm-carto uses 
to generate data. Mapbox is now on version 6 of vtile structure, so it 
is obviously a hard problem to nail down. [4] We could make it 
compatible, or come up with a different structure.

TBD: vtile structure


A style designed for showing off OSM data and mapper feedback will put 
different data into vector tile different ways than one designed for 
Wikimedia's purposes. If osm-carto was vector tile based, we'd probably 
be changing the vector tile definitions every 6-12 weeks, and these 
would be backwards-incompatible changes.



#3 Vectors -> (a) PNG server side and (b) WebGL client side
If #2 produces Mapbox-compatible vtiles, we can easily reuse all the 
open source styles Mapbox produced, both MSS and WebGL, or create new 
ones. The problem here is that MSS & WebGL are different languages, 
and keeping them in sync with Mapbox Studios (Classic & GL) is hard. 
Richard Fairhurst is working on Glug[5] to simplify this.  Eventually 
it could be just the WebGL style and Kartotherian could render WebGL 
on the server side if needed.


Speaking as someone who has worked with two styles which were supposed 
to look the same, but had a different structure internally, writing the 
same style twice should be avoided at all costs. It's not twice the 
work, it's often more. Some kind of common language that gets processed 
to both Mapnik XML and GL JSON is needed. I don't know what this 
language will end up being, but it's too easy to write styles which have 
too many cascaded rules in CartoCSS so it probably won't be that. On the 
other hand, CartoCSS has multiple implementations which talk to multiple 
rendering engines, so it might win out for that reason.


I'm not sure Richard has any intent of targeting Mapnik XML with Glug.

If it weren't for minutely updates and the need to handle planet-sized 
files, I'd probably go with tilemaker[1] for vector tile generation. 
This would completely replace SQL tables. Keep in mind, I've been a 
PostGIS consultant, am a maintainer of osm2pgsql, and know writing SQL 
for layer definitions like few others do, and I'm prepared to throw that 
away for the advantages of how tilemaker transforms its data. 
Unfortunately, for what I'm interested in, I need minutely updates and 
planet dump handling.


I have a blog post on style complexity coming out when I gather some 
more numbers.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Yuri Astrakhan]

Thanks for all the responses! Are there any people on the stylesheet team
interested in working on this?  It might be possible to adapt osm-carto
work directly, since it is based on the same underlying technology.

Jochen, Kartotherian can dynamically alter/recombine tiles, e.g. adding
layers from multiple sources (e.g. language), or adding raster layers (both
raster and vector data can be stored in the same vtile). So we could create
a set of well-defined layers, stored either together or separate, and
combined on the fly.

Oleksiy, the true multilingual support requires Mapnik hstore support, see
#1113 [1]. Wikimedia DE might be able to help us with it, but it's still in
the air.  Another issue is Mapbox's WebGL cannot shape connected fonts yet
(e.g. Arabic), so they will still rely on the server-side renderig.

Martin:  If WebGL is a problem for mobile, they can fallback to PNGs
generated by Kartotherian, but we should consider the energy+time needed to
download high DPI PNG tiles vs smaller vector tiles + local rendering, plus
per-client customization aspects (multilingual, user-centric icons, etc)

Christoph, I agree that our current map is nowhere near as feature rich as
default OSM - we spent most of our time building the tile serving platform,
and spent very little time on the map itself.  We have even considered
using Natural Earth for the low zooms, possibly with an extra labels layer.

Benjamin, we had to alter Mapbox's Bright to use the free Google Noto
fonts. Can you help with automating WebGL setup & font generation for
Kartotherian? See [2]

Colin, we considered raster-layer combining for languages, but it might be
bad for caching, and would still require a lot of storage due to per-item
overhead. Storing all available languages in one vtile, and deciding at
render time seems more efficient.

Christoph, re multiple projections - if Mapnik implements vector tile
merging, we could in theory dynamicaly recombine tiles and change
projections, would't we? At a performance cost of course.  And we could
store some low-complexity data in the original format, converting it to the
needed tile on the fly, in any projection.

Thanks!!

[1] https://github.com/mapnik/mapnik/issues/1113
[2] https://github.com/kartotherian/osm-bright.tm2


On Sat, Oct 31, 2015 at 6:45 PM, Stadin, Benjamin <
benjamin.sta...@heidelberg-mobil.com> wrote:

> Hi Peter,
>
> Most what you said is already solved by data driven style possibilities
> and clustering that you have already with some existing WebGL libraries.
> E.g. which city label takes priority when zooming out over another city¹s
> label (by default the one with highest population), or whether city labels
> take priority over icons.
>
> I do not see why there would be any difference doing this on client or
> server side, besides you¹d have to adapt a few things at first of course.
> For an example, if you have an algorithm that places your labels depending
> on language and geometrical shape of labels, then this algorithm can
> likely be applied in realtime on the client side.
> If you have different anchor points for labels depending on language, then
> this is even less a problem: Decide by a filter rule in realtime which
> labels to render, each label may have a different position. This can be as
> simple as a single line in your style rule.
>
> Regards
> Ben
>
>
> Am 31.10.15 09:32 schrieb "Peter Wendorff" unter
> :
>
> >Hi Colin,
> >
> >just stitching a labels-layer and a base layer together would work, but
> >most often would not look well.
> >
> >Map rendering involves deciding which labels to show considering more
> >than just the labels, but icons and shields etc. as well to avoid
> >overlapping between all of them.
> >
> >You don't want to get a map where the label overlaps icons.
> >
> >As names in different languages differ in their need for geometrical
> >space on the map, these placement decisions have to be taken for each
> >language differently, thus the labels layer would include icons and
> >stuff like that as well - even oneway and so on.
> >
> >Of course it would be possible to do, but the result would look worse
> >than our current maps.
> >
> >regards
> >Peter
> >
> >Am 30.10.2015 um 12:41 schrieb Colin Smale:
> >> Can't we have a multi-lingual map by overlaying a base tile with a
> >> transparent text layer in the chosen language? We wouldn't *need* vector
> >> tiles for that, just a bit more storage (bitmap-based text layers should
> >> compress nicely) and clients which can handle the selection and display
> >> of the extra layer, which is pretty commonplace these days anyway).
> >>
> >> --colin
> >>
> >> On 2015-10-30 12:29, Oleksiy Muzalyev wrote:
> >>
> >>> I've heard from the OSM operations team's member at the conference
> >>> that it is the question of the servers' infrastructure cost.
> >>>
> >>> But if we have a vector-based map itself language selection, that we
> >>> could just add tags /name:fr/ or 

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Christoph Hormann]

On Saturday 31 October 2015, Yuri Astrakhan wrote:
>
> Christoph, re multiple projections - if Mapnik implements vector tile
> merging, we could in theory dynamicaly recombine tiles and change
> projections, would't we? At a performance cost of course.

First of all if you don't use the tiles as tiles but merge them into a 
full data set again there is no point in generating them in the first 
place of course.

Apart from that regarding the abstract possibility: probably no for a 
number of reasons:

- since mercator tiles don't include areas near the poles you will not 
be able to handle any projections that include the poles although 
better display of polar regions is of course one of the primary reasons 
for using other projections.
- in general the tile edges in mercator will not be straight lines in 
other projections.  You would have to specifically take this into 
account when generating and processing the tiled data, otherwise you 
can't reproject the tiles without generating artefacts or invalid 
geometries.
- you cannot map the zoom levels in one projection to those in another, 
especially not with the strongly variable scale of the mercator 
projection so unless your tiles are not zoom level specific you are 
screwed.

These points apply to all generic geometry data in the tiles, any data 
specifically generated for rendering like label positions etc. is 
normally projection specific anyway.

-- 
Christoph Hormann
http://www.imagico.de/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Yuri Astrakhan]

Christoph, thanks for the explanation! Yes, sounds like each projection
would have to be generated and stored separatelly. Not that big of a deal
from Kartotherian perspective because it handles each tile source
independently, but it will require extra space and cpu. Also, it will
require osm2pgsql or other db update tool to generate multiple invalidation
tilesets in different projections.

On Sat, Oct 31, 2015 at 10:55 PM, Christoph Hormann 
wrote:

> On Saturday 31 October 2015, Yuri Astrakhan wrote:
> >
> > Christoph, re multiple projections - if Mapnik implements vector tile
> > merging, we could in theory dynamicaly recombine tiles and change
> > projections, would't we? At a performance cost of course.
>
> First of all if you don't use the tiles as tiles but merge them into a
> full data set again there is no point in generating them in the first
> place of course.
>
> Apart from that regarding the abstract possibility: probably no for a
> number of reasons:
>
> - since mercator tiles don't include areas near the poles you will not
> be able to handle any projections that include the poles although
> better display of polar regions is of course one of the primary reasons
> for using other projections.
> - in general the tile edges in mercator will not be straight lines in
> other projections.  You would have to specifically take this into
> account when generating and processing the tiled data, otherwise you
> can't reproject the tiles without generating artefacts or invalid
> geometries.
> - you cannot map the zoom levels in one projection to those in another,
> especially not with the strongly variable scale of the mercator
> projection so unless your tiles are not zoom level specific you are
> screwed.
>
> These points apply to all generic geometry data in the tiles, any data
> specifically generated for rendering like label positions etc. is
> normally projection specific anyway.
>
> --
> Christoph Hormann
> http://www.imagico.de/
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Stadin, Benjamin]

Hi Peter,

Most what you said is already solved by data driven style possibilities
and clustering that you have already with some existing WebGL libraries.
E.g. which city label takes priority when zooming out over another city¹s
label (by default the one with highest population), or whether city labels
take priority over icons.

I do not see why there would be any difference doing this on client or
server side, besides you¹d have to adapt a few things at first of course.
For an example, if you have an algorithm that places your labels depending
on language and geometrical shape of labels, then this algorithm can
likely be applied in realtime on the client side.
If you have different anchor points for labels depending on language, then
this is even less a problem: Decide by a filter rule in realtime which
labels to render, each label may have a different position. This can be as
simple as a single line in your style rule.

Regards
Ben


Am 31.10.15 09:32 schrieb "Peter Wendorff" unter
:

>Hi Colin,
>
>just stitching a labels-layer and a base layer together would work, but
>most often would not look well.
>
>Map rendering involves deciding which labels to show considering more
>than just the labels, but icons and shields etc. as well to avoid
>overlapping between all of them.
>
>You don't want to get a map where the label overlaps icons.
>
>As names in different languages differ in their need for geometrical
>space on the map, these placement decisions have to be taken for each
>language differently, thus the labels layer would include icons and
>stuff like that as well - even oneway and so on.
>
>Of course it would be possible to do, but the result would look worse
>than our current maps.
>
>regards
>Peter
>
>Am 30.10.2015 um 12:41 schrieb Colin Smale:
>> Can't we have a multi-lingual map by overlaying a base tile with a
>> transparent text layer in the chosen language? We wouldn't *need* vector
>> tiles for that, just a bit more storage (bitmap-based text layers should
>> compress nicely) and clients which can handle the selection and display
>> of the extra layer, which is pretty commonplace these days anyway).
>>
>> --colin
>>
>> On 2015-10-30 12:29, Oleksiy Muzalyev wrote:
>>
>>> I've heard from the OSM operations team's member at the conference
>>> that it is the question of the servers' infrastructure cost.
>>>
>>> But if we have a vector-based map itself language selection, that we
>>> could just add tags /name:fr/ or /name: ru/ and have the OSM map of
>>> say New York in French or in Russian for tourists. Or map of Siberia
>>> in German also for tourists, Middle East in English, etc. without any
>>> additions cost on the server side. It is not that difficult to add
>>> such multi-language tags. There could be also the map in Basque,
>>> Catalan, Kurdish, Scots and other smaller (by number of speakers)
>>> languages without any additional cost and without a civil conflict.
>>>
>>> I realize that mobile hardware is not enough advanced for that yet and
>>> vector-based technology is only in an development stage.
>>>
>>> brgds
>>> Oleksiy
>>>
>>> On 30.10.2015 12:06, Maarten Deen wrote:
 On 2015-10-30 11:53, Oleksiy Muzalyev wrote:
> One of the advantages of the the vector-based map would be the
> multilingualism.
>
> For instance at the moment the OSM map of the Middle East is
>basically
> useless for me as I do not know the Arabic alphabet yet. But as far
>as
> I understand and as I heard at the conference the vector-based map
> would allow the choice of a language of the map itself.

 I do not see how that can not be solved with png-based tiles. You
 only have to render the tiles.
 The method for detecting which tileset/language to show is the same.

 BTW: it is still not as simple as rendering "in a different
 language". Then you start rendering a map in English and see names
 like "Cologne" or "Brussels"  show up on the map.

 Regards,
 Maarten

 ___
 dev mailing list
 dev@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/dev
>>>
>>>
>>> ___
>>> dev mailing list
>>> dev@openstreetmap.org 
>>> https://lists.openstreetmap.org/listinfo/dev
>>
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>
>
>___
>dev mailing list
>dev@openstreetmap.org
>https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Yuri Astrakhan]

I'm on IRC as "yurik", if anyone wants to chat.

For vectors and style development, the easiest way is to install Mapbox
Studio Classic [1] and use a local Postgres DB with OSM data (full or
partial). Kartotherian can use the data source and the style without any
modifications. The studio uses the same key components as Kartotherian, but
Kartotherian is oriented towards massive web serving, whereas the studio is
geared towards development. Regardless, it should be very quick to set it
up. Usually Kartotherian source pipeline [2] should include some caching
source to store vtiles - either in Cassandra, Postgres, mbtiles or as
files, but you can configure it to generate from SQL on the fly (very slow).

As a path forward, I see 3 key areas and 3 descisions to make (assuming
positive answer to "do we actually need this?")

#1 OSM download -> SQL tables
We have used osm2pgsql to produce default table structure. Yet, it is not
the most efficient way to parse data afterwards, and Paul has been working
on the new ClearTables structure [3].
TBD: We would need to agree on a prefered table structure for this project.

#2 SQL -> Vectors
This is very similar to the set of SQL statements that osm-carto uses to
generate data. Mapbox is now on version 6 of vtile structure, so it is
obviously a hard problem to nail down. [4] We could make it compatible, or
come up with a different structure.
TBD: vtile structure

#3 Vectors -> (a) PNG server side and (b) WebGL client side
If #2 produces Mapbox-compatible vtiles, we can easily reuse all the open
source styles Mapbox produced, both MSS and WebGL, or create new ones. The
problem here is that MSS & WebGL are different languages, and keeping them
in sync with Mapbox Studios (Classic & GL) is hard. Richard Fairhurst is
working on Glug[5] to simplify this.  Eventually it could be just the WebGL
style and Kartotherian could render WebGL on the server side if needed.
TBD: styling language

For Wikipedia, we would obviously need a cleaner map, so that editors can
add article-specific info to it. But it could be drawn from the same vtiles
as for osm.

[1] https://www.mapbox.com/mapbox-studio-classic
[2] https://github.com/kartotherian/kartotherian-core
[3] https://github.com/pnorman/ClearTables
[4] https://www.mapbox.com/developers/vector-tiles/mapbox-streets-v6/
[5] https://github.com/systemed/glug

On Sat, Oct 31, 2015 at 11:14 PM, nebulon42  wrote:

> Then there would be the need for some infrastructure, some test rendering
> stack, which does not need to be super fast but should have global coverage.
>



>
> I think it is not possible to port osm-carto and keep the port in sync
> with the current style. So it has to be clear from the beginning that this
> would be a new style (which does not need to be really different visually)
> but nevertheless being separate from osm-carto.
>
> The specification of goals is rather important as using up a lot of
> volunteer effort and having no real outcome would not be good. I know it is
> hard to make promises, that's why it would be good for the project to be
> backed (at least partly) by some kind of grant.
>
> The more time people from the "stylesheet team" would spent on this
> project, the less time there would be for improving osm-carto, which could
> use still some improvements.
>
> If this starts as a grassroots project it would need one or two dedicated
> people to bootstrap it. From there on a team could take it further as was
> done with osm-carto. Otherwise, this should be backed by the OSMF. There
> should be discussion if the OSMF wants it, until when they want it and what
> can be done to support it e.g. if the OSMF could provide the
> infrastructure, donations or could (help) apply for a grant to do it.
>
> Above thoughts are for newly specifying the vector tiles for the OSM stack
> and styling them by using the Kartotherian platform. If you are talking
> about porting osm-carto to an existing vector tile source or about helping
> with improving the already existing vector tiles and style this would of
> course be a lot less effort.
>
> nebulon42
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[nebulon42]

I think it would be good to have a testbed for it that could then evolve 
towards replacing the current stack, but as it is quite a complex and 
ambitious project there should be some sort of roadmap for it, what are 
the goals and how they should be reached.


Some more thoughts from the top of my head (rather from a project 
perspective than from a technical viewpoint):


I'm not sure if this is exclusively something for the stylesheet team 
(what is it anyway, contributors to osm-carto? maintainers?) as the 
vector tiles need to be specified and styled, which is a bit more than 
what is done today since vector tiles form a rather separate re-usable 
component. The vector tile specification will also not only affect one 
style but several styles in the long run. Actually, this would be two 
projects in one.


Then there would be the need for some infrastructure, some test 
rendering stack, which does not need to be super fast but should have 
global coverage.


I think it is not possible to port osm-carto and keep the port in sync 
with the current style. So it has to be clear from the beginning that 
this would be a new style (which does not need to be really different 
visually) but nevertheless being separate from osm-carto.


The specification of goals is rather important as using up a lot of 
volunteer effort and having no real outcome would not be good. I know it 
is hard to make promises, that's why it would be good for the project to 
be backed (at least partly) by some kind of grant.


The more time people from the "stylesheet team" would spent on this 
project, the less time there would be for improving osm-carto, which 
could use still some improvements.


If this starts as a grassroots project it would need one or two 
dedicated people to bootstrap it. From there on a team could take it 
further as was done with osm-carto. Otherwise, this should be backed by 
the OSMF. There should be discussion if the OSMF wants it, until when 
they want it and what can be done to support it e.g. if the OSMF could 
provide the infrastructure, donations or could (help) apply for a grant 
to do it.


Above thoughts are for newly specifying the vector tiles for the OSM 
stack and styling them by using the Kartotherian platform. If you are 
talking about porting osm-carto to an existing vector tile source or 
about helping with improving the already existing vector tiles and style 
this would of course be a lot less effort.


nebulon42


Am 2015-10-31 um 19:43 schrieb Yuri Astrakhan:

Thanks for all the responses! Are there any people on the stylesheet
team interested in working on this?  It might be possible to adapt
osm-carto work directly, since it is based on the same underlying
technology.

Jochen, Kartotherian can dynamically alter/recombine tiles, e.g. adding
layers from multiple sources (e.g. language), or adding raster layers
(both raster and vector data can be stored in the same vtile). So we
could create a set of well-defined layers, stored either together or
separate, and combined on the fly.

Oleksiy, the true multilingual support requires Mapnik hstore support,
see #1113 [1]. Wikimedia DE might be able to help us with it, but it's
still in the air.  Another issue is Mapbox's WebGL cannot shape
connected fonts yet (e.g. Arabic), so they will still rely on the
server-side renderig.

Martin:  If WebGL is a problem for mobile, they can fallback to PNGs
generated by Kartotherian, but we should consider the energy+time needed
to download high DPI PNG tiles vs smaller vector tiles + local
rendering, plus per-client customization aspects (multilingual,
user-centric icons, etc)

Christoph, I agree that our current map is nowhere near as feature rich
as default OSM - we spent most of our time building the tile serving
platform, and spent very little time on the map itself.  We have even
considered using Natural Earth for the low zooms, possibly with an extra
labels layer.

Benjamin, we had to alter Mapbox's Bright to use the free Google Noto
fonts. Can you help with automating WebGL setup & font generation for
Kartotherian? See [2]

Colin, we considered raster-layer combining for languages, but it might
be bad for caching, and would still require a lot of storage due to
per-item overhead. Storing all available languages in one vtile, and
deciding at render time seems more efficient.

Christoph, re multiple projections - if Mapnik implements vector tile
merging, we could in theory dynamicaly recombine tiles and change
projections, would't we? At a performance cost of course.  And we could
store some low-complexity data in the original format, converting it to
the needed tile on the fly, in any projection.

Thanks!!

[1] https://github.com/mapnik/mapnik/issues/1113
[2] https://github.com/kartotherian/osm-bright.tm2


On Sat, Oct 31, 2015 at 6:45 PM, Stadin, Benjamin
> wrote:

Hi Peter,

Most what you said is already 

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Tom Hughes]

On 30/10/15 01:11, Yuri Astrakhan wrote:


At the State of the Map conference, some OSM developers were discussing
the possibility of  migrating osm.org  to the
vector-based backend. I would like to get some feedback on feasibility
and desirability of this effort.


Well I think everybody thinks it's desirable and will happen at some 
point now that there are various options starting to mature as far as 
underlying technology stacks go.


Really though it's up to the stylesheet team - they're the ones that are 
going to have to do a lot of work and only once we have a style would we 
need to consider what stack to run it on.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Oleksiy Muzalyev]

I've heard from the OSM operations team's member at the conference that 
it is the question of the servers' infrastructure cost.


But if we have a vector-based map itself language selection, that we 
could just add tags /name:fr/ or /name: ru/ and have the OSM map of say 
New York in French or in Russian for tourists. Or map of Siberia in 
German also for tourists, Middle East in English, etc. without any 
additions cost on the server side. It is not that difficult to add such 
multi-language tags. There could be also the map in Basque, Catalan, 
Kurdish, Scots and other smaller (by number of speakers) languages 
without any additional cost and without a civil conflict.


I realize that mobile hardware is not enough advanced for that yet and 
vector-based technology is only in an development stage.


brgds
Oleksiy

On 30.10.2015 12:06, Maarten Deen wrote:

On 2015-10-30 11:53, Oleksiy Muzalyev wrote:
One of the advantages of the the vector-based map would be the 
multilingualism.


For instance at the moment the OSM map of the Middle East is basically
useless for me as I do not know the Arabic alphabet yet. But as far as
I understand and as I heard at the conference the vector-based map
would allow the choice of a language of the map itself.


I do not see how that can not be solved with png-based tiles. You only 
have to render the tiles.

The method for detecting which tileset/language to show is the same.

BTW: it is still not as simple as rendering "in a different language". 
Then you start rendering a map in English and see names like "Cologne" 
or "Brussels"  show up on the map.


Regards,
Maarten

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Maarten Deen]

On 2015-10-30 11:53, Oleksiy Muzalyev wrote:
One of the advantages of the the vector-based map would be the 
multilingualism.


For instance at the moment the OSM map of the Middle East is basically
useless for me as I do not know the Arabic alphabet yet. But as far as
I understand and as I heard at the conference the vector-based map
would allow the choice of a language of the map itself.


I do not see how that can not be solved with png-based tiles. You only 
have to render the tiles.

The method for detecting which tileset/language to show is the same.

BTW: it is still not as simple as rendering "in a different language". 
Then you start rendering a map in English and see names like "Cologne" 
or "Brussels"  show up on the map.


Regards,
Maarten

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Christoph Hormann]

On Friday 30 October 2015, Jochen Topf wrote:
>
> This is not a failure of the vector tile approach in general. It
> might be a problem of how the vector tiles are generated today, but
> not a general problem of the vector tile approach. A vector tile can
> contain any data that you need for rendering any kind of style. The
> question is how the data will get into that vector tile. Currently
> the tool chain that does this is not mature enough, I agree. It
> doesn't take into account different sources of data (needed for
> special maps) and doesn't do generalization well enough. But this is
> something we can and should work on.

We need to distinguish a number of different things here:

1. the idea of storing and transporting geometries in a tiled form in 
general
2. the concept of current vector tile stacks
3. the idea of processing geometry on a level that is not generally 
possible to do on the fly during rendering

I was exclusively talking about 2. here, in particular because in 
contrast to the classical non-vector-tile stacks theys are not suited 
to efficiently deal with the geometric complexity issues without 3.  As 
i have mentioned many times i think 3. is important and grossly 
neglected.  Vector tiles might bring an additional incentive to address 
this of course.

The one important thing speaking against tiled geometry storage and 
transport in general is handling of different map projections.  Any 
such system will require you to select one projection from the very 
beginning and you stay bound to it or need fully separate setups for 
every projection you want to offer.

-- 
Christoph Hormann
http://www.imagico.de/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Colin Smale]

 

Can't we have a multi-lingual map by overlaying a base tile with a
transparent text layer in the chosen language? We wouldn't *need* vector
tiles for that, just a bit more storage (bitmap-based text layers should
compress nicely) and clients which can handle the selection and display
of the extra layer, which is pretty commonplace these days anyway). 

--colin 

On 2015-10-30 12:29, Oleksiy Muzalyev wrote: 

> I've heard from the OSM operations team's member at the conference that it is 
> the question of the servers' infrastructure cost.
> 
> But if we have a vector-based map itself language selection, that we could 
> just add tags _name:fr_ or _name: ru_ and have the OSM map of say New York in 
> French or in Russian for tourists. Or map of Siberia in German also for 
> tourists, Middle East in English, etc. without any additions cost on the 
> server side. It is not that difficult to add such multi-language tags. There 
> could be also the map in Basque, Catalan, Kurdish, Scots and other smaller 
> (by number of speakers) languages without any additional cost and without a 
> civil conflict.
> 
> I realize that mobile hardware is not enough advanced for that yet and 
> vector-based technology is only in an development stage.
> 
> brgds
> Oleksiy
> 
> On 30.10.2015 12:06, Maarten Deen wrote: On 2015-10-30 11:53, Oleksiy 
> Muzalyev wrote: 
> One of the advantages of the the vector-based map would be the 
> multilingualism. 
> 
> For instance at the moment the OSM map of the Middle East is basically 
> useless for me as I do not know the Arabic alphabet yet. But as far as 
> I understand and as I heard at the conference the vector-based map 
> would allow the choice of a language of the map itself. 
> I do not see how that can not be solved with png-based tiles. You only have 
> to render the tiles. 
> The method for detecting which tileset/language to show is the same. 
> 
> BTW: it is still not as simple as rendering "in a different language". Then 
> you start rendering a map in English and see names like "Cologne" or 
> "Brussels"  show up on the map. 
> 
> Regards, 
> Maarten 
> 
> ___ 
> dev mailing list 
> dev@openstreetmap.org 
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev ___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Christoph Hormann]

On Friday 30 October 2015, Martin Koppenhoefer wrote:
> >
> > - improved tile serving efficiency
> > - a larger bandwidth of style variations
> > - tighter contraints in basic styling decisions beyond what is
> > already imposed by the OSM data model
>
> some additional points that come into mind:
>
> [...]

Note these are issues with client side rendering, not with vector 
tiles - those are ultimately independent things.  That client side 
rendering is often not very efficient if you consider overall costs is 
a given.

Especially when serving different styles from the same infrastructure 
(as most vector tiles map providers do) there is certainly potential 
for energy saving by caching the vector representations on the server 
side.  If the way this is done with current vector tiles stacks is the 
most efficient approach for that is a different matter of course.

-- 
Christoph Hormann
http://www.imagico.de/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Daniel Koć]

W dniu 30.10.2015 11:10, Martin Koppenhoefer napisał(a):


- vector maps are consuming more energy to display (because have to be
calculated/rendered) -> problem on mobile devices, but also generally


Mobile applications like OsmAnd show that while it really is slower, 
it's still usable.


I would not like to "migrate" osm.org to anything - including vector 
tiles - right now. I would rather like to have vector playground to test 
it and draw some conclusions myself. Yet at this moment I'm not aware of 
such experimental service.


--
"The train is always on time / The trick is to be ready to put your bags 
down" [A. Cohen]


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Christoph Hormann]

On Friday 30 October 2015, Yuri Astrakhan wrote:
>
> Kartotherian , the
> Mapnik+Mapbox-based vector service has been implemented and
> trial-launched  at Wikipedia. The
> service itself is fairly stable, but the styles can use some
> improvements - both the sql->vtile
>  and
> vtile->image .
> Hopefully this work can be used as the basis for the osm.org style.
> Once the vtiles are ready, we can easily move to client side WebGL
> rendering.

From my perspective this, i.e. imposing a certain technological 
framework on designers based on technological considerations, is the 
wrong approach.  I wrote about this on my blog recently from a slightly 
different angle[1].  For a high quality style, design development has 
to mandate the technological framework, not the other way round.

If you look at design problems recently discussed in the osm-carto style 
development[2] you will see most of them have nothing to do with vector 
tiles, they would not be made any easier to address with such an 
approach.  On the other hand there are a multitude of things the 
current style handles fairly gracefully, especially the problem of 
reducing geometric complexity, that would be much harder to deal with 
in a vector tiles system[3].

In general it seems to me vector tiles are today often carried as some 
kind of religious mantra promising to be the solution of all problems 
while in reality they certainly are not.  It is better to look for and 
identify actual design problems and see what technological means are 
available to solve them.  So far use of vector tiles seem to primarily 
have lead to the following effects:

- improved tile serving efficiency
- a larger bandwidth of style variations
- tighter contraints in basic styling decisions beyond what is already 
imposed by the OSM data model

In short: from a design perspective vector tiles so far brought more 
variety in map styles but they ultimately all look very similar beyond 
superficial aspects.  Nearly all of the more unusual maps that 
currently exist are not vector tile based.

[1] http://blog.imagico.de/map-design-economics/
[2] https://github.com/gravitystorm/openstreetmap-carto
[3] For another example of where wikimedia maps fails miserably here 
see:
https://maps.wikimedia.org/#18/47.99579/7.85194
http://www.openstreetmap.org/#map=18/47.99579/7.85194

-- 
Christoph Hormann
http://www.imagico.de/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Oleksiy Muzalyev]

One of the advantages of the the vector-based map would be the 
multilingualism.


For instance at the moment the OSM map of the Middle East is basically 
useless for me as I do not know the Arabic alphabet yet. But as far as I 
understand and as I heard at the conference the vector-based map would 
allow the choice of a language of the map itself.


There are so many disputes about languages, so an ability to just select 
whatever language one wants makes it not just a new feature but speaking 
figuratively an advent of a new brave world.


brgds
Oleksiy




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Martin Koppenhoefer]

2015-10-30 10:52 GMT+01:00 Christoph Hormann :

> So far use of vector tiles seem to primarily
> have lead to the following effects:
>
> - improved tile serving efficiency
> - a larger bandwidth of style variations
> - tighter contraints in basic styling decisions beyond what is already
> imposed by the OSM data model
>


some additional points that come into mind:

- vector maps are consuming more energy to display (because have to be
calculated/rendered) -> problem on mobile devices, but also generally a
problem because every client has to spend energy on calculating the "same"
image (admittedly depends how many different styles there are, and how many
people are looking at them until the underlying data changes), so globally
(i.e. wikipedia use and not some "niche" usecases) this means really a lot
of wasted energy.

- are likely slower to display (for the same reason), although this is
depending on different factors (e.g. if you have internet connection and
how fast it is (vector maps likely scale better for offline use), how
complex the style sheet is, etc.) -> generally vector maps require more
ressources, newer more capable client devices.

Cheers,
Martin
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Jochen Topf]

On Fr, Okt 30, 2015 at 10:52:40 +0100, Christoph Hormann wrote:
> If you look at design problems recently discussed in the osm-carto style 
> development[2] you will see most of them have nothing to do with vector 
> tiles, they would not be made any easier to address with such an 
> approach.  On the other hand there are a multitude of things the 
> current style handles fairly gracefully, especially the problem of 
> reducing geometric complexity, that would be much harder to deal with 
> in a vector tiles system[3].

This is not a failure of the vector tile approach in general. It might be a
problem of how the vector tiles are generated today, but not a general
problem of the vector tile approach. A vector tile can contain any data
that you need for rendering any kind of style. The question is how the
data will get into that vector tile. Currently the tool chain that does
this is not mature enough, I agree. It doesn't take into account different
sources of data (needed for special maps) and doesn't do generalization
well enough. But this is something we can and should work on.

And one more thing: I suggest we start thinking about not one set of vector
tiles, but many. Each one can contain different data (from OSM or not) derived
in different ways. Vector tiles from different tile sets can then be merged
(either in an extra step before delivering to client, or during rendering).
This will give us enormous flexiblity and decouple different parts of the
tool chain to allow for easier experimentation.

As an example: Somebody can have a very specialized process for generalizing,
say, railway infrastructure for low zoom levels. This can then be combined with
the standard tiles for rendering specialized maps without requiring the
maker of the specialized map to have a full OSM tool chain working.

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.jochentopf.com/  +49-351-31778688

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Simon Poole]

Am 30.10.2015 um 11:10 schrieb Martin Koppenhoefer:
>
> 2015-10-30 10:52 GMT+01:00 Christoph Hormann  >:
>
> So far use of vector tiles seem to primarily
> have lead to the following effects:
>
> - improved tile serving efficiency
> - a larger bandwidth of style variations
> - tighter contraints in basic styling decisions beyond what is already
> imposed by the OSM data model
>
>
>
> some additional points that come into mind:
>
> - vector maps are consuming more energy to display (because have to be
> calculated/rendered) -> problem on mobile devices, but also generally
> a problem because every client has to spend energy on calculating the
> "same" image (admittedly depends how many different styles there are,
> and how many people are looking at them until the underlying data
> changes), so globally (i.e. wikipedia use and not some "niche"
> usecases) this means really a lot of wasted energy.
>
> - are likely slower to display (for the same reason), although this is
> depending on different factors (e.g. if you have internet connection
> and how fast it is (vector maps likely scale better for offline use),
> how complex the style sheet is, etc.) -> generally vector maps require
> more ressources, newer more capable client devices.

I don't think anybody is contemplating moving completely away from
server-side tile rendering osm.org at this point in time,  using vector
tiles pre-rendering will likely simply be a further option.

I kind of half agree with Christoph, but I think the perspective that
the vector tiles are a complete replacement for a style specific
rendering database is likely simply wrong. So while vector tiles allow
to produce a group of related styles where you previously simply had
one, if you are doing something completely different you will not be
able to use the same tile set. Naturally, in a wide sense of the word,
there is likely a substantial amount of economic pressure to reuse what
is already there which might lead to really different styles becoming
rarer than they are now . Obviously Andy, Christoph, Richard et al know
a lot more about this than I do. 

Simon


signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Stadin, Benjamin]

One such service would be MapBox [1] itself. And in my opinion
mapbox-gl-js would be a natural choice for the client part. MapBox is
doing a really good job, and this is getting really solid.

I wrote a tiny script to create a mapbox-gl-js package for own deployment
[2]. It doesn't do much useful at the moment, other than doing a basic
host config of the build and generating and packaging some of the required
assets. No custom style integration, and no font generation (I wanted to
add this to the script, so they'd be generated from freetype fonts [3]
using fontnik [4]).

~Ben

[1] https://www.mapbox.com/mapbox-gl-js/examples/
[2] https://github.com/benstadin/mapbox-gl-js-packager
[3] https://fedorahosted.org/liberation-fonts/
[4] https://github.com/mapbox/node-fontnik



Am 30.10.15 12:50 schrieb "Daniel Koć" unter :

>W dniu 30.10.2015 11:10, Martin Koppenhoefer napisał(a):
>
>> - vector maps are consuming more energy to display (because have to be
>> calculated/rendered) -> problem on mobile devices, but also generally
>
>Mobile applications like OsmAnd show that while it really is slower,
>it's still usable.
>
>I would not like to "migrate" osm.org to anything - including vector
>tiles - right now. I would rather like to have vector playground to test
>it and draw some conclusions myself. Yet at this moment I'm not aware of
>such experimental service.
>
>-- 
>"The train is always on time / The trick is to be ready to put your bags
>down" [A. Cohen]
>
>___
>dev mailing list
>dev@openstreetmap.org
>https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Simon Poole]

Am 30.10.2015 um 13:31 schrieb Maarten Deen:
>
> So you have to have some kind of mechanisme to decide when to show
> name:en (which I want to see for e.g. Afghanistan) and when to show name.
>
That is simple, we live with some people not being completely happy :-) *

Seriously the whole point is exposing questionable tag values then they
get fixed, so if Gravelines has name:nl=Grevelingen tagged but in
reality that should be old_name:nl=Grevelingen the best way to get it
fixed is to show the wrong value on the map.

Simon

* even an imperfect language specific map is likely going to make more
people happy than now




signature.asc
Description: OpenPGP digital signature
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Maarten Deen]

On 2015-10-30 12:29, Oleksiy Muzalyev wrote:


But if we have a vector-based map itself language selection, that we
could just add tags _name:fr_ or _name: ru_ and have the OSM map of
say New York in French or in Russian for tourists. Or map of Siberia
in German also for tourists, Middle East in English, etc. without any
additions cost on the server side. It is not that difficult to add
such multi-language tags. There could be also the map in Basque,
Catalan, Kurdish, Scots and other smaller (by number of speakers)
languages without any additional cost and without a civil conflict.


Again, that does not work. I am Dutch. There are a lot of name:nl tags 
in the database that really are too obscure to put on a map.
There is a place "Grevelingen" which we associate with a lake in the 
southwest of the country. But it is also the old dutch name for the 
french town of Gravelines which nobody uses.
But similarly, I don't want to see the name:en tag for of cities 
because, as in my example, I will see Cologne or Brussels or Hook of 
Holland, which are all not what I want to see.
So you have to have some kind of mechanisme to decide when to show 
name:en (which I want to see for e.g. Afghanistan) and when to show 
name.




I realize that mobile hardware is not enough advanced for that yet and
vector-based technology is only in an development stage.

brgds
Oleksiy

On 30.10.2015 12:06, Maarten Deen wrote:


On 2015-10-30 11:53, Oleksiy Muzalyev wrote:


One of the advantages of the the vector-based map would be the
multilingualism.

For instance at the moment the OSM map of the Middle East is
basically
useless for me as I do not know the Arabic alphabet yet. But as
far as
I understand and as I heard at the conference the vector-based map

would allow the choice of a language of the map itself.


I do not see how that can not be solved with png-based tiles. You
only have to render the tiles.
The method for detecting which tileset/language to show is the same.


BTW: it is still not as simple as rendering "in a different
language". Then you start rendering a map in English and see names
like "Cologne" or "Brussels"  show up on the map.

Regards,
Maarten

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Migrating osm.org to vectors/Kartotherian

[Oleksiy Muzalyev]

In Dutch, in English and other languages based on the Latin alphabet 
/Afghanistan/ is spelled about the same. But for example in Greek it is: 
/Αφγανιστάν/, in Ukrainian: /Афганістан/, in Kazakh: /Ауғанстан/, etc. 
Even though people study foreign languages in school, the absolute 
majority speak, read and write realistically only in a native language. 
The name of the country /Afghanistan /even in Latin letters is not 
understandable to a lot of people.


Certainly, there is a lot of imperfection in a language tagging yet, as 
it has not been that important. But if there is a robust comprehensive 
multilingual map solution similar to say the Gettext for text websites, 
then mappers will pay more attention to this type of tags.


brgds
Oleksiy

On 30.10.2015 13:31, Maarten Deen wrote:
So you have to have some kind of mechanisme to decide when to show 
name:en (which I want to see for e.g. Afghanistan) and when to show name.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Migrating osm.org to vectors/Kartotherian

[Yuri Astrakhan]

At the State of the Map conference, some OSM developers were discussing the
possibility of  migrating osm.org to the vector-based backend. I would like
to get some feedback on feasibility and desirability of this effort.

Kartotherian , the
Mapnik+Mapbox-based vector service has been implemented and trial-launched
 at Wikipedia. The service itself is fairly
stable, but the styles can use some improvements - both the sql->vtile
 and vtile->image
. Hopefully this work can
be used as the basis for the osm.org style. Once the vtiles are ready, we
can easily move to client side WebGL rendering.

P.S. Paul Norman has applied

for
a grant to help us with style work. Please add your support and comments to
his proposal.

Thanks!
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev