Re: [OSM-dev] Restarting the EWG

[Paul Norman via dev]

I'm just going to reiterate the call for interested people to contact 
me. I'm not on the board, but restarting or forming a working group 
isn't something that needs to the board to start off


On 2020-11-19 8:09 a.m., Paul Norman via dev wrote:
The OSMF Board is looking at restarting the Engineering Working Group 
with a revised scope to include handling paid software development. 
This scope needs to be developed with existing and new volunteers, but 
my ideas are that it would include


- Google Summer of Code,
- managing development to be paid by the OSMF by contracts and grants, 
and

- collaborating with other organizations for multi-organization grants.

It would do this by by
- placing calls for proposals for paid work on topics like top ten tasks;
- accepting other proposals;
- defining an approximate distribution of OSMF funds for development 
between primary/secondary/tertiary services, external services, and 
new services;
- encouraging applications from skilled individuals who aren't 
professional developers, professional contractors, companies, and others.


Once the scope and funding distribution guidelines are defined we 
would want to start with low-risk projects involving experienced 
people who need less management.


If you are interested in changing the EWG to handle these roles, 
please let me know.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

As Mapbox GL has been un-open sourced, I'm putting this project on hold 
until I figure out a suitable rendering engine.


See https://github.com/pnorman/openstreetmap-cartographic/issues/7

On 2020-05-24 5:34 p.m., Paul Norman via dev wrote:

I've been working on a new project, OpenStreetMap Cartographic. This is
a client-side rendering based on OpenStreetMap Carto. This is an
ambitious project, as OpenStreetMap Carto is an extremely complex style
which shows a large number of features. The technical choices I'm making
are designed so the style is capable of handling the load of osm.org
with minutely updates.

I've put up a world-wide demo at 
https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html,
using data from 2020-03-16, and you can view the code at 
https://github.com/pnorman/openstreetmap-cartographic.


Incomplete parts


Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am
working my way down.

Admin boundaries are not implemented. OpenStreetMap Carto uses
Mapnik-specific tricks to deduplicate the rendering of these. I know how
I can do this, but it requires the changes I intend to make with the flex
backend.

Landuse, vegetation, and other natural features are not rendered until
zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and these
features first appear at zoom 5. There are numerous problems with
unprocessed OpenStreetMap data at these scales. OpenStreetMap Carto gets
a result that looks acceptable but is poor at conveying information by
tweaking Mapnik image rasterizing options. I'm looking for better options
here involving preprocessed data, but haven't found any.

I'm still investigating how to best distribute sprites.

Technology
==

The technology choices are designed to be suitable for a replacement for
tile.osm.org. This means minutely updates, high traffic, high 
reliability,

and multiple servers. Tilekiln, the vector tile generator, supports all
of these. It's designed to better share the rendering results among
multiple servers, a significant flaw with renderd + mod_tile and the
standard filesystem storage. It uses PostGIS' ST_AsMVT, which is very
fast with PostGIS 3.0. On my home system it generates z0-z8 in under
40 minutes.

Often forgotten is the development requirements. The style needs to
support multiple developers working on similar areas, git merge conflicts
while maintaining an easy development workflow. I'm still figuring this
out. Mapbox GL styles are written in JSON and most of the tools
overwrite any formatting. This means there's no way to add comments to
lines of codes. Comments are a requirement for a style like this, so
I'm investigating minimal pre-processing options. The downside to this
will make it harder to use with existing GUI editors like Fresco or 
Maputnik.


Cartography
===

The goal of this project isn't to do big cartography changes yet, but
client-side rendering opens up new tools. The biggest immediate change
is zoom is continuous, no longer an integer or fixed value. This means
parameters like sizes can smoothly change as you zoom in and out,
specified by their start and end size instead of having to specify each
zoom.

Want to help?
=

Have a look at https://github.com/pnorman/openstreetmap-cartographic and
have a go at setting it up and generating your own map. If you have
issues, open an issue or pull request. Or, because OpenStreetMap
Cartographic uses Tilekiln have a look at 
https://github.com/pnorman/tilekiln/issues.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] New minutely replication test

[Paul Norman via dev]

On 2020-11-20 10:08 a.m., Paul Norman via dev wrote:

We are looking to have users test this new feed for correctness to make
sure it works with their update processes and software. To use it
adjust your replication URL from 
https://planet.openstreetmap.org/replication/minute/

to https://planet.openstreetmap.org/replication/test/minute/. We intend
to offer hourly and daily update streams like are provided right now. 


The hourly and daily streams are now available at 
https://planet.openstreetmap.org/replication/test/hour/

and https://planet.openstreetmap.org/replication/test/day/.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] New minutely replication test

[Paul Norman via dev]

One of the services the Operations Working Group (OWG) runs is the
replication feed for OSM changes. This is used to broadcast changes
which are picked up by data consumers. We are running a test of new
software to power the replication feed, osmdbt 
(https://github.com/openstreetmap/osmdbt).

The existing software, osmosis, relies on quirks of PostgreSQL that do
not apply to more recent versions.

We are looking to have users test this new feed for correctness to make
 sure it works with their update processes and software. To use it
 adjust your replication URL from 
https://planet.openstreetmap.org/replication/minute/

 to https://planet.openstreetmap.org/replication/test/minute/. We intend
 to offer hourly and daily update streams like are provided right now.
 When using the new diffs, there are several things to be aware of

- There is no simple relationship between the old sequence numbers and
  the new ones. Use the timestamps to work out what the new sequence
  number is.
- These are experimental and we're trying to find bugs. It might be
  necessary to stop them or run other maintenance.
- The current URL is not a long-term one. It will change.

We are particularly interested in reports from people who have written
custom OSM diff fetching code, as they may find different bugs in
osmdbt or their own code that the standard software does not.

If you have questions or issues, please comment on
https://github.com/openstreetmap/operations/issues/475 or here.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Restarting the EWG

[Paul Norman via dev]

The OSMF Board is looking at restarting the Engineering Working Group 
with a revised scope to include handling paid software development. This 
scope needs to be developed with existing and new volunteers, but my 
ideas are that it would include


- Google Summer of Code,
- managing development to be paid by the OSMF by contracts and grants, and
- collaborating with other organizations for multi-organization grants.

It would do this by by
- placing calls for proposals for paid work on topics like top ten tasks;
- accepting other proposals;
- defining an approximate distribution of OSMF funds for development 
between primary/secondary/tertiary services, external services, and new 
services;
- encouraging applications from skilled individuals who aren't 
professional developers, professional contractors, companies, and others.


Once the scope and funding distribution guidelines are defined we would 
want to start with low-risk projects involving experienced people who 
need less management.


If you are interested in changing the EWG to handle these roles, please 
let me know.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[osmosis-dev] Duplicate keys in replication diffs

[Paul Norman via osmosis-dev]

I believe I've found a bug with how osmosis handles replication diffs,
control characters, and duplicate keys. Because it involves control
characters, this bug can't be seen by viewing the objects in the
browser, so I've supplied shell commands that will show them.

In changeset 90486873 a node was introduced with the tags

   
   k="shelter_type^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?^?" 
v="lean_to"/>


where the ^? are the DEL character (U+0001). This was obviously not
what the user intended, but is valid OSM XML and correctly reproduced
by the API if you query for the node (e.g. with the shell command
`curl -s https://www.openstreetmap.org/api/0.6/node/7881523719/3 | less')

This was transmitted out in the minutely replication feed with sequence
number 004183898. If this is examined with
`curl -s 
https://planet.openstreetmap.org/replication/minute/004/183/898.osc.gz | 
zcat | less -p 90486873'

this reveals the XML

  
  

This XML does not contain the DEL characters, and has two keys which are
the same. We found this because it was resulting in a duplicate key
error, but aside from that Osmosis is producing results inconsistent
with what is in the database.


___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev

[OSM-dev] Planet and ironbelly updates

[Paul Norman via dev]

Last month there were some problems with the weekly planet dump which
impacted the website and API.

Ironbelly, the primary site planet server, was upgraded to Ubuntu 20.04.
This revealed a bug in planet-dump-ng, the software that generates the
weekly planet dumps, which caused it to consume excessive memory,
exhausting resources. Ironbelly also serves as the NFS server for GPX
traces. The resource exhaustion caused NFS to stop responding, leading
to stalled website processes which caused an outage.

A series of actions were taken as a result of this

- The bug with planet-dump-ng was fixed

- Resource limits were placed on the planet dump generation process,
  preventing bugs from consuming all the RAM on the machine

- Planned maintenance was done on ironbelly on August 15th to upgrade
  the kernel and firmware

Medium term the existing plan is to move GPX traces to an object store,
removing the dependency of the website on ironbelly. This is waiting on
Rails 6.1 being released.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Deprecation of Trac and SVN

[Paul Norman via dev]

The Operations Working Group is moving forward with the retirement of 
trac.openstreetmap.org and svn.openstreetmap.org. In August 2018 we 
reached out to some of the developers using 
svn.openstreetmap.org to warn them of the retirement of 
svn.openstreetmap.org.


We will only be creating new SVN accounts if they are necessary for 
transitioning source code away from SVN, and we will be switching Trac 
to read-only at the end of July. SVN will be made read only in August 2020.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

I've been busy with

On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote:

Thank you for making this, it looks like a lot of work!

These client-side vector tiles at z8? 
(https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html)


My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears 
to use some effort to show areas with a large amount of data,
For example if I view England + Wales on z8 and then move over to the 
Netherlands and then to Germany, at z7 and z8,
CPU usage goes up to >100% for a time, and there is a 
noticeable delay. Perhaps part of this is a delay in serving the tiles?


I turned on Content-Encoding based compression which cuts the bytes 
transferred in half. This helps significantly.


In general vector tiles have more large tiles, even if the average tile 
size is the same. The largest z8 tile in the area is 1MB transfer size 
while an average weighted by requests on the tile.osm.org level 2 cache 
is 189k uncompressed. For comparison, four z9 raster tiles from 
tile.osm.org are about 200kb, a fifth of the bytes transferred. Retina 
tiles would decrease the size different.





___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

On 2020-05-25 1:15 a.m., Maarten Deen wrote:
Still, it looks to be a very simplified subset of the complete data. 
So I'd be interested to see how this works on a full dataset.


No, it has the full data except for admin boundaries and bay/straight 
names on zoom 7 and 8. When you compare it to OpenStreetMap Carto 
they're pretty similar. There are some labeling differences with sizes 
of labels and when labels come in, but those don't have anything to do 
with performance or size.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

On 2020-05-25 9:25 a.m., Jason Remillard wrote:

Hi Paul,

The Mapbox GL styles are very low level, and are like the mapnik XML 
input files. The Mapbox GL editing tools available don't seem to 
provide abstractions over the underlying specification.


Yes, the specification and tools are quite clearly not designed for 
human readability. There has been some work at languages that compile to 
Mapbox GL styles, but nothing that supports expressions yet.


Also, rewriting from scratch the main OpenStreetMap Carto style sheet 
is not appealing. There has been a lot of work put into it, and 
whatever happens next, everybody is going to be living with it for a 
long time.


I'm a maintainer of OpenStreetMap Carto so I'm familiar with its 
development. I don't think this is a problem. It's not from scratch, 
it's implementing existing cartography in a new language.


I have done some projects with Mapbox GL with client side rendering, 
and the performance on my tiny style sheet was unacceptable on mobile. 
Having the main style sheet use client side rendering will need to 
wait for newer and faster phones.


Performance with normal basemaps and small stylesheets should be 
acceptable. The question is, how do you get normal sized tiles at low 
zooms from OSM data, how big will the tiles be at high zooms, and will 
the stylesheet be too complex.



Going back to the problems. Long term it is clear that it would be a 
good thing if OSM had a client side rendering option. Also, right now, 
the existing tile servers are overloaded. A victim of the projects 
ongoing success and CartoCSS baked in performance issues.


No, the issues have been with the tile CDN, not the tile servers. 
CartoCSS has never been a performance issue.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote:

Thank you for making this, it looks like a lot of work!

These client-side vector tiles at z8? 
(https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html)


My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears 
to use some effort to show areas with a large amount of data,
For example if I view England + Wales on z8 and then move over to the 
Netherlands and then to Germany, at z7 and z8,
CPU usage goes up to >100% for a time, and there is a 
noticeable delay. Perhaps part of this is a delay in serving the tiles?


Some of the tiles are decently large so could be slow to download, but 
based on your CPU it's not that. More likely it's the high number of 
vegetation and landuse features at those zooms. There are some things I 
might be able to do reduce the feature count.


It would be nice to see a demonstration of this applied server-side to 
produce raster tiles. Are there any samples of that yet?


There are a few examples of producing raster tiles from vector tiles and 
a Mapbox GL style. Mapbox does this with Mapbox GL Native on the server, 
but I don't believe they've released the exact software they use. 
https://github.com/maptiler/tileserver-gl/ is open-source software that 
generates raster tiles on demand.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto

[Paul Norman via dev]

I've been working on a new project, OpenStreetMap Cartographic. This is
a client-side rendering based on OpenStreetMap Carto. This is an
ambitious project, as OpenStreetMap Carto is an extremely complex style
which shows a large number of features. The technical choices I'm making
are designed so the style is capable of handling the load of osm.org
with minutely updates.

I've put up a world-wide demo at 
https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html,
using data from 2020-03-16, and you can view the code at 
https://github.com/pnorman/openstreetmap-cartographic.


Incomplete parts


Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am
working my way down.

Admin boundaries are not implemented. OpenStreetMap Carto uses
Mapnik-specific tricks to deduplicate the rendering of these. I know how
I can do this, but it requires the changes I intend to make with the flex
backend.

Landuse, vegetation, and other natural features are not rendered until
zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and these
features first appear at zoom 5. There are numerous problems with
unprocessed OpenStreetMap data at these scales. OpenStreetMap Carto gets
a result that looks acceptable but is poor at conveying information by
tweaking Mapnik image rasterizing options. I'm looking for better options
here involving preprocessed data, but haven't found any.

I'm still investigating how to best distribute sprites.

Technology
==

The technology choices are designed to be suitable for a replacement for
tile.osm.org. This means minutely updates, high traffic, high reliability,
and multiple servers. Tilekiln, the vector tile generator, supports all
of these. It's designed to better share the rendering results among
multiple servers, a significant flaw with renderd + mod_tile and the
standard filesystem storage. It uses PostGIS' ST_AsMVT, which is very
fast with PostGIS 3.0. On my home system it generates z0-z8 in under
40 minutes.

Often forgotten is the development requirements. The style needs to
support multiple developers working on similar areas, git merge conflicts
while maintaining an easy development workflow. I'm still figuring this
out. Mapbox GL styles are written in JSON and most of the tools
overwrite any formatting. This means there's no way to add comments to
lines of codes. Comments are a requirement for a style like this, so
I'm investigating minimal pre-processing options. The downside to this
will make it harder to use with existing GUI editors like Fresco or 
Maputnik.


Cartography
===

The goal of this project isn't to do big cartography changes yet, but
client-side rendering opens up new tools. The biggest immediate change
is zoom is continuous, no longer an integer or fixed value. This means
parameters like sizes can smoothly change as you zoom in and out,
specified by their start and end size instead of having to specify each
zoom.

Want to help?
=

Have a look at https://github.com/pnorman/openstreetmap-cartographic and
have a go at setting it up and generating your own map. If you have
issues, open an issue or pull request. Or, because OpenStreetMap
Cartographic uses Tilekiln have a look at 
https://github.com/pnorman/tilekiln/issues.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v5.0.0

[Paul Norman via dev]

Dear all,

Today, v5.0.0 of the OpenStreetMap Carto stylesheet (the default
stylesheet on the OSM website) has been released. Once changes are deployed
on the openstreetmap.org it will take couple of days before all tiles
show the new rendering.

Changes include
- An update to Lua tag transforms, setting line vs polygon decisions
  for new tags

- Added upper way_area limits to most features using ST_PointOnSurface
  to avoid performance problems from large polygons

- Moved MSS files into their own directory

- Removed rendering of power=cable features

- Removed overlay pattern for natural=sand

- Reduced landcover fading at mid-low zoom levels

- Removed rendering of barrier=kerb

Thanks to all the contributors for this release.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v4.25.0...v5.0.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v4.22.0

[Paul Norman via dev]

Dear all,

Today, v4.22.0 of the OpenStreetMap Carto stylesheet (the default
stylesheet on the OSM website) has been released. Once changes are deployed
on the openstreetmap.org it will take couple of days before all tiles
show the new rendering.

Changes include

- Shop label fixes and use ST_PointOnSurface for building label placement

  This fixes some bugs and makes building label placement consistent 
with shop

  label placement.

- Use `cache-feature: true` to improve performance of layers with 
attachments


- Use retail colour fill on malls

- Drop `highway=steps` from zoom 13

  This makes step rendering consistent with footways

- Render `place=locality` from zoom 16

  This fits current usage of the tag and what it is normally tagged on.

- Render `natural=bay` from linear ways

- Render administrative boundary labels from relations only

- Stop rendering natural=marsh

  It is recommended marshes are tagged with `natural=wetland` + 
`wetland=marsh`


- Use a whitelist for barrier rendering, and render `historic=citywalls` 
like

  `barrier=city_wall`.

- Support new Tibetan font name

  Noto has renamed Noto Sans Tibetan to Noto Serif Tibetan. The old name is
  still supported.

- Code cleanups to increase reuse and improve consistency


Thanks to all the contributors for this release, including daveol and
btwhite92, new contributors, and jeisenbe, a new maintainer.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v4.21.0...v4.22.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v4.21.0

[Paul Norman via dev]

Dear all,

Today, v4.21.0 of the OpenStreetMap Carto stylesheet (the default
stylesheet on the OSM website) has been released. Once changes are deployed
on the openstreetmap.org it will take couple of days before all tiles
show the new rendering.

Changes include
- Removed unused world_boundaries-spherical.tgz file from scripts

- Switch to osmdata.openstreetmap.de for water & icesheet shapefiles

- Started using ST_PointOnSurface for some label placements

- Adjusted index for military areas

- Adjusted starting zooms for labeling of administrative areas.

- Revert rendering of healthcare key

- Stop place some place labels when the objects become too big or at 
high zooms.


- Only render capes as points and render them like other points.

- Only render ferry lines from ways, not relations

- Improved developer internal documentation



Thanks to all the contributors for this release including Nakaner, a new 
contributor.


For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v4.20.0...v4.21.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Pre-rendered OpenStreetMap Carto tiles

[Paul Norman via dev]

On 2019-02-21 6:06 p.m., Anish Mangal wrote:


Is your new tileset raster or vector? I intend to go about making a 
base OSM tileset in vector, + high zoom regional vector tilesets. Also 
intend to render srtm data as contour+hillshade. I was looking at 
openmaptiles for the same.


I do not find vector tiles and client-side rendering suitable for 
archival purposes. I am confident that in 5-10 years there will be 
software that can display xyz tiles on current computers, and in 30-50 
years PNG tiles will still be readable. I do not expect the same to be 
true for any of the client-side rendering libraries working with an old 
style.


I thought I'd reply to this email if there is a common goal here, or 
whether you have any thoughts on the plans outlined above. The current 
OSM vector tiles (low zoom) made available on the Internet-in-a-Box 
may be seen here - http://iiab.me/modules/en-worldmap-10/map.html



The script I have at https://github.com/pnorman/lz-prerender might be 
useful for you. I'm considering making pg_dump files available, which 
could be useful if you're using something that takes the standard 
osm-carto schema.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Pre-rendered OpenStreetMap Carto tiles

[Paul Norman via dev]

On 2019-01-06 1:21 p.m., Paul Norman wrote:
Many people have asked about downloading low-zoom tiles of 
OpenStreetMap Carto. I’ve completed my project to script automatic 
downloading, importing, and pre-rendering of an OpenStreetMap Carto 
database to make it available for others, and have put the results on 
one of my servers at http://legolas.paulnorman.ca/prerender/



I've moved them to https://pnorman.dev.openstreetmap.org/lz-prerender/.

I want to add some more files to the tarballs documenting how they're 
generated, their contents, how to use them, and other material useful if 
archiving them for a large number of years. I doubt I'll do this soon.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Pre-rendered OpenStreetMap Carto tiles

[Paul Norman]

Many people have asked about downloading low-zoom tiles of OpenStreetMap 
Carto. I’ve completed my project to script automatic downloading, 
importing, and pre-rendering of an OpenStreetMap Carto database to make 
it available for others, and have put the results on one of my servers 
at http://legolas.paulnorman.ca/prerender/


The tar files contain zoom 0 to zoom 6, 8, 9 or 10, depending on the 
file name.


These are most useful for someone who wants to render the world but 
doesn’t need high zooms and doesn’t want to set up a database server. 
Because the tiles are just tiles in a directory structure there’s no 
need for anything except a web server to serve them.


Are these useful for you? Let me know, and I’ll provide them on an 
ongoing basis. These particular files contain OSM data from December 
17th rendered with OpenStreetMap Carto v4.18.0



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v4.12.1

[Paul Norman]

Dear all,

Today, v4.12.1 of the OpenStreetMap Carto stylesheet (the default
stylesheet on the OSM website) has been released. Once changes are deployed
on the openstreetmap.org it will take couple of days before all tiles
show the new rendering.

The sole change is dropping rendering of the surface tag because the method
used was causing unacceptable performance problems. Anyone running v4.12.0
should immediately update, particularly if they have a large database.

You can view the previous changes with v4.12.0 at 
https://github.com/gravitystorm/openstreetmap-carto/blob/master/CHANGELOG.md#v4120---2018-06-22


As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Bolder - Starting a new client-side OpenStreetMap style

[Paul Norman]

On Apr 30, 2018, at 04:03 AM, Christoph Hormann  wrote:

I think that is a good idea - i in particular like that the toolchain 
used seems to be free of NodeJS. ;-)


This was in the back of my mind, but more important to me was being free of 
Mapnik. Mapnik offers great output if generating images, but it's always seemed 
more work than it's worth if you're not using any of those features.


As you know i doubt serious overall mapper feedback is possible at the 
moment with client side rendered vector tiles but the possibility of 
additional QA functionality based on data in the vector tiles is an 
interesting option.


I know your views on simplification and mapper feedback, but I don't find them 
to be issues. I suspect the answer is that we'll have to see to be sure. Even 
if some difficulty in feedback from simplification happens, I feel that QA 
based on data in vector tiles is more valuable. I have to figure out what 
causes a label far more often than I have to look at the precise details of a 
geometry.
 
I have not actually tested it yet but you seem to be using stone age 
Natural Earth data at z0-z5 - which is a big step back from OSM-carto 
IMO.


I didn't find noticeable differences between the Natural Earth and OSM data I'm 
using at these very low zooms
 
Do you have practical experience in deployments with loading external 
data, in particular the coastlines, in PostGIS with regular (daily) 
updates? Certainly this is possible but it is also kind of and extreme 
case of using PostGIS for something it is not intended for.


I've been using the same script with my server demoing the new WMF style work. 
It doesn't run automatically there, but i have run it a lot. It takes about 30 
seconds to load and process the water polygons for the coastline on my home 
server, and that includes clustering and index building. It takes longer to 
download the file.

The script makes sure to create a table optimised for rendering, with the 
contents clustered, indexes created for a read-only case, and some other 
tweaks. This avoids bloated tables.

How do you consider this case extreme?
 
In the client style the method to specify colors seems rather odd to 
me - is this some kind of standard in this field or is this just an 
invention of you?


I've been experimenting with different colour selection methods, and have found 
using a limited colour palette helpful. I want to specify colours in LCH, not 
sRGB, and Tangram scene files, like most style files, only allow direct input 
of the latter.___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Bolder - Starting a new client-side OpenStreetMap style

[Paul Norman]

I managed to forget a very important link, that of the repository: 
https://github.com/pnorman/bolder

On Apr 30, 2018, at 03:06 AM, Paul Norman <penor...@mac.com> wrote:

This is a copy of my diary entry, which has a picture: 
https://www.openstreetmap.org/user/pnorman/diary/43814

I’ve started work on a new client-side style for OpenStreetMap data, and feel 
it’s reached the point where I can release it to the public. My goal is to make 
a style that shows a rich selection of the data OSM has, and to make use of 
most of the colour space, rather than a style designed for overlaying other 
data on top of.

As a new style, I’ve been able to approach a lot from scratch, looking at 
avoiding mistakes of previous projects, and using best practices while building 
on existing work. All the components are open-source, and no assumptions are 
made about using closed-source software or particular commercial solutions.

Technical overview
The style is rendered with Tangram, which allows for client-side rendering. 
Server-side rendering is possible but is a secondary target. Closely coupled 
with the client-side style is a set of vector tile definitions, handled by 
Tegola, a vector tile server. It pulls from an osm2pgsql database in the 
OpenStreetMap Carto schema, with additional data like ocean polygons loaded in 
by a script.

Cartographic target
The goal of Bolder is to be a general-purpose style, filling a target similar 
to OpenStreetMap Carto, while also being a better “default” for people wanting 
an OSM map. Being a client-side style, it’s easier to turn off classes of 
features like some POIs if a map with fewer features is needed.

The style should still be useful for mapper feedback, and some ways will become 
more useful. Vector tiles can associate OSM feature IDs with objects in many 
cases, helping debugging “where did that label come from”.

Setup
The style has two arts that are installed, one for the vector tiles, and the 
other for displaying the client-side style. The documentation for both of them 
has been tested by users who hadn’t seen it before, so it should be possible to 
set up for anyone reasonably experienced in style authoring.

Limitations
As a new project, Bolder has limitations. The biggest limitation is that only a 
small number of features are rendered, and many things have to be added. I’ve 
also been doing lots of new stuff with Tegola, and have uncovered a number of 
critical bugs, most of which should be fixed next Tegola release.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Bolder - Starting a new client-side OpenStreetMap style

[Paul Norman]

This is a copy of my diary entry, which has a picture: 
https://www.openstreetmap.org/user/pnorman/diary/43814

I’ve started work on a new client-side style for OpenStreetMap data, and feel 
it’s reached the point where I can release it to the public. My goal is to make 
a style that shows a rich selection of the data OSM has, and to make use of 
most of the colour space, rather than a style designed for overlaying other 
data on top of.

As a new style, I’ve been able to approach a lot from scratch, looking at 
avoiding mistakes of previous projects, and using best practices while building 
on existing work. All the components are open-source, and no assumptions are 
made about using closed-source software or particular commercial solutions.

Technical overview
The style is rendered with Tangram, which allows for client-side rendering. 
Server-side rendering is possible but is a secondary target. Closely coupled 
with the client-side style is a set of vector tile definitions, handled by 
Tegola, a vector tile server. It pulls from an osm2pgsql database in the 
OpenStreetMap Carto schema, with additional data like ocean polygons loaded in 
by a script.

Cartographic target
The goal of Bolder is to be a general-purpose style, filling a target similar 
to OpenStreetMap Carto, while also being a better “default” for people wanting 
an OSM map. Being a client-side style, it’s easier to turn off classes of 
features like some POIs if a map with fewer features is needed.

The style should still be useful for mapper feedback, and some ways will become 
more useful. Vector tiles can associate OSM feature IDs with objects in many 
cases, helping debugging “where did that label come from”.

Setup
The style has two arts that are installed, one for the vector tiles, and the 
other for displaying the client-side style. The documentation for both of them 
has been tested by users who hadn’t seen it before, so it should be possible to 
set up for anyone reasonably experienced in style authoring.

Limitations
As a new project, Bolder has limitations. The biggest limitation is that only a 
small number of features are rendered, and many things have to be added. I’ve 
also been doing lots of new stuff with Tegola, and have uncovered a number of 
critical bugs, most of which should be fixed next Tegola release.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] about Carto and SQL

[Paul Norman]

On 3/20/2018 1:40 PM, sav123 wrote:

Kompza covered the variables you mentioned, but I if you're looking at

benchmarking, I recommend you set up the full rendering stack rather
than trying to generate queries yourself. The latter can be tricky to
get right, and you have to handle parallelism the same as the full stack
to get meaningful results. It's just easier to use renderd and feed it a
list of tiles with render_list than to write code that will take that

list, generate SQL, and run it.


This doesn't answer the question.
If you read the carto file, you will see that there are natural 
criteria , not only geometry.
Indeed, it is a bad idea to partition on coordinates, unless if a 
request applies to a small region. 


I'm not speaking of partitioning, but of how to best benchmark 
performance, which is what you're planning to do to see if partitioning 
is worthwhile.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] about Carto and SQL

[Paul Norman]

On 3/20/2018 9:13 AM, sav123 wrote:
I hope to be able to grab enough information to partition some or all 
the postgresql tables and to publish the resulting comparison boards.


I know that it is possible to setup logs and to check them, but this 
method may miss rare calls ...


Kompza covered the variables you mentioned, but I if you're looking at 
benchmarking, I recommend you set up the full rendering stack rather 
than trying to generate queries yourself. The latter can be tricky to 
get right, and you have to handle parallelism the same as the full stack 
to get meaningful results. It's just easier to use renderd and feed it a 
list of tiles with render_list than to write code that will take that 
list, generate SQL, and run it.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] GSOC Project: "Make the website use the API"

[Paul Norman]

I wrote a blog post on how to get started with the API projects: 
http://paulnorman.ca/blog/2018/02/make-the-website-use-the-api-gsoc-project/


I recommend steps 2, 4, and 5 for anyone applying for a project which 
interacts with the API.



On 2/13/2018 9:08 AM, IMT2016050 Biswesh Mohapatra wrote:
Hello, My name is  Biswesh. I am a second year Computer Science 
student and am very keen on contributing to OSM for GSOC 2018. I have 
been looking into the idea list for GSOC 2018 and found out the 
project - “ Make the website use the API “ proposed by Paul Norman 
<https://wiki.openstreetmap.org/wiki/User:Pnorman> to be very 
interesting as the concept can be very useful for OSM and also because 
it seems to be a bit challenging. I have some prior knowledge of 
Javascript, Ruby on Rails and also have some basic knowledge of REST api.


Although I don’t have much knowledge about implementing the idea, I 
searched a bit about it and found out some useful articles which 
showed some advantages of changing the website to rely on API calls 
instead of directly accessing the database. I have given the links here:


http://solnic.eu/2011/08/01/making-activerecord-models-thin.html

http://jamesgolick.com/2010/3/14/crazy-heretical-and-awesome-the-way-i-write-rails-apps.html

I have also searched a bit about implementing the project and found 
out that the following can be useful:


Active rest client - https://github.com/whichdigital/active-rest-client

Active resource - https://github.com/rails/activeresource

I would like to be guided on this project idea so that I can prepare 
myself better for it. Also if I could be provided some better sources 
through which I can have a better understanding of the problem then I 
would be grateful.


On a side note - I would also like to ask if we can explore about 
migrating the entire rest based workflow to some of the emerging 
technologies like graphQL.


My GitHub account:
https://github.com/biswesh456


Regards
Biswesh Mohapatra


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Synchronous mode

[Paul Norman]

On 2/20/2018 12:07 PM, Darafei "Komяpa" Praliaskouski wrote:
This suggests that currently apidb replicas aren't in synchronous 
mode. Can it be switched to synchronous?


https://www.postgresql.org/docs/9.5/static/warm-standby.html#SYNCHRONOUS-REPLICATION 



Given the absolutely miserable performance of waiting for the data to 
hit the disk on replicas, this is a bad idea and not normally 
recommended, although we could use remote_apply to reduce some of the 
problems.


The data loss potential is very minimal, particularly since anything 
that brings down the DB server is likely to cause much larger 
interruptions to mapping than the loss of 5 seconds of PostgreSQL 
replication.


It's also not an issue for an editor. The API provides a diff response, 
so the editor doesn't need to make another request.


In fact, this is the first API call I've seen where it is a potential 
issue, and we don't even know if it is one for sure. If MapRoulette is 
doing this as a batch process, it's probably easier to use the changeset 
diffs anyways.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Read-only API calls, fair usage

[Paul Norman]

On 2/20/2018 2:56 AM, Frederik Ramm wrote:

Hi,

On 20.02.2018 10:51, Michał Brzozowski wrote:

You can use changeset replication with ChangesetMD to query your own DB.

I was tempted to respond the same, but I imagine the idea is that as
soon as the user clicks on the "I have fixed this" button, MapRoulette
checks what has been committed... and replication could mean up to one
minute delay.


There's no guarantee that the read-only changeset call will go to the 
same backend server as the write call. 
https://github.com/openstreetmap/iD/issues/1646 shows the problem caused 
by iD not parsing the diff response and relying on redownloading.


In the normal editing workflow a 5s (or even 60s) delay is not an issue. 
Someone is likely to spend longer than that in their editor changing data.


I would say this is a reasonable use of the OSM API, as it is part of 
editing, but is not the best for technical reasons. If you don't need up 
to the second changeset data, you can consume the diff stream every 
minute. If you do need up to the second data, then the API doesn't 
guarantee it.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Working with OSM data with less or no metadata

[Paul Norman]

On 2/14/2018 8:17 AM, Roland Olbricht wrote:


- How about simply asking the users for consent? We could then
-- make a clear-cut last complete history dump before the date
-- start with a planet dump without history before that date 
afterwards that then accumulates history only from users that have 
given consent 


Consent is revocable. If we didn't have to deal with people revoking 
consent and account deletion requests, it would all be much easier.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Extra-large metatiles

[Paul Norman]

I'm working on a script to do some pre-rendering of a Mapnik-based tile 
layer. Conventional meta-tile sizes would be 4x4 or 8x8, but these are 
selected to balance latency and throughput or efficiency. I'm using 
mapproxy, which stores the tiles as individual files on-disk, not 
mod_tile, which stores metatiles. This means that I can easily change 
meta-tile size, and I don't even have to worry about differing latency 
from slicing PNGs.


Latency does not matter with a pre-render task, so I've been considering 
using extra-large metatiles. Has anyone tried this before?


Based on experience from metatiles 1-8 tiles across, as size increases I 
expect query time and time to render to increase, but time per tile to 
decrease. I don't know at what point this stops being true.



I believe Mapnik has problems with very large images, but I'm not sure 
when that happens. I think at least 16k px wide is doable.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSMand Live can steal your money

[Paul Norman]

On 1/12/2018 6:03 AM, Andy Allan wrote:

In general, I'd like to disable HTTP Basic Auth to our API, and only
use OAuth. This removes any need to share your OSM password with third
parties. However, developers often find it easier to build
integrations using basic auth, so I can imagine some opposition to
this.


Do we need some terms for the API covering this kind of stuff? Right now 
it's not clear that a service that stores your OSM password server-side 
is violating anything.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Advantages PostgreSQL 10.0 + PostGIS 2.4

[Paul Norman]

On 10/8/2017 11:03 AM, Sven Geggus wrote:

Hello,

Debian stable 9.x comes with PostgreSQL 9.6 and Postgis 2.3.

However the now released PostgreSQL 10.0 and Postgis 2.4 are an easy
backport.

Will the newer version provide any advantage for a osm2pgsql database use case?


Yes, but nothing game-changing for a osm2pgsql database and a standard 
production workload according to the release notes. There's a lot of 
improvements related to replication. The following are likely to help


- Faster GiST index inserts and updates
- Faster GIN index vacuuming
- General performance improvements

The parallel b-tree index usage could help updates, but this is seldom 
an issue, and may even be undesirable under heavy rendering load. I've 
also seen no data on if it helps or not for the particular queries 
osm2pgsql runs.


In short, I'd recommend the upgrade for any new deployments, but it's 
not likely to be worth an out of schedule reload.


If you're doing analysis with an osm2pgsql database, it's likely to be 
more useful. Query-level parallelism is more useful there because 
there's generally not so many connections running in parallel at the 
application level. This makes the parallelism improvements useful, if 
they apply to the types of queries that are being run.


The multi-column statistics have the potential to be interesting for 
analysis queries.


If upgrading to 10.x, make sure to set the parallel configuration 
options if you want to see those performance gains.


PostGIS 2.4 is a similar situation. Stylesheets are unlikely to require 
it yet, so it won't enable anything new for rendering servers for some 
time. If a new function helps with your analysis use-case, it will be 
useful.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql 0.94.0 release candidate

[Paul Norman]

On 9/18/2017 2:03 PM, Paul Norman wrote:

We're looking at releasing osm2pgsql 0.94.0 soon and could use testing
of the 0.94.0-RC1 version. 


We've tagged 0.94.0-RC2. This fixes a problem where an invalid geometry 
could end up in the tables if it was valid in EPSG 4326. This is a 
particular problem at the edge of projections.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql 0.94.0 release candidate

[Paul Norman]

We're looking at releasing osm2pgsql 0.94.0 soon and could use testing
of the 0.94.0-RC1 version. Testing from packagers is appreciated,
particularly if you are doing something different with libosmium.

Major changes

- Store unprojected coordinates in slim tables and use osmium dense file
  array for flat nodes

  This fixes a number of projection-related issues

- Use libosmium for geometry building instead of geos

  This offers speed increases, improves code, and avoids relying on a
  large library for a small portion of its functionality. As a
  consequence of this change, invalid geometries are no longer ever
  created or fixed

Changes

- Rewrite of tile expiry

  A bug in the old code silently dropped a significant portion of tiles.
  This has been fixed, increasing the size of the expiry list. For
  expiry arguments that cover multiple zooms, the behavior has been
  changed. This is best visualized with
https://github.com/openstreetmap/osm2pgsql/issues/709#issuecomment-302674031
  rather than trying to explain it in text. There is more discussion in
  the issue.

- Don't store node tags in slim tables

  This reduces space requirements when importing with --slim and no
  --flat-nodes or --drop

- Don't assume a default database name of gis

  If you want this database, either add -d gis to your command line or
  set PGDATABASE.

- Various bugfixes with processing relations, tag transforms, and 
related issues


Future changes

We expect this to be the last osm2pgsql release that supports old-style
multipolygons with the "C" transform.

Overall this release should be faster, with a 15% performance increase
on CPU-bound machines

You can see a complete list of commits at 
https://github.com/openstreetmap/osm2pgsql/compare/0.92.1...0.94.0-RC1



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v4.3.0

[Paul Norman]

Dear all,

Today, v4.3.0 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released.

Changes include

- Moving ford and emergency phone to a new tagging scheme
- Moving natural=tree to higher zoom level (z18+)
- Changing embassy color to brown
- Rendering name for waterway=dock
- The same line wrap of amenities for all zoom levels
- Fixing combined railway/highway ordering regression
- Fixing line wrapping bug in Docker
- Some documentation and code cleaning
- Improve ferry line text legibility
- Hide small theme parks and zoos
- Use solid lines for admin borders at low zooms

Thanks to all the contributors for this release, including stevenLAD, a
new contributor.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v4.2.0...v4.3.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Questions regarding carto 4.0 release and German style fork

[Paul Norman]

On 5/28/2017 1:59 PM, Paul Norman wrote:
This is not supported, so you'll have to figure out the details 
yourself. I recommend writing a script that reads in two osm2pgsql 
.style files.


I should add that this was a conscious decision by the osm-carto 
developers, and we discussed a number of options before settling on what 
we're doing as the best option.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto v3.3.0 release

[Paul Norman]

Dear all,

Today, v3.3.0 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released.

This may not be immediately rolled out to the openstreetmap.org servers,
but that is up to the OSM sysadmin team, not the openstreetmap-carto
maintainers.

Changes include
- Most shops are now rendered as dots z17 to deal with overcrowding
- Font selection is moved to its own file to make customization easier,
  and to make it easier for other styles to reuse our font work
- Rare CJK characters outside the BMP should now render better
- Waterway tunnels in forests and lakes are clearer

This version is unusual because the next version scheduled for release
is v4.0.0, which will have the same cartography as this version.

Thanks to all the contributors for this release.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v3.2.0...v3.3.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto schema change updates

[Paul Norman]

We've decided to drop support for old-style multipolygons in 
OpenStreetMap Carto 4.0.0. This has resolved our outstanding issues, and 
we now feel the work is ready to merge. The big changes are


- database schema change;

- disjoint area handling;

- different tag columns; and

- multipolygon handling changes

Issues should be raised at 
https://github.com/gravitystorm/openstreetmap-carto/pull/2533



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql minor releases

[Paul Norman]

osm2pgsql 0.92.1 and 0.90.2 have been released.

0.92.1 contains two changes from 0.92.0

- A fix to the large relation bug encountered in March and documented in 
https://lists.openstreetmap.org/pipermail/dev/2017-March/029770.html. 
This is an important fix if you are using minutely updates, and still 
could impact users with normal imports.


- A fix to a critical data loss issue using updates with the multi 
backend (--append --output multi). If you were doing so, you should 
update and reimport your database. If you aren't using both the multi 
backend and append this does not affect you.


0.90.2 contains the above fixes, plus minor accumulated code fixes.

If you are on Debian Stretch a new version has been packaged with the 
fixes. If dpkg -l osm2pgsql reports you have 0.92.0+ds-2 then you have 
the bug fixes.


As always, you can get the code, documentation, and report issues for 
osm2pgsql at https://github.com/openstreetmap/osm2pgsql



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v3.2.0

[Paul Norman]

Dear all,

Today, v3.20 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released.

Changes include

- Render aeroway terminal buildings like other buildings
- Removed rendering of landuse=farm
- Added rendering for arts centre, fitness centre, plant nursery, mixed
  lift aerialways
- Rendering for fens changed
- Typography for point road-related features, addresses, and water
  features changed
- Removed rendering of waterway=canal as an area
- Take text properties of roads under construction from the type of road
  they will be

Thanks to all the contributors for this release including Richard Fairhurst
and jnachtigall, new contributors.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v3.2.0...v3.1.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] PHP vs. Ruby in OSM projects

[Paul Norman]

On 3/21/2017 8:02 AM, Hakuch wrote:

On 21.03.2017 02:50, Paul Norman wrote:

On 3/20/2017 9:22 AM, Hakuch wrote:
It depends. For projects that produce webpages, there's probably more
Ruby activity than PHP activity. Python is also pretty popular.

did you want to say "projects that do NOT produce webpages" ? Then I
would agree with you, because usually you only do webpages with PHP


No - this was about projects which produced web pages, with the second 
paragraph about projects which involved APIs over HTTP(s). Given the 
initial question mentioned rails, I focused on stuff which is accessible 
in some way over HTTP(s).


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] PHP vs. Ruby in OSM projects

[Paul Norman]

On 3/20/2017 9:22 AM, Hakuch wrote:

What do you think, are there more php or more ruby developers in OSM ?
Normally within open source projects people tend to use ruby, but many
times I saw small projects coded in php. Iam currently unsure if I
should start a new project with php/symfony or ruby/rails.


It depends. For projects that produce webpages, there's probably more 
Ruby activity than PHP activity. Python is also pretty popular.


For web-based APIs, I don't see PHP often used in OSM. Ruby on rails and 
Python are both common, and you also see Go, nodejs, and C++.


All of these languages have advantages and disadvantages and none of 
them is appropriate for everything.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] API project idea

[Paul Norman]

On 3/18/2017 2:23 AM, Subhani Munasinghe wrote:

Hi everyone,

Can someone please tell me how to get contacted with a mentor of 
the project idea "Make the website use the API". I would like to know 
more about that project idea, because I would love to work with Java 
since I am skilled with it.


I'm the person who has suggested it. The project would mainly be 
Javascript and Ruby, not Java.


If you're still interested I would recommend you:

1. Register for osm.org and map a bit. The editors use the API, so you 
need to know how to use them.
2. Start JOSM from the command line to have a console open, and edit 
some more. This will show the API calls used
3. Read the parts on https://wiki.openstreetmap.org/wiki/API_v0.6 about 
element GET calls

4. Read http://www.asklater.com/matt/blog/2015/11/18/the-road-to-api-07/
5. Use the various methods to get object information. This includes the 
pages like https://www.openstreetmap.org/way/ but also the history 
view in JOSM (View->History) and 
http://osmlab.github.io/osm-deep-history/. The pages on 
openstreetmap.org don't use the API but the JOSM tool and osm deep 
history do. The goal of this project is to make osm website pages use 
the API.


Hopefully that's a good start for you or anyone else interested in it.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] GSoC API mentoring help needed

[Paul Norman]

On 2/20/2017 12:19 AM, Andy Allan wrote:

I can see the purpose of this, but I've never seen it as being as high
a priority as other developers do.


For me the concerns all stem from code duplication, principally leading 
to more optimization work on one path, so cgimap is much faster than 
browse pages. You can see a good example of this with relation history 
pages that time out while the cgimap powered API call is much faster.


I do consider this less important than my other API proposals, but I 
have students interested in it



For example, I can see why the
browse pages shouldn't have access to the data in a manner that's not
exposed in the API, but that to me suggests improving the API, rather
than forcing a lowest-common-denominator approach of forcing the
browse pages to use the API.


I'm not advocating reducing the functionality of the browse pages. Part 
of the work with this is identifying what is lacking in the API. I hope 
to propose a new call to start the discussion before GSoC.



I would avoid the pure-javascript approach, as it's just rewriting for
the sake of rewriting. My suggestion would be to just change the
existing browse pages code slightly - the controllers should make
(internal) calls to the API endpoints, to replace their direct db
access. Even better if those internal API calls are processed by
cgimap-ruby!


As I see it, either way accomplishes the goal of having the requests for 
object information go through a common path and come with tradeoffs.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] GSoC API mentoring help needed

[Paul Norman]

Much to my surprise, I have 2-4 students issued in my API-related GSoC 
proposals. This is more than one person or even two people can mentor, 
so I'm asking for help. The cgimap-ruby, cgimap read-only support, 
cgimap write support, and make the website use the API


Some of the proposals involve cgimap, and I'm probably the best suited 
to mentor those because I'm one of the three significant cgimap 
contributors. Instead, there's two proposals well suited to someone else.


1. cgimap-ruby

I don't yet have a student interested in this, but I'd like to see if 
one of the ones who has contacted me is. This could use a mentor who has 
dealt with ruby gems before, which I haven't. I have a feeling this part 
of the work isn't enough for a full project, so it could pull in 
something from a different API-related proposal.


2. website from API

This is a project to change parts of the website to call the API instead 
of the database for object information. Good candidate pages would be 
the object browse ones (/node/, etc). There are two different 
technical approaches to this, and depending on which route a student 
takes, the mentor might need different skills. The first of these is to 
do the processing of API results on the server and return HTML to the 
client, like http://osm.mapki.com/history/ does. The second is to do the 
processing of API results on the client with Javascript, like 
http://osmlab.github.io/osm-deep-history/ does. For the first, the 
student would be reproducing existing HTML output so the mentor would 
need knowledge of ruby and API calls. For the second, client-side 
javascript would be needed, but less ruby.


If you're interested and available to help mentor, please contact me 
off-list.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql default database change

[Paul Norman]

The behavior of osm2pgsql when no database is specified is changing in 
the next release. New releases will rely on libpq to pick the database 
if nothing is explicitly specified, typically either resulting in a 
database of what the PGDATABASE environment variable is set to, or the 
user's name.


If you are writing scripts that call osm2pgsql and rely on the default 
behavior, you should add --database gis to your command line. This is a 
good practice, even if you don't plan on updating.


Details on the change are at 
https://github.com/openstreetmap/osm2pgsql/pull/685



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] GSoC 2017

[Paul Norman]

On 2/1/2017 10:34 PM, Ronak Jain wrote:
After going through the list of possible idea's for this summer, I 
would like to consider

Full cgimap write support
Full cgimap read-only support
I have chosen these as I have good experience with the languages being 
used in the project also I have worked with Postgres earlier. I will 
choose the idea after speaking with the mentor and deciding what could 
be additionally added to the project.


I would like to know,
how can I get started with the project?


I would start by reading 
http://www.asklater.com/matt/blog/2015/11/18/the-road-to-api-07/. This 
places the GSoC work in the context of the problems and other work needed.


Then, as you've mentioned, map a bit. All the editors interact with the 
API. I would start with iD, the default web-based editor, but then try 
JOSM, which is a java-based desktop editor. It is much easier to 
investigate the API calls with JOSM.


After mapping normally a bit, look over 
https://wiki.openstreetmap.org/wiki/API_v0.6, in particular the Elements 
part (2.4) and Changesets (2.2). Then start JOSM from the command line 
so that there's a java console and map some more, looking at the API 
calls JOSM makes, and how they related to what you're doing.



are there any bite-sized project or bugs that could be fixed?


https://github.com/zerebubuth/openstreetmap-cgimap is the repository for 
cgimap, and it's issue tracker is on GitHub. #90 and #84 seem like easy 
ones where most of the work will be getting familiar with the project.


https://github.com/openstreetmap/openstreetmap-website is the other 
project to be familiar with (It's called "The Rails Port"). It powers 
the website, and for various reasons, all the API calls cgimap does are 
also implemented in it. cgimap-ruby aims to end this duplication.



what are the steps I could take to increase my acceptance rate?


Asking in advance is good, as many students don't ;)

Working on a plan would be good. I recommend giving some consideration to

- Which API calls you will work on
- How you will test the cgimap part
- How you will test that cgimap and the rails port have the same behavior

None of these need to be complete, but having given some thought to what 
to do puts you well ahead of most potential students.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto database schema change

[Paul Norman]

One of the long-running OpenStreetMap Carto projects has been a database 
schema change, 
https://github.com/gravitystorm/openstreetmap-carto/pull/2533. Included 
in this are


- database schema change;

- disjoint area handling;

- different tag columns; and

- multipolygon handling changes

This is now in a state where the output would benefit from review, not 
just the code, and I have set up a demo at 
https://lua.osm-carto.paulnorman.ca/. Please read the issue and provide 
feedback on Github.


Feedback is especially needed from people who deploy OpenStreetMap 
Carto, as the database schema change requires a database reload. To 
allow people to switch over, cartographic compatibility between the 3.x 
and 4.x series will be maintained for some time. The time has not yet 
been determined.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v3.1.0

[Paul Norman]

Dear all,

Today, v3.1.0 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released.

Changes include

- Added coffee shop rendering
- Added health clinic rendering
- Adjusted place label typography
- Road shield rendering improvements
- Internal code cleanups

Thanks to all the contributors for this release.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v3.0.1...v3.1.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] size of postgresql database - without flat file

[Paul Norman]

On 1/28/2017 3:00 PM, Walter Nordmann wrote:


i'm just installing my brand new server to get my applications 
(missing boundaries and many more) running again. The server has 2x 
960GB SSD and 1x 2TB SATA disk installed.


I'm doing a full planet import using osm2pgsql 0.92. This time i'm 
doing the import without flat file, because i would like to use the 
nodes in planet_osm_nodes for some new queries.


But: planet_osm_nodes got an actual size of 785 GB, which is much much 
more than i exspected. Is that the usual size or may be, i'm doing 
something wrong? 


With the version of osm2pgsql you have when --extra-attributes is 
specified then tags are added to each object with the username, uid, 
version, timestamp, and changeset. This adds about 100 bytes per node to 
the table, which should add about 350GB. I would expect the size of 
planet_osm_nodes with a plain --slim import to be about 175GB, plus indexes.


Does your 785GB figure include indexes? If so, this sounds about right.

The latest versions of osm2pgsql do not have this issue because the node 
slim table schema has changed to no longer store tags, because they are 
never needed.


It's worth noting that 100 bytes per object for metadata is remarkably 
inefficient because it gets stored in a text[] array like 
"osm_user,lokejul,osm_uid,2034065,osm_version,11,osm_timestamp,2015-04-12T18:39:22Z,osm_changeset,30169891". 
This uses more bytes because it stores "tag" names, and because there 
are more efficient ways to store numbers and timestamps than as text.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] tile generation performance

[Paul Norman]

On 1/12/2017 12:29 PM, Joseph Armbruster wrote:

Do you or anyone else know someone that has performed tile generation
for some of the lower levels / scales recently, say for levels 15 -
20?


Higher zooms (12+) typically take about 1 second per 8x8 metatile per 
CPU thread. 15+ tiles are not pre-rendered for the world, they are 
rendered on demand and cached. The OSMF pre-renders z0-z12 tiles, and it 
takes about a day to do this.


Depending on your hardware, these times could easily change by a factor 
of 2x.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Incoming osm2pgsql change without migrations

[Paul Norman]

On 1/6/2017 11:28 AM, Walter Nordmann wrote:

Hi Paul,

in other words: because i'm using the diff-process i have to reload 
the full planet once, right?


When will this osm2pgsql be availiabe?

I'm asking because my osm database server has broken some days ago and 
starting end of next week i must reload the full planet to get an 
actual live database.


Of course this should be done by the newest version.



The changes have been merged to master, and we have no ETA on when we 
will release a new version. There's a good chance we'll break backwards 
compatibility of the internals in some other way before release. It's 
good to have all those changes in one release.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Incoming osm2pgsql change without migrations

[Paul Norman]

There is an incoming change to osm2pgsql which changes the format of 
slim tables. This will change the schema of the slim tables, but have no 
impact on the rendering tables.


The change is to use unprojected coordinates in slim tables, and an 
osmium dense file array instead of flat nodes. There is no migration for 
old databases.


It only impacts you if you

- Run an osm2pgsql database that updates with diffs; or
- Develop software which relies on the internals of the slim nodes table 
or the flat nodes file (don't do this)


Nothing will change if you

- Only ever load data with --slim --drop or without --slim;
- Only ever update by reloading the database; or
- Only write software/styles which access the normal rendering tables

The pull request with details is 
https://github.com/openstreetmap/osm2pgsql/pull/668


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap Carto release v3.0.0

[Paul Norman]

On 12/22/2016 4:57 AM, Sven Geggus wrote:

Christoph Hormann  wrote:


No, the new project.mml is just a renamed project.yaml.

*argh*

So it would be best to do a git mv project.yaml project.mml before trying to
merge?


On our side project.yaml was moved to project.mml in one commit by 
itself, so it worked fairly well with git when I had to merge the 
changes into our lua branch.


I'd have rather kept the name the same, but CartoCSS requires a MML 
suffix for its layer definition, regardless of if its JSON* or YAML.


* Strictly speaking all MML files are YAML now since JSON is a subset of 
it and they're passed through a YAML parser, not a JSON one.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v3.0.0

[Paul Norman]

Dear all,

Today, v3.0.0 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released.

Major changes include

- Mapnik 3 is now required
- CartoCSS 0.16.x is now required
- Official Tilemill support is dropped
- Shapefiles are downloaded with a new python script

Changes include

- Noto Naskh is now used for Arabic
- Visual impact of campsites and quarries reduced below z13
- Wilderness huts rendered
- Subway entrances rendered

Thanks to all the contributors for this release including jojo4u, a new 
contributor.


For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v2.45.1...v3.0.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues

See also http://www.openstreetmap.org/user/pnorman/diary/40114 for diary 
version



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql 0.92.0-RC1 release

[Paul Norman]

On 12/7/2016 2:57 PM, Paul Norman wrote:
osm2pgsql 0.92.0-RC1 has been tagged in preperation for a release. 
Feedback is welcome, in particular from downstream packagers.


You can get the release from git or 
https://github.com/openstreetmap/osm2pgsql/releases/tag/0.92.0-RC1. 


0.92.0 has been tagged and released.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Turning boundaries to linestrings

[Paul Norman]

On 12/13/2016 9:04 AM, Martin Koppenhoefer wrote:
that's a nice hack which will generally work, but having overlapping 
boundaries (different ways on the same position) is not a clear 
"error" in OSM, so it is not 100% reliable.


No. I looked at the data visually with transparency to find overlaps, 
and they are exceedingly rare. As a data consumer, I have no hesitation 
in calling what you have described an error.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Turning boundaries to linestrings

[Paul Norman]

On 12/13/2016 4:33 AM, Christoph Hormann wrote:

On Tuesday 13 December 2016, Paul Norman wrote:

Feedback from anyone interested in using this output is welcome, as
well as any additional information that should be added to the
linestrings.

You have some conflicting goals here - you want to maintain the OSM
tagging information on the way level so you need to maintain the
individual ways - which however limits what you can do in terms of
rendering - dashing is not consistent for example where a boundary is
split and you have problems doing geometry processing (like the all
popular ST_Simplify()).


Maintaining the individual ways is not a goal. The primary goal is being 
able to render disputed boundaries, which this achieves, although not 
perfectly. The dashing problems are most visible comparing dashed 
borders on straight line borders in remote Canada vs several US states. 
In the former, the border tends to be one long linestring because it's 
unsplit and doesn't have many junction points, while in US states 
junction points are more common.



IMO a process that merges the ways where it is non-ambiguous would be
more useful.  And most cases where tagging varies in an admin boundary
without there being a junction that is a tagging error.


Most places where two admin boundary ways join are either part of a very 
long stretch (e.g. BC-AB border) where merging isn't a huge issue, or 
junction points of some kind where >2 admin boundary ways join. The 
second poses a problem for merging and simplification, which can be 
solved in a three ways


1. Merging on linestring admin_level and simplification by amounts that 
don't visibly break topology, even if examining the data on a sub-pixel 
accuracy shows breaks. More than this is a problem, not because of 
opening up small gaps, but having ways appear to extend to far. The 
former will end up interpreted as part of the dashes most of the time, 
but the latter can look odd.


2. Doing merging that varies based on rendered admin_levels. This would 
need to do a topology-aware simplification based on the admin_levels 
that are being rendered. If done in preprocessing by osmborder, this 
would probably need to result in 4-6 files being generated, each going 
from admin_level 2 to a different maximum.


3. Simplification without merging. This can cause topology problems, but 
not at junction points. Unfortunately, this won't help much with data 
volume or dashes


Given current priorities, I can't see putting topology awareness into 
osmborder or doing it at query time. If that happens, it will probably 
be when working on generated vector tile size for low-bandwidth devices. 
It's probably going to be simplification without merging at query time 
initially, and at some point I'll look at linestring merging for roads, 
and boundaries will be easy to do after that.



The maritime attribute based on tagging is currently of very limited use
since it is not consistently applied.  On admin_level 2 a better way
would be to classify this based on topology - i.e. all boundaries that
are only part of one admin_level 2 relation are maritime boundaries.
On the higher admin levels this is more complicated of course and not
easy to solve.


Yes, I've been considering if I should do something like that by 
counting number of parent relations, but not because of incomplete data. 
The data is pretty good for admin_level=2, and less consistent for 4.


The question I was considering is how I want to render boundaries in 
places like the Georgia Straight or Great Lakes, e.g. 
https://cloud.githubusercontent.com/assets/1190866/21168186/faeecaae-c166-11e6-8cd1-7488fffddca4.png. 
If I want those to be rendered, I have to count number of parent 
relations and style based on that.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Turning boundaries to linestrings

[Paul Norman]

A common problem with rendering administrative boundaries from
OpenStreetMap data is the need to create non-overlapping linestrings
from the admin relations. This is necessary for many styles,
particularly any which want to use dashes. Ideally the information would
be present in on way tags, but this is inconsistently done. A second
need is to figure out which boundaries are disputed, which is
information on the ways.

For my work on the Wikimedia map styles I've written OSMBorder, software
which extracts the admin boundary data from the planet file and assmbles
it into linestrings along with

- the lowest admin_level value from the boundary relation that boundary
  line section is in

- if the boundary line is disputed

- if the boundary line is a maritime one

The output is a CSV file which can be loaded into PostgreSQL. I've put an
example up on my server at 

along with a Shapefile version at 
.


Instructions for loading the data are at 
https://github.com/pnorman/osmborder#output


I'm still looking for somewhere to regularly run the conversion and host
the output, and expect to find some bugs and change output details.

Feedback from anyone interested in using this output is welcome, as well
as any additional information that should be added to the linestrings.

The code is at https://github.com/pnorman/osmborder, as well as
documentation on the tags used and how to run it. Parts of the code are
based on OSMCoastline by Jochen Topf.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql 0.92.0-RC1 release

[Paul Norman]

osm2pgsql 0.92.0-RC1 has been tagged in preperation for a release. 
Feedback is welcome, in particular from downstream packagers.


You can get the release from git or 
https://github.com/openstreetmap/osm2pgsql/releases/tag/0.92.0-RC1.


The big changes are

- PostgreSQL 9.1 + PostGIS 2.0 or later are now required, which has lead 
to performance improvements and cleanups
- EPSG 3857 is now default. You can get the old behavior by manually 
specifying 900913.

- All invalid and empty geometries should now be excluded

Other changes are
- A new option to change the max bbox at which polygons will expire all 
the tiles in them, not just the boundary. Only used when generating 
expired tile lists. See https://github.com/openstreetmap/osm2pgsql/pull/570

- Behavior fixes for C transforms and tables with no columns
- More numeric datatypes are allowed for table columns in C tagtransforms
- Lua is now required by default
- Code fixes, particularly replacement of C memory management

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto dependency changes

[Paul Norman]

OpenStreetMap Carto has changed it's dependencies, which will impact any 
third

parties using the style. The changes are

- Mapnik 3 is now required

- CartoCSS >= 0.16.0 is now required

- Official Tilemill support is dropped. Tilemill does not meet the above 
requirements


- project.mml is now a YAML file which is passed directly to CartoCSS 
without any

  preprocessing

If you are maintaining a fork, rebasing the changes onto your branch 
should be fairly

easy, but you can contact me off-list if you need git help.

The next version released will be v3.0.0. The v3 series is expected to 
be a much
shorter than v2, as we are working on 
https://github.com/gravitystorm/openstreetmap-carto/issues/1504

which will start the v4 series.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] A PostgreSQL extension for real-time simplification of objects in OSM API database

[Paul Norman]

On 11/16/2016 5:37 AM, RTOSM DOOPAS wrote:

The rtosm map call (not that exact, but the framework is clear):

[...]
6. send back the  simplified objects to request.


The purpose of the API and the apidb is for editing. How would someone 
edit the simplified object?


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tilemaker v1.4

[Paul Norman]

On 11/7/2016 9:28 AM, Richard Fairhurst wrote:

Hi all,

Really pleased to announce a new version of Tilemaker on github:

https://github.com/systemed/tilemaker

Tilemaker is a command-line application that makes vector tiles in one 
hit, directly from an .osm.pbf file. There is no database and no 
stack. Everything is kept in RAM so it's best suited for country-size 
extracts.


v1.4 is many times faster than previous versions thanks to Shunsuke 
Shimizu's great work on making tile output multithreaded. 


To quantify this, it is 3x as fast on my 4 core system and uses about 
the same RAM. It takes about 3 minutes for a 400MB PBF and a very simple 
conversion, with 4GB of RAM usage. Larger extracts or more complex 
vector tiles will take longer and more RAM.


The real advantage of tilemaker with small areas is that you can run it 
and upload the tiles to virtual any web host and the ops side is easy.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v2.44.1

[Paul Norman]

Dear all,

Today, v2.44.1 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released. Also, v2.44.0 was
released last month without an email, so this email includes changes
in both.

v2.44.0 has been rolled out to the openstreetmap.org servers, but v2.44.1
has not yet.

Major changes are
- Rendering of restricted access roads and paths significantly changed
- Changed to use Noto fonts for all languages

Other changes in both versions include
- A code of conduct adopted, based on the Go code of conduct
- Adjustments to city wall rendering
- Revised low zoom place rendering
- Render both house name and number if address has both

Thanks to all the contributors for this release, in particular Lukas
Sommer, Hsiao-Ting Yu and vholten for work in debugging complex font
issues with the Noto CJK fonts.

For a full list of commits from both releases, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v2.43.0...v2.44.1

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] How does osm2pgsql handle multiple matches of same key?

[Paul Norman]

On 9/16/2016 5:37 PM, Spencer Gardner wrote:
It's not clear to me how osm2pgsql handles multiple values for the 
same key on an OSM feature. For example, imagine I have a way with the 
following tags:

parking:lane:right=perpendicular
parking:lane:left=parallel


"parking:lane:right" and "parking:lane:left" are different keys. What 
happens depends what your tag transform is doing. The default C tag 
transforms does nothing special with those keys, so doesn't change the 
values or anything. A lua tag transform or a multi-backend style does 
what you have defined it to.



 1. How does the "parking" column reflect this after the import
assuming that my style file has an entry for parking?



If those are the only tags and there is not a "parking" key on the 
feature, the column is null with the C tag transforms.




 1. Is there a way to define styles that separate these? (i.e. one
column for parking:lane:right and another column for
parking:lane:left?

My understanding of hstore is that it will suffer from the same 
problem--if there's a duplicate entry for the same key it takes the 
first and ignores the others. If this is not the case I'll investigate 
using that for my purposes.


If you want a column for parking:lane:right and parking:lane:left, add 
both to your style file.


If you want to do something more sophisticated, you need to use Lua 
transforms. You could have a column which indicates which sides there is 
parking on. https://github.com/ClearTables/ClearTables/issues/51 is 
about something similar for maxspeed:forward and maxspeed:backward
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Paul Norman]

On 9/15/2016 1:53 PM, Stefan Keller wrote:

Is it OK to do API lookups like this
https://www.osm.org/api/0.6/nodes?nodes=59906080,4400821613  even for
minutely diffs? Any alternatives?


No, particularly if your code takes off and multiple people start 
running it. I would guess that it would hit rate limits and be 
automatically blocked.


If all you need is node positions this can be done fairly efficiently 
for the entire planet with something like osm2pgsql flat-nodes. If you 
need full information this takes more space. But you should consider if 
you're just rebuilding augmented diffs.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Road Speed In South Africa Geotab

[Paul Norman]

No one has replied, likely because it's not a really a dev@ subject, so 
I'll provide some basic info.



On 9/13/2016 2:34 AM, Riaan Grobler wrote:


*Problem* : We are experiencing more and more road that do not have 
updated road speed, see below as an example attached




Road speed coverage will vary road class and region, and also if a 
mapper in the region really cares about mapping all the maxspeed data. I 
wouldn't expect it to be complete.


   The other problem is that once a update runs it 
over writes all the users information.




That sounds like a technical problem with the process you're using to 
update. In general you want a process which will allow you to update 
your data without manual intervention.


From the information here it sounds like you're making what's called a 
"derivative database" in the license, in which case it is licensed under 
the ODbL like OSM data is and you have to be able to provide that 
database to others.


*Resolution* :  I know you can make the changes on the site but I know 
you it needs to be validates first and this takes time.


   Is there a way to speed up the process if we 
supply you with all the details  with pictures.




There isn't a validation step between a user making changes and the data 
being available to everyone. OpenStreetMap is crowd-sourced project 
built around individuals mapping areas they know. Nor is permission 
required to map except in exceptional cases like imports and mechanical 
edits.


If you have noticed a problem with our map data, for example a road is 
missing or your address, the best way to proceed is to join the 
OpenStreetMap community and add or repair the data yourself.


The pictures aren't necessary for OpenStreetMap itself, but would be 
useful to both the OpenStreetView and Mapillary projects, both of which 
collect geotagged road photos.


   Is the above files the correct places to get 
the most up to date maps for South Africa ?




The Geofabrik data extracts are provided as a service to the community 
by Geofabrik, a consulting and software firm specializing in 
OpenStreetMap. Their extracts are operated daily. It's possible to get 
more frequent updates directly from planet.openstreetmap.org, but 
significantly more work is required if you only want South Africa.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] RFC: OSM data format MIME types

[Paul Norman]

I'm planning on registering MIME types for OSM formats in the vendor 
tree and could use feedback. If you're unfamiliar with registering MIME 
types, https://github.com/mapbox/vector-tile-spec/issues/48 is a 
reasonable overview for a registration that took place with a different 
format.



The obvious type for OSM XML is application/vnd.osm+xml, but there are a 
few unanswered questions


- Is .osc a different type? I'm think yes. Tools that parse one don't 
normally accept the other in its place, they normally have different 
file suffixes, and the top-level XML elements are different.


- Is osc returned from the server a different type from what is 
uploaded? I'm inclined towards no. This is more a matter of selection of 
valid identifiers.


- Where does history fit in

I'm inclined towards

- application/vnd.osm+xml for OSM XML

- application/vnd.osm.osc+xml for OsmChange XML

- application/vnd.osm.osh+xml for historical XML

- application/vnd.osm.osmpbf for current data PBFs. pbf is not a MIME 
suffix.


- application/vnd.osm.oshpbf for historical PBFs.

- application/vnd.osm.o5m

- application/vnd.osm.o5c

I'm not sure where to slot changeset data into this. Changeset data can 
be present in an OSM XML file alongside other data (e.g. with planet 
dumps) or standalone.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] PostGIS query "Crossing ways"

[Paul Norman]

On 9/12/2016 9:56 AM, Mike N wrote:
I did use the --slim option when importing, and see the node table, 
but I don't see topology in a direct table view.


The ways table has node membership information. Joining the table on 
w1.nodes && w2.nodes and differing way IDs will find ways that share 
nodes. You'll need to define your requirements more precisely than this 
and use something more sophisticated, but this should get you started.


  If I use something like the extension  postgis_topology , would I 
still need to write a tool to populate from OSM data? 


I'm not aware of any tools which import using postgis topology. You 
might be able to do what you want with pgrouting and osm2pgrouting, but 
I haven't used it myself.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] PostGIS query "Crossing ways"

[Paul Norman]

On 9/11/2016 8:20 AM, Mike N wrote:
Given a PostGIS database populated from OSM data by osm2pgsql, and 2 
sets of lines  (such as the selection of all footways and the 
selection of all roads)   what function or series of functions will 
result in a list of locations where footways cross roads without any 
OSM connecting node?  (the equivalent of JOSM's "Crossing Ways" warning)


  According to the PostGIS documentation, ST_Intersect() includes 
"Touches", which I assume would be sharing an OSM node.


  ST_Crosses() might be what I want, but does it exclude the crossings 
that share a node?   If not, do I just remove all the ST_Intersect() 
results that are spatially close to ST_Crosses()? 


PostGIS databases do not have topology so there is no notion of 
connected linestrings. You can tell if two ways cross each other with 
ST_Intersects, and you can tell if two share points by turning the 
linestrings into points, but this doesn't tell you if the two share 
nodes. If you want that you need either the osm2pgsql slim tables, 
pgsnapshot, or some other schema with node information or topology.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] PostgreSQL 9.6 coming out

[Paul Norman]

PostgreSQL 9.6 RC1 has been released, and depending on bugs found, 9.6 
should be released soon. There are a few changes which should help 
anyone running PostGIS workloads with OSM data.


From the release notes[1] I've extracted some relevant stuff for this list

Big new stuff:

- Parallel queries. A single query can now do more work in parallel. 
This won't help production tile rendering for various reasons but could 
help analysis where the bottleneck is not geometry computations and 
perhaps rendering latency on unloaded servers.


- Performance improvements, particularly with scalability

- Replication improvements

- GIN index builds can use more than 1 GB maintenance_work_mem. 
osm2pgsql slim tables and others schemas have large GIN tables


- GIN indexes can have their pending list manually cleaned up with 
gin_clean_pending_list. This could allow fastupdate with osm2pgsql GIN 
indexes


Other new stuff:

- Better statistics for columns with many nulls

[1]: https://www.postgresql.org/docs/9.6/static/release-9-6.html


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v2.43.0

[Paul Norman]

Dear all,

Today, v2.43.0 of the openstreetmap-carto stylesheet (the default
stylesheet on openstreetmap.org) has been released. It has not yet
been rolled out to the openstreetmap.org servers.

Changes include

- Adjust alotments pattern
- Whitespace cleanups of code
- Adjust colours of dog parks and construction sites
- Increase font size of addresses
- Fix combination of long names and oneway arrows

Thanks to all the contributors for this release, including Ircama and 
measad, both new contributors.


For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v2.42.0...v2.43.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Evaluate layer list from CartoCSS style

[Paul Norman]

On 8/28/2016 8:07 AM, Stadin, Benjamin wrote:

we’re defining the data structure for the next version of our 3D map, a
basic feature of it is the indoor capability. I could write more about the
background, but to keep it short for now: I need to generate an ordered
list of layers with their table names in the rendering db, so ordered from
„bottom layer to top layer“, out of a Carto CSS style definition (e.g.
from the actual OSM style def.
https://github.com/gravitystorm/openstreetmap-carto).


The typical osm2pgsql rendering database does not have layers. It has 
tables for points, lines, polygons, and one for data commonly used at 
low zooms.


OpenStreetMap Carto comes with layers, but these are not layers in the 
traditional GIS sense. They are layers which are rendered and the layer 
definitions are not intended to be meaningful outside of the specific 
style. These regularly change between style versions and copying them is 
a bad idea unless you want to recreate that specific style.


If you do want to do that, the style definitions are in YAML and should 
be easy to parse.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Experimenting with ClearTables, self-hosted vector tiles, and Tangram client-side rendering

[Paul Norman]

On 8/25/2016 12:45 AM, Oleksiy Muzalyev wrote:
I looked at several cities at the Tangram map. I like especially 
Crosshatch style, - very impressive. It takes a split second to find 
say the Eiffel Tower on the Tangram map: 
https://tangrams.github.io/carousel/?crosshatch#16/48.8616/2.3082 and 
in general to get a layout of a place recognizable later on the ground 
via these tall reference points.


To be clear, styles other than the one I linked are not by me and 
unrelated to my style. They use the same client-side rendering engine, 
but that's it.


Since Tangram is "drawing vector tiles live in a web browser" is it 
possible to make a comprehensive selection of a language?


MVT, the tile format I'm using, and Mapnik, the vector tile generator do 
not have a suitable datatype for names in an arbitrary list of 
languages: https://github.com/mapbox/vector-tile-spec/issues/75. 
TopoJSON might be more flexable but I haven't considered it.


One more thing, for some reason the search of the Tangram map does not 
find Odessa, Ukraine.


Tangram doesn't have a search functionality, what's on that page is 
provided by some geocoding provider. It's not my page so I don't know 
what it's using, but given Tangram is developed by Mapzen, it's probably 
Mapzen's geocoder.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Experimenting with ClearTables, self-hosted vector tiles, and Tangram client-side rendering

[Paul Norman]

I've been experimenting with generating my own vector tiles and
client-side rendering with Tangram[1] in order to figure out how to best
write its styling language.

Tangram is a GL-based renderer written by Mapzen and normally used with
their Tilezen[2] vector tiles, but I'm interested in being able to make
my own vector tiles and different cartographic choices. I also consider
diversity of vector tile schemas important. I hope to avoid a situation
where only large players in the market can get involved like we have
right now.

For a toolchain I used osm2pgsql with ClearTables[3] and Mapnik via
Kosmtik[4] to write vector tiles. On the demo I'm serving the tiles with
Apache but in development I used Kosmtik because it's xray functionality
is useful. For development I worked in Tangram Play, a web-based editor
that automatically reloads the map when you change the style.

The cartography and vector tile definitions are loosely based on OSM
Clear[5], a demo style I wrote. As it's a learning exercise I don't
consider the style complete or free of bugs.

The demo page is on my server at
http://tangram-clear-demo.faramir.paulnorman.ca/ with the style and
vector tile code at https://github.com/ClearTables/tangram-clear-demo.

I'm not sure what direction I'm going to take next as I don't have any
particular style goals right now, or collaborators.

[1] https://mapzen.com/products/tangram/
[2] https://mapzen.com/projects/vector-tiles/
[3] https://github.com/ClearTables/ClearTables
[4] https://github.com/kosmtik/kosmtik
[5] https://github.com/ClearTables/osm-clear


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Polygon inner/outer relation in osm file

[Paul Norman]

On 8/23/2016 6:58 AM, Martin Koppenhoefer wrote:

recently we stumbled upon problematic behavior of current multipoligon 
processing in osm-carto


OpenStreetMap Carto does not do any multipolygon processing. That is all 
done by osm2pgsql, and the behavior of osm2pgsql has changed 
significantly from the version used on the OSMF tile servers.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Nominatim Index bloat? Try pgindexrebuild, a production friendly index debloater

[Paul Norman]

On 8/19/2016 4:30 AM, Rory McCann wrote:

Although I'm using this often, I'm always open to suggestions about
braindead things I might be doing. ☺ Suggestions welcome!


It's good to see commands for a concurrent reindex scripted.

For rendering tables I'd recommend pausing updates, creating a new 
ordered table, building indexes on the new table, then putting the new 
table in place. The reclustering adds performance beyond what a reindex 
does. 
http://paulnorman.ca/blog/2016/06/improving-speed-with-reclustering/ has 
example SQL.


For slim tables, I'd probably just pause updates and REINDEX.

http://reorg.github.io/pg_repack/ would be interesting on the slim 
tables but can't operate on the rendering tables.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto issues of interest

[Paul Norman]

There are some OpenStreetMap Carto issues which might be interesting to 
a larger audience


Improving the water colour: 
https://github.com/gravitystorm/openstreetmap-carto/issues/1781


There's been a discussion of options for improving the water colour to 
improve contrast with a number of other features.


Change access rendering: 
https://github.com/gravitystorm/openstreetmap-carto/pull/2257


This needs more people to try the PR in their local area. Access 
representations is a *hard* problem to solve and we're not going to get 
it perfect or satisfy everyone, but hopefully we can do something a lot 
better


Plan database re-import (aka hstore and lua): 
https://github.com/gravitystorm/openstreetmap-carto/issues/1504


This is one of the big long-term planning issues for the database 
reload, changing the schema, and enabling osm2pgsql --hstore. It needs 
more people testing. You can see additional issues: 
https://github.com/gravitystorm/openstreetmap-carto/issues?q=is%3Aopen+is%3Aissue+label%3Alua


Review onboarding: 
https://github.com/gravitystorm/openstreetmap-carto/issues/2291


If you're reading this on the lists and aren't following the issues 
already, the onboarding experience is relevant to you if you want to get 
involved. It's hard for the maintainers to evaluate; we've all been 
working at map styles for some time.


Comments are best posted on the GitHub issues, otherwise they'll 
probably get lost when reviewing the discussion.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Diff issues: minutely overwritten by later data/incompl. daily diffs

[Paul Norman]

On 8/14/2016 2:56 AM, mmd wrote:

Processing daily diffs is in fact a very convenient to load 4 years
worth of OSM data (needed for full history), rather than downloading
more than 2 Mio. minutely diffs.


I don't know about the data issues, but I suggest you use the full 
history. Over half the data in OSM is covered by those diffs. The full 
history is available as a faster to parse PBF format, and an initial 
import is generally faster than consuming diffs.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tile usage without proper identification

[Paul Norman]

On 8/12/2016 11:27 AM, Matthijs Melissen wrote:

I would also like to point out that there might be a conflict of
interest here between commercial operators of OpenStreetMap services,
and the rest of the community.


While in theory there could be a conflict, most of the admins do not 
sell tile services.


The website involved is also one that I would *not* want as a client and 
if I sold tile services I'd want them to remain on osm.org instead of 
risking that they'd come to me.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tile usage without proper identification

[Paul Norman]

On 8/12/2016 4:34 AM, Maarten Deen wrote:
I have wondered about this. Does rate limiting now not occur? I have 
seen a lot of times that the map in the JOSM downloadscreen gets slow 
if you pan around a lot or use it a lot to download lots of small 
areas. Also the mapnik overlay has this issue. 


This is unlikely to be from rate limiting, but more likely from the 
limits on the number of simultaneous downloads, or your connection.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Tile usage without proper identification

[Paul Norman]

The usage policy for tile.openstreetmap.org requires that users send a
valid user-agent[1], or, in the case of a web browser, a HTTP Referer.
Ops are looking into automatically rate limiting clients violating this
part of the policy.[2]

The reason for rate limiting instead of blocking is that if you access a
single tile from the browser you do not send a referer, but requests are
made slowly when doing that. Rate limiting has become necessary because
of abusive websites and apps which hide the origin of their requests and
have been consuming excessive resources.

Websites will not be affected unless they using a referrer policy to
mask the origin.[3]

If you develop an app, please make sure you are setting a User-agent to
identify your application, and if applicable the referer. You must not
use the default User-agent for your HTTP library.

As a reminder, it is highly recommended that you do not code the URL
tile.openstreetmap.org into an app because usage restrictions may change
subject to the needs of the project. The priority is its usage as a
resource for mappers to view their work as part of the editing cycle.

[1] 
http://wiki.openstreetmap.org/wiki/Tile_usage_policy#Technical_Usage_Requirements

[2] https://github.com/openstreetmap/chef/pull/79
[3] https://www.w3.org/TR/referrer-policy


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql 0.90.1 release

[Paul Norman]

On 7/18/2016 6:07 AM, Sebastiaan Couwenberg wrote:

Important for context is that "something" was mapnik (from 2.2 to 3.0),
which is non-trivial due to breaking its reverse dependencies.


I believe I also asked about osm2pgsql, but I'm left in a situation 
where I don't have the permissions to do it myself or a route to move 
forwards.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql 0.90.1 release

[Paul Norman]

On 7/18/2016 4:32 AM, Komяpa wrote:

Hello,

Is it available for Ubuntu Xenial?



It should compile fine, but Xenial has osm2pgsql 0.88.1 by default. I 
haven't backported all the changes to the 0.88.x branch, but I could.


The chances of Ubuntu getting the bug fixes is close to zero. I asked 
for a backport from Xenial to Trusty of something and got absolutely no 
response.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql 0.90.1 release

[Paul Norman]

osm2pgsql 0.90.1 has been released

This is a maintainance release without new features

- outdated stuff cleanup

- null pointer dereference fix

- don't unicode sort geohashes

- process relations for line tables with the multi-backend

- generic projection fix

Upgrading is recommended. The geohash sort change improves performance 
by up to 5%, particularly on imports small relative to RAM.



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OpenStreetMap Carto release v2.41.0

[Paul Norman]

Dear all,

Today, v2.41.0 of the openstreetmap-carto stylesheet (the default stylesheet
on openstreetmap.org) has been released.

Changes include
* More consistent fonts for POI labels
* Less saturated stadiums
* Rendering obelisks and dog parks
* An updated list of font packages
* Cleaning up the font list
* Rewriting the road colours script for easier changes
* Various bug fixes

Thanks to all the contributors, including jdhoek, a new contributor.

For a full list of commits, see
https://github.com/gravitystorm/openstreetmap-carto/compare/v2.40.0...v2.41.0

As always, we welcome any bug reports at
https://github.com/gravitystorm/openstreetmap-carto/issues.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Taginfo files from osm2pgsql style files

[Paul Norman]

On 7/13/2016 2:11 AM, Sven Geggus wrote:

Hm, looks like this does not make that much sense in case of a hstore-only
database like the one I am using in the german style setup.


Yes, I designed it for the defaults. If you add hstore options any tag 
can be
used and there's no way to indicate that a taginfo project uses all 
tags.It's

also not useful to say a project uses every possible tag.


One should at least pass the "nocolumn" entries to reflect, that these are
considered important tags. Not adding a column does not make them unused.


I considered adding delete, phstore, and nocolumn but decided against it.


I think the tool which will generate taginfo json should actually be
added to node-carto rarther than osm2pgsql.


I'm not adding it to osm2pgsql because it's very specialized. It's not a
tool for Mapnik styles or CartoCSS so it doesn't belong in node-carto.

Generating taginfo project JSON for a style is a much harder problem
andI don't think it's possible because tags are transformed when
turned into PostgreSQL columns, then again transformed by Mapnik layer
SQL queries, then some values can be used in the Mapnik XML or CartoCSS.[1]

I am working on generation of Taginfo project JSON for an osm2pgsql
multi-backend style[2] which is hard but might be possible.

[1]: 
https://github.com/gravitystorm/openstreetmap-carto/issues/961#issuecomment-56249411

[2]: https://github.com/ClearTables/ClearTables/issues/42

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Taginfo files from osm2pgsql style files

[Paul Norman]

Taginfo allows projects to define a list of tags they use, which then 
appear in http://taginfo.openstreetmap.org/projects. Osm2pgsql style 
files define a list of tag keys that are used by osm2pgsql, if the keys 
are for nodes, ways, or both, and if the keys cause an object to be an area.


I wrote some code to turn style files into taginfo project json: 
https://github.com/osmlab/osm2pgsql_taginfo


Any project using osm2pgsql with default settings is using a subset of 
those tags.


It's not the most useful utility but there's a chance it might be useful 
for someone else.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Modeling OSM sidewalk data

[Paul Norman]

On 7/12/2016 3:44 AM, Andy Townsend wrote:

On 12/07/2016 08:42, Paul Norman wrote:

Rather than having sidewalk data available in a column, I instead with 
a different highway value for "this road has a usable sidewalk".  We 
already have a plethora of highway values, so the extra work to 
support isn't high, and has the advantage that you don't need to mess 
about with the schema (though you do need to reload the database) if 
anything changes:


https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua#L258 



Don't you quickly start needing to have many many values for your 
highway column as you combine sidewalks, link status, cycleways, etc?


For me a sidewalk column is much easier to do than expand the highway 
class values and it avoids users downstream of me having to deal with 
sidewalks if they're not using them in their analysis or rendering.


My initial goal was to be able to render sidewalks on unclassified and 
tertiary roads.  Note that there are more values in that list than 
you're probably expecting - that was what was found to be in the data 
when I looked locally. 


I also see you're treating cycleways as sidewalks. I'm planning on 
handling them separately, which is another reason I'm using more columns.


From the readme of https://github.com/ClearTables/ClearTables it 
sounds like you're trying to process OSM data as it actually exists, 
not "as it ought to be tagged", which means you might have to do 
something "clever" to detect poorly mapped examples like the one 
above, perhaps based on proximity.  I'd be interested to see what you 
come up with...


I'm dealing with "tagging as it exists", which is slightly different 
than "data as it exists", so I don't have to worry about data errors 
like those disconnected ways. This is partially due to technical 
constraints, but even without them I'm not sure if I'd handle that case.


It's also going to a PostGIS database so there's no notion of if two 
linestrings are connected, only if they're touching.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Modeling OSM sidewalk data

[Paul Norman]

On 7/12/2016 2:15 AM, Komяpa wrote:


Both ways are only applicable to cities with blocky structure.
In ex-USSR, architecture of cities is different, and footways / 
sidewalks graph is more complicated than a replica of roads for cars, 
so all the sidewalks have to be drawn separately.


Yes, I should of been clearer, sidewalks in this context are highways 
tagged with a sidewalk=* tag.


It does bring up the question of how to handle something like 
highway=footway sidewalk=right - a footway with a sidewalk.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Modeling OSM sidewalk data

[Paul Norman]

I'm working on putting OSM sidewalk data into PostgreSQL with 
ClearTables (https://github.com/ClearTables/ClearTables/issues/39) and 
looking at two different ways, and wondering if anyone has experience 
modeling it.


There are two obvious options

1. Have a sidewalk column with an enum type with both/left/right/neither

2. Have sidewalk_left and sidewalk_right boolean columns

I can see advantages and disadvantages to both.

Does anyone have experience consuming OSM sidewalk data and have 
thoughts about what would be better to work with?


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using API 0.6 to create a new node

[Paul Norman]

On 6/14/2016 8:41 AM, toni hernández wrote:

Hi everyone,

I am trying to display OSM data into my web map as well as other 
custom layers.
One of the goals of my web application is to upload data from my 
application to the osm database.  I have been reading this 
http://wiki.openstreetmap.org/wiki/API_v0.6#Elements but still I do 
not understand how a PUT request functions. I have so much to learn


After authentificating with osmauth.js I try this code without any 
success. I get a 401 error.


var xml_string = ' version="0.6" generator="MyOpenstreetmapApp">lat="41.983910" lon="2.816094">v="supermarket"/>';


ajaxurl= "http://www.openstreetmap.org/api/0.6/node/create;; 


I do not recommend the node/create API endpoint. Instead, use the diff 
upload API call documented at 
http://wiki.openstreetmap.org/wiki/API_v0.6#Diff_upload:_POST_.2Fapi.2F0.6.2Fchangeset.2F.23id.2Fupload.


You will have to use this endpoint anyways for modifying existing data.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Mukti and Gargi fonts for openstreetmap-carto

[Paul Norman]

On 6/11/2016 11:44 AM, Sven Geggus wrote:

Hm so can't we just go for "No Tofu" fonts?

They are supported by Google and are available as a Debian Package:
https://www.google.com/get/noto/
https://packages.debian.org/sid/fonts-noto


Well, no one has done much research into issue #1067 which covers this. 
Comments are welcome there.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Mukti and Gargi fonts for openstreetmap-carto

[Paul Norman]

On 6/7/2016 2:44 AM, Sven Geggus wrote:

Hello,

I'm trying to install all required fonts for openstreetmap-carto on debian
stable.

Two fonts seem to be troublesome:
"Mukti Narrow Bold" and "gargi Medium"

While the former seems to be there
(/usr/share/fonts/truetype/fonts-beng-extra/MuktiNarrowBold.ttf) but is not
found by fc-match -s for some reason, the other one just seems to have gone
from the Debia/Ubuntu repositories. It seems to have been replace by
"Gargi Regular".

Any hint on how to install?

I worked around the gargi font problem by downgrading the fonts-deva-extra
package to 2.0, but I havee no Idea about the Mukti dont.


As Ubuntu and Debian packages are mostly common, I expect that the 
package name problems I had when switching Ubuntu versions are present 
on Debian too. A lot of apt-cache searching might be required to find 
the current names.


For the specific issue of gargi, on Ubuntu 14.04 it is 
/usr/share/fonts/truetype/ttf-indic-fonts-core/gargi.ttf from 
ttf-indic-fonts-core, and /usr/share/fonts/truetype/Gargi/Gargi.ttf is 
supplied from fonts-gargi.


fc-query names the former gargi and the latter Gargi. They are not the 
same weight either, gargi is Medium with a weight of 100 and Gargi is 
Regular with a weight of 80. gargi has better coverage and more 
capabilities, but a lower version.


Something I've found when researching fonts for Asian languages is that 
the versioning, releases, name consistency, and other releasing 
engineering matters are often inadequate. Coupled to this, many webpages 
for fonts disappear after a few years.


If someone can come up with appropriate font names and package names 
that work on Ubuntu 14.04, 16.04, Debian stable, and Debian testing, 
please open an issue: 
https://github.com/gravitystorm/openstreetmap-carto/issues


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM + iD + Nominatim Setup instructions

[Paul Norman]

On 6/4/2016 9:34 AM, Stadin, Benjamin wrote:


Hi Bryan,

thank you for these pointers.

A somehwat unrelated question: The FAQ mentions cgimap. Is this 
performance optimized map API implementation still relevant and is it 
still in use? Will a production system benefit from this 
significantly, when it has to deal with lots of requests?


cgimap is still used in production. It is far faster when dealing with 
requests which return lots of data, e.g. big map calls, relation/full 
calls. It is also faster on small calls but you're less likely to notice 
that.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Updated simplified osm2pgsql database dump available

[Paul Norman]

On 5/25/2016 2:32 PM, Sven Geggus wrote:

I have always been wondering if it would be possible to get rid of the
special tables used for the sole purpose of keeping the rendering database
up-to-date by using a scheme like this:

osm2pgsql database with special tables  -> some psql replication scheme

--> many slave databases without special tables kept up-to date using psql
 some replication scheme

This would have the advantage of a lower storage foot-print (only 4
rendering tables left).


You'd still have the full storage footprint on one machine, it just 
helps with the requirements on your read-only replica slaves.


This can be done a few ways right now

- pglogical or trigger-based (e.g. slony) replication

Streaming replication doesn't work here since it replicates the entire 
cluster including slim tables


- PostgreSQL FDW magic

It should be fairly easily to post-import move the rendering tables to 
another cluster and use any replication method with that cluster.


- Allow connection information to be set per-table in osm2pgsql

This may be possible with the multi- backend. Adding it to all the 
backends shouldn't be too hard as osm2pgsql already uses independent 
connections for each table. The slim tables and rendering tables could 
then be directly placed on databases on different clusters


What is not possible is to centralize this. When doing updates what goes 
to the rendering tables depends on the osm2pgsql style, so your 
rendering table updates will not be the same as other people with a 
different style.




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Updated simplified osm2pgsql database dump available

[Paul Norman]

On 5/25/2016 3:11 AM, Mateusz Konieczny wrote:

I am guessing that data is from 25-04-2016. Is it a correct guess?


Yes.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Updated simplified osm2pgsql database dump available

[Paul Norman]

To help with some OpenStreetMap carto development work, I've created a 
dump of the rendering tables with certain features removed, for testing 
at low zoom. This allows someone to load the database if using a machine 
incapable of importing the planet, and the dropped features cut the 
database size in half.


It is available at http://tile.paulnorman.ca/planet-lz-160425.dump but 
before downloading please read the notes below


- Buildings without a name, amenity, shop, or similar tag have been removed

- Residential roads have been removed

- Don't try to download this from your browser, as it is 26GB. If you 
need to check the download, the md5sum is 5c771789b0820b4f7c4f705156633c7f


- The dump has been generated with pg_dump from PostgreSQL 9.4 in the 
pg_dump "custom" format and -Z9. To get 9.4 on Debian or Ubuntu based 
systems, see https://wiki.postgresql.org/wiki/Apt


- The indexes at 
https://github.com/gravitystorm/openstreetmap-carto/blob/master/indexes.sql 
have been added, except with a fillfactor of 100.


- The database is 78GB after loading and index creation. Data only is 56GB.

To restore, do

createdb gis
psql -d gis -c 'create extension postgis;'
pg_restore -d gis -O -j4 planet-lz-160425.dum

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Switch2OSM graphic design help needed

[Paul Norman]

Switch2OSM is undergoing a rewrite to allow a different contribution 
model. As part of this we had to switch from the existing WordPress 
design to one using Jekyll. This requires a new visual design, and we're 
looking for help from a graphic designer.


This is independent of content changes, so you don't need to know 
anything about the subjects the guides cover.


There's some more information at 
https://github.com/switch2osm/switch2osm.github.io/issues/3, and the 
content can be seen at http://switch2osm.github.io/ with just a default 
Jekyll style.


If you're interested in helping, please get in touch. A company could 
also help by letting one of their graphic designers work on it.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [josm-dev] Auto center new node

[Paul Norman]

On 3/22/2016 11:35 AM, Frederik Ramm wrote:

The new mode allowed me to keep the mouse pointer roughly somewhere
right of centre, and between clicks I would only need a minimal
adjustment to move the mouse pointer onto the fence line.

But maybe this use case is much rarer today, and/or other JOSM input
modes are meanwhile available to cater to that use case.


I've tried it and didn't find it particularly efficient for anything 
these days.


I tend to use refine way for cases like that - draw a very rough way 
then refine it.


If it's no longer normally used, should it be removed?

___
josm-dev mailing list
josm-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/josm-dev