[OSM-dev] A spreadsheet to generate Mapnik stylesheets
For the past few years I have used a slightly modified version of the OSM Mapnik stylesheet on my own website. With the recent change to CartoCSS I transfered my patches across. Both the original raw XML format and the newer CSS type format are quite complicated to modify (although the new format is easier than the old one). To make creating Mapnik stylesheets even easier I have created a new tool that is somewhat like Spreadnik [1] in that it uses a spreadsheet to define the formatting. Compared to Spreadnik though it is somewhat simpler because one line in the spreadsheet can define everything about a single object type (for example highway=motorway). The spreadsheet has four types of style definition sheet: * shapefiles - for low zoom shapefiles, coastlines etc. * areas - for polygons with optionally: solid fill, pattern fill, outline, label. * lines - for linear features with optionally: casing, fill, tunnel and bridge styles, oneway arrows, up to two labels. * points - for nodes (and optionally polygons) with optionally: symbol, up to two labels. The Perl script that processes the text files (exported from the spreadsheet) will automatically generate a reasonably optimised SQL statement for a group of objects and the styles and layers to draw everything required for each zoom level. I have written a longer description on a web page [2] which also includes a download of the Perl script and the spreadsheet that re-implements ~99% of the OSM map styles (with a few style changes). The download is a bit rough at the moment, more like a dump of my working version. There is also a side-by-side slippy map comparison [3] of the standard OSM stylesheet and this new stylesheet (for the UK only). [1] https://wiki.openstreetmap.org/wiki/Spreadnik [2] http://www.gedanken.org.uk/mapping/custom-maps/ [3] http://www.gedanken.org.uk/mapping/custom-maps/comparison.html -- Andrew. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Stephan Knauss o...@stephans-server.de writes: On 06.02.2013 21:25, Andrew M. Bishop wrote: Does anybody know if there is a released version of Mapnik that supports ids up to (2^32)-1 rather than requiring an unreleased 64-bit version? could you give details about a use case where mapnik needs the osm_id? The official styles do not contain a reference to osm_id, it's an internal thing in the database. Might be worth being clarified on the wiki page. Currently it reads as mapnik is broken in general. I asked the question based on the new wiki page which says that the unreleased Mapnik version 2.2 is required for 64-bit ids. My question could have been more accurately stated as: If using the standard toolchain of osm2psql, postgresql and mapnik what is the minimum software versions that are needed to continue creating maps after ids reach 2^31-1? The wiki page says that osm2psql version 0.81.1 is required but the version that I have reports itself as osm2pgsql SVN version 0.80.0 (32bit id space). This suggests to me that it will work up until id 2^32-1 appears but this may be wishful thinking on my part. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Ilya Zverev zve...@textual.ru writes: Andrew M. Bishop wrote: Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. I've started it at http://wiki.openstreetmap.org/wiki/64-bit_Identifiers Please update with any software I've missed. That's very helpful, I have added my software (Routino) to the list. Does anybody know if there is a released version of Mapnik that supports ids up to (2^32)-1 rather than requiring an unreleased 64-bit version? -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Paul Norman penor...@mac.com writes: As of right now, we are 5.31 million nodes away from hitting our 2^31th node. This is likely to cause some software to break. Hopefully everything major has been tested, but if you haven't checked your software now would be a good time to do so. I think coastcheck will break in substantial untested ways, but I haven't checked. Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Status of the Mapnik stylesheets
SomeoneElse li...@mail.atownsend.org.uk writes: The thing that I was trying to have a stab at England and Wales designation rendering, a bit like this question: http://forum.openstreetmap.org/viewtopic.php?id=19032 However, after a lot of buggering about it had stopped raining, so I went back outside (where mkgmap-based Garmin maps, with muppet-proof style file support, can render absolutely any OSM feature without problems!). This sounds like Trac ticket 3968[1] for which I added a link[2] to a proof-of-concept Mapnik stylesheet a week ago. [1] https://trac.openstreetmap.org/ticket/3968 [2] http://www.gedanken.org.uk/mapping/custom-maps/#H_1_3_4 -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Patch for osm2pgsql duplicate keys
Nick Whitelegg nick.whitel...@solent.ac.uk writes: I know on my (rather old) version of osm2pgsql, append fails with duplicate ways. Will have to try the new version and see if I get the same problem. Sorry, this is with slim mode only. Dup ways give no error with regular mode. Does the patch relate to slim mode or regular mode? Yes, I use slim mode and this patch was written for slim mode. I didn't realise that regular mode was different in this respect but I don't personally have enough RAM to use that anyway. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql failure with low-end server
Martijn van Oosterhout klep...@gmail.com writes: On 8 December 2011 11:13, Nick Whitelegg nick.whitel...@solent.ac.uk wrote: 700 MB is a tiny machine, but then the Finland data set isn't that large either... it must be possible somehow ;) Incidentally why would it run out of memory in slim mode? I had the same problem too some time ago when trying to import the whole of England, back in the days before the county extracts. I did some research into this a while back and it has to do with the code that goes over all pending ways after the import, to deal with polygons. It looks like the number of pending ways is a lot more than it used to be. Osm2pgsql requests all the pending ways in a single query which fails on small machines. Can someone check if there is really the case. i.e. show the number of pending ways after a simple import. With this command line: osm2pgsql --create --database GIS --slim --cache 128 great_britain.osm I get this output: osm2pgsql great_britain.osm Reading in file: great_britain.osm Processing: Node(32899k) Way(4022k) Relation(80855) parse time: 3481s Node stats: total(32899023), max(1541436207) Way stats: total(4022307), max(140763013) Relation stats: total(80855), max(1905258) Going over pending ways processing way (1639k) Going over pending relations node cache: stored: 8047817(24.46%), storage efficiency: 47.97%, hit rate: 25.52% ... osm2pgsql SVN version 0.70.5 osm2pgsql great_britain.osm This runs on a VPS with ~512MB RAM (I think that this is the guaranteed level but I have certainly used ~50% more at times). -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Patch for osm2pgsql duplicate keys
;
+else
+ osmdata-action = ACTION_CREATE;
}
else if (!strcmp(name, osmChange))
{
@@ -280,7 +289,10 @@
realloc_members(osmdata);
} else if (!strcmp(name, add) ||
!strcmp(name, create)) {
-osmdata-action = ACTION_MODIFY; // Turns all creates into modifies, makes it resiliant against inconsistant snapshots.
+if(osmdata-allow_dups)
+ osmdata-action = ACTION_MODIFY;
+else
+ osmdata-action = ACTION_CREATE;
} else if (!strcmp(name, modify)) {
osmdata-action = ACTION_MODIFY;
} else if (!strcmp(name, delete)) {
Index: parse-xml2.c
===
--- parse-xml2.c (revision 27196)
+++ parse-xml2.c (working copy)
@@ -50,7 +50,12 @@
actions_t new_action = ACTION_NONE;
xmlChar *action = xmlTextReaderGetAttribute( reader, BAD_CAST action );
if( action == NULL )
-new_action = ACTION_CREATE;
+{
+if(osmdata-allow_dups)
+ new_action = ACTION_MODIFY;
+else
+ new_action = ACTION_CREATE;
+}
else if( strcmp((char *)action, modify) == 0 )
new_action = ACTION_MODIFY;
else if( strcmp((char *)action, delete) == 0 )
@@ -73,7 +78,11 @@
if (xmlStrEqual(name, BAD_CAST osm))
{
osmdata-filetype = FILETYPE_OSM;
-osmdata-action = ACTION_CREATE;
+
+if(osmdata-allow_dups)
+ osmdata-action = ACTION_MODIFY;
+else
+ osmdata-action = ACTION_CREATE;
}
else if (xmlStrEqual(name, BAD_CAST osmChange))
{
@@ -205,7 +214,10 @@
xmlFree(xtype);
} else if (xmlStrEqual(name, BAD_CAST add) ||
xmlStrEqual(name, BAD_CAST create)) {
-osmdata-action = ACTION_MODIFY; // Turns all creates into modifies, makes it resiliant against inconsistant snapshots.
+if(osmdata-allow_dups)
+ osmdata-action = ACTION_MODIFY;
+else
+ osmdata-action = ACTION_CREATE;
} else if (xmlStrEqual(name, BAD_CAST modify)) {
osmdata-action = ACTION_MODIFY;
} else if (xmlStrEqual(name, BAD_CAST delete)) {
Note: This patch was created against osm2pgsql-intarray since I use
Postgres 8.3 which isn't supported by the newer osm2pgsql but I
would imagine that it should apply against both.
--
Andrew.
--
Andrew M. Bishop a...@gedanken.demon.co.uk
http://www.gedanken.org.uk/mapping/
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Patch for osm2pgsql duplicate keys
Martijn van Oosterhout klep...@gmail.com writes: On 11 December 2011 10:51, Andrew M. Bishop a...@gedanken.demon.co.uk wrote: My own personal need is to import great_britain.osm and then add ireland.osm both using data from geofabrik. Either file can be imported on its own and the only duplicate data comes from the tiny overlap of the two data sets. With this patch I can run osm2pgsql twice, the first of which is fast because there is no duplicate data to worry about and the second of which is fast because although there is duplicate data there are not as many objects in the second file. With this command line option it is faster to import now than it was with both patches applied. osm2pgsql --create [...] great_britain.osm osm2pgsql --append --allow-dups [...] ireland.osm Ok, that's weird because --append is supposed to allow duplicates already, that's why the MODIFY flag exists already. Otherwise it would be a bit pointless. I invite you to try it like I have. The SVN code does not use MODIFY instead of CREATE for the places that matter in this case. The 2010 patch from the mailing list that I referred to in my original e-mail is the one that does this. Just to be clear, using these two commands (without the --allow-dups option) with the SVN version of the intarray branch gives this error: Reading in file: osm-data/ireland.osm Processing: Node(3020k) Way(0k) Relation(0)COPY_END for COPY planet_osm_nodes FROM STDIN; failed: ERROR: duplicate key value violates unique constraint planet_osm_nodes_pkey CONTEXT: COPY planet_osm_nodes, line 22424: 21043717 735445482 -55818613 \N Secondly, your patch changes some places where it does MODIFY now to using CREATE, so I imagine it's going to break some stuff in normal use. Yes, I did say that in my original e-mail. I reverted the 2008 patch from the mailing list and included that change only when enabled by my new command line option. What was originally written about the 2008 patch by you (Martijn van Oosterhout) was: : Umm, yeah. There's that. The way I solved it was with the patch below, : which is a gross hack but it works. Basically it turns every create : into a modify so it deletes any conflicting rows before inserting. It : may be the only way, but I'm still thinking on it... If you look at the diff to the code you will see that this patch applies to only one out of the three cases (the one nearest the bottom of the file). In 2008 it was a gross hack but perhaps is now considered part of the normal operation - in that case you might want to leave that change out of the new command line option. If using MODIFY instead of CREATE is slower then making it selectable on the command line would be an improvement but, as you say, it would change the current behaviour. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
Tom Hughes t...@compton.nu writes: On 21/06/10 17:19, Kai Krueger wrote: My guess would be that Nic meant: Fork (i.e. copy) the git rails_port repository. Add the desired functionality and then once done try and get it merged back into trunc. So the usual development way to get new functionality into the main site. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. In that case it may even be possible for people to choose which routing engine they want to use. Be it gosmore, some other yet to be written opensource routing engine, open route service, or even a commercial provider like cloudmade's routing. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. Nominatim is the model here - a separate server with an API the rails code can call to get what it needs. There is an OSM wiki page that lists many online routers (presumably meaning they already have a web interface), several of which are free software: http://wiki.openstreetmap.org/wiki/Routing/online_routers They offer a wide variety of features (according to the comparison table) and nearly all of them offer results as GPX format. Before I finish I should declare an interest here: I wrote one of the free software routers listed on that wiki page. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Recursive relations
Shaun McDonald sh...@shaunmcdonald.me.uk writes: On 13 Sep 2009, at 10:56, Andrew M. Bishop wrote: In OSM a relation can contain other relations but it seems that there is nothing to check that a relation doesn't contain itself. I can't think of any legitimate reason that it should be allowed though. From a data and technical point of view this has always been possible in the API intentionally. It's just that no one has come up with a good use for it yet. Since I can't see any purpose that it can serve for a route or a boundary I have edited the relations listed in my original e-mail and removed the self-recursion. As others have pointed out JOSM warned me that the relations were self-recursive when I edited them and offered to automatically fix the problem. The JOSM validator doesn't check for this condition though. For reference I created two changesets, one for route relations 12179, 15852, 80545, 101440, 163368, 165638, 167468, 170290 and one for boundary relation 57535. There are no other self-recursive relations within the UK. For the route relations the changeset is 2478746 and for the boundary relation the changeset is 2478757. Within the UK alone the following route relations all contain themselves: 12179 15852 80545 101440 163368 165638 165638 167468 168189 170290 170290 Also within the UK there is one boundary relation that contains itself: 57535 -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.demon.co.uk/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Recursive relations
In OSM a relation can contain other relations but it seems that there is nothing to check that a relation doesn't contain itself. I can't think of any legitimate reason that it should be allowed though. For example relation 15852 has contained itself since version 108 which was created at the beginning of July (changeset 1754423). Within the UK alone the following route relations all contain themselves: 12179 15852 80545 101440 163368 165638 165638 167468 168189 170290 170290 Also within the UK there is one boundary relation that contains itself: 57535 Obviously it is possible to ignore these when parsing the data (which is what I do and what alerted me to them in the first place) but it would be better if it didn't happen. This is probably something that the OSM editors should warn users about when they try to create such a relation. Is there somebody that can find and fix all such relations on the server (if they have no legitimate use)? Obviously self-recursion is easy to find; there might be mutually recursive relations as well but I haven't looked for them. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk UK OpenStreetMap Route Relations: http://www.gedanken.org.uk/mapping/osm-routes/route.html UK OpenStreetMap Boundary Relations: http://www.gedanken.org.uk/mapping/osm-boundaries/boundary.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Routino - a router for OpenStreetMap data
Steve Hosgood st...@tallyho.bc.nu writes: Claudius wrote: Am 16.04.2009 18:52, Andrew M. Bishop: An online demonstration of the router for the UK is available as well: http://www.gedanken.org.uk/mapping/router/router.html The map div of your demo page doesn't show at all in Opera browser. This is probably caused by loads of /td and /tr missing. Use the W3C HTML validator to clean up your HTML source: http://validator.w3.org/ ...and keep your good work. Looks primising. Yeah - I'll second that. I see the map display is working now, so obviously the missing tags are fixed. Thanks for the words of encouragement. It wasn't missing tags (they aren't missing, they are optional in HTML 4.01 - the page validated OK) it was a difference of opinion between browsers on how to interpret the HTML/CSS. Apparently an HTML DIV marked with height: 100% inside a table row marked as height: 100% inside a table marked as height: 100% inside an absolutely positioned DIV that fills the window height is only shown as tall as possible in Firefox v3.x and zero height otherwise. I have reverted the HTML/CSS combination to an earlier one that worked. Useful effects of this router are being able to spot broken connectivity - especially in things I've not spotted using mkgmap to create Garmin routable maps for my in-car GPS unit. For instance: things wrong with the cycleways. As soon as the OSM - 0.6 API kerfuffle is over, I've got a few cycleway edits for Swansea! Yes, I have found several of those while developing it. Also roundabouts not tagged as such (it was routing the wrong way round). I had thought about using it to find roads that are not connected to the highway network. Start somewhere that obviously is connected and try routing to one node on each other road way. I don't know how you would interpret the data though. It also wouldn't find breaks that can be routed round. May I offer a couple of comments please: 1) The positioning of the 'start' and 'end' markers for the route are a bit over-critical. The slippy map provided doesn't let you zoom in enough to place them accurately enough. Either please add a further layer of zoom to the zoom-tool, or increase the tolerance for searching for a road near the 'start' or 'end' markers. Currently it finds the closest highway node of any type. I have been thinking about limiting it to finding only nodes that are connected to highway types that you have selected to use. This should solve your problem. 2) Would be nice to mark a road type as avoid if possible. For instance, when finding a bicycle route you might like to mark primary roads in that way. I obtained a similar result by setting the speed limit for a primary as 5kph, then selecting find fastest but of course the estimated arrival time would be skewed by doing that. I'd might expect to do 20kph on a bike on a primary if I was on such a road, but would rather not be there in the first place. Sometimes, there's just no avoiding riding the bike on a primary - or even a trunk. This is already a work in progress - replacing the yes/no selection with a percentage preference. Other than that - great work, neat GUI, interesting results. Keep it up, and thanks for what you've done so far. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Routino - a router for OpenStreetMap data
[This has already been announced on the talk-gb list, but obviously there is likely to be interest here as well.] I have written a routing algorithm that uses OSM format data as its input and calculates either the shortest or quickest route between two points. You can select from any of the major OSM transport types (foot, bicycle, horse, motorbike, motorcar, goods, hgv, psv). For each of the OSM highway types (motorway, trunk, primary, secondary, tertiary, unclassified, residential, service, track, path, bridleway, cycleway, footway) you can select whether to use them and if so what speed limit. Restrictions on one-way streets, weight, height, width and length are also options. The router takes into account private/public/permissive restrictions on highways as well as tagged speed limits. What it doesn't do is barriers (gates, bollards) and turn restriction relations (which I have heard about but never seen). More information about the software and source code (Affero GPLv3 license) can be downloaded from here: http://www.gedanken.org.uk/software/routino/ An online demonstration of the router for the UK is available as well: http://www.gedanken.org.uk/mapping/router/router.html -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] JOSM Jumbo patch v8 (JOSM and validator enhancements)
Dave Hansen [EMAIL PROTECTED] writes: There was something wrong with my publishing scripts and I was copying old versions of the validator plugin from _my_ end. Can you try version 014? http://dev.openstreetmap.org/~daveh/josm/014/ That runs fine without the validator crashing - thanks for sorting it out and I hope that you keep up the good work with new validator tests. -- Andrew. -- Andrew M. Bishop [EMAIL PROTECTED] http://www.gedanken.demon.co.uk/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
Re: [OSM-dev] JOSM Jumbo patch v8 (JOSM and validator enhancements)
Dave Hansen [EMAIL PROTECTED] writes: On Wed, 2008-01-30 at 19:27 +, Andrew M. Bishop wrote: I have downloaded and tried version 10 and version 11 but they both fail in the validator with a Java error: java.lang.NoSuchMethodError: org.openstreetmap.josm.data.osm.Node.waysUsing()Ljava/util/Collection; at org.openstreetmap.josm.plugins.validator.tests.DuplicateNode.visit(DuplicateNode.java:121) at org.openstreetmap.josm.data.osm.Node.visit(Node.java:44) at org.openstreetmap.josm.plugins.validator.Test.visit(Test.java:126) at org.openstreetmap.josm.plugins.validator.ValidateAction.doValidate(ValidateAction.java:93) at org.openstreetmap.josm.plugins.validator.ValidateAction.actionPerformed(ValidateAction.java:41) at org.openstreetmap.josm.plugins.validator.ValidatorDialog.actionPerformed(ValidatorDialog.java:200) If I disable all tests on nodes then it is OK but this means the motorway tests and the duplicated nodes tests. Are you using both my version of JOSM _and_ the validator plugin? I added some new support to JOSM that my validator requires to run. Yes, I am fairly sure that I am - but how to tell? You haven't changed the version number string which means that the dialog box warning about the problem still refers to the original author. The version number in the preferences dialog is still the same as well. Perhaps you should add a specific version number string of your own to make this clearer. On the other hand the error message stacktrace seems (to me) to be saying that it is one of the new JOSM functions that it is not finding. The original validator.jar file DuplicateNode.class doesn't call the function Node.waysUsing() but the one in your new validator.jar does. This means that I must be using the new validator plugin. I also get the options to check motorway junctions which is one of your new options. I am sure that I am using the new JOSM as well because I noticed some new information (time remaining) on the upload progress dialog box. I also get loads of debugging messages that are not in the normal JOSM. Running 'java -version' says that I am using: java version 1.6.0 Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing) This is on a Linux 32-bit system that is up to date as far as I know. I haven't had any problems running the mainline JOSM versions. I have just downloaded and tried version 12 and that is the same. I have deleted every mention of validator from the preferences file and then re-enabled it but that is the same. -- Andrew. -- Andrew M. Bishop [EMAIL PROTECTED] http://www.gedanken.demon.co.uk/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
