Re: [OSM-talk] Coastline updates
On Thu, Nov 21, 2013 at 09:24:46PM -0800, Paul Norman wrote: From: Jochen Topf [mailto:joc...@remote.org] Sent: Monday, November 18, 2013 8:13 AM To: talk@openstreetmap.org Subject: [OSM-talk] Coastline updates Hi! For the last about two weeks we haven't gotten any coastline updates through. The problem is that every day there is something broken somewhere with the coastline. Often problems get fixed the same day, but new problems show up the next. Can't osmcoastline localize these problems to a small area? Coastcheck kept any errors from impacting entire continents, even when there were high numbers of errors. I remember it finding 2k errors when I started running it regularly! Osmcoastline fixes small gaps in the data. I don't know of anything more coastcheck did. If anybody can give me any algorithms how to fix the data or localize these problems, please do. We're always going to have local coastline errors, so we have to figure out how to recover from them I think we have to face it: The approach of allowing the user to add anything to the map and later fix it has failed. I strongly believed in it for the longest time. But I don't any more. We have to have stronger guarantees in the data structures and more checks in editors and in the API. Remember, the coastline is one of the easier problems there. Boundaries are much more complicated. Having spent months already on this easy problem, I don't see how our limited ressources are going to solve this if we don't tackle the problem in a very different way. Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
Hi, I think we have to face it: The approach of allowing the user to add anything to the map and later fix it has failed. I strongly believed in it for the longest time. But I don't any more. We have to have stronger guarantees in the data structures and more checks in editors and in the API. Remember, the coastline is one of the easier problems there. Boundaries are much more complicated. Having spent months already on this easy problem, I don't see how our limited ressources are going to solve this if we don't tackle the problem in a very different way. A first step might be to disallow upload of certain easily detected errors (e.g. self intersecting version=1 ways with coastline tags) either in the editor(s) or in the API. This is certainly not going to magically fix all coastline problems but it will lessen the workload of the advanced mappers who try to fix coastline problems. Regards, Patrick Petschge Kilian ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
From: Jochen Topf [mailto:joc...@remote.org] Sent: Friday, November 22, 2013 1:51 AM To: Paul Norman Cc: talk@openstreetmap.org Subject: Re: [OSM-talk] Coastline updates On Thu, Nov 21, 2013 at 09:24:46PM -0800, Paul Norman wrote: From: Jochen Topf [mailto:joc...@remote.org] Sent: Monday, November 18, 2013 8:13 AM To: talk@openstreetmap.org Subject: [OSM-talk] Coastline updates Hi! For the last about two weeks we haven't gotten any coastline updates through. The problem is that every day there is something broken somewhere with the coastline. Often problems get fixed the same day, but new problems show up the next. Can't osmcoastline localize these problems to a small area? Coastcheck kept any errors from impacting entire continents, even when there were high numbers of errors. I remember it finding 2k errors when I started running it regularly! Osmcoastline fixes small gaps in the data. I don't know of anything more coastcheck did. If anybody can give me any algorithms how to fix the data or localize these problems, please do. A particular square (400x400 mercator km, I think) containing coastline can be evaluated without considering the rest of the continent thanks to the directionality of coastline ways mattering. You'd periodically get squares that would end up incorrect, but it wouldn't generally cause incorrect results outside that square. I don't know any details of the algorithms used since I didn't write or maintain them, I just know the results. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Friday 22 November 2013, Paul Norman wrote: A particular square (400x400 mercator km, I think) containing coastline can be evaluated without considering the rest of the continent thanks to the directionality of coastline ways mattering. You'd periodically get squares that would end up incorrect, but it wouldn't generally cause incorrect results outside that square. I suggested something similar some time ago - processing in tiles would allow partial updates of the valid areas and the invalid tiles would simply be left in the previous state. Given the dynamics of the errors in the OSM inspector this would probably not cause any of the tiles to get overly old due to persistent errors. This is however not trivial to implement, it would not produce full unsplit polygons (which are sometimes needed - not for normal rendering though) and it would occasionally lead to small artefacts at the tile boundaries when one tile is changed and the other tile is not. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Wed, Nov 20, 2013 at 10:49:36PM +0100, Jochen Topf wrote: BTW it seems that the inspector doesn't show icons on errors in the lowest zoomlevel (experienced with Firefox and Opera). Works for me. What you are seeing might be due to a different known bug with OSMI: If the 180° line is in the view, it doesn't show the overlay at all. Of course this happens more often in lower zoom levels, because the chances are greater that the 180° line is inside your window. Try moving the map around or making your window smaller. Hm, you are right. I have seen the same problem now. Strange. Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
From: Jochen Topf [mailto:joc...@remote.org] Sent: Monday, November 18, 2013 8:13 AM To: talk@openstreetmap.org Subject: [OSM-talk] Coastline updates Hi! For the last about two weeks we haven't gotten any coastline updates through. The problem is that every day there is something broken somewhere with the coastline. Often problems get fixed the same day, but new problems show up the next. Can't osmcoastline localize these problems to a small area? Coastcheck kept any errors from impacting entire continents, even when there were high numbers of errors. I remember it finding 2k errors when I started running it regularly! We're always going to have local coastline errors, so we have to figure out how to recover from them ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
Jochen Topf wrote: I have now changed the coastline update process from once a day to once every 4 hours. This means updates show up quicker in the OSM Inspector at http://tools.geofabrik.de/osmi/?view=coastline . Some mappers still break the coastlines faster than you check them. I looked at a lot of the 4h-updates of your inspector and there was always an error. BTW it seems that the inspector doesn't show icons on errors in the lowest zoomlevel (experienced with Firefox and Opera). Thomas ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Wed, Nov 20, 2013 at 06:53:15PM +0100, malenki wrote: Jochen Topf wrote: I have now changed the coastline update process from once a day to once every 4 hours. This means updates show up quicker in the OSM Inspector at http://tools.geofabrik.de/osmi/?view=coastline . Some mappers still break the coastlines faster than you check them. I looked at a lot of the 4h-updates of your inspector and there was always an error. I have seen the same thing. :-( BTW it seems that the inspector doesn't show icons on errors in the lowest zoomlevel (experienced with Firefox and Opera). Works for me. What you are seeing might be due to a different known bug with OSMI: If the 180° line is in the view, it doesn't show the overlay at all. Of course this happens more often in lower zoom levels, because the chances are greater that the 180° line is inside your window. Try moving the map around or making your window smaller. Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Monday 18 November 2013, Jochen Topf wrote: Hi! For the last about two weeks we haven't gotten any coastline updates through. The problem is that every day there is something broken somewhere with the coastline. Often problems get fixed the same day, but new problems show up the next. I have now changed the coastline update process from once a day to once every 4 hours. This means updates show up quicker in the OSM Inspector at http://tools.geofabrik.de/osmi/?view=coastline . Sounds good. This should also increase the chance that if you see an error in the OSMI it is not already fixed (which is very common at the moment and somewhat annoying of course). How sensitive is your system by the way in discarding a coastline as broken? Obviously any unfixable error in one of the four large continental coastlines will trigger it. Same for large islands like Greenland and New Guinea i suppose. But there probably is a limit. There have for example been many coastline errors in the Philippines recently due to recent mapping activity there but many on fairly small islands. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates still running?
On Wed, 2012-04-11 at 20:41 +0100, OJ W wrote: Is there some problem with the coastline at Doha airport? The new coastline (changed since February) doesn't yet appear in rendered maps, but looks reasonable in the Edit view: http://www.openstreetmap.org/?lat=25.24915lon=51.61024zoom=15layers=M I have just deployed an updated set of coastline shapefiles which look like they will fix the problem once the area is rendered again. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
- Original Message - From: Jochen Topf joc...@remote.org To: OSM Talk talk@openstreetmap.org Sent: Friday, February 25, 2011 4:03 PM Subject: [OSM-talk] Coastline updates Hi! I have edited coastlines in Cornwall last summer and they are still not updated on the main map. (Easy to see because the border and coastline share the same location in the data but show up differently on the map.) The coastline error checker says: Last update of coastline errors: Wed Apr 14 13:19:17 UTC 2010 It also says it needs new hosting. I guess thats the problem. Can we get this going again? What exactly is needed? The error checker hasn't been working for a long time, but I'm sure that coastline edits I made in January have since shown up on the map David Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
- Original Message - From: Jochen Topf joc...@remote.org To: OSM Talk talk@openstreetmap.org Sent: Friday, February 25, 2011 4:03 PM Subject: [OSM-talk] Coastline updates Hi! I have edited coastlines in Cornwall last summer and they are still not updated on the main map. (Easy to see because the border and coastline share the same location in the data but show up differently on the map.) Can you gave an example location? David The coastline error checker says: Last update of coastline errors: Wed Apr 14 13:19:17 UTC 2010 It also says it needs new hosting. I guess thats the problem. Can we get this going again? What exactly is needed? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Fri, 25 Feb 2011 17:39:46 -, David Groom wrote: I have edited coastlines in Cornwall last summer and they are still not updated on the main map. (Easy to see because the border and coastline share the same location in the data but show up differently on the map.) Can you gave an example location? Same problem here: http://osm.org/go/xZHqTlPn http://www.openstreetmap.org/browse/way/33900429 The shape was changed in January, the other revisions are for added nodes (slipways or other similar things). -- . ''`. Debian developer | http://wiki.debian.org/DavidPaleino : :' : Linuxer #334216 --|-- http://www.hanskalabs.net/ `. `'` GPG: 1392B174 | http://deb.li/dapal `- 2BAB C625 4E66 E7B8 450A C3E1 E6AA 9017 1392 B174 signature.asc Description: PGP signature ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
Is there any reason to still have the NPE layer accessible from the editors. It was useful in the pre-OS/Bing days but seems like a liability now? Kevin On 25 February 2011 17:47, Ed Loach e...@loach.me.uk wrote: I have edited coastlines in Cornwall last summer and they are still not updated on the main map. (Easy to see because the border and coastline share the same location in the data but show up differently on the map.) Can you gave an example location? I had a quick look around the coastline, but the first couple of discrepancies I found are recent edits (one was 6th February, the other in January http://www.openstreetmap.org/browse/way/95126825 - this one I found interesting as the coastline that was based on the boundary has been updated to NPE which is over 50 years old) Ed ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
- Original Message - From: David Paleino da...@debian.org To: talk@openstreetmap.org Sent: Friday, February 25, 2011 5:57 PM Subject: Re: [OSM-talk] Coastline updates David the way you refer to was last edited on 1 Feb 2011, I would not necessarily expect to see the results of those changes reflected yet in the coastline on the Mapnik layer. see http://wiki.openstreetmap.org/wiki/Coastline#Main_Mapnik_layer David ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Fri, 25 Feb 2011 18:35:41 -, David Groom wrote: David the way you refer to was last edited on 1 Feb 2011, [..] You probably haven't read my whole message :) The last changes are just added nodes, there is no change in the shape of the coastline. The last changes in shape happened in January. David -- . ''`. Debian developer | http://wiki.debian.org/DavidPaleino : :' : Linuxer #334216 --|-- http://www.hanskalabs.net/ `. `'` GPG: 1392B174 | http://deb.li/dapal `- 2BAB C625 4E66 E7B8 450A C3E1 E6AA 9017 1392 B174 signature.asc Description: PGP signature ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
Hi! Falmouth: http://www.openstreetmap.org/?lat=50.15364lon=-5.0639zoom=17layers=M Looks ok now in this zoom level. Still some errors if you zoom in. I had a look at it again and marked some tile manually as dirty and I get the correct rendering then. So the problem seems not to be the coastline stuff but that there are tiles that haven't been re-rendered for half a year!? Jochen On Fri, Feb 25, 2011 at 05:39:46PM -, David Groom wrote: Date: Fri, 25 Feb 2011 17:39:46 - From: David Groom revi...@pacific-rim.net To: OSM Talk talk@openstreetmap.org Subject: Re: [OSM-talk] Coastline updates - Original Message - From: Jochen Topf joc...@remote.org To: OSM Talk talk@openstreetmap.org Sent: Friday, February 25, 2011 4:03 PM Subject: [OSM-talk] Coastline updates Hi! I have edited coastlines in Cornwall last summer and they are still not updated on the main map. (Easy to see because the border and coastline share the same location in the data but show up differently on the map.) Can you gave an example location? David The coastline error checker says: Last update of coastline errors: Wed Apr 14 13:19:17 UTC 2010 It also says it needs new hosting. I guess thats the problem. Can we get this going again? What exactly is needed? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Fri, 2011-02-25 at 20:01 +0100, Jochen Topf wrote: Hi! Falmouth: http://www.openstreetmap.org/?lat=50.15364lon=-5.0639zoom=17layers=M Looks ok now in this zoom level. Still some errors if you zoom in. I had a look at it again and marked some tile manually as dirty and I get the correct rendering then. So the problem seems not to be the coastline stuff but that there are tiles that haven't been re-rendered for half a year!? I updated the coastline shapefiles used by the mapnik layer last night. The previous update was about 4 weeks ago. If no one edits any data in an area after the coastline shapefiles are changed then it may be a long time before the tiles are rendered again to pick up the changes (as you are seeing here). Normally I force the zoom 0 - 12 tiles to re-render across the world each time I update the coastline shapefiles but it takes too long to do all zoom levels on each update. Back in January I ran a process to re-rendered all tiles last rendered prior to 1st Oct 2010 across the full 0-18 zoom range. This took a few weeks to complete and should have picked you older changes. I will consider running it again with a more recent cut-off point. Perhaps some of the other discrepancies are due to newer edits to the coastlines? Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On 25-2-2011 17:03, Jochen Topf wrote: The coastline error checker says: Last update of coastline errors: Wed Apr 14 13:19:17 UTC 2010 It also says it needs new hosting. I guess thats the problem. The osm.org tile server generates its own coastline shapefiles. It doesn't use the ones from the coastline error checker. FWIW: curl --head http://tile.openstreetmap.org/processed_p.tar.bz2 Last-Modified: Fri, 25 Feb 2011 09:24:39 GMT curl --head http://tile.openstreetmap.org/shoreline_300.tar.bz2 Last-Modified: Fri, 25 Feb 2011 09:22:31 GMT I don't know if the caching influences these dates. Can we get this going again? What exactly is needed? A server with sufficient storage and bandwidth to process a weekly planet, daily diffs, and many hours of single task processing per coastline run. -- Lennard ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On Fri, Feb 25, 2011 at 09:24:19PM +0100, Lennard wrote: Can we get this going again? What exactly is needed? A server with sufficient storage and bandwidth to process a weekly planet, daily diffs, and many hours of single task processing per coastline run. We have a server that has planet file kept current each day for Taginfo. Maybe it can do the coastline check, too. It only has PBF files, though which the coastline checker doesn't understand. But we could work around that or change it. Storage should also not be a problem, but RAM could be. The machine has only 16 GB. Anybody know how much RAM the coastline checker needs? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Coastline updates
On 25-2-2011 22:02, Jochen Topf wrote: We have a server that has planet file kept current each day for Taginfo. Maybe it can do the coastline check, too. It only has PBF files, though which the coastline checker doesn't understand. But we could work around that or change it. Storage should also not be a problem, but RAM could be. The machine has only 16 GB. Anybody know how much RAM the coastline checker needs? The first stage of the coastline checker tool chain is a modified copy of an old revision of osm2pgsql. It could be adapted for pbf, same as the current osm2pgsql. It does two passes over the data, that's why a simple pbf2osm pipe doesn't work. The rest of the tool chain works on the resulting interim files, and wouldn't need to be adapted for a pbf planet. 16GB should be plenty. Last year I could still do a complete coastline run with a 4GB server, and it probably needs less than that. -- Lennard ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
