[OSM-talk] Something Might be Broken
Take a look at this boundary where a forest and national park meet: http://osm.org/go/TwUljNo-- Notice that the boundaries don't line up. This is because the national park is in slightly the wrong place. The national park is this changeset uploaded yesterday: http://www.openstreetmap.org/browse/changeset/1980439 Today I moved the national park into the correct position. The changeset was closed at 31 Jul 00:09: http://www.openstreetmap.org/browse/changeset/1989864 I then marked the tile you are looking at as dirty. It was apparently rendered by Mapnik on 31 Jul 03:21: http://a.tile.openstreetmap.org/12/772/1608.png/status As you can see the data from my new changeset has not been used. On 31 Jul 01:33 I added a new changeset with some trails: http://www.openstreetmap.org/browse/changeset/1990063 This was rendered with trails at 31 Jul 03:13: http://a.tile.openstreetmap.org/13/1567/3318.png/status If data I uploaded at 01:33 was rendered at 3:13, how come data I uploaded at 00:09 has not been rendered at the time of writing this? (03:21)? One clue might be that the trails are new data but the movement of nodes was not. Also JOSM gave me an error of "unexpected end of file" when the changeset was closing, but the changeset is listed in my edits as being closed anyway. It also has all 23573 nodes. I have cleared my browser cache and tried two browsers. I have two other examples of different data/changesets that I just cannot get Mapnik to render it. In both cases some of the data is rendered. One of those I've asked for help on here and the Mapnik list with no solution. I've tried everything I can think of. I don't know what the Osmarender update speed is or how to mark tiles as dirty or find out when they were rendered, so I am unsure if Osmarender tiles can be directly compared. Any help is greatly appreciated, otherwise I am losing confidence. Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On Thu, Jul 30, 2009 at 7:29 PM, Andrew Ayre wrote: > Take a look at this boundary where a forest and national park meet: > > http://osm.org/go/TwUljNo-- > > Notice that the boundaries don't line up. This is because the national > park is in slightly the wrong place. The national park is this changeset > uploaded yesterday: > > http://www.openstreetmap.org/browse/changeset/1980439 > > Today I moved the national park into the correct position. The changeset > was closed at 31 Jul 00:09: > > http://www.openstreetmap.org/browse/changeset/1989864 > > I then marked the tile you are looking at as dirty. It was apparently > rendered by Mapnik on 31 Jul 03:21: > > http://a.tile.openstreetmap.org/12/772/1608.png/status > > As you can see the data from my new changeset has not been used. > > On 31 Jul 01:33 I added a new changeset with some trails: > > http://www.openstreetmap.org/browse/changeset/1990063 > > This was rendered with trails at 31 Jul 03:13: > > http://a.tile.openstreetmap.org/13/1567/3318.png/status > > If data I uploaded at 01:33 was rendered at 3:13, how come data I > uploaded at 00:09 has not been rendered at the time of writing this? > (03:21)? > > One clue might be that the trails are new data but the movement of nodes > was not. Also JOSM gave me an error of "unexpected end of file" when the > changeset was closing, but the changeset is listed in my edits as being > closed anyway. It also has all 23573 nodes. > > I have cleared my browser cache and tried two browsers. > > I have two other examples of different data/changesets that I just > cannot get Mapnik to render it. In both cases some of the data is > rendered. One of those I've asked for help on here and the Mapnik list > with no solution. I've tried everything I can think of. > > I don't know what the Osmarender update speed is or how to mark tiles as > dirty or find out when they were rendered, so I am unsure if Osmarender > tiles can be directly compared. > > Any help is greatly appreciated, otherwise I am losing confidence. > > Andy > > If the boundary is a relation, that may be the reason. (Since you said it has 23573 nodes, then it must be a boundary relation.) AFAIK, Mapnik (or more properly, osm2pgsql) currently doesn't process relations for diffs. You'll have to wait until the planet reload after next Wed to see the border update. Karl ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
I don't know what the Osmarender update speed is or how to mark tiles as dirty or find out when they were rendered, so I am unsure if Osmarender tiles can be directly compared. osmarender doesn't work currently for large areas defined by relation boundaries there is a lonly white tile http://tah.openstreetmap.org/Browse/?layer=tile&z=12&x=729&y=1638 you can mark them as dirty on the above page or more convenient in zoom12 at http://www.informationfreeway.org/ Any help is greatly appreciated, otherwise I am losing confidence. Andy If the boundary is a relation, that may be the reason. (Since you said it has 23573 nodes, then it must be a boundary relation.) AFAIK, Mapnik (or more properly, osm2pgsql) currently doesn't process relations for diffs. You'll have to wait until the planet reload after next Wed to see the border update. can't confirm this. I am doing some corrections on large parks. made an update of a relation and added one more. no changes on any node. ties are marked as rerendered within < 10min but the update of the area tag didn't change. Neither for the area with a relation nor the area with a polygon and an area tag It varies also widely in different zoom levels. some observations so far status is updated earlier than the tiles. guess there is a lag until the tiles is available on all tile caches. I give it some time after a dirty request and reload .png to clean it from the local browser cache even after weeks it's not rendered. some tiles are new others aren't for the exact same polygon an example is here, same relation one tile is rendered today with new pattern one still has the old pattern. both are reported clean only relation was updated not way or node changes http://a.tile.openstreetmap.org/16/10469/25279.png http://a.tile.openstreetmap.org/16/10469/25280.png zoom level > 16 are rendered zoom level < 16 are not rendered both are reported clean Karl ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On 31 Jul 2009, at 04:41, Karl Newman wrote: On Thu, Jul 30, 2009 at 7:29 PM, Andrew Ayre wrote: Take a look at this boundary where a forest and national park meet: http://osm.org/go/TwUljNo-- Notice that the boundaries don't line up. This is because the national park is in slightly the wrong place. The national park is this changeset uploaded yesterday: http://www.openstreetmap.org/browse/changeset/1980439 Today I moved the national park into the correct position. The changeset was closed at 31 Jul 00:09: http://www.openstreetmap.org/browse/changeset/1989864 I then marked the tile you are looking at as dirty. It was apparently rendered by Mapnik on 31 Jul 03:21: http://a.tile.openstreetmap.org/12/772/1608.png/status As you can see the data from my new changeset has not been used. On 31 Jul 01:33 I added a new changeset with some trails: http://www.openstreetmap.org/browse/changeset/1990063 This was rendered with trails at 31 Jul 03:13: http://a.tile.openstreetmap.org/13/1567/3318.png/status If data I uploaded at 01:33 was rendered at 3:13, how come data I uploaded at 00:09 has not been rendered at the time of writing this? (03:21)? One clue might be that the trails are new data but the movement of nodes was not. Also JOSM gave me an error of "unexpected end of file" when the changeset was closing, but the changeset is listed in my edits as being closed anyway. It also has all 23573 nodes. I have cleared my browser cache and tried two browsers. I have two other examples of different data/changesets that I just cannot get Mapnik to render it. In both cases some of the data is rendered. One of those I've asked for help on here and the Mapnik list with no solution. I've tried everything I can think of. I don't know what the Osmarender update speed is or how to mark tiles as dirty or find out when they were rendered, so I am unsure if Osmarender tiles can be directly compared. Any help is greatly appreciated, otherwise I am losing confidence. Andy If the boundary is a relation, that may be the reason. (Since you said it has 23573 nodes, then it must be a boundary relation.) AFAIK, Mapnik (or more properly, osm2pgsql) currently doesn't process relations for diffs. You'll have to wait until the planet reload after next Wed to see the border update. Wrong, osm2pgsql does process relations properly. If they aren't then Jon Burgess is happy to take a look to see if he can fix the problem with osm2pgsql. Second there has been no planet reload for a few weeks now. As http://www.openstreetmap.org/browse/changeset/1989864 is such a large changeset, what has probably happened is that it took so long to process that it missed the minute diffs, thus didn't get into any downstream services that use the minute diffs. This is a problem that is known and work is in progress to change the way that the minute diffs are generated so that you don't get this problem any more. How long did it take to process the upload? Don't worry the data is saved in the main OSM database. Shaun___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
> Wrong, osm2pgsql does process relations properly. If they aren't then > Jon Burgess is happy to take a look to see if he can fix the problem > with osm2pgsql. Second there has been no planet reload for a few weeks > now. There's definitely something wrong here: http://www.openstreetmap.org/?lat=49.93906&lon=10.9213&zoom=17&layers=B000FTF The building called "Angewandte Informatik" is a multipolygon, which has been moved one and a half weeks ago. Both the old and the new shape are rendered now, and the hole is filled too. I know that there have been problems with multipolygons and diffs. Are they supposed to be fixed? Regards, Marc -- Neu: GMX Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate für nur 19,99 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
2009/7/31 "Marc Schütz" : >> Wrong, osm2pgsql does process relations properly. If they aren't then >> Jon Burgess is happy to take a look to see if he can fix the problem >> with osm2pgsql. Second there has been no planet reload for a few weeks >> now. > > There's definitely something wrong here: > http://www.openstreetmap.org/?lat=49.93906&lon=10.9213&zoom=17&layers=B000FTF > > The building called "Angewandte Informatik" is a multipolygon, which has been > moved one and a half weeks ago. Both the old and the new shape are rendered > now, and the hole is filled too. > > I know that there have been problems with multipolygons and diffs. Are they > supposed to be fixed? Please file a trac ticket with the details and assign it to me. Lots of issues have been fixed but there are still several possible reasons why things some times don't work correctly. It takes time to analyse, diagnose & fix each example. -- Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
> > The building called "Angewandte Informatik" is a multipolygon, which has > been moved one and a half weeks ago. Both the old and the new shape are > rendered now, and the hole is filled too. > > > > I know that there have been problems with multipolygons and diffs. Are > they supposed to be fixed? > > Please file a trac ticket with the details and assign it to me. Lots > of issues have been fixed but there are still several possible reasons > why things some times don't work correctly. It takes time to analyse, > diagnose & fix each example. Done: http://trac.openstreetmap.org/ticket/2116 BTW, the link I gave was wrong; here is the correct one: http://www.openstreetmap.org/?lat=49.926286&lon=11.585866&zoom=18&layers=B000FTF Regards, Marc -- Neu: GMX Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate für nur 19,99 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Jon Burgess wrote: > 2009/7/31 "Marc Schütz" : >>> Wrong, osm2pgsql does process relations properly. If they aren't then >>> Jon Burgess is happy to take a look to see if he can fix the problem >>> with osm2pgsql. Second there has been no planet reload for a few weeks >>> now. >> There's definitely something wrong here: >> http://www.openstreetmap.org/?lat=49.93906&lon=10.9213&zoom=17&layers=B000FTF >> >> The building called "Angewandte Informatik" is a multipolygon, which has >> been moved one and a half weeks ago. Both the old and the new shape are >> rendered now, and the hole is filled too. >> >> I know that there have been problems with multipolygons and diffs. Are they >> supposed to be fixed? > > Please file a trac ticket with the details and assign it to me. Lots > of issues have been fixed but there are still several possible reasons > why things some times don't work correctly. It takes time to analyse, > diagnose & fix each example. Done. See: http://trac.openstreetmap.org/ticket/2118 I add add tickets for the other two issues I referred to later today. Thanks! Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On Fri, 2009-07-31 at 09:36 -0700, Andrew Ayre wrote: > Done. See: > >http://trac.openstreetmap.org/ticket/2118 > > I add add tickets for the other two issues I referred to later today. > Thanks! As Shaun mentioned in another email, this seems to be another instance of nodes missing from the minutely diffs. This is a known issue but I'm not sure if we have a trac ticket for it. I have put more details into the trac ticket. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Jon Burgess wrote: > On Fri, 2009-07-31 at 09:36 -0700, Andrew Ayre wrote: >> Done. See: >> >>http://trac.openstreetmap.org/ticket/2118 >> >> I add add tickets for the other two issues I referred to later today. >> Thanks! > > As Shaun mentioned in another email, this seems to be another instance > of nodes missing from the minutely diffs. This is a known issue but I'm > not sure if we have a trac ticket for it. I have put more details into > the trac ticket. > > Jon Is there a history of when full imports where made listed somewhere, or is it on a predictable schedule? I'm wondering if some other things I've seen are the same problem or something else. Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Jon Burgess wrote: > On Fri, 2009-07-31 at 09:36 -0700, Andrew Ayre wrote: >> Done. See: >> >>http://trac.openstreetmap.org/ticket/2118 >> >> I add add tickets for the other two issues I referred to later today. >> Thanks! > > As Shaun mentioned in another email, this seems to be another instance > of nodes missing from the minutely diffs. This is a known issue but I'm > not sure if we have a trac ticket for it. I have put more details into > the trac ticket. > > Jon Jon, Thanks for the details added to the ticket. I think if the OSM API was improved so it could accept large changesets faster then that would greatly help out the people who are trying to add large amounts of data. So far I haven't see a fast and reliable method. Perhaps people can apply for access to a second API for large changesets only? Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On 31/07/09 23:55, Andrew Ayre wrote: > I think if the OSM API was improved so it could accept large changesets > faster then that would greatly help out the people who are trying to add > large amounts of data. So far I haven't see a fast and reliable method. Excuse me a minute while I find my magic wand... [ time passes ] ...found it! There you go, changeset processing is now 100 times faster. > Perhaps people can apply for access to a second API for large changesets > only? How exactly is that supposed to help? Will this API have access to some magic accelerator technology that the current API doesn't use? Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Tom Hughes wrote: > On 31/07/09 23:55, Andrew Ayre wrote: > >> I think if the OSM API was improved so it could accept large changesets >> faster then that would greatly help out the people who are trying to add >> large amounts of data. So far I haven't see a fast and reliable method. > > Excuse me a minute while I find my magic wand... > > [ time passes ] > > ...found it! There you go, changeset processing is now > 100 times faster. > >> Perhaps people can apply for access to a second API for large changesets >> only? > > How exactly is that supposed to help? Will this API have access to some > magic accelerator technology that the current API doesn't use? It would help because people could upload large data sets as fast as they can prepare them, then tweak any problems quickly and efficiently. I was perhaps thinking of a dedicated machine/bandwidth for large uploads. But then I don't know what the bottleneck is in the current system. Is it users, bandwidth, processing power, something else or a combination? A couple of weeks ago JOSM told me it would take 15 hours to upload a 5Mb OSM file. It seems a bit better recently though. Anyway, just throwing out ideas borne of frustration. Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On Fri, 2009-07-31 at 15:58 -0700, Andrew Ayre wrote: > > As Shaun mentioned in another email, this seems to be another > instance > > of nodes missing from the minutely diffs. This is a known issue but > I'm > > not sure if we have a trac ticket for it. I have put more details > into > > the trac ticket. > > > > Jon > > Is there a history of when full imports where made listed somewhere, > or > is it on a predictable schedule? There is no log of the full imports. They generally happen every few weeks on a Thursday or Friday. > I'm wondering if some other things I've > seen are the same problem or something else. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
On 01/08/09 01:08, Andrew Ayre wrote: >> How exactly is that supposed to help? Will this API have access to >> some magic accelerator technology that the current API doesn't use? > > It would help because people could upload large data sets as fast as > they can prepare them, then tweak any problems quickly and efficiently. I understand why a faster API helps. What I don't understand is why you think there is some magic switch we can flip that would instantly make it faster - if such a switch existed we would already have flipped it! > I was perhaps thinking of a dedicated machine/bandwidth for large > uploads. But then I don't know what the bottleneck is in the current > system. Is it users, bandwidth, processing power, something else or a > combination? I suspect the limitation is primarily how fast you can do all the processing of the changeset and the required database queries and having a different client machine won't make the slightest difference to that. You might be able to get some speedup in the XML parsing but not much more than that I suspect. The real way to speed it up (in so far as is possible given the constraints of the database queries that need to be run) is to rewrite the API in a more efficient language, which would benefit everybody rather than a select elite. There has been some work done towards such a rewrite, but only for the map call so far. > A couple of weeks ago JOSM told me it would take 15 hours to upload a > 5Mb OSM file. It seems a bit better recently though. That number must be plucked out of thin air though - there is no reasonable way that JOSM can know how long it will take. I've never seen a changeset take anywhere near that long to apply though. Tom -- Tom Hughes (t...@compton.nu) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Tom Hughes wrote: > Excuse me a minute while I find my magic wand... > > [ time passes ] > > ...found it! There you go, changeset processing is now > 100 times faster. > > Once you are done with your magic wand, do you mind lending it to me, I have a server to optimize. Emilie Laffray signature.asc Description: OpenPGP digital signature ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Something Might be Broken
Jon Burgess wrote: > On Fri, 2009-07-31 at 15:58 -0700, Andrew Ayre wrote: >>> As Shaun mentioned in another email, this seems to be another >> instance >>> of nodes missing from the minutely diffs. This is a known issue but >> I'm >>> not sure if we have a trac ticket for it. I have put more details >> into >>> the trac ticket. >>> >>> Jon >> Is there a history of when full imports where made listed somewhere, >> or >> is it on a predictable schedule? > > There is no log of the full imports. They generally happen every few > weeks on a Thursday or Friday. > >> I'm wondering if some other things I've >> seen are the same problem or something else. Thanks Jon. I'll wait until the next full import to see if the other issues I have are still there. In either case do you want the details? Andy -- Andy PGP Key ID: 0xDC1B5864 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
