Re: [OSM-talk] Tile refresh on openstreetmap.org
On 02/03/14 12:12, Christoph Hormann wrote: On Sunday 02 March 2014, Tom Hughes wrote: http://a.tile.openstreetmap.org/11/1577/243.png/status yevaud: Dirty. Last rendered at Sat Feb 22 03:13:25 2014 orm: Clean. Last rendered at Sun Mar 02 11:43:19 2014. http://b.tile.openstreetmap.org/12/3154/483.png/status yevaud: Dirty. Last rendered at Sun Feb 23 02:00:44 2014. orm: Clean. Last rendered at Sun Mar 02 11:43:20 2014. Based on the status i got from the tile server these two were last rendered on Jan 11 and Jan 30 before the 11:43 render today. The third one mentioned in the other mail is still on Jan 11 (don't know how long it will stay of course) If they have been rendered in between the update did not make it to the tile distribution apparently. There seem to be some bugs - firstly the monthly rerender has not been triggering, which I think I have now fixed: http://git.openstreetmap.org/chef.git/commitdiff/009639159796063757101d372fc5d0da8c250169 Secondly, it looks like the z11+12 renders on a stylesheet change have not been completing. I think the fix is this: https://github.com/openstreetmap/mod_tile/commit/13da3afe709434e2237111bb2bfd34ee4d17d08c So I have asked Kai if he can do a new build with that in. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Sunday 02 March 2014, Tom Hughes wrote: > > > > http://a.tile.openstreetmap.org/11/1577/243.png/status > > yevaud: Dirty. Last rendered at Sat Feb 22 03:13:25 2014 > orm: Clean. Last rendered at Sun Mar 02 11:43:19 2014. > > > http://b.tile.openstreetmap.org/12/3154/483.png/status > > yevaud: Dirty. Last rendered at Sun Feb 23 02:00:44 2014. > orm: Clean. Last rendered at Sun Mar 02 11:43:20 2014. Based on the status i got from the tile server these two were last rendered on Jan 11 and Jan 30 before the 11:43 render today. The third one mentioned in the other mail is still on Jan 11 (don't know how long it will stay of course) If they have been rendered in between the update did not make it to the tile distribution apparently. > The four recent updates - v2.9.0, v2.9.1, v2.10.0 and v2.10.1 have > come so quickly that the z11+12 renders probably haven't been > finishing before the next one rolls out. That is highly abnormal > though. > > Andy has gone away for a few days now though, so this one should get > a chance to finish ;-) Ok - no sweat. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 02/03/14 11:43, Christoph Hormann wrote: On Sunday 02 March 2014, Tom Hughes wrote: That pretty much explains my observations. By having one or several stylesheet updates within the monthly update cycle the actual update of some tiles is stretched from a month to the ~7 weeks i observed. I have no idea how you reach that conclusion - literally everything up to z12 has been rerendered at least four times in the last month. http://a.tile.openstreetmap.org/11/1577/243.png/status yevaud: Dirty. Last rendered at Sat Feb 22 03:13:25 2014 orm: Clean. Last rendered at Sun Mar 02 11:43:19 2014. http://b.tile.openstreetmap.org/12/3154/483.png/status yevaud: Dirty. Last rendered at Sun Feb 23 02:00:44 2014. orm: Clean. Last rendered at Sun Mar 02 11:43:20 2014. So orm has already rendered both using v2.10.1 which was deployed on Friday evening while yevaud last rendered one of them with v2.9.0 and the other with v2.9.1. The four recent updates - v2.9.0, v2.9.1, v2.10.0 and v2.10.1 have come so quickly that the z11+12 renders probably haven't been finishing before the next one rolls out. That is highly abnormal though. Andy has gone away for a few days now though, so this one should get a chance to finish ;-) Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Sunday 02 March 2014, Tom Hughes wrote: > > > > That pretty much explains my observations. By having one or > > several stylesheet updates within the monthly update cycle the > > actual update of some tiles is stretched from a month to the ~7 > > weeks i observed. > > I have no idea how you reach that conclusion - literally everything > up to z12 has been rerendered at least four times in the last month. Seems updates are progressing fast now - the previously mentioned tiles are now refreshed. Still got: http://c.tile.openstreetmap.org/11/1592/246.png/status from Sat Jan 11 04:17:46 2014 -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Sunday 02 March 2014, Tom Hughes wrote: > > > > That pretty much explains my observations. By having one or > > several stylesheet updates within the monthly update cycle the > > actual update of some tiles is stretched from a month to the ~7 > > weeks i observed. > > I have no idea how you reach that conclusion - literally everything > up to z12 has been rerendered at least four times in the last month. http://a.tile.openstreetmap.org/11/1577/243.png/status http://b.tile.openstreetmap.org/12/3154/483.png/status -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 02/03/14 11:20, Christoph Hormann wrote: On Sunday 02 March 2014, Tom Hughes wrote: Well - in principle i think re-rendering of very old tiles (like >2 weeks age) at the low zooms should take precedence over style induced updates at the high zooms. Not sure how feasible it is to implement this. They do. The algorithm goes something like this: * Update stylesheet, remembering time * Render z0 to z10 in the background * Mark planet as dirty, using saved time * Start rendering z11 to z12 in the background [...] Note that nothing in z0 to z12 is marked dirty as a result of changes to the data, so they only re-render when the style changes, or once a month as a background job. That pretty much explains my observations. By having one or several stylesheet updates within the monthly update cycle the actual update of some tiles is stretched from a month to the ~7 weeks i observed. I have no idea how you reach that conclusion - literally everything up to z12 has been rerendered at least four times in the last month. For tiles up to z12 a style change actually causes it to be rerendered more quickly than would otherwise happen. Anything beyond z12 is entirely dependent on how busy the tile server you are using is and, other than the last two weeks when we have had four style updates, there should have been no problem with rerendering it if necessary. > The solution i could envision would be to have a single render > queue for the pre-rendered tiles that is prioritized based on age > of the existing tile, possibly modulated with the zoom level (i.e. > the age of the lowest zooms could be considered more severe than > z11 and z12). Given that all tiles up to z12 have basically the same age, because they are only rerendered as part of a batch update, it's not clear to me how exactly this is supposed to work. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Sunday 02 March 2014, Tom Hughes wrote: > > > > Well - in principle i think re-rendering of very old tiles (like >2 > > weeks age) at the low zooms should take precedence over style > > induced updates at the high zooms. Not sure how feasible it is to > > implement this. > > They do. The algorithm goes something like this: > > * Update stylesheet, remembering time > * Render z0 to z10 in the background > * Mark planet as dirty, using saved time > * Start rendering z11 to z12 in the background > > [...] > > Note that nothing in z0 to z12 is marked dirty as a result of changes > to the data, so they only re-render when the style changes, or once a > month as a background job. That pretty much explains my observations. By having one or several stylesheet updates within the monthly update cycle the actual update of some tiles is stretched from a month to the ~7 weeks i observed. The solution i could envision would be to have a single render queue for the pre-rendered tiles that is prioritized based on age of the existing tile, possibly modulated with the zoom level (i.e. the age of the lowest zooms could be considered more severe than z11 and z12). This means the queue always renders the tile which has the highest need for update at the moment and you do not have the problem that several style sheet updates induce rapid re-renders of some tiles but at the same time delay updates of other tiles. In fact such a system would be fully agnostic to style sheet changes but still make sure they show up in the map as fast as possible. Not sure if this is possible with the current render stack of course. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 02/03/14 10:03, Christoph Hormann wrote: On Sunday 02 March 2014, Tom Hughes wrote: The load from the re-rendering job is adjustable. Of course, if you go for a low load, then the re-rendering job will take a lot longer. Not really - the load we are talking about is not the pre-rendering of low zoom, rather it is the load from normal tile requests triggering re-rendering of high zoom tiles. Well - in principle i think re-rendering of very old tiles (like >2 weeks age) at the low zooms should take precedence over style induced updates at the high zooms. Not sure how feasible it is to implement this. They do. The algorithm goes something like this: * Update stylesheet, remembering time * Render z0 to z10 in the background * Mark planet as dirty, using saved time * Start rendering z11 to z12 in the background So only at the third step will user requests starting triggering a rerender for tiles which are older then than the stylesheet. At that point z0 to z10 are already done (only takes a few hours). The rendering of z11 to z12 does overlap, but we normally arrange for that to happen overnight so that it can be done before load picks up in the morning. Both background renders are limited to one quarter of the available render threads, and will suspend themselves if the load gets too high. Note that nothing in z0 to z12 is marked dirty as a result of changes to the data, so they only re-render when the style changes, or once a month as a background job. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Sunday 02 March 2014, Tom Hughes wrote: > > > > The load from the re-rendering job is adjustable. Of course, if you > > go for a low load, then the re-rendering job will take a lot > > longer. > > Not really - the load we are talking about is not the pre-rendering > of low zoom, rather it is the load from normal tile requests > triggering re-rendering of high zoom tiles. Well - in principle i think re-rendering of very old tiles (like >2 weeks age) at the low zooms should take precedence over style induced updates at the high zooms. Not sure how feasible it is to implement this. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 02/03/14 04:49, Paul Norman wrote: Would there be a way to avoid large rendering queues with every update? Rolling out the stylesheets region by region might solve this problem, but that would probably be nearly impossible to set up and maintain. I don't really see how we can avoid this problem, but perhaps anyone else has an idea? The load from the re-rendering job is adjustable. Of course, if you go for a low load, then the re-rendering job will take a lot longer. Not really - the load we are talking about is not the pre-rendering of low zoom, rather it is the load from normal tile requests triggering re-rendering of high zoom tiles. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
> From: Matthijs Melissen [mailto:i...@matthijsmelissen.nl] > Sent: Saturday, March 01, 2014 5:28 PM > To: talk@openstreetmap.org > Subject: Re: [OSM-talk] Tile refresh on openstreetmap.org > > Would there be a way to avoid large rendering queues with > every update? Rolling out the stylesheets region by region might solve > this problem, but that would probably be nearly impossible to set up > and maintain. I don't really see how we can avoid this problem, but > perhaps anyone else has an idea? The load from the re-rendering job is adjustable. Of course, if you go for a low load, then the re-rendering job will take a lot longer. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 28 February 2014 15:26, Tom Hughes wrote: > On 28/02/14 15:05, Christoph Hormann wrote: >> I have the impression the rerendering of tiles on openstreetmap.org is >> very slow at the moment - according to /status various z=11 tiles are >> from January 11. Also /dirty no more seems to work (at least not on >> the lower zoom levels). >> Has there been a change in configuration recently that caused this? > > There have been several stylesheet updates in the last week so the servers > are very busy and are likely to serve old tiles if they are too busy to > rerender them. The stylesheets have seen a lot of development effort recently: both the number of new bugs and features suggested and the number of new users in the repository have increased a lot. It can therefore be expected that we will keep seeing regular stylesheet updates in the future. Would there be a way to avoid large rendering queues with every update? Rolling out the stylesheets region by region might solve this problem, but that would probably be nearly impossible to set up and maintain. I don't really see how we can avoid this problem, but perhaps anyone else has an idea? -- Matthijs ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On Friday 28 February 2014, Tom Hughes wrote: > > There have been several stylesheet updates in the last week so the > servers are very busy and are likely to serve old tiles if they are > too busy to rerender them. I see, thanks. -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile refresh on openstreetmap.org
On 28/02/14 15:05, Christoph Hormann wrote: I have the impression the rerendering of tiles on openstreetmap.org is very slow at the moment - according to /status various z=11 tiles are from January 11. Also /dirty no more seems to work (at least not on the lower zoom levels). Has there been a change in configuration recently that caused this? There have been several stylesheet updates in the last week so the servers are very busy and are likely to serve old tiles if they are too busy to rerender them. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Tile refresh on openstreetmap.org
I have the impression the rerendering of tiles on openstreetmap.org is very slow at the moment - according to /status various z=11 tiles are from January 11. Also /dirty no more seems to work (at least not on the lower zoom levels). Has there been a change in configuration recently that caused this? -- Christoph Hormann http://www.imagico.de/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
