Re: [OSM-talk] Mapnik rendering update
On 29 Jan 2008, at 23:40, Jon Burgess wrote: On Fri, 2008-01-25 at 23:28 +, Jon Burgess wrote: On Fri, 2008-01-25 at 22:35 +, Robert (Jamie) Munro wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jon Burgess wrote: | Once this first pass of the rendering is complete the mod_tile setup | will have all the existing tiles fully rendered with the current planet | dump and current map style. At that point I will switch over the URLs on | the server to make the new tiles live. | | The new system has already rendered 5M tiles in just under 24 hours. | This covers all of zooms 0 - 12 and about halfway through zoom 13. At | the current rate all the old 30M tiles will be rendered by about | Wednesday next week. I've made the new tiles from mod_tile live on the main OSM tile server (on the default Mapnik layer). The initial impression I get is that the tiles are being served to more responsively and the on-access rendering seems to be working well too. Hopefully this will all continue smoothly. Fantastic. * The new colour reduction algorithm causes some odd colour changes in a few tiles. 99% are fine but a few have a background colour or road fill which is slightly wrong. I had a look and there are quite a few of wrong tile. I committed changes that seems to fix this in r625. Great work! Artem ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
On Fri, 2008-01-25 at 23:28 +, Jon Burgess wrote: On Fri, 2008-01-25 at 22:35 +, Robert (Jamie) Munro wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jon Burgess wrote: | Once this first pass of the rendering is complete the mod_tile setup | will have all the existing tiles fully rendered with the current planet | dump and current map style. At that point I will switch over the URLs on | the server to make the new tiles live. | | The new system has already rendered 5M tiles in just under 24 hours. | This covers all of zooms 0 - 12 and about halfway through zoom 13. At | the current rate all the old 30M tiles will be rendered by about | Wednesday next week. I've made the new tiles from mod_tile live on the main OSM tile server (on the default Mapnik layer). The initial impression I get is that the tiles are being served to more responsively and the on-access rendering seems to be working well too. Hopefully this will all continue smoothly. There are a couple of issues that I've noticed so far. I don't think any of these are serious enough to force us to go back to the old rendering setup. They should all be resolved in due course. * The new colour reduction algorithm causes some odd colour changes in a few tiles. 99% are fine but a few have a background colour or road fill which is slightly wrong. * Some of the z16/z17 tiles from the old system have not been rendered yet. They should get rendered and appear shortly after you access them. The new system uses a 8x8 meta tile rendering approach so overall we actually have more tiles than the old system (40M vs 30M previously). * The 'Expires' HTTP header has been broken causing tiles not to be cached as well as they should be in your browser (or other proxy/cache). * The munin stats for Apache volume have broken, you can see the real throughput is still OK by looking at the eth0 stats. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
On Freitag 25 Januar 2008, Gervase Markham wrote: Jon Burgess wrote: There is a major update to the way tiles are rendered occurring at the moment, let me explain... snip Makes perfect sense. Thanks for the update, and your hard work. I look forward to seeing the shiny new map next Wednesday or so :-) A great thanks from Germany too !! -- Joerg ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jon Burgess wrote: | On Fri, 2008-01-25 at 22:35 +, Robert (Jamie) Munro wrote: | -BEGIN PGP SIGNED MESSAGE- | Hash: SHA1 | | Jon Burgess wrote: | | Once this first pass of the rendering is complete the mod_tile setup | | will have all the existing tiles fully rendered with the current planet | | dump and current map style. At that point I will switch over the URLs on | | the server to make the new tiles live. | | | | The new system has already rendered 5M tiles in just under 24 hours. | | This covers all of zooms 0 - 12 and about halfway through zoom 13. At | | the current rate all the old 30M tiles will be rendered by about | | Wednesday next week. | | Now that you've done the slowest-to-render low zoom tiles, Why not just | bin the old DB and backlog and let the rest of the tiles render on demand? | | I might do this once all the z16 or z17 tiles are done but with only | z0-12 I fear I'll get lots of complaints about stuff suddenly | disappearing from the map. The zoom levels 15 - 17 are by far the most | numerous according to the histogram of tiles per zoom. Are you able to say how many times each high zoom tile has been viewed? I suspect that it most cases it will be 1 if they are rendered on demand, and 0 if they are rendered from a backlog. There will be tiles that have been viewed a lot - tiles in important cities or places with lots of OSM activity, for example, but I suspect a lot of the 8 million z17 tiles will never be looked again, at least not until their content has been changed. If we do get a backlog, then it needs to give priority to tiles that have been viewed more than once, so that on a second viewing, a tile will be rendered live, even if the system is a bit busy (but not a lot busy, as it were). If you do get complaints, you can always switch back. Out of interest, do you have rough stats on how long tiles at each zoom level take to render on average? Robert (Jamie) Munro -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHmzviz+aYVHdncI0RAtMhAJ9uY+Nvw8NgRllh1VErN2wdAV+jiACgoulq CxZ2MbuLSGPJFK1qR+SfeJU= =knlt -END PGP SIGNATURE- ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
[OSM-talk] Mapnik rendering update
Anyone know why Mapnik still hasn't rendered the data I put here: http://www.openstreetmap.org/?lat=54.6134lon=-2.6175zoom=12layers=B0FT over a week ago? Osmarender has it fine. I've tried using the rerender command on informationfreeway. (Is that Osmarender only?) I've tried waiting until next Wednesday and then waiting a bit longer. I've tried going to the individual tile URL and appending /dirty; /status then saying it's awaiting rerender, and later it says it no longer is (so presumably it's been done) but yet, no change :-( Sorry if I'm being thick. What am I missing? Gerv ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
On Fri, 2008-01-25 at 17:49 +, Thomas Wood wrote: I believe the mapnik tile server has been a little overloaded recently - queuing of new tiles was disabled for a time. Rerender on ifw is for osmarender only (and isn't really required much, now that changed-tiles is being run to pick up data changes) On Jan 25, 2008 5:36 PM, Gervase Markham [EMAIL PROTECTED] wrote: Anyone know why Mapnik still hasn't rendered the data I put here: http://www.openstreetmap.org/?lat=54.6134lon=-2.6175zoom=12layers=B0FT over a week ago? Osmarender has it fine. I've tried using the rerender command on informationfreeway. (Is that Osmarender only?) I've tried waiting until next Wednesday and then waiting a bit longer. I've tried going to the individual tile URL and appending /dirty; /status then saying it's awaiting rerender, and later it says it no longer is (so presumably it's been done) but yet, no change :-( Sorry if I'm being thick. What am I missing? There is a major update to the way tiles are rendered occurring at the moment, let me explain... As Thomas mentions the rendering on the tile server has been overloaded recently. I've been working on developing an improved version of the software rendering framework called mod_tile for a few months and I took the plunge this week to install and run it on tile. The old system used a Mysql database to store the tiles, the new one uses the filesystem. I am currently running the list of 30M tiles from the MySQL database through the new mod_tile rendering engine to build up the new tiles from scratch. Right now all the tiles you are seeing are coming from the MySQL DB. The tiles are still being rendered with the Mapnik renderer so the new map will look the same as the old one. Many major changes have been made to the other parts of the rendering setup to massively improve the overall rendering speed. Once this first pass of the rendering is complete the mod_tile setup will have all the existing tiles fully rendered with the current planet dump and current map style. At that point I will switch over the URLs on the server to make the new tiles live. The new system has already rendered 5M tiles in just under 24 hours. This covers all of zooms 0 - 12 and about halfway through zoom 13. At the current rate all the old 30M tiles will be rendered by about Wednesday next week. Given how well things are progressing with the rendering I don't want to attempt any hybrid schemes to mix the new and old tiles or transfer the old tiles from the database to the filesystem. The new rendering rate is about 5 times faster than the old setup and allow us to be able to keep the tiles more up to date in future. The weekly import of the planet dump data will still be a limitation for the foreseeable future, but you should see updates appearing on the map much sooner after the Wednesday each week. Hopefully you can all live with a few more days delay in the updates before you get a nice new set of tiles with faster updates in future. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
On Fri, 2008-01-25 at 22:35 +, Robert (Jamie) Munro wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jon Burgess wrote: | Once this first pass of the rendering is complete the mod_tile setup | will have all the existing tiles fully rendered with the current planet | dump and current map style. At that point I will switch over the URLs on | the server to make the new tiles live. | | The new system has already rendered 5M tiles in just under 24 hours. | This covers all of zooms 0 - 12 and about halfway through zoom 13. At | the current rate all the old 30M tiles will be rendered by about | Wednesday next week. Now that you've done the slowest-to-render low zoom tiles, Why not just bin the old DB and backlog and let the rest of the tiles render on demand? I might do this once all the z16 or z17 tiles are done but with only z0-12 I fear I'll get lots of complaints about stuff suddenly disappearing from the map. The zoom levels 15 - 17 are by far the most numerous according to the histogram of tiles per zoom. ZoomNumber of tiles in database 0 1 1 4 2 16 3 64 4 256 5 1024 6 4096 7 16384 8 65539 9 262163 10 1039716 11 1507579 12 1010639 13 1495247 14 2333230 15 5110638 16 5017451 17 8192471 18 3793287 Total = 29849805 Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jon Burgess wrote: | Once this first pass of the rendering is complete the mod_tile setup | will have all the existing tiles fully rendered with the current planet | dump and current map style. At that point I will switch over the URLs on | the server to make the new tiles live. | | The new system has already rendered 5M tiles in just under 24 hours. | This covers all of zooms 0 - 12 and about halfway through zoom 13. At | the current rate all the old 30M tiles will be rendered by about | Wednesday next week. Now that you've done the slowest-to-render low zoom tiles, Why not just bin the old DB and backlog and let the rest of the tiles render on demand? Robert (Jamie) Munro -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHmmRMz+aYVHdncI0RApeyAJ4izpTGTi2tn8u3X5vPZHfFRsSbVQCdGqC5 WcenOewOb/FpqChyFeULhBk= =/LLL -END PGP SIGNATURE- ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik rendering update
I believe the mapnik tile server has been a little overloaded recently - queuing of new tiles was disabled for a time. Rerender on ifw is for osmarender only (and isn't really required much, now that changed-tiles is being run to pick up data changes) On Jan 25, 2008 5:36 PM, Gervase Markham [EMAIL PROTECTED] wrote: Anyone know why Mapnik still hasn't rendered the data I put here: http://www.openstreetmap.org/?lat=54.6134lon=-2.6175zoom=12layers=B0FT over a week ago? Osmarender has it fine. I've tried using the rerender command on informationfreeway. (Is that Osmarender only?) I've tried waiting until next Wednesday and then waiting a bit longer. I've tried going to the individual tile URL and appending /dirty; /status then saying it's awaiting rerender, and later it says it no longer is (so presumably it's been done) but yet, no change :-( Sorry if I'm being thick. What am I missing? Gerv ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk -- Regards, Thomas Wood (Edgemaster) ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
