Re: [OSM-talk] Tile caching (osm startpage)
Hi That was a long mail. How about some lines in apache setup? cache: 1 year rest: same as www.osm.org For me I solved the tile cache problem I had here in Phnom Penh. I have a super fast proxy. The actual problem: I made a javascript map library for webkit that is much faster than openlayers. Navigation is more visual and namefinder is not so important anymore. But to have a good visual experience, the tiles should be in cache. Using my proxy I can really see the difference. The fast zoom gives a new dimension and more tiles are needed. conclusio A tileserver with longer cache time would be nice. Bernhard Kai Krueger schrieb: > On 01/-10/-28163 08:59 PM, Michal Migurski wrote: > >> On Mar 5, 2010, at 11:34 AM, John Smith wrote: >> >> >>> On 6 March 2010 01:24, Bernhard zwischenbrugger >>> wrote: >>> Google Cache Time: Cache-Control: public, max-age"22 //feels like one month (I didn't calculate) >>> I'd say it's a bad idea to specify a cache time, instead there is >>> other caching mechanisms to tell if a tile has changed: >>> >>> ETag: "d096ddafba32c0da609007e224530ccd" >>> This way if a tile never changes you never need to refresh. >>> >> For what it's worth, the current tile server does specify a cache time as >> well as an ETag. >> >> % curl -sI "http://tile.openstreetmap.org/14/2627/6331.png"; >> HTTP/1.1 200 OK >> Date: Sun, 07 Mar 2010 02:19:30 GMT >> Server: Apache/2.2.8 (Ubuntu) >> ETag: "93087c5713c17d9939cac9e341fdd14c" >> Content-Length: 26595 >> Cache-Control: max-age36 >> Expires: Sun, 07 Mar 2010 02:36:46 GMT >> Content-Type: image/png >> >> 1,000 sec. max age there is a little over 15 minutes, though when I repeat >> this request I get expiry times all over the place, from a few minutes to >> many hours. What currently decides on the cache expiration time? >> > > mod_tile, the apache module used to server the tiles, has a fairly > sophisticated mechanism to decided the expiry times, driven by a bunch > of heuristics. As with the minutely rendering, we don't have a periodic > update cycle anymore, there is no real good way of setting the expiry > times, as one would need to guess when in the future this tile might > change. As that is obviously not possible, we need to trade off between > caching time (reducing server resources and client side latency) and > up-to-dateness to not loose the benefits of the minutely updates. > > The heuristics currently supported (and used) are the following. > > At a first instance it decides if the tile is known to be "dirty" i.e. > outdated. If the tile server is overloaded, or the rendering takes > longer than 3 seconds, mod_tile will serve an old tile rather than wait > until the on-the-fly rendering will finish. (Again a trade-off between > client side latency and up-to-dateness) At that point, given that we > know the tile will soon change, the max-age cache parameter is set very > low. 15 minutes + a 7 minute random jitter. > > If the tile served is not stale, there are another 3 heuristics > A zoom level based heuristic, a last modified heuristic and a known > planet update cycle if it exists. > > The zoom level based heuristic allows to set the minimum max-age caching > time based on if the tile served is a low zoom, medium zoom or high zoom > tile. The idea behind this is that low zoom tiles (even though they are > effected by all changes) don't appear to change much. Thus it seems > reasonable to allow clients to cache these much longer as the effect of > a stale tile from cache is probably less. > > The current setup of tile.osm.org, I think, doesn't use this heuristic > though and setts the minimum max-age caching to 3 hours + 3 hours random > jitter for all zoom levels, even though the minutely tile expiry doesn't > actually expire low zoom tiles and thus only change if manually > requested. So I think it would be good to increase the time to cache low > zoom tiles, as in the current setup it shouldn't affect things > negatively. > > The last modified heuristic tries to guess how likely it is for a tile > to change. E.g. a tile in the middle of the pacific is probably not > going to change anytime soon. So it wouldn't matter to give e.g. a > max-age of a week. A tile perhaps in central Berlin is more likely to > change. So the heuristic guesses how likely it is to change in the > future based on how long it has been since it last changed. It then > specifies a linear scaling of max-age to last modified time with a > tunable slope parameter. As it is fairly unclear how well this heuristic > works, I believe the osm tile server still has this at its default, i.e. > turned off completely. > > The last "heuristic", is that based on planet update cycles. For those > servers that have a planet update cycle (i.e. not tile.osm.org), you > don't have to guess and can just s
Re: [OSM-talk] Tile caching (osm startpage)
On 01/-10/-28163 08:59 PM, Michal Migurski wrote: > > On Mar 5, 2010, at 11:34 AM, John Smith wrote: > >> On 6 March 2010 01:24, Bernhard zwischenbrugger wrote: >>> Google Cache Time: >>> Cache-Control: public, max-age"22 //feels like one month (I >>> didn't calculate) >> >> I'd say it's a bad idea to specify a cache time, instead there is >> other caching mechanisms to tell if a tile has changed: >> >>> ETag: "d096ddafba32c0da609007e224530ccd" >> >> This way if a tile never changes you never need to refresh. > > > For what it's worth, the current tile server does specify a cache time as > well as an ETag. > > % curl -sI "http://tile.openstreetmap.org/14/2627/6331.png"; > HTTP/1.1 200 OK > Date: Sun, 07 Mar 2010 02:19:30 GMT > Server: Apache/2.2.8 (Ubuntu) > ETag: "93087c5713c17d9939cac9e341fdd14c" > Content-Length: 26595 > Cache-Control: max-age36 > Expires: Sun, 07 Mar 2010 02:36:46 GMT > Content-Type: image/png > > 1,000 sec. max age there is a little over 15 minutes, though when I repeat > this request I get expiry times all over the place, from a few minutes to > many hours. What currently decides on the cache expiration time? mod_tile, the apache module used to server the tiles, has a fairly sophisticated mechanism to decided the expiry times, driven by a bunch of heuristics. As with the minutely rendering, we don't have a periodic update cycle anymore, there is no real good way of setting the expiry times, as one would need to guess when in the future this tile might change. As that is obviously not possible, we need to trade off between caching time (reducing server resources and client side latency) and up-to-dateness to not loose the benefits of the minutely updates. The heuristics currently supported (and used) are the following. At a first instance it decides if the tile is known to be "dirty" i.e. outdated. If the tile server is overloaded, or the rendering takes longer than 3 seconds, mod_tile will serve an old tile rather than wait until the on-the-fly rendering will finish. (Again a trade-off between client side latency and up-to-dateness) At that point, given that we know the tile will soon change, the max-age cache parameter is set very low. 15 minutes + a 7 minute random jitter. If the tile served is not stale, there are another 3 heuristics A zoom level based heuristic, a last modified heuristic and a known planet update cycle if it exists. The zoom level based heuristic allows to set the minimum max-age caching time based on if the tile served is a low zoom, medium zoom or high zoom tile. The idea behind this is that low zoom tiles (even though they are effected by all changes) don't appear to change much. Thus it seems reasonable to allow clients to cache these much longer as the effect of a stale tile from cache is probably less. The current setup of tile.osm.org, I think, doesn't use this heuristic though and setts the minimum max-age caching to 3 hours + 3 hours random jitter for all zoom levels, even though the minutely tile expiry doesn't actually expire low zoom tiles and thus only change if manually requested. So I think it would be good to increase the time to cache low zoom tiles, as in the current setup it shouldn't affect things negatively. The last modified heuristic tries to guess how likely it is for a tile to change. E.g. a tile in the middle of the pacific is probably not going to change anytime soon. So it wouldn't matter to give e.g. a max-age of a week. A tile perhaps in central Berlin is more likely to change. So the heuristic guesses how likely it is to change in the future based on how long it has been since it last changed. It then specifies a linear scaling of max-age to last modified time with a tunable slope parameter. As it is fairly unclear how well this heuristic works, I believe the osm tile server still has this at its default, i.e. turned off completely. The last "heuristic", is that based on planet update cycles. For those servers that have a planet update cycle (i.e. not tile.osm.org), you don't have to guess and can just set the expiry time to when the next update cycle begins. This is the most efficient from a caching point of view, but doesn't work with minutely updates. The final max-age handed out by the server for clean tiles is then the maximum time of any of the 3 heuristics capped to a week. The random jitter factor is there mostly for if you have weekly update cycles, to not expire all tiles at exactly the same time and then overwhelm your tile server when suddenly all cached tiles expire. Since a couple of hours, the mod_tile code would now also support a tile expiry based on hostname header, so it would theoretically be possible to do something like cache.tile.osm.org handing out expiry headers of e.g. a month. But it isn't clear how one would decided who to send to a hypothetical cache.tile and who to the normal tile server. I
Re: [OSM-talk] Tile caching (osm startpage)
On Mar 5, 2010, at 11:34 AM, John Smith wrote: > On 6 March 2010 01:24, Bernhard zwischenbrugger wrote: >> Google Cache Time: >> Cache-Control: public, max-age= //feels like one month (I >> didn't calculate) > > I'd say it's a bad idea to specify a cache time, instead there is > other caching mechanisms to tell if a tile has changed: > >> ETag: "d096ddafba32c0da609007e224530ccd" > > This way if a tile never changes you never need to refresh. For what it's worth, the current tile server does specify a cache time as well as an ETag. % curl -sI "http://tile.openstreetmap.org/14/2627/6331.png"; HTTP/1.1 200 OK Date: Sun, 07 Mar 2010 02:19:30 GMT Server: Apache/2.2.8 (Ubuntu) ETag: "93087c5713c17d9939cac9e341fdd14c" Content-Length: 26595 Cache-Control: max-age=1036 Expires: Sun, 07 Mar 2010 02:36:46 GMT Content-Type: image/png 1,000 sec. max age there is a little over 15 minutes, though when I repeat this request I get expiry times all over the place, from a few minutes to many hours. What currently decides on the cache expiration time? The Phnom Penh issue all sounds like a job for a CDN like Akamai's or a caching proxy (i.e. squid-cache.org) closer to Cambodia. Bernhard, these are not difficult to set up for yourself if you are interested, and require little knowledge of the actual map. -mike. michal migurski- m...@stamen.com 415.558.1610 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile caching (osm startpage)
On 6 March 2010 23:29, John Smith wrote: > Forcing everyone to view dirty cached tiles is going to make others unhappy. If you're a visitor, i.e. are not logged in, it will most likely make you happier. There will be the obligatory 5% internet users in a different situation, but for 95% this would be a gain imho. Re: awards there are the lolcatz of awsumness and wiki barnstars. Cheers ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile caching (osm startpage)
On 6 March 2010 17:52, Bernhard zwischenbrugger wrote: > In browser the load time is even worse. What you really need is your own tile server, or a specalised proxy that serves up what ever tiles it has cached and checks in idle time if requested tiles have changed. Forcing everyone to view dirty cached tiles is going to make others unhappy. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile caching (osm startpage)
John Smith schrieb: > On 6 March 2010 01:24, Bernhard zwischenbrugger wrote: > >> Google Cache Time: >> Cache-Control: public, max-age= //feels like one month (I >> didn't calculate) >> > > I'd say it's a bad idea to specify a cache time, instead there is > other caching mechanisms to tell if a tile has changed: > > >> ETag: "d096ddafba32c0da609007e224530ccd" >> > > This way if a tile never changes you never need to refresh. > But the browser sends the ETag to the server all the time. The server answers "304 not modified". Phnom Penh load times: To get the HTTP header of a picture takes here at the moment 1.3 seconds. To load the picture it's about 2.5 seconds for a single picture. An image that is already in cache takes about 1.3 seconds to load (must wait for 304 header). This values are measured using "wget" and "HEAD" at commandline. In browser the load time is even worse. see: http://wiki.openstreetmap.org/wiki/File:Imageloadtime.png Bernhard ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile caching (osm startpage)
On 6 March 2010 01:24, Bernhard zwischenbrugger wrote: > Google Cache Time: > Cache-Control: public, max-age= //feels like one month (I > didn't calculate) I'd say it's a bad idea to specify a cache time, instead there is other caching mechanisms to tell if a tile has changed: > ETag: "d096ddafba32c0da609007e224530ccd" This way if a tile never changes you never need to refresh. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Tile caching (osm startpage)
Hi Bernhard, all, Very interesting proposal. I think people in a position to do so should give attentive consideration to your situation, facing Internet infrastructure limitations in a developing country. (Beside your personal situation and dedication which would certainly deserve a "veteran's medal for OSM cause" if such a thing existed - or does it exist yet ?). I imagine that the largest number of potential users of OSM in the world must be facing similar technical conditions (based on population numbers). So your forced rest time in Phnom Penh could be an opportunity for OSM to tune its offer for this majority of mankind. Crisis situations could also benefit from progress in this area. For example, I was in contact via irc last night with someone in Concepción, Chile, who only had a very weak intermittent wifi connection to a remote hot spot (and could not move because of curfew). This could be an opportunity for osm start page and basic services (MapOSMatic, Walking Papers, OpenStreetBugs, etc...) to improve themselves if possible, for people who, on average, might not have an understanding of technical problems as insightful as yours. Kind regards, Jean-Guilhem Bernhard zwischenbrugger a écrit : > Hi all > > I'm sitting in Phom Penh and have very slow internet. > The tiles come in very very slow. > > I'm working on a new javascript map... > full story: http://www.mail-archive.com/d...@openstreetmap.org/msg10669.html > > The internet is too slow here for developing, so I had to install a > local image proxy. > Now it's fast as hell. It's faster than at home. > > If a tile is changed I will see the old version for a long time. > But it's fast and usable. > > > HTTP HEAD > > Google Cache Time: > Cache-Control: public, max-age= //feels like one month (I > didn't calculate) > > OSM Cache Time: > eTag (no cache) > > - Story > > In many countrys (like cambodia) the ping is very long. It can be easily > some seconds. > osm.org is slow slow slow. > It takes long time until all the tile for the osm startpage are loaded. > But then the problem starts. > Every move I make on the map is s slow. > Even if I was looking at the same spot on the map a couple of days ago. > > > For visitors of osm.org a good caching is important. > For editors of osm.org it's important to have the newest version of the > tiles. > > For visitors (not logged in) there could be a diffent caching than for > editors (logged in). > > Caching is important if you sit somewhere in a developig country with > slow internet connection. > > Implementation > > Tileserver visiter > cache.openstreetmap.org > > Tileserver editor > tiles.openstreetmap.org > > The Servers should deliver headers for long and short caching time. > The map for logged in users should load images from > tiles.openstreetmap.org. > (extensible rules...) > > > Never mind, I was just thinking about that and want to distribute it. > > Wish you a nice day > > Bernhard > > > > > > > > --OSM: > > HEAD http://a.andy.sandbox.cloudmade.com/tiles/cycle/17/37480/50153.png > 200 OK > Cache-Control: max-age=440434 > Connection: close > Date: Fri, 05 Mar 2010 14:37:11 GMT > ETag: "d096ddafba32c0da609007e224530ccd" > Server: Apache/2.2.8 (Ubuntu) > Content-Length: 45409 > Content-Type: image/png > Expires: Wed, 10 Mar 2010 16:57:46 GMT > Client-Date: Fri, 05 Mar 2010 14:37:11 GMT > Client-Peer: 75.101.147.222:80 > Client-Response-Num: 1 > > > -GOOGLE: > > HEAD "http://mt1.google.com/vt/lyr...@118&hl=de&x=3&y=6&z=4&s=Galileo"; > 200 OK > Cache-Control: public, max-age= > Connection: close > Date: Fri, 05 Mar 2010 14:45:02 GMT > Server: maptiles-versatile > Content-Length: 19810 > Content-Type: image/png > Client-Date: Fri, 05 Mar 2010 14:45:02 GMT > Client-Peer: 64.233.189.136:80 > Client-Response-Num: 1 > X-Content-Type-Options: nosniff > X-XSS-Protection: 0 > > > Sorry, I posted something similar bevor: > http://www.mail-archive.com/talk@openstreetmap.org/msg23461.html > > > Bernhard > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > > > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Tile caching (osm startpage)
Hi all I'm sitting in Phom Penh and have very slow internet. The tiles come in very very slow. I'm working on a new javascript map... full story: http://www.mail-archive.com/d...@openstreetmap.org/msg10669.html The internet is too slow here for developing, so I had to install a local image proxy. Now it's fast as hell. It's faster than at home. If a tile is changed I will see the old version for a long time. But it's fast and usable. HTTP HEAD Google Cache Time: Cache-Control: public, max-age= //feels like one month (I didn't calculate) OSM Cache Time: eTag (no cache) - Story In many countrys (like cambodia) the ping is very long. It can be easily some seconds. osm.org is slow slow slow. It takes long time until all the tile for the osm startpage are loaded. But then the problem starts. Every move I make on the map is s slow. Even if I was looking at the same spot on the map a couple of days ago. For visitors of osm.org a good caching is important. For editors of osm.org it's important to have the newest version of the tiles. For visitors (not logged in) there could be a diffent caching than for editors (logged in). Caching is important if you sit somewhere in a developig country with slow internet connection. Implementation Tileserver visiter cache.openstreetmap.org Tileserver editor tiles.openstreetmap.org The Servers should deliver headers for long and short caching time. The map for logged in users should load images from tiles.openstreetmap.org. (extensible rules...) Never mind, I was just thinking about that and want to distribute it. Wish you a nice day Bernhard --OSM: HEAD http://a.andy.sandbox.cloudmade.com/tiles/cycle/17/37480/50153.png 200 OK Cache-Control: max-age=440434 Connection: close Date: Fri, 05 Mar 2010 14:37:11 GMT ETag: "d096ddafba32c0da609007e224530ccd" Server: Apache/2.2.8 (Ubuntu) Content-Length: 45409 Content-Type: image/png Expires: Wed, 10 Mar 2010 16:57:46 GMT Client-Date: Fri, 05 Mar 2010 14:37:11 GMT Client-Peer: 75.101.147.222:80 Client-Response-Num: 1 -GOOGLE: HEAD "http://mt1.google.com/vt/lyr...@118&hl=de&x=3&y=6&z=4&s=Galileo"; 200 OK Cache-Control: public, max-age= Connection: close Date: Fri, 05 Mar 2010 14:45:02 GMT Server: maptiles-versatile Content-Length: 19810 Content-Type: image/png Client-Date: Fri, 05 Mar 2010 14:45:02 GMT Client-Peer: 64.233.189.136:80 Client-Response-Num: 1 X-Content-Type-Options: nosniff X-XSS-Protection: 0 Sorry, I posted something similar bevor: http://www.mail-archive.com/talk@openstreetmap.org/msg23461.html Bernhard ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
