Hello,
Kai, thank you for your prompt and informative response.
On Thu, 3 Oct 2013, Kai Krueger wrote:
do you have any more details of which tile layers are getting hit? Is it
low or high zoom tiles? What referes / user-agents do they come from? Is
it the tiles that get served through mod_tile, the hillshading tiles or
the tiles for the wiki mini atlas?
I can send you some example log lines of the throttled IPs:
2013/10/04 08:11:30 [error] 28822#0: *53658093 limiting requests, excess: 55.240 by zone "hikebike", client:
213.73.96.44, server: toolserver.org, request: "GET /tiles/hikebike/15/17169/11177.png HTTP/1.1", host:
"toolserver.org", referrer: "http://www.gpsies.com/map.do?fileId=gbcojbhrdfqlglbc";
2013/10/04 08:11:02 [error] 28822#0: *53650597 limiting requests, excess: 55.120 by zone
"hikebike", client: 85.0.37.63, server: toolserver.org, request: "GET /tiles/
osm/3/4/0.png HTTP/1.1", host: "c.www.toolserver.org", referrer:
"http://toolserver.org/~kolossos/openlayers/embed.html?layer=mapnik&bbox=39.68865498340449,43.5524339
5214844,39.778011016595514,43.614232047851566&marker=43.583333,39.733333
If you want to get more logs I'd send them to you in private.
It seems to relate especially hikebike/cmarq tools.
Too high load from individual clients has been an issue on many other
tileservers as well. Mostly it comes from various mobile apps, that
offer their users to download large areas (e.g. Germany) for offline
use. These areas then cover potentially millions of tiles, that the
clients then try and download as fast as the connection allows.
Sounds plausible.
For that reason, the tileservers on osm.org have a significant list of
user-agents that they block completely and in addition they also have an
automatic rate limiting per IP. There is also a specific tile usage
policy ( https://wiki.openstreetmap.org/wiki/Tile_usage_policy ) that
gouverns how you are allowed to technically access the tile servers
(once you have it downloaded, the use is freely gouverned by the
CC-BY-SA licence)
Other tileservers like the opencyclemap, equally have restrictions and
mod_tile (the apache module used to deliver tiles) has a number of
features available to limit traffic. mod_tile also has a complex system
to try and ensure maximum cachability of tiles while still ensuring
up-to-dateness. This system can furthermore be tuned either towards
fresshness or cacheability as needed.
My impression was so far this has never been an issue with the
toolserver and I wasn't aware of any explicit policies of how the
toolserver tiles are allowed to be accessed, so I never activated any of
the limiting features. But if it is becoming an issue we can see how
best to compat the issue.
gpsies.com stated they use the cache-control header which is sometimes not set
reasonably probably as far as I tried to see - i had a look at these hikebike
URL delivered from cmarq.
The will look at the problem closer on their side so I expect some more
details in the next days.
I could set cache control headers in the nginx which acts as load balancer for TS for tiles where it makes sense.
Do you have any advices on this? Which tiles dont change for what time about?
At least on the munin graphs for ptolemy, I don't see much increased
load. But if it is the hillshading tiles, or the WMA tiles, those don't
get served through ptolemy as far as I am aware.
Seems these tiles are delivered via ortelius/wolfsbane. Unfortunatelly the high
load times lead to munin not graphing anymore so I dont have any accutal data
but the error logs/loads via console.
I dont want to have this option configured forever - I rather hope we
can do something about caching or give the pictures they need to the
projects themselves (I doubt we have to deliver hill shading pictures
for everyone - this is Toolserver)
Cheers
nosy
_______________________________________________
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
