Hi Wolfram and others, On Sat, 25 Jan 2020 at 10:32, Wolfram Schneider <[email protected]> wrote: > > I wanted to download the latest planet from planet.osm.org and the > download rate is less than 450KByte/s. In the past I got > 30-60Mbyte/s. It takes now more than 36 hours to download the planet > (was 15min). All mirrors are outdated, you have to wait until Sat > morning to see a new planet.osm.pbf file. >
I am part of the OpenStreetMap Operations team and Operations Working Group. I am the person who temporarily instigated the download rate limit. https://github.com/openstreetmap/chef/commit/1647548590878bfd153b8b8c74e73eda0b8a2cd3 Some background... www.openstreetmap.org + API, planet and many other services primarily run in our Amsterdam data center (Equinix AM6). See servers here: https://hardware.openstreetmap.org/#equinix-am6 Our uplink from these servers to the internet is a 1Gbps link to Cogent. Bandwidth usage: https://munin.osm.org/openstreetmap.org/switch1.openstreetmap.org/if_bytes/if_49.html Due to recent increases in outbound bandwidth usage (partially due to switch to BBR congestion control https://github.com/openstreetmap/chef/commit/67c05c6c4b484a6cdac9473b4e6951e5c76c42a3), we started to exceed 1Gbps... Once we exceed the limit, packets get dropped. We initially did not understand what was causing the packets to be dropped. We suspected a flaw with switches and/or GBIC modules and/or fibre link. The dropped packet caused DNS failures and reliability issues with the API, which were reported in many channels: email, Slack osm-us, irc etc. The planet downloads are the main reason the upstream link is being swamped. 60Mbyte/s alone is nearly 50% of our available upstream capacity. Normally around 30 to 60 simultaneous requests. I've started to prep for upgrading our upstream link to 10Gbps. But we've had ongoings with Cogent peering and would prefer to switch to using a new provider upstream. https://twitter.com/OSM_Tech/status/1220276733359919104 The 20GB+ planet .osm.bz2 files are automatically redirected to the https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/ mirror But unfortunately due to flaw in my mirror update script the redirect was now working for PBF files. Pull requests welcome: https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb I have manually fixed the PBF file mirroring now (for current files). Once we have fixed the upstream link issue, I will remove the download rate limits. For the moment, please use a mirror for full speed downloads. The minutely, hourly and daily diff files will only take a second or 2 extra to download with the current late limits. Kind regards, Grant _______________________________________________ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
