Re: [OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
On 02/02/2020 16:39, Yuri Astrakhan wrote: * Anyone working on an evolving project like OpenMapTiles would attest that the import schema constantly changes. Indeed, but ... Every time schema changes, one needs to download newest planet, import it based on the new schema, and run diffs from that point. ... that's why I said "... and there are ways of keeping a *.pbf* up to date" in my message. - the idea is to avoid large downloads wherever possible (emphasis new in this message; won't make it to plain text archive). For more info see: https://docs.osmcode.org/pyosmium/latest/tools_uptodate.html or if that's somehow not an option: https://wiki.openstreetmap.org/wiki/User:EdLoach#Osmosis_to_keep_a_local_copy_of_a_.osm_file_updated (from a few years back) * Automation / easy adaptation. Providing an out-of-the box way to set up your own server is much easier if you have a tool that automatically downloads and validates the planet file or a portion of it, Sure - an automated process is much easier to follow than a manual do-it-yourself one, but the fact that a process is automated doesn't mean that it has to be wasteful. Ultimately, someone's paying for the bandwidth that downloads from each of the mirrors at https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors use, so it seems only fair to not be profligate with it. Best Regards, Andy ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
While keeping a planet file up to date is really easy, it is probably not the first idea that comes to mind when you first plan to do something with the data. This service looks like a good idea, but filtering abusers must be kept in mind anyway. Yves ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
Andy, two major reasons: * Anyone working on an evolving project like OpenMapTiles would attest that the import schema constantly changes. Every time schema changes, one needs to download newest planet, import it based on the new schema, and run diffs from that point. * Automation / easy adaptation. Providing an out-of-the box way to set up your own server is much easier if you have a tool that automatically downloads and validates the planet file or a portion of it, rather than forcing each user to find the proper mirror, wait for an hour to download it, only to find out that somehow that mirror has invalid data (my tool finds when one mirror has slightly off file lengths and ignores it - already happened several times, and it also auto-validates the file with md5) So yes, not having a tool is worse than having it for the above reasons. Also, considering that others have already twitted about it, and a lot of other OSM/geo community has liked it, it seems the tool has value to at least some people, thus warrants inclusion in a newsletter. On Sun, Feb 2, 2020 at 11:02 AM Andy Townsend wrote: > > For those who download OSM data regularly, there is now a simple way to > reduce the load on the primary OSM servers, while also making download much > faster and ensure the data is correct. > > Apologies if this has been done to death already, but surely if you are > downloading the entire planet regularly you are quite simply "doing it > wrong"? > > Minutely (and other frequency) diffs exist, and there are ways of keeping > a .pbf up to date (both osmium-based and osmosis-based, if for some reason > the former doesn't work for you). > > A "tool" that downloads from all mirrors in parallel just leads to a > "tragedy of the commons" situation, and isn't a solution to the underlying > problem of scarce resources. > > Best Regards, > Andy > > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
> For those who download OSM data regularly, there is now a simple way to reduce the load on the primary OSM servers, while also making download much faster and ensure the data is correct.Apologies if this has been done to death already, but surely if you are downloading the entire planet regularly you are quite simply "doing it wrong"?Minutely (and other frequency) diffs exist, and there are ways of keeping a .pbf up to date (both osmium-based and osmosis-based, if for some reason the former doesn't work for you).A "tool" that downloads from all mirrors in parallel just leads to a "tragedy of the commons" situation, and isn't a solution to the underlying problem of scarce resources.Best Regards,Andy ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
The news mentions that downloads from Planet OSM are currently rate limited to 400 kB/s and suggest to use mirrors, but does not mention the related announcement about the new tool to simplify such downloads. I think it will help anyone downloading, and it might be worth including in the next weekly. Original announcement: For those who download OSM data regularly, there is now a simple way to reduce the load on the primary OSM servers, while also making download much faster and ensure the data is correct. OpenMapTiles new tool downloads the planet from all mirrors in parallel. It usually takes just a few minutes, and it automatically verifies md5 checksum. The tool will not use the primary planet source by default. The tool can also download and validate regional extracts from geofabrik, bbbike, and osmfr. Internally the tool uses aria2c. Easiest is to use it with the docker -- share current dir with docker and save the file there. Anything after the "--" is passed to aria2c. Here's a Linux/Mac command, but should be runnable from Windows with the minor command adjustment. docker run --rm -it -v $PWD:/download openmaptiles/openmaptiles-tools download-osm planet -- -d /download Use --dry-run (-n) to run it without the actual download (i.e. to see which file it would download and from what mirrors). You may also add --verbose (-v) docker run --rm -it openmaptiles/openmaptiles-tools download-osm planet --dry-run Use --help for all arguments. See source and documentation here: https://github.com/openmaptiles/openmaptiles-tools#multi-streamed-osm-data-downloader P.S. Conceptually, the script is doing for OSM data what torrents were designed to do, but sadly there is no well established web of torrents that would offer similar functionality. On Sun, Feb 2, 2020 at 9:29 AM weeklyteam wrote: > The weekly round-up of OSM news, issue # 497, > is now available online in English, giving as always a summary of a lot of > things happening in the openstreetmap world: > > http://www.weeklyosm.eu/en/archives/12814/ > > Enjoy! > > Did you know that you can also submit messages for the weeklyOSM? Just log > in to https://osmbc.openstreetmap.de/login with your OSM account. Read > more about how to write a post here: > http://www.weeklyosm.eu/this-news-should-be-in-weeklyosm > > weeklyOSM? > who: https://wiki.openstreetmap.org/wiki/WeeklyOSM#Available_Languages > where?: > https://umap.openstreetmap.fr/en/map/weeklyosm-is-currently-produced-in_56718#2/8.6/108.3 > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] weeklyOSM #497 2020-01-21-2020-01-27
The weekly round-up of OSM news, issue # 497, is now available online in English, giving as always a summary of a lot of things happening in the openstreetmap world: http://www.weeklyosm.eu/en/archives/12814/ Enjoy! Did you know that you can also submit messages for the weeklyOSM? Just log in to https://osmbc.openstreetmap.de/login with your OSM account. Read more about how to write a post here: http://www.weeklyosm.eu/this-news-should-be-in-weeklyosm weeklyOSM? who: https://wiki.openstreetmap.org/wiki/WeeklyOSM#Available_Languages where?: https://umap.openstreetmap.fr/en/map/weeklyosm-is-currently-produced-in_56718#2/8.6/108.3 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
