Andy, I agree that being frugal with bandwidth is important. Yet, there is a significant operations cost involved here, that I suspect very few will actually be willing to spend, unless it is made trivial -- the cost of setting up an independent planet file update pipeline - i.e. a docker image that could be pointed to a planet file, and has an easy way to suspend and resume update when the file needs to be static during the loading process.
I think having an optimized download tool that can both download+validate planet/areas, and could provide other services like diff updating would solve both goals. If the current process is manually pick a mirror, manually validate, and re-download if failed, vs use a dedicated tool that will optimize the download and crash recovery, all the better. Especially if it offers us a way to add more functionality later, like patch updating. Also, lets not fall into the premature optimization trap, as that only achieves local rather than global maximums. As a theoretical example - what is better - wider OSM adaption with higher number of planet downloads (i.e. some wasted bandwidth), or lower adaption and more optimized download process? I would think OSM project would be better off from wider adaption at a cost of a 10-50% extra bandwidth, right? If the startup cost (human hours) is lower, more people would participate. My numbers could be totally off, but keeping the eyes on bigger picture is very important when deciding such matters. Convenience/low manual labor cost is a big contributor to wider adaption. P.S. Am I correct to assume that every data consumer would need to download each diff file twice -- once for planet file update, and once to update the PostgreSQL database? On Sun, Feb 2, 2020 at 4:17 PM Andy Townsend <ajt1...@gmail.com> wrote: > ... that's why I said "... and there are ways of keeping a *.pbf* up to > date" in my message. - the idea is to avoid large downloads wherever > possible (emphasis new in this message; won't make it to plain text > archive). > Andy, > For more info see: > > https://docs.osmcode.org/pyosmium/latest/tools_uptodate.html > > or if that's somehow not an option: > > > https://wiki.openstreetmap.org/wiki/User:EdLoach#Osmosis_to_keep_a_local_copy_of_a_.osm_file_updated > > (from a few years back) > > * Automation / easy adaptation. Providing an out-of-the box way to set up > your own server is much easier if you have a tool that automatically > downloads and validates the planet file or a portion of it, > > Sure - an automated process is much easier to follow than a manual > do-it-yourself one, but the fact that a process is automated doesn't mean > that it has to be wasteful. Ultimately, someone's paying for the bandwidth > that downloads from each of the mirrors at > https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors use, so > it seems only fair to not be profligate with it. > > Best Regards, > > Andy > > > >
_______________________________________________ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
