Interesting! That sounds like a huge amount of work, if you are validating every changeset in the globe, though so far there is no schedule to release this frequently.
Thank you for following the Open Database license by releasing this to the public. Is facebook planning to create new planet files like this on a frequent basis, for internal use? If so, those should be released to the public on a regular schedule, if I understand the ODbL correctly. Can I confirm that facebook is editing Openstreetmap data to fix the errors which are found? Or is this only happening in some cases? The diary post gave links that explain how this was made (both from September 2019): - https://engineering.fb.com/ml-applications/mars/ "MaRS: How Facebook keeps maps current and accurate" "We calculate the absolute diff between the two OSM versions (from option 1 above). We then split this diff into a set of LoChas that can be individually applied (from option 2)." ... "At a high level, a map change is vetted by a reviewer," - https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html "“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at Facebook": This is a video presentation, so I haven't seen it all, but the abstract says: "we have created an automated ingestion and integrity framework for OSM data that allows us to selectively update parts of the map instead of doing a full snapshot change all at once. "Decomposing a large set of changes in this way gives us the flexibility to rapidly ingest our own additions to the map, focus on geographical areas of importance to downstream products, and allows us to quickly apply hotfixes whenever egregious problems do arise. "With millions of tiny changes happening every week, we have created a system that is built on per-feature approval and preprocessing, that allows us to ingest changes at scale, while creating rules to automatically process logical changesets and enforce integrity constraints (e.g. anti-vandalism, anti-profanity etc.). "Due to the contextual nature of some of the changes in OpenStreetMap, the system combines Human Approval, necessary for highly visible features such as names of large administrative areas, with Automated AI/ML-based approval: for example, using computer vision techniques to reconcile newly created features against satellite imagery ground truth, or applying NLP techniques to determine whether other user-visible string changes are sensible and valid. These components are combined to create a continuous ingest-validate-deploy cycle for OSM map data." Lot's of buzz words there! But it sounds like it is a combination of computer algorithms and human checking for vandalism and errors. - Joseph Eisenberg On 3/10/20, Michal Migurski <m...@teczno.com> wrote: > Hi everyone, > > I’m writing to let you know about a new OpenStreetMap project Facebook just > released. It’s called Daylight Map Distribution. Daylight is a complete, > downloadable preview of OpenStreetMap data we plan to start using in a > number of our public maps: > > https://www.openstreetmap.org/user/migurski/diary/392416 > > Facebook uses maps to let our users find friends, businesses, groups and > more. OpenStreetMap (OSM) has a substantial global footprint of map data > built and maintained by a dedicated community of global mappers and it’s a > natural choice for us. Every day, OSM receives millions of contributions > from the community. Some of these contributions may have intentional and > unintentional edits that are incompatible with our needs. Our mapping teams > work to scrub these contributions for consistency and quality. > > What’s Included in the Daylight Map Distribution: > > • A PBF planet file composed of 100% OSM data, released under the terms > of > the Open Database License. > • Only those edits which have been validated to contain no malicious > vandalism or unintentional errors so we can show them in our display maps > > This is just an initial first release, and we’re looking for feedback from > the community to decide what would be useful to release in the future and > how frequently. I’d be interested to hear any response you might have about > it! > > -mike. > > ---------------------------------------------------------------- > michal migurski- contact info and pgp key: > sf/ca http://mike.teczno.com/contact.html > > > _______________________________________________ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > _______________________________________________ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
