Hi Joseph, thanks for your questions! We are unsure yet about the public release schedule we want to commit to, because a lot of it depends on community feedback which we’re getting now. I hope we will do this again!
We do correct OSM upstream for the errors that we find. When our human review process catches a map error, we do two things: 1) Hold it back from release to our display maps 2) Fix the error upstream in OSM.org Ideally, the fix subsequently passes our process on the next round. -mike. > On Mar 9, 2020, at 5:47 PM, Joseph Eisenberg <joseph.eisenb...@gmail.com> > wrote: > > Interesting! That sounds like a huge amount of work, if you are > validating every changeset in the globe, though so far there is no > schedule to release this frequently. > > Thank you for following the Open Database license by releasing this to > the public. > > Is facebook planning to create new planet files like this on a > frequent basis, for internal use? If so, those should be released to > the public on a regular schedule, if I understand the ODbL correctly. > > Can I confirm that facebook is editing Openstreetmap data to fix the > errors which are found? Or is this only happening in some cases? > > The diary post gave links that explain how this was made (both from > September 2019): > - https://engineering.fb.com/ml-applications/mars/ "MaRS: How > Facebook keeps maps current and accurate" > > "We calculate the absolute diff between the two OSM versions (from > option 1 above). > We then split this diff into a set of LoChas that can be individually > applied (from option 2)." > ... > "At a high level, a map change is vetted by a reviewer," > > - > https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html > "“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at > Facebook": > > This is a video presentation, so I haven't seen it all, but the > abstract says: "we have created an automated ingestion and integrity > framework for OSM data that allows us to selectively update parts of > the map instead of doing a full snapshot change all at once. > > "Decomposing a large set of changes in this way gives us the > flexibility to rapidly ingest our own additions to the map, focus on > geographical areas of importance to downstream products, and allows us > to quickly apply hotfixes whenever egregious problems do arise. > > "With millions of tiny changes happening every week, we have created a > system that is built on per-feature approval and preprocessing, that > allows us to ingest changes at scale, while creating rules to > automatically process logical changesets and enforce integrity > constraints (e.g. anti-vandalism, anti-profanity etc.). > > "Due to the contextual nature of some of the changes in OpenStreetMap, > the system combines Human Approval, necessary for highly visible > features such as names of large administrative areas, with Automated > AI/ML-based approval: for example, using computer vision techniques to > reconcile newly created features against satellite imagery ground > truth, or applying NLP techniques to determine whether other > user-visible string changes are sensible and valid. These components > are combined to create a continuous ingest-validate-deploy cycle for > OSM map data." > > Lot's of buzz words there! But it sounds like it is a combination of > computer algorithms and human checking for vandalism and errors. > > - Joseph Eisenberg > > On 3/10/20, Michal Migurski <m...@teczno.com> wrote: >> Hi everyone, >> >> I’m writing to let you know about a new OpenStreetMap project Facebook just >> released. It’s called Daylight Map Distribution. Daylight is a complete, >> downloadable preview of OpenStreetMap data we plan to start using in a >> number of our public maps: >> >> https://www.openstreetmap.org/user/migurski/diary/392416 >> >> Facebook uses maps to let our users find friends, businesses, groups and >> more. OpenStreetMap (OSM) has a substantial global footprint of map data >> built and maintained by a dedicated community of global mappers and it’s a >> natural choice for us. Every day, OSM receives millions of contributions >> from the community. Some of these contributions may have intentional and >> unintentional edits that are incompatible with our needs. Our mapping teams >> work to scrub these contributions for consistency and quality. >> >> What’s Included in the Daylight Map Distribution: >> >> • A PBF planet file composed of 100% OSM data, released under the terms >> of >> the Open Database License. >> • Only those edits which have been validated to contain no malicious >> vandalism or unintentional errors so we can show them in our display maps >> >> This is just an initial first release, and we’re looking for feedback from >> the community to decide what would be useful to release in the future and >> how frequently. I’d be interested to hear any response you might have about >> it! >> >> -mike. >> >> ---------------------------------------------------------------- >> michal migurski- contact info and pgp key: >> sf/ca http://mike.teczno.com/contact.html >> >> >> _______________________________________________ >> talk mailing list >> talk@openstreetmap.org >> https://lists.openstreetmap.org/listinfo/talk >> > ---------------------------------------------------------------- michal migurski- contact info and pgp key: sf/ca http://mike.teczno.com/contact.html _______________________________________________ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
