Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Wed, 2017-09-27 at 09:18 +0200, Jo wrote: > 2017-09-27 8:30 GMT+02:00 Safwat Halaby: > > > On Tue, 2017-09-26 at 11:46 +0200, Jo wrote: > > > > > Then load that in PostGIS and create scripts to read GTFS into > > > PostGIS. > > > > > > Then compare the data in the DB and produce output and ideally a > > > UI. > > > > > > I started doing something like that here: > > > > > > https://github.com/osmbe/public_transport > > > > > > Let me know if you see ways of working on that or another way to > > > tackle the > > > problem together. > > > > > > Jo > > > > I will check the project out. Thanks for the link. Would you mind > > explaining what it is capable of? The readme is not so descriptive. > > > > For the time being not so much yet. Before the summer I made some > progress > migrating scripts I had created for an import of Belgian bus stops > and > lines, but during the summer I got a bit 'distracted' by mentoring > the > PT_Assistant plugin. > > The basic idea is to take operator data, either GTFS or a dump of > their > internal DB. > > Then compare all the stops regarding tags and position. For the stops > the > other tables routes, trips and segments, can be used to calculate > route_ref. > > Then create OSM route relations based on their data and compare to > what is > present in OSM. > > This is where it becomes trickier to keep track of their versions and > ours, > especially if the intent is to both give feedback to the operators > and have > a platform showing mappers which stops, routes and route_masters are > in > need of attention. > > Polyglot That's very interesting. My script is far less ambitious (no routes, only plain stops). ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
I think my last two replies never got through and were sent privately instead. Here's a rephrasing. (which is possibly better anyways). __ On Wed, 2017-09-27 at 10:53 +0200, Marc Gemis wrote: > isn't it possible that the 2017 contains data from e.g. 2014, which > has not been reviewed in the meantime ? > Which would then mean that the 2015 edit is more recent. > > Or is there a review date in the "database" ? There is no internal review date, and what you describe could happen. This is the only drawback I can think of. The first run would be imperfect in that regard, but the government publishes new GTFS dumps on a daily basis, so after the first run this wouldn't be an issue at all, and the "review date" would be "yesterday" or at worst "during the last few weeks". This is still better than any practically possible manual work considering our mapper density and the amount of stops. No one has been able to manually review or edit all those stops and it's been 5 years and the stops are horribly stale. Note that: 1. extra tags (shelter, wheelchair) will be preserved despite the drawback 2. the new database is considered pretty good quality, so issues will likely be negligible or nonexistent. __ On Wed, 2017-09-27 at 11:07 +0200, Jo wrote: > If there is a conflict regarding position or tags, they should be > resolved > by a human mapper. If I were to apply the newer is better approach, > we > would constantly be reverting back to the positions the operators > think > their stops are at. > > It's important to respect the mappers work, because without mappers > OSM > quickly dies off. > > It might be complex to put such a solution in place, but it should be > obvious that if data flows in 2 directions, too much simplification > doesn't > work. The script will not "constantly revert". 1. Government publishes database with bad edit X: script adds it 2. user fixes it with fix Y 3. Goverment publishes database which still has bad edit X: The script ignores 3, because "newer is better" and the user edit is newer. No constant reverts are involved. However, in the following scenario, the script would indeed introduce a bad edit to OSM: 1. Government publishes database with bad edit X: script adds it 2. user fixes it with fix Y 3. Goverment publishes database with a NEW BAD EDIT Z. So the script rests on the assumption that whenever a government updates a bus stop, it's because the bus stop has actually changed or because a newer better measurement was made, so Z is always expected to be better than X (and usually better than Y because it's newer in time) and this scenario shouldn't exist. Z is always assumed better. If a provider is unreliable such that it makes random edits in its new database that are LESS accurate than its older database, then this script is not suitable for dealing with such a provider and the complexity of your project is needed. This does not seem to be the case in Israel. To quote anonymous_gushdan_mapper from the forums: > Israel has something like 30,000 bus stops, > and they change daily all across the country. > There's no way human mappers could ever verify > the accuracy of all of them, unless you have > someone working full-time on > this. However, the data is considered extremely > accurate, and inaccuracies are quite rare. > We do have a system that announces the name > of the next stop, which uses this data. > I think people from other countries don't realize that this > is not a single, private operator data, nor it's a single > city data - it's government generated data that controls > the entire public transportation network in the country. > If a bus stop is not in this dataset, it doesn't exist. > There will never be anything more accurate for bus stops > in Israel than this dataset. > If accuracy is important to us, we *must* implement this > importing script, otherwise the data on OSM will get > stale quickly - just like the current data is stale > and shows a lot of bus stops that have been > since then moved or canceled. source: https://forum.openstreetmap.org/viewtopic.php?pid=665570#p665570 So, I could probably naively always trust the GTFS and it'll still be mostly fine. But I still want to give mappers the ability to fix coordinates and such, without overriding their edits the next run (unless they've been updated by the government to newer values during that time). Lastly, I'd like to point out that no local mappers are complaining about this and the feedback is so far positive. Thanks. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
If there is a conflict regarding position or tags, they should be resolved by a human mapper. If I were to apply the newer is better approach, we would constantly be reverting back to the positions the operators think their stops are at. It's important to respect the mappers work, because without mappers OSM quickly dies off. It might be complex to put such a solution in place, but it should be obvious that if data flows in 2 directions, too much simplification doesn't work. Jo 2017-09-27 10:46 GMT+02:00 Safwat Halaby: > On Wed, 2017-09-27 at 09:12 +0200, Jo wrote: > > Deleting data on OpenStreetMap and replacing it by imported data is > > obviously never the acceptable approach. > > > > What I don't understand is why you don't create something that > > compares the > > latest version of all the bus stops in OSM with the latest version of > > the > > GTFS data from upstream. > > > > Why compare with an inbetween version? > > I'm taking a "Newer is better" approach for conflict resolution. > > Let's say a user made a 2015 edit for a coordinate (We'll call this > version X). And your 2017 database has a different coordinate (We'll > call this version Y). How do you determine which coordinate to keep? > > If you had a 2012 database that would be easy. > > Case 1: > 2012 database: Y > 2015 user edit: X > 2017 database: Y > > This means the Y has not been updated since 2012. Newer is better, so > trust the user edit and set the coordinates to X. > > Case 2: > 2012 database: T > 2015 user edit: X > 2017 database: Y > > This means Y is the newest value, and we should override the user edit. > > Without two database, we'd have to guess or resort to manual user > intervention via fancy web services. > > > > What I think is needed is a (web) service that stands between the > > operator > > data, be it GTFS or DB dumps and OpenStreetMap where comparison is > > made and > > which can be used by mappers to either improve OSM, or to send > > feedback to > > the operators that there are issues with the data they provide. Or > > where > > the operators can request flagged stops in bulk. > > > > I am a huge advocate of simplicity. In my humble opinion, web services > and SQL databases are overkill for what I'm trying to do. Your project > spans dozens of files while mine is a single .js file for the JOSM > scripting plugin, which reads two textual stops.txt files from the old > and the newer GTFS databases. > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Wed, 2017-09-27 at 09:12 +0200, Jo wrote: > Deleting data on OpenStreetMap and replacing it by imported data is > obviously never the acceptable approach. > > What I don't understand is why you don't create something that > compares the > latest version of all the bus stops in OSM with the latest version of > the > GTFS data from upstream. > > Why compare with an inbetween version? I'm taking a "Newer is better" approach for conflict resolution. Let's say a user made a 2015 edit for a coordinate (We'll call this version X). And your 2017 database has a different coordinate (We'll call this version Y). How do you determine which coordinate to keep? If you had a 2012 database that would be easy. Case 1: 2012 database: Y 2015 user edit: X 2017 database: Y This means the Y has not been updated since 2012. Newer is better, so trust the user edit and set the coordinates to X. Case 2: 2012 database: T 2015 user edit: X 2017 database: Y This means Y is the newest value, and we should override the user edit. Without two database, we'd have to guess or resort to manual user intervention via fancy web services. > What I think is needed is a (web) service that stands between the > operator > data, be it GTFS or DB dumps and OpenStreetMap where comparison is > made and > which can be used by mappers to either improve OSM, or to send > feedback to > the operators that there are issues with the data they provide. Or > where > the operators can request flagged stops in bulk. > I am a huge advocate of simplicity. In my humble opinion, web services and SQL databases are overkill for what I'm trying to do. Your project spans dozens of files while mine is a single .js file for the JOSM scripting plugin, which reads two textual stops.txt files from the old and the newer GTFS databases. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
2017-09-27 8:30 GMT+02:00 Safwat Halaby: > On Tue, 2017-09-26 at 11:46 +0200, Jo wrote: > > > Then load that in PostGIS and create scripts to read GTFS into > > PostGIS. > > > > Then compare the data in the DB and produce output and ideally a UI. > > > > I started doing something like that here: > > > > https://github.com/osmbe/public_transport > > > > Let me know if you see ways of working on that or another way to > > tackle the > > problem together. > > > > Jo > > I will check the project out. Thanks for the link. Would you mind > explaining what it is capable of? The readme is not so descriptive. > For the time being not so much yet. Before the summer I made some progress migrating scripts I had created for an import of Belgian bus stops and lines, but during the summer I got a bit 'distracted' by mentoring the PT_Assistant plugin. The basic idea is to take operator data, either GTFS or a dump of their internal DB. Then compare all the stops regarding tags and position. For the stops the other tables routes, trips and segments, can be used to calculate route_ref. Then create OSM route relations based on their data and compare to what is present in OSM. This is where it becomes trickier to keep track of their versions and ours, especially if the intent is to both give feedback to the operators and have a platform showing mappers which stops, routes and route_masters are in need of attention. Polyglot ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Deleting data on OpenStreetMap and replacing it by imported data is obviously never the acceptable approach. What I don't understand is why you don't create something that compares the latest version of all the bus stops in OSM with the latest version of the GTFS data from upstream. Why compare with an inbetween version? What I think is needed is a (web) service that stands between the operator data, be it GTFS or DB dumps and OpenStreetMap where comparison is made and which can be used by mappers to either improve OSM, or to send feedback to the operators that there are issues with the data they provide. Or where the operators can request flagged stops in bulk. The same goes for the lines (route_master) and the various itineraries (route relations). Polyglot 2017-09-27 7:42 GMT+02:00 Safwat Halaby: > Hi John Whelan, > > As implied in the forum thread, not wanting to destroy user data is > exactly why I'm building a relatively complex script. The naive > approach is to destroy all-bus stops are re-import, everytime a GTFS > update is released. But I don't want that. > > Instead of doing that, the script preserves all user-submitted data > such as shelter, wheelchair, and GPS coordinate fixes and provides > incremental updates rather than the destruction of all bus stops per > import. The incremental updates do not destroy user changes. In order > to achieve that, the older and the newer GTFS database need to be > compared per update. > > Yesterday I didn't have the database which was used in the old 2012 > import, so I couldn't locally test-run my script. which is what lead to > my original question in this thread. But now I managed to reverse-build > the old GTFS database from version 3 of the bus stops in Israel by > downloading the bus stops through the relevant changesets. > > Here's some of the discussion: > > >Here's a merging idea. > > > >Problem: Dealing with conflicts between mapper > edits and gtfs data. > >Solution: "The most recent version is the correct > version" philosophy. > > > >- The first gtfs update would update everything. > Conflicts are > > resolved by prioritizing the gtfs file's version. This > is a > > "necessary evil" but is >only needed once. (edit: I might be > > > able to mitigate this by tracing bus stop OSM history). > >- Some time > passes, and users update some of the bus stops. > >- The ministry of > transportation updates some bus stops in > > its database and publishes a > new gtfs file. > >- The next gtfs update would inspect the difference > between > > the new gtfs file and the older gtfs file. Only bus stops > > > that have had their data (in >the gtfs file) changed since the > > last > file are updated. So, conflicts are resolved by prioritizing > > the gtfs > file version, but only for the bus stops that were changed > > by the > ministry since the last update. The rest of the bus stops > > are left > intact. > > (source: https://forum.openstreetmap.org/viewtopic.php?id=16738=3) > > Also, we know the data can be used in OSM. > > https://forum.openstreetmap.org/viewtopic.php?pid=244096#p244096 > > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 11:46 +0200, Jo wrote: > Then load that in PostGIS and create scripts to read GTFS into > PostGIS. > > Then compare the data in the DB and produce output and ideally a UI. > > I started doing something like that here: > > https://github.com/osmbe/public_transport > > Let me know if you see ways of working on that or another way to > tackle the > problem together. > > Jo I will check the project out. Thanks for the link. Would you mind explaining what it is capable of? The readme is not so descriptive. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Thanks for the info, mmd! ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Hi John Whelan, As implied in the forum thread, not wanting to destroy user data is exactly why I'm building a relatively complex script. The naive approach is to destroy all-bus stops are re-import, everytime a GTFS update is released. But I don't want that. Instead of doing that, the script preserves all user-submitted data such as shelter, wheelchair, and GPS coordinate fixes and provides incremental updates rather than the destruction of all bus stops per import. The incremental updates do not destroy user changes. In order to achieve that, the older and the newer GTFS database need to be compared per update. Yesterday I didn't have the database which was used in the old 2012 import, so I couldn't locally test-run my script. which is what lead to my original question in this thread. But now I managed to reverse-build the old GTFS database from version 3 of the bus stops in Israel by downloading the bus stops through the relevant changesets. Here's some of the discussion: >Here's a merging idea. > >Problem: Dealing with conflicts between mapper edits and gtfs data. >Solution: "The most recent version is the correct version" philosophy. > >- The first gtfs update would update everything. Conflicts are > resolved by prioritizing the gtfs file's version. This is a > "necessary evil" but is >only needed once. (edit: I might be > able to mitigate this by tracing bus stop OSM history). >- Some time passes, and users update some of the bus stops. >- The ministry of transportation updates some bus stops in > its database and publishes a new gtfs file. >- The next gtfs update would inspect the difference between > the new gtfs file and the older gtfs file. Only bus stops > that have had their data (in >the gtfs file) changed since the > last file are updated. So, conflicts are resolved by prioritizing > the gtfs file version, but only for the bus stops that were changed > by the ministry since the last update. The rest of the bus stops > are left intact. (source: https://forum.openstreetmap.org/viewtopic.php?id=16738=3) Also, we know the data can be used in OSM. https://forum.openstreetmap.org/viewtopic.php?pid=244096#p244096 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
One last point have you confirmed the Open Data license is acceptable to OSM? It took me about five years and a great deal of effort to ensure and confirm the licenses were compatible to get my GTFS bus stops in. The OSM legal working group eventually agreed they were but noted they would want to confirm every other Canadian municipal license Open Data licence before agreeing they were compatible. The LWG are volunteers and I understand there is a small backlog. Cheerio John On 26 Sep 2017 5:18 pm, "john whelan"wrote: > Just a clarification, I'm not against imports and I have been involved in > a number as I'm sure Frederik will recall but it's important to me that the > data to be imported is of good quality, is correctly licensed and is merged > with existing data. In the case of bus stops that means including tags > previously mapped such as shelter and wheelchair=yes and not just deleting > the old and dropping in the new. > > Mappers who have spent time tagging deserve their efforts should be > recorded. Although I sometimes have my doubts about a mapper who has > mapped once for half an hour and added a very inaccurate building tagged as > an apartment block in a small African village but in general it should be > followed. > > Cheerio John > > On 26 Sep 2017 5:08 pm, "john whelan" wrote: > >> >We are well aware that GTFS accuracy is not perfect. (Though nobody Reported >> a 100-meter deviation so far). A potential solution, which could also >> work globally, is discussed below. >> >> Some University researchers who were looking at the GTFS files noted the >> problem sometime ago in the US. One provider was a couple of hundred >> metres out on bus stop location. Locally they were a little better but it >> was only when the bus announcement system came in and they very very >> carefully measured the location of each bus stop to comply with the >> provincial instructions that the GTFS file became at all usable and even >> then it took a dedicated team to do it and it wasn't a straight import. >> >> Personally unless you can confirm the accuracy of the bus stops my >> feeling is no data is better than poor data that cannot be trusted. >> >> Please do not do this globally. >> >> Cheerio John >> >> ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Just a clarification, I'm not against imports and I have been involved in a number as I'm sure Frederik will recall but it's important to me that the data to be imported is of good quality, is correctly licensed and is merged with existing data. In the case of bus stops that means including tags previously mapped such as shelter and wheelchair=yes and not just deleting the old and dropping in the new. Mappers who have spent time tagging deserve their efforts should be recorded. Although I sometimes have my doubts about a mapper who has mapped once for half an hour and added a very inaccurate building tagged as an apartment block in a small African village but in general it should be followed. Cheerio John On 26 Sep 2017 5:08 pm, "john whelan"wrote: > >We are well aware that GTFS accuracy is not perfect. (Though nobody Reported > a 100-meter deviation so far). A potential solution, which could also > work globally, is discussed below. > > Some University researchers who were looking at the GTFS files noted the > problem sometime ago in the US. One provider was a couple of hundred > metres out on bus stop location. Locally they were a little better but it > was only when the bus announcement system came in and they very very > carefully measured the location of each bus stop to comply with the > provincial instructions that the GTFS file became at all usable and even > then it took a dedicated team to do it and it wasn't a straight import. > > Personally unless you can confirm the accuracy of the bus stops my feeling > is no data is better than poor data that cannot be trusted. > > Please do not do this globally. > > Cheerio John > > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 17:08 -0400, john whelan wrote: > Please do not do this globally. > Of course not. I wouldn't deploy anything globally without prior discussion. What I meant was the script is not Israel-specific, and anyone who knows their provider has some reasonable accuracy could use it. It allows incremental GTFS updates where mapper-submitted accuracy fixes are not overridden by the next update. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
>We are well aware that GTFS accuracy is not perfect. (Though nobody Reported a 100-meter deviation so far). A potential solution, which could also work globally, is discussed below. Some University researchers who were looking at the GTFS files noted the problem sometime ago in the US. One provider was a couple of hundred metres out on bus stop location. Locally they were a little better but it was only when the bus announcement system came in and they very very carefully measured the location of each bus stop to comply with the provincial instructions that the GTFS file became at all usable and even then it took a dedicated team to do it and it wasn't a straight import. Personally unless you can confirm the accuracy of the bus stops my feeling is no data is better than poor data that cannot be trusted. Please do not do this globally. Cheerio John ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 08:50 -0400, john whelan wrote: > I suggest pulling in the country file and if need be chop it up to > load > into JOSM. Load up the latest bus stops in a different layer then > use the > todo plugin and go through them one at a time cleaning them up that > way. > > Note that the GTFS file bus stop location may not be accurate, some > bus > stops are known to be 100 meters out, it depends on the provider. If > you > have a system that announces bus stops for partially sighted people > then > that is an indication they should be fairly good but check with the > provider. Even when they are accurate they can still be out by a few > meters, locally one was in the middle of a road junction so they do > need > verifying. > We are well aware that GTFS accuracy is not perfect. (Though nobody reported a 100-meter deviation so far). A potential solution, which could also work globally, is discussed below. I already have a (somewhat) working prototype. I'll look for other solutions too. https://forum.openstreetmap.org/viewtopic.php?id=16738=3 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
link fix: https://forum.openstreetmap.org/viewtopic.php?id=16738 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
I'd like to point out (as I pointed out in the sub-threads), that I solved this problem simply by fetching the changeset which introduced v3. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 14:12 +0200, Frederik Ramm wrote: > > Are all these automatic edits at least discussed on the Israeli > mailing > list, or is this a free-for-all where anyone who knows how to write a > script runs over all bus stops in Israel again and again? > > Bye > Frederik > Why are you automatically assuming the worst? I have, and I am, extensively discussing fixing the import mess. Even the previously messy imports by the other users were discussed and were fairly coordinated. See this: https://forum.openstreetmap.org/viewtopic.php?id =16738 ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Am 26.09.2017 um 12:39 schrieb Safwat Halaby: > On Tue, 2017-09-26 at 11:46 +0200, Jo wrote: >> >> Overpass API is definitely your friend. Version 3 doesn't mean much >> though. >> Do you mean all bus stops with a public_transport tag? > > Thanks for the reply. > > By version 3, I mean a particular historic version of the OSM element. > > For instance, see "Version #3" of this bus stop: > > https://www.openstreetmap.org/node/1802982884/history > > I need all "version #3" of all bus stops that have a > "source=israel_gtfs_v1" in Israel. As far as I know. Overpass can only > fetch the latest version. Can overpass directly fetch version #3? > That's not yet possible with release 0.7.54. A future Overpass release will allow you to filter for a specific version of an object, which exists at a given point in time. That's either NOW, or the date you specify via [date: ...]. A query for all version 3 objects would only work, if you can find such a point in time, where all of those objects are valid. If there isn't such date, you need to try several queries with varying date. Here's some to try out in the meantime: http://overpass-turbo.eu/s/rZc -- ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
I suggest pulling in the country file and if need be chop it up to load into JOSM. Load up the latest bus stops in a different layer then use the todo plugin and go through them one at a time cleaning them up that way. Note that the GTFS file bus stop location may not be accurate, some bus stops are known to be 100 meters out, it depends on the provider. If you have a system that announces bus stops for partially sighted people then that is an indication they should be fairly good but check with the provider. Even when they are accurate they can still be out by a few meters, locally one was in the middle of a road junction so they do need verifying. Cheerio John On 26 September 2017 at 08:12, Frederik Rammwrote: > Hi, > > On 26.09.2017 13:31, Safwat Halaby wrote: > > V3 was an automatic import for all nodes. it was made pretty quickly > > after v1 and v2 > > This whole import is a terrible mess: > > https://www.openstreetmap.org/node/1802982151/history > > It starts with the version 1 already being titled "attempt 3", then in > version 2 a "fixme = check/adjust position and/or merge with existing > stop if exists" is added which of course is another wording for "this > import should not have happened in the first place"! Later all name tags > were duplicated in name:he and the latest three versions are also some > form of automated doctoring. > > A month ago you've removed the "fixme=..." from 30k nodes and replaced > it with a "gtfs:reviewed=no", creating yet another version of the whole > data set in our database. What makes you think that, after the "fixme" > tags have *not* been acted on since the original import 5 years ago, > changing this to "gtfs:reviewed=no" will suddenly have an effect? > > Are all these automatic edits at least discussed on the Israeli mailing > list, or is this a free-for-all where anyone who knows how to write a > script runs over all bus stops in Israel again and again? > > Bye > Frederik > > -- > Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" > > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Hi, On 26.09.2017 13:31, Safwat Halaby wrote: > V3 was an automatic import for all nodes. it was made pretty quickly > after v1 and v2 This whole import is a terrible mess: https://www.openstreetmap.org/node/1802982151/history It starts with the version 1 already being titled "attempt 3", then in version 2 a "fixme = check/adjust position and/or merge with existing stop if exists" is added which of course is another wording for "this import should not have happened in the first place"! Later all name tags were duplicated in name:he and the latest three versions are also some form of automated doctoring. A month ago you've removed the "fixme=..." from 30k nodes and replaced it with a "gtfs:reviewed=no", creating yet another version of the whole data set in our database. What makes you think that, after the "fixme" tags have *not* been acted on since the original import 5 years ago, changing this to "gtfs:reviewed=no" will suddenly have an effect? Are all these automatic edits at least discussed on the Israeli mailing list, or is this a free-for-all where anyone who knows how to write a script runs over all bus stops in Israel again and again? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
It is a bit odd that you are not interested in the latest versions of these objects, but OK. (As long as you don't plan to use them for an updated import to OSM, of course) Overpass API also makes it possible to fetch data for a given point in time. Jo 2017-09-26 13:31 GMT+02:00 Safwat Halaby: > On Tue, 2017-09-26 at 12:15 +0100, Tom Hughes wrote: > > On 26/09/17 11:41, Safwat Halaby wrote: > > > On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote: > > > > Hi, > > > > > > > > Am 2017-09-26 um 09:19 schrieb SwiftFast: > > > > > I need to download all version 3 bus stops in a country which > > > > > has > > > > > about > > > > > 30,000 bus stops. Version 3 exists since a 2012 import. What's > > > > > the > > > > > recommended way to accomplish this? I have two ways in mind. > > > > > > > > Version 3 of what? > > > > > > > > Best regards > > > > > > > > > > Version 3 of all Israeli bus stops that have > > > "source=israel_gtfs_v1". > > > See this: https://www.openstreetmap.org/node/1802982884/history and > > > look at "version #3". > > > > But how do you know it's version 3 you want for every one? > > > > It sounds like you're making an assumption that all these objects > > have > > only been edited in a particular way by some automated process and > > that > > nobody has ever touched one by hand and hence added an extra version. > > > > Tom > > > > V3 was an automatic import for all nodes. it was made pretty quickly > after v1 and v2 and it's very unlikely someone edited during that time. > But I solved my problems simply by fetching the responsible changeset. > I was overcomplicating this problem. All I needed was: > > get https://api.openstreetmap.org/api/0.6/changeset/14265835/download > > Thanks for the help! > > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 12:15 +0100, Tom Hughes wrote: > On 26/09/17 11:41, Safwat Halaby wrote: > > On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote: > > > Hi, > > > > > > Am 2017-09-26 um 09:19 schrieb SwiftFast: > > > > I need to download all version 3 bus stops in a country which > > > > has > > > > about > > > > 30,000 bus stops. Version 3 exists since a 2012 import. What's > > > > the > > > > recommended way to accomplish this? I have two ways in mind. > > > > > > Version 3 of what? > > > > > > Best regards > > > > > > > Version 3 of all Israeli bus stops that have > > "source=israel_gtfs_v1". > > See this: https://www.openstreetmap.org/node/1802982884/history and > > look at "version #3". > > But how do you know it's version 3 you want for every one? > > It sounds like you're making an assumption that all these objects > have > only been edited in a particular way by some automated process and > that > nobody has ever touched one by hand and hence added an extra version. > > Tom > V3 was an automatic import for all nodes. it was made pretty quickly after v1 and v2 and it's very unlikely someone edited during that time. But I solved my problems simply by fetching the responsible changeset. I was overcomplicating this problem. All I needed was: get https://api.openstreetmap.org/api/0.6/changeset/14265835/download Thanks for the help! ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
I think I've needlessly overcomplicated this. All I need to do was to do is fetch changeset #14265835, which introduced those historic nodes. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On 26/09/17 11:41, Safwat Halaby wrote: On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote: Hi, Am 2017-09-26 um 09:19 schrieb SwiftFast: I need to download all version 3 bus stops in a country which has about 30,000 bus stops. Version 3 exists since a 2012 import. What's the recommended way to accomplish this? I have two ways in mind. Version 3 of what? Best regards Version 3 of all Israeli bus stops that have "source=israel_gtfs_v1". See this: https://www.openstreetmap.org/node/1802982884/history and look at "version #3". But how do you know it's version 3 you want for every one? It sounds like you're making an assumption that all these objects have only been edited in a particular way by some automated process and that nobody has ever touched one by hand and hence added an extra version. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 11:31 +0200, Michael Reichert wrote: > Hi, > > Am 2017-09-26 um 09:19 schrieb SwiftFast: > > I need to download all version 3 bus stops in a country which has > > about > > 30,000 bus stops. Version 3 exists since a 2012 import. What's the > > recommended way to accomplish this? I have two ways in mind. > > Version 3 of what? > > Best regards > Version 3 of all Israeli bus stops that have "source=israel_gtfs_v1". See this: https://www.openstreetmap.org/node/1802982884/history and look at "version #3". ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
On Tue, 2017-09-26 at 11:46 +0200, Jo wrote: > Hi Safwat, > > Overpass API is definitely your friend. Version 3 doesn't mean much > though. > Do you mean all bus stops with a public_transport tag? Thanks for the reply. By version 3, I mean a particular historic version of the OSM element. For instance, see "Version #3" of this bus stop: https://www.openstreetmap.org/node/1802982884/history I need all "version #3" of all bus stops that have a "source=israel_gtfs_v1" in Israel. As far as I know. Overpass can only fetch the latest version. Can overpass directly fetch version #3? ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Hi Safwat, Overpass API is definitely your friend. Version 3 doesn't mean much though. Do you mean all bus stops with a public_transport tag? I would go about it in a slightly different way and download everything related to public transport, not just the stops using the query found on this page: https://wiki.openstreetmap.org/wiki/JOSM/Plugins/PT_Assistant/Mapping_Public_Transport_with_JOSM#Downloading_data Then load that in PostGIS and create scripts to read GTFS into PostGIS. Then compare the data in the DB and produce output and ideally a UI. I started doing something like that here: https://github.com/osmbe/public_transport Let me know if you see ways of working on that or another way to tackle the problem together. Jo 2017-09-26 9:19 GMT+02:00 SwiftFast: > I need to download all version 3 bus stops in a country which has about > 30,000 bus stops. Version 3 exists since a 2012 import. What's the > recommended way to accomplish this? I have two ways in mind. > > 1. Fetch all stops with Overpass API, then, for each >stop id, fetch version 3 from the main API. >- Pros: I already know how to do that >- Cons: Potentially too much load on the OSM server. >Will I be rate-limited? >Requires some script writing which could take >more time than the way below. > > 2. Download a history-supporting planet file, >and extract the desired bus stops through Osmosis. >- Pros: Less API load >- Cons: I'm not sure what the proper Osmosis commands are, >and if this is a supported operation. Also, I need >to download a big file (the oldest history supporting >import is 40GB from 2013). > > 3. Other suggestions? > > Why I need this: > > I am creating a script[1] for incrementally updating GTFS imports. > It'll initially be used in Israel but it can be adopted worldwide. We > have an import from 2012 which was "dumped and forgotten" and is now > very stale. I want to forever fix this. > > The ministry of transportation publishes new GTFS files daily. My > script compares an older GTFS file with newer one, and only updates the > bus stops that were changed/added/deleted. The idea is to feed it a > newer GTFS database daily or weekly. However, I do not possess the file > that was used in the 2012 import, so, in order to bootstrap the script, > I need to recreate that file from the old OSM data. > > [1] https://forum.openstreetmap.org/viewtopic.php?id=16738=3 > (note: SwiftFast and SafwatHalaby are both me) > > Thanks in advance. > > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading Version 3 of all bus stops in a country
Hi, Am 2017-09-26 um 09:19 schrieb SwiftFast: > I need to download all version 3 bus stops in a country which has about > 30,000 bus stops. Version 3 exists since a 2012 import. What's the > recommended way to accomplish this? I have two ways in mind. Version 3 of what? Best regards Michael -- Per E-Mail kommuniziere ich bevorzugt GPG-verschlüsselt. (Mailinglisten ausgenommen) I prefer GPG encryption of emails. (does not apply on mailing lists) signature.asc Description: OpenPGP digital signature ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk