Re: [Talk-us] MassGIS Building Import - process
On Thu, Dec 13, 2012 at 11:12 PM, Jason Remillard remillard.ja...@gmail.com wrote: Hi, I update the wiki http://wiki.openstreetmap.org/wiki/MassGIS_Buildings_Import Jason, While I think it's great you want to help OSM in this way, I have a few concerns. I think it's great that you've documented the process, but even after following a lot of links that point to other links that point to Google Docs, I can't find the scripts you use to do the conversion. If those script are available at all, they're hard to find and review. And the same goes for the timeframe. It seems you documented this on Friday and then ran it on Sunday? I don't think I'm the only one who sometimes takes email vacations on the weekend, and I didn't feel I had time to look over any of the process. I suspect others may feel the same way. - Serge ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
Hi, The script has been getting move around a bit as switched over to having all of the OSM files in one zip file. I put the script in the google docs directory, outside of the big zip. Thanks Jason. On Sun, Dec 16, 2012 at 7:41 PM, Serge Wroclawski emac...@gmail.com wrote: On Thu, Dec 13, 2012 at 11:12 PM, Jason Remillard remillard.ja...@gmail.com wrote: Hi, I update the wiki http://wiki.openstreetmap.org/wiki/MassGIS_Buildings_Import Jason, While I think it's great you want to help OSM in this way, I have a few concerns. I think it's great that you've documented the process, but even after following a lot of links that point to other links that point to Google Docs, I can't find the scripts you use to do the conversion. If those script are available at all, they're hard to find and review. And the same goes for the timeframe. It seems you documented this on Friday and then ran it on Sunday? I don't think I'm the only one who sometimes takes email vacations on the weekend, and I didn't feel I had time to look over any of the process. I suspect others may feel the same way. - Serge ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
If anyone has a problem relating to this I don't mind fixing it. The way to do it is simply to download all the data within the bounding box, then use either the search or the authors dialogue to select all the data uploaded by that particular user and remove. Then the data can be re-uploaded :) Cheers, ingalls On Sat, Dec 15, 2012 at 1:18 AM, Paul Norman penor...@mac.com wrote: I notice that several of you are using uploads that involve 50k nodes per changeset. ** ** What is your revert plan if one of these long extended uploads gets interrupted? The JOSM reverter plugin will not handle this correctly. ** ** *From:* Jason Remillard [mailto:remillard.ja...@gmail.com] *Sent:* Thursday, December 13, 2012 8:12 PM *To:* talk-us@openstreetmap.org; impo...@openstreetmap.org *Subject:* [Talk-us] MassGIS Building Import - process ** ** Hi, I update the wiki http://wiki.openstreetmap.org/wiki/MassGIS_Buildings_Import It includes all of the emails from tonight. - The import account will be restricted to directly uploading the data. Any post fixes should be done with the normal account. - There will be a new upload tomorrow with simplified data, and the full massgis data osm replaced with an overlapping osm. This will be saved away for later incase anybody wants to do merges after this project is done. I will do Groton for sure Only two remaining issue - If we should restrict manual importing to a small group of people. I have gotten several feedback messages that letting everybody do it has *** * turned out not to be a great idea on other imports. We are up to 5 people now, all of them have zillion changes to the map. This is not a problem yet. Honestly, I am not sure if people will be beating down the doors to help or not... - How hard do we work on the manual imports. Do we stop doing them if the data looks really good, and just run the script, or do we keep doing the manual imports until we tire of it. Running the script sooner, gets the map better quicker, and lighter OSM users can probably handle cleaning up the small 1/500 errors. After this, I think we are ready to roll this up and get to work. Thanks Jason. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
Hi, On 14.12.2012 05:12, Jason Remillard wrote: - How hard do we work on the manual imports. Do we stop doing them if the data looks really good, and just run the script, or do we keep doing the manual imports until we tire of it. Running the script sooner, gets the map better quicker, and lighter OSM users can probably handle cleaning up the small 1/500 errors. My usual lament: 1. The map is not better just because more is on it. 2. If you cannot be bothered to fix problems then why should others be? What is your plan for growing the community to a point where it can maintain the data you plan to dump onto OSM? It is a common scenario to see importers aim for a quick win (look how much better we've made the map) and then let others clean up the mess. In programming, this is called technical debt - in order to be able to present nice results quickly you add some problems to your code which have to be cleaned out later. Sometimes there's a good reason for incurring such debt (e.g. because a deadline forces you to presents results quickly no matter how rotten your code is on the inside). What is your external deadline that would prevent you from doing things right? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
Paul - I've added a few comments and questions about changeset size and revert policies on the Import Guidelines Plan Outline wiki pages. Are there any recommended changeset size limits and/or revert plan practices? - Jeff On Sat, Dec 15, 2012 at 6:43 AM, nicholas ingalls nicholas.inga...@gmail.com wrote: If anyone has a problem relating to this I don't mind fixing it. The way to do it is simply to download all the data within the bounding box, then use either the search or the authors dialogue to select all the data uploaded by that particular user and remove. Then the data can be re-uploaded :) Cheers, ingalls On Sat, Dec 15, 2012 at 1:18 AM, Paul Norman penor...@mac.com wrote: I notice that several of you are using uploads that involve 50k nodes per changeset. ** ** What is your revert plan if one of these long extended uploads gets interrupted? The JOSM reverter plugin will not handle this correctly. ** ** *From:* Jason Remillard [mailto:remillard.ja...@gmail.com] *Sent:* Thursday, December 13, 2012 8:12 PM *To:* talk-us@openstreetmap.org; impo...@openstreetmap.org *Subject:* [Talk-us] MassGIS Building Import - process ** ** Hi, I update the wiki http://wiki.openstreetmap.org/wiki/MassGIS_Buildings_Import It includes all of the emails from tonight. - The import account will be restricted to directly uploading the data. Any post fixes should be done with the normal account. - There will be a new upload tomorrow with simplified data, and the full massgis data osm replaced with an overlapping osm. This will be saved away for later incase anybody wants to do merges after this project is done. I will do Groton for sure Only two remaining issue - If we should restrict manual importing to a small group of people. I have gotten several feedback messages that letting everybody do it has ** ** turned out not to be a great idea on other imports. We are up to 5 people now, all of them have zillion changes to the map. This is not a problem yet. Honestly, I am not sure if people will be beating down the doors to help or not... - How hard do we work on the manual imports. Do we stop doing them if the data looks really good, and just run the script, or do we keep doing the manual imports until we tire of it. Running the script sooner, gets the map better quicker, and lighter OSM users can probably handle cleaning up the small 1/500 errors. After this, I think we are ready to roll this up and get to work. Thanks Jason. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us -- Jeff Meyer Global World History Atlas www.gwhat.org j...@gwhat.org 206-676-2347 ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
On Sat, Dec 15, 2012 at 7:50 AM, Frederik Ramm frede...@remote.org wrote: 2. If you cannot be bothered to fix problems then why should others be? What is your plan for growing the community to a point where it can maintain the data you plan to dump onto OSM? What's an acceptable error rate? 0? What's the error rate for manual entry? I'd suggest that if Jason hits 1 error in 500 whatevers, that that's lower than manual entry. What are some specific examples of costs expected to be incurred by importing the building outlines the Massachusetts OSM community has identified? Could concerns about errors be addressed by a manual, building-by-building QA process that might address any error issues? What can he *do* to address your concerns? Rather than addressing the imports=good vs. imports=bad discussion, what are the steps Jason hasn't addressed that should be included in order to make sure that we are minimizing the potential for introducing problems into the OSM database. - Jeff ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
On 15 December 2012 20:09, Jeff Meyer j...@gwhat.org wrote: Paul - I've added a few comments and questions about changeset size and revert policies on the Import Guidelines Plan Outline wiki pages. Are there any recommended changeset size limits and/or revert plan practices? One good practice is not to revert data that is not known to be wrong. If a big changeset fails halfway through it's possible to fix the remaining part to use the nodes that have been uploaded and continue, rather than delete the 1000s of nodes just to create new ones in the same places. You can probably now do that in JOSM by downloading the changeset containing the orphaned nodes, opening in JOSM together with the data being uploaded and telling the validator to fix all duplicate nodes. Myself I've been using the python scripts at http://wiki.openstreetmap.org/wiki/Upload.py in such situations, although the api is much more stable now than it was a couple years ago. The other good practice, but possibly not usable with JOSM alone, is not to let the program upload the naked nodes in bulk and then the buildings in bulk. You can sort the elements in such a way that every 50k element changeset contains say 45k nodes and 5k ways. The scripts let you limit the number of elements in a chunk which the next chunk depends on to the minimum (optimally 0), this way there's no risk of a passer by spotting orphan nodes and deleting some causing you conflicts in your next chunk. Cheers ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
On 15 December 2012 22:45, Paul Norman penor...@mac.com wrote: From: andrzej zaborowski [mailto:balr...@gmail.com] Subject: Re: [Talk-us] MassGIS Building Import - process On 15 December 2012 20:09, Jeff Meyer j...@gwhat.org wrote: Paul - I've added a few comments and questions about changeset size and revert policies on the Import Guidelines Plan Outline wiki pages. Are there any recommended changeset size limits and/or revert plan practices? One good practice is not to revert data that is not known to be wrong. If a big changeset fails halfway through it's possible to fix the remaining part to use the nodes that have been uploaded and continue, rather than delete the 1000s of nodes just to create new ones in the same places. You can probably now do that in JOSM by downloading the changeset containing the orphaned nodes, opening in JOSM together with the data being uploaded and telling the validator to fix all duplicate nodes. This hasn't come up for me on an import but I've tried it with normal mapping. I don't believe you can do a fix all on duplicate nodes and instead have to resolve them all individually With a current JOSM it seems you can select all the errors and click fix, or you can select the Other duplicate nodes category and click fix, I just checked. But JOSM will add each pair of merged nodes as an individual operation in undo history. I noticed it also merges the dupe nodes when you're merging two layers rather than copy from one layer and paste onto another. Myself I've been using the python scripts at http://wiki.openstreetmap.org/wiki/Upload.py in such situations, although the api is much more stable now than it was a couple years ago. What's your workflow? Do changes in JOSM, save and then pass to the scripts? Yes. The osm2change script understands the JOSM format and produces and .osc. With the TIGER name expansion the bot produced .osc files directly which were reviewed in a text editor. The other good practice, but possibly not usable with JOSM alone, is not to let the program upload the naked nodes in bulk and then the buildings in bulk. You can sort the elements in such a way that every 50k element changeset contains say 45k nodes and 5k ways. The scripts let you limit the number of elements in a chunk which the next chunk depends on to the minimum (optimally 0), this way there's no risk of a passer by spotting orphan nodes and deleting some causing you conflicts in your next chunk. The best way to do this in JOSM alone is to only merge in 1-5k nodes+ways at a time, review them, then upload. This also avoids most of the problems above. I would only ever do a 50k object changeset in very limited circumstances where I am confident that it is safe to do so. Even then I'd try to keep it under 25k normally. For normal imports I'd suggest 10k as a soft limit. Yes, good points. Cheers ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
Jeff Meyer j...@gwhat.org writes: On Sat, Dec 15, 2012 at 7:50 AM, Frederik Ramm frede...@remote.org wrote: 2. If you cannot be bothered to fix problems then why should others be? What is your plan for growing the community to a point where it can maintain the data you plan to dump onto OSM? What's an acceptable error rate? 0? What's the error rate for manual entry? I'd suggest that if Jason hits 1 error in 500 whatevers, that that's lower than manual entry. Well put. Before the import, a vast number of buildings that actually exist were not shown on the map. To me, that is a greater problem than a tiny number of non-buildings rendered as buildings. One can argue about the weight of actual buildings not shown vs non-buildings shown in computing a total map quality function, but by all accounts of people who have looked at the data the quality is very high. And the non-buildings shown are things are typically thing that are structure-like but not really structures, which is exactly the kind of thing that I've added by hand as builings when maybe I shouldn't. The community in Mass is actually growing, and I haven't heard from anyone local who is opposed to what's happening. This process is causing us to know each other better. So I think random complaining at Jason for you are importing so you must be harming the community is uncalled for, unsubstantiated, and harmful to building community. In my view, community is about whether people want to belong and want to work with each other, and I see the current situation as a net plus. pgpSKUvwmGOiS.pgp Description: PGP signature ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
Jeff Meyer j...@gwhat.org writes: Paul - I've added a few comments and questions about changeset size and revert policies on the Import Guidelines Plan Outline wiki pages. Are there any recommended changeset size limits and/or revert plan practices? Tools support for reverting is a fair point. But, I'd argue that it's a bug in JOSM if one can upload a single changeset that the reverter can't undo, and if so (I may be confused) this should be fixed in JOSM rather than be a special manual requirement. pgpphR8cNggpV.pgp Description: PGP signature ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] MassGIS Building Import - process
From: Greg Troxel [mailto:g...@ir.bbn.com] Subject: Re: [Talk-us] MassGIS Building Import - process Jeff Meyer j...@gwhat.org writes: Paul - I've added a few comments and questions about changeset size and revert policies on the Import Guidelines Plan Outline wiki pages. Are there any recommended changeset size limits and/or revert plan practices? Tools support for reverting is a fair point. But, I'd argue that it's a bug in JOSM if one can upload a single changeset that the reverter can't undo, and if so (I may be confused) this should be fixed in JOSM rather than be a special manual requirement. It's not a bug in JOSM. JOSM gives you lots of way to shoot yourself in the foot, this is due to an interaction between large changesets, the rails port, database/serialization times on the changeset /download call and imports. Because it's pretty much import-specific it's not too serious an issue if people are following the guidelines because they require you to have the knowledge of how to fix your mistakes. The checklist suggests developing a revert plan and the guidelines specifically state If you don't know how to revert an import, don't do the import in the first place. All of this being said, an item on my to-do list is to propose reducing the maximum changeset size. Right now it is too easy to make a changeset that only specialized ill-documented tricky to use tools can download. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
[Talk-us] MassGIS Building Import - process
Hi, I update the wiki http://wiki.openstreetmap.org/wiki/MassGIS_Buildings_Import It includes all of the emails from tonight. - The import account will be restricted to directly uploading the data. Any post fixes should be done with the normal account. - There will be a new upload tomorrow with simplified data, and the full massgis data osm replaced with an overlapping osm. This will be saved away for later incase anybody wants to do merges after this project is done. I will do Groton for sure Only two remaining issue - If we should restrict manual importing to a small group of people. I have gotten several feedback messages that letting everybody do it has turned out not to be a great idea on other imports. We are up to 5 people now, all of them have zillion changes to the map. This is not a problem yet. Honestly, I am not sure if people will be beating down the doors to help or not... - How hard do we work on the manual imports. Do we stop doing them if the data looks really good, and just run the script, or do we keep doing the manual imports until we tire of it. Running the script sooner, gets the map better quicker, and lighter OSM users can probably handle cleaning up the small 1/500 errors. After this, I think we are ready to roll this up and get to work. Thanks Jason. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
