Re: [Talk-us] NHD conversions to OSM
* Josh Doe [2011-06-09 15:43 -0400]: > I suppose one of the waterway=stream's should be changed to > waterway=river, and only one of the waterway=riverbank objects should be > used. Ian covered the medium/high and area/flowline difference already. As for the waterway=river issue, the NHD doesn't make a distinction between waterway=stream and waterway=river the way OSM does. If a waterway is wide enough, it gets represented in the area tables with FCode 46006, which we map to waterway=riverbank, and in the flowline tables with FCode 55800 (ARTIFICAL PATH), which we map to waterway=stream[0]. If there's no representation in the area tables, a waterway with year-round flowing water gets FCode 46006 in the flowline tables, regardless of its width. Basically, OSM draws different distinctions among waterways than the NHD does, so no mapping between the two tagging systems is going to be correct in all cases. [0] And we can't just map all 55800 FCodes to waterway=river because the same FCode is used *anywhere* a waterway flows through a wider area of water, including things like streams that flow in and out of ponds. -- ...computer contrarian of the first order... / http://aperiodic.net/phil/ PGP: 026A27F2 print: D200 5BDB FC4B B24A 9248 9F7A 4322 2D22 026A 27F2 --- -- If the aborigine drafted an IQ test, all of Western civilization would presumably flunk it. -- Stanley Garn --- -- ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On Thu, Jun 9, 2011 at 4:17 PM, Richard Weait wrote: > On Thu, Jun 9, 2011 at 5:08 PM, Ian Dees wrote: > > > If I remember correctly the converter is keeping as little in-memory as > > possible (so it can convert huge shapefiles). Also, it will split ways at > > 2000 nodes and turn things into multipolygons. > > If that 2000 could be cranked down lower, it would leave room for > local mappers to improve a geometry without also having to split the > way at the 2k boundary right away. Best practice iirc is to limit > ways to 1000 nodes when creating them. Good call. I'll file a bug for that. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On Thu, Jun 9, 2011 at 5:08 PM, Ian Dees wrote: > If I remember correctly the converter is keeping as little in-memory as > possible (so it can convert huge shapefiles). Also, it will split ways at > 2000 nodes and turn things into multipolygons. If that 2000 could be cranked down lower, it would leave room for local mappers to improve a geometry without also having to split the way at the 2k boundary right away. Best practice iirc is to limit ways to 1000 nodes when creating them. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On Thu, Jun 9, 2011 at 4:00 PM, Ben Supnik wrote: > Hi Ian, > > > On 6/9/11 1:34 PM, Ian Dees wrote: > >> 1) The XML converter should split up the data so that ways and their >> nodes are in the same XML file. I *think* my shp-to-osm tries to do >> this. If you end up with a bunch of XML files that have 50k node >> elements in them then you're doing it wrong [0]. >> >> 2) Each XML file should contain fewer than the absolute maximum number >> of changes allowed by the API and should be uploaded using the diff >> upload API endpoint. >> > > I'm using osm-to-shp.jar and it sort of does this. Each file contains its > own nodes, and --maxnodes will help create a larger number of small files. > If I use --maxnodes 5 then most files end up with 50k or fewer nodes, > but I found one with about 80k nodes. > > Is there an option to limit total changes of any type? If I remember correctly the converter is keeping as little in-memory as possible (so it can convert huge shapefiles). Also, it will split ways at 2000 nodes and turn things into multipolygons. One of the side effects is that it has to guess at how many elements will be created for a particular way before deciding to start converting it into OSM format. It will split into a new file if it guesses that the next shape will create a file with more changes than what you specified with --maxnodes. It will not split a single geometry in the shapefile into separate files. This is probably why you have a file with 80+ kilonodes (one big, complex multipolygon). I'd love to hear about better ways to do it. Code is here: https://github.com/iandees/shp-to-osm > > > 3) The upload script should keep track of the response information to >> see what nodes actually get uploaded. If I remember correctly there's no >> way to know which elements in a diff upload were successful and which >> weren't if you receive an error in the reply from the server. >> > > I don't have an upload script - my theory is that we could post the set of > xml diff files, ready for editing and let people manually integrate them > using an editor (e.g. JOSM) then upload once the NHD data and existing data > have been integrated. > > When I was doing my NHD uploads I used JOSM with mixed success. It didn't do a good job saving the results of diff uploads at the time. Maybe that's changed now. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On 6/9/2011 5:00 PM, Ben Supnik wrote: I don't have an upload script - my theory is that we could post the set of xml diff files, ready for editing and let people manually integrate them using an editor (e.g. JOSM) then upload once the NHD data and existing data have been integrated. Even JOSM doesn't properly handle the error cases from what I last knew - it depends on the user to resolve what has been uploaded after an error.I suppose a proper error recovery handler for JOSM could be submitted as a patch. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
Hi Ian, On 6/9/11 1:34 PM, Ian Dees wrote: 1) The XML converter should split up the data so that ways and their nodes are in the same XML file. I *think* my shp-to-osm tries to do this. If you end up with a bunch of XML files that have 50k node elements in them then you're doing it wrong [0]. 2) Each XML file should contain fewer than the absolute maximum number of changes allowed by the API and should be uploaded using the diff upload API endpoint. I'm using osm-to-shp.jar and it sort of does this. Each file contains its own nodes, and --maxnodes will help create a larger number of small files. If I use --maxnodes 5 then most files end up with 50k or fewer nodes, but I found one with about 80k nodes. Is there an option to limit total changes of any type? 3) The upload script should keep track of the response information to see what nodes actually get uploaded. If I remember correctly there's no way to know which elements in a diff upload were successful and which weren't if you receive an error in the reply from the server. I don't have an upload script - my theory is that we could post the set of xml diff files, ready for editing and let people manually integrate them using an editor (e.g. JOSM) then upload once the NHD data and existing data have been integrated. cheers ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
Hi Josh, On 6/9/11 2:37 PM, Josh Doe wrote: In the test data you provided I saw a number of objects that didn't have any tags that weren't in the gnis or nhd namespace (i.e no waterway tag). Is there use in importing these features? Is the thinking that maybe in the future we can add sensible tags to these? IMHO we either leave them out or give them sensible tags, even if this import is the only one that uses them. I don't think there's an easy way to do this with osm-to-shp.jar, short of removing _all_ gnis and nhd namespace tags. Again, I punt to others on the dictionary for conversion. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
Hi Josh, First: the wiki - I was planning to document the data once we had a final download location, either an official-ish OSM server or anything else we come up with. I will update the wiki page once we know where the data will live I've also looked at the data, and I must say I'm confused! One river is represented in the flh file as waterway=stream, in arhi as waterway=riverbank, in flmd as waterway=stream (but with worse spatial accuracy than flh), and then in armd as waterway=riverbank! Right - there are two resolutions in the NHD extracts the USGS sent me, medium and high. I am stripping out the medium res ones per Ian's comments, which will make things more consistent. I suppose one of the waterway=stream's should be changed to waterway=river, and only one of the waterway=riverbank objects should be used. Well, only one of the _layers_ should be used, e.g. all high or all medium, Ian saying all high. Whether the waterway should be stream or river, I defer to others who have deep insights or strong opinions on such things; these extracts are based on a conversion key built by others. Is this really what the NHD data is like? It will take more time than I thought to properly merge these these 6 or 7 layers. Well, once it's only high res, it's 5 layers that are spatially consistent. cheers ben ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On Thu, Jun 9, 2011 at 2:43 PM, Josh Doe wrote: > I've gone ahead and cleaned up the NHD page a bit. > > I've also looked at the data, and I must say I'm confused! One river > is represented in the flh file as waterway=stream, in arhi as > waterway=riverbank, in flmd as waterway=stream (but with worse spatial > accuracy than flh), and then in armd as waterway=riverbank! I suppose > one of the waterway=stream's should be changed to waterway=river, and > only one of the waterway=riverbank objects should be used. > > Is this really what the NHD data is like? It will take more time than > I thought to properly merge these these 6 or 7 layers. > > Yes. You should only be using "high" resolution. "High" resolution is already a bit too low for OSM and "medium" is even worse. The "area" dataset consists of polygons that represent rivers wider than a certain cut-off width. Below that width they are represented only as flowlines. Above that width they are represented as flowlines of type "artificial" and a polygon in the "area" dataset. I would suggest reading the NHD data description. It's only a few pages of text if you skip some of the code/type mappings. > If I am confused, then please enlighten me. Preferably by putting it > in the wiki :) > http://wiki.openstreetmap.org/wiki/NHD > ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
I've gone ahead and cleaned up the NHD page a bit. I've also looked at the data, and I must say I'm confused! One river is represented in the flh file as waterway=stream, in arhi as waterway=riverbank, in flmd as waterway=stream (but with worse spatial accuracy than flh), and then in armd as waterway=riverbank! I suppose one of the waterway=stream's should be changed to waterway=river, and only one of the waterway=riverbank objects should be used. Is this really what the NHD data is like? It will take more time than I thought to properly merge these these 6 or 7 layers. If I am confused, then please enlighten me. Preferably by putting it in the wiki :) http://wiki.openstreetmap.org/wiki/NHD -Josh On Thu, Jun 9, 2011 at 2:37 PM, Josh Doe wrote: > Can we improve the documentation on the NHD wiki page in terms of the > import process? I've been wanted to add hydrography to my area > (Virginia, HUC_6 02070) for a while now, but have been trying to > obtain local datasets that presumably could be of higher quality. That > doesn't seem to be panning out, so it's back to NHD. > > It would be great to mention on the wiki that you're doing this, as > not everyone is on the mailing list. For that matter, it would be > great to consolidate all the tips/suggestions on the wiki: > http://wiki.openstreetmap.org/wiki/NHD > > In the test data you provided I saw a number of objects that didn't > have any tags that weren't in the gnis or nhd namespace (i.e no > waterway tag). Is there use in importing these features? Is the > thinking that maybe in the future we can add sensible tags to these? > IMHO we either leave them out or give them sensible tags, even if this > import is the only one that uses them. > > Thanks for the work, looking forward to getting all the waterways in > my area added! > > -Josh > > On Thu, Jun 9, 2011 at 1:27 PM, Ben Supnik wrote: >> Hi Y'all, >> >> I apologize for going NORDO for a while there. :-( I have integrated James' >> latest NHD conversion rules. >> >> Test data: >> >> http://dev.x-plane.com/download/01090001.zip >> >> The entire process is running now...not sure how long it will take to >> complete - it looks like it could take up to a week to convert all of HUCs. >> >> If anyone has feedback on that converted set of XML files, please let me >> know, otherwise I'll let it keep running. >> >> Based on the threads about broken imports that took place over the last few >> weeks, it looks like there is still an additional problem: even given >> pre-converted data, users who want to do a import may get killed when their >> import fails, leaving a splattering of broken nodes all over the place. :-( >> >> cheers >> ben >> >> ___ >> Talk-us mailing list >> Talk-us@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/talk-us >> > ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
Can we improve the documentation on the NHD wiki page in terms of the import process? I've been wanted to add hydrography to my area (Virginia, HUC_6 02070) for a while now, but have been trying to obtain local datasets that presumably could be of higher quality. That doesn't seem to be panning out, so it's back to NHD. It would be great to mention on the wiki that you're doing this, as not everyone is on the mailing list. For that matter, it would be great to consolidate all the tips/suggestions on the wiki: http://wiki.openstreetmap.org/wiki/NHD In the test data you provided I saw a number of objects that didn't have any tags that weren't in the gnis or nhd namespace (i.e no waterway tag). Is there use in importing these features? Is the thinking that maybe in the future we can add sensible tags to these? IMHO we either leave them out or give them sensible tags, even if this import is the only one that uses them. Thanks for the work, looking forward to getting all the waterways in my area added! -Josh On Thu, Jun 9, 2011 at 1:27 PM, Ben Supnik wrote: > Hi Y'all, > > I apologize for going NORDO for a while there. :-( I have integrated James' > latest NHD conversion rules. > > Test data: > > http://dev.x-plane.com/download/01090001.zip > > The entire process is running now...not sure how long it will take to > complete - it looks like it could take up to a week to convert all of HUCs. > > If anyone has feedback on that converted set of XML files, please let me > know, otherwise I'll let it keep running. > > Based on the threads about broken imports that took place over the last few > weeks, it looks like there is still an additional problem: even given > pre-converted data, users who want to do a import may get killed when their > import fails, leaving a splattering of broken nodes all over the place. :-( > > cheers > ben > > ___ > Talk-us mailing list > Talk-us@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk-us > ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On 6/9/2011 1:34 PM, Ian Dees wrote: 3) The upload script should keep track of the response information to see what nodes actually get uploaded. If I remember correctly there's no way to know which elements in a diff upload were successful and which weren't if you receive an error in the reply from the server. This is the source of many blank nodes. The correct solution is to have the upload script query the last changeset on error to see what was actually uploaded and resume from that point. An "error status" on the last upload chunk might not indicate that the data upload failed. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On Thu, Jun 9, 2011 at 12:27 PM, Ben Supnik wrote: > Hi Y'all, > > I apologize for going NORDO for a while there. :-( I have integrated > James' latest NHD conversion rules. > > Test data: > > http://dev.x-plane.com/download/01090001.zip > > The entire process is running now...not sure how long it will take to > complete - it looks like it could take up to a week to convert all of HUCs. > > If anyone has feedback on that converted set of XML files, please let me > know, otherwise I'll let it keep running. > > Based on the threads about broken imports that took place over the last few > weeks, it looks like there is still an additional problem: even given > pre-converted data, users who want to do a import may get killed when their > import fails, leaving a splattering of broken nodes all over the place. :-( > > There isn't a whole lot we can do to outright *prevent* this, but there is some mitigation: 1) The XML converter should split up the data so that ways and their nodes are in the same XML file. I *think* my shp-to-osm tries to do this. If you end up with a bunch of XML files that have 50k node elements in them then you're doing it wrong [0]. 2) Each XML file should contain fewer than the absolute maximum number of changes allowed by the API and should be uploaded using the diff upload API endpoint. 3) The upload script should keep track of the response information to see what nodes actually get uploaded. If I remember correctly there's no way to know which elements in a diff upload were successful and which weren't if you receive an error in the reply from the server. [0] This is how a lot of the original NHD imports got screwed up and left behind thousands of orphaned nodes. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD conversions to OSM
On 6/9/11 1:27 PM, Ben Supnik wrote: Based on the threads about broken imports that took place over the last few weeks, it looks like there is still an additional problem: even given pre-converted data, users who want to do a import may get killed when their import fails, leaving a splattering of broken nodes all over the place. :-( it's a reason why imports like this should only be done by OSMers who are willing to commit to doing the necessary QC and cleanup. richard ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
[Talk-us] NHD conversions to OSM
Hi Y'all, I apologize for going NORDO for a while there. :-( I have integrated James' latest NHD conversion rules. Test data: http://dev.x-plane.com/download/01090001.zip The entire process is running now...not sure how long it will take to complete - it looks like it could take up to a week to convert all of HUCs. If anyone has feedback on that converted set of XML files, please let me know, otherwise I'll let it keep running. Based on the threads about broken imports that took place over the last few weeks, it looks like there is still an additional problem: even given pre-converted data, users who want to do a import may get killed when their import fails, leaving a splattering of broken nodes all over the place. :-( cheers ben ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
