Re: [Talk-us] NHD import and conversion - sample data
> Hi James, > > > I did some more corrections on the rules files and I think that it > > covers all the left over points I saw (including adding > > waterway:stream to one of FCODE for Connectors that wasn't working). > > Just to confirm, these changes are the ones I see on the wiki, right? > Yes > > In terms of untagged ways, if we don't import ComID's and the rest of > > the additional NHD tags. > > But ComIDs are still on the wiki. I can add -t in case there are > untagged nodes/ways, but I don't have a strong opinion re: keeping or > nuking back-references like ComIDs. My opinion is not strong, which is why I didn't nuke them there. I had stopped using them, as the prospect of ever referencing back to the NHD seemed nearly infinitessimal. Also, it would make conversion much easier ;). > > cheers > ben James ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
Hi Guys, On 4/29/11 12:10 PM, Ian Dees wrote: I've been meaning to add a "node dedupe" step to the shp-to-osm suite. I'd love to see what you had to modify to get it to work. I'd also like to add a "way dedupe" at some point, too (to handle abutting areas like boundaries). Well, that code for the CORINE imports is very one-off...I had to make two mods: - Changed the tag delimetor from " to ' - Change the 'offset' in the node record for lat/lon from 3/5 to 5/7. In other words, it's hard code to particular text output, it's not a general XML parser. There is an option (I think it's the "-t" flag?) to shp-to-osm to only include OSM primitives that had tags applied. Oh _that's_ what that does. :-) Hrm...but I'm not seeing untagged data in this latest import. Is there an easy way to find such things in JOSM? It looks to me that, because the ComID is imported as a tag, pretty much everything is going to get a tag, except for ways and nodes that are sub-parts of relations and ways, respectively. cheers ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
On Fri, Apr 29, 2011 at 10:49 AM, Ben Supnik wrote: > > 3. Duplicate nodes need to be merged, especially in the flowline files. >> The >> data currently have duplicate nodes a every point where one reach >> intersects >> another. This creates many duplicate nodes. The Corine data import folks >> wrote a java tool to remove these >> ( >> http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import >> ) >> which I use to remove the duplicate nodes. >> > > With some modification, that seems to have worked. > I've been meaning to add a "node dedupe" step to the shp-to-osm suite. I'd love to see what you had to modify to get it to work. I'd also like to add a "way dedupe" at some point, too (to handle abutting areas like boundaries). > g. In points and lines files there were many things that had no >> osm tags >> including gauging stations and nonearthen shores. We should not import >> things >> that have no osm tags. >> > > Hrm...interesting. The tools don't currently strip untagged data, do they? > Is there an existing tool that strips 'null' data? > > There is an option (I think it's the "-t" flag?) to shp-to-osm to only include OSM primitives that had tags applied. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
Hi Y'all, So first: a new extraact is attached with... - Dupes removed. - GNIS names fixed. - One file per export. http://dev.x-plane.com/download/01090001.zip (Be sure to nuke caches to get the build from today's date...) On 4/29/11 9:51 AM, James Umbanhowar wrote: 2. You should use one file per attribute. If you are using Ian's shp-to-osm tool, crank up the changeset size to some very large value to accomplish this. That seems to work so far...I had to boost heap size to cope with the flow line files. 3. Duplicate nodes need to be merged, especially in the flowline files. The data currently have duplicate nodes a every point where one reach intersects another. This creates many duplicate nodes. The Corine data import folks wrote a java tool to remove these (http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import) which I use to remove the duplicate nodes. With some modification, that seems to have worked. 4. The conversion rules you are using seem to have a lot of issues. I see some of these on the default rules for shp-to-osm wiki page and I have corrected them where I see them, but others I can't identify their source: So first, those rules did come off the wiki. For some reason, capitalization in the shapefiles is...a bit funky. I tried to fix some, but may have missed them. The GNIS names are just me being stupid - I fixed all capitalization issues except for that one. The other issues (specific codes) are due to the limits of the rule set I took. Since the posted sub-basin dump has the original .shp files, would it be simpler for you to simply correct the rules and then send me back the rule sheet you used for inclusion directly in my process? f. It would probably be good to have a discussion of how much of the additional NHD/GNIS tags we want to import. I think we should import the FCODE and That's beyond my pay grade...I'll trust whatever y'all come up with. :-) g. In points and lines files there were many things that had no osm tags including gauging stations and nonearthen shores. We should not import things that have no osm tags. Hrm...interesting. The tools don't currently strip untagged data, do they? Is there an existing tool that strips 'null' data? 5. Once we set this up, we'll need to give a fair bit of guidance on what sorts of things to look for. One thing that is quite tricky is importing coastline. One has to make sure that you delete all the previously existing coastline in the area and to connect the new coastline at the edges of its extent. Given the poor resolution of the PGS and the general disagreement between it and USGS on whether tidal riverbanks are coastline or not this can be quite a bit of work. Right...I would go a step further and say that I'd be terrified of the results of users attempting to use these extracts to create coastline without a lot of experience. In some cases, the cut-over where OSM switches from coastline edge marking to closed water bodies might not be in the same place as NHD. cheers ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
On Thursday 28 April 2011 20:11:39 Ben Supnik wrote: > Hi Y'all, > > This is one sub-basin (01090001 - go Sox :-), isolated from the latest > NHD data, then converted via the latest rule set off of the wiki (with > shapefile key capitalizations fixed). > > http://dev.x-plane.com/download/01090001.zip > > If anyone has done NHD imports before, please take a quick look and let > me know if this looks useful. If so, I can batch the rest of the HUC8s. > > This particular archive contains every layer from the source in both shp > and osm format for comparison. > > cheers > Ben Hi Ben, I've done a fair bit of importing of NHD so I'll have a fair bit of critique given my prior mistakes that I've had to correct. Here are my opinions: 1. The medium resolution is far too coarse. Use high resolution. 2. You should use one file per attribute. If you are using Ian's shp-to-osm tool, crank up the changeset size to some very large value to accomplish this. 3. Duplicate nodes need to be merged, especially in the flowline files. The data currently have duplicate nodes a every point where one reach intersects another. This creates many duplicate nodes. The Corine data import folks wrote a java tool to remove these (http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import) which I use to remove the duplicate nodes. 4. The conversion rules you are using seem to have a lot of issues. I see some of these on the default rules for shp-to-osm wiki page and I have corrected them where I see them, but others I can't identify their source: a I didn't see any names. If there is a GNIS_name this should be the source for the name. b. The 55800 connectors should have a waterway tag. I typically default them to waterway:stream and then change them to waterway:river if their name has a "River" in it. c. 55600 should be tagged natural:coastline d. 36400 is foreshore and should either be additionally tagged as water:tidal or ignored. e. 46000/46006 should be tagged as waterway:riverbank if it is a polygon f. It would probably be good to have a discussion of how much of the additional NHD/GNIS tags we want to import. I think we should import the FCODE and g. In points and lines files there were many things that had no osm tags including gauging stations and nonearthen shores. We should not import things that have no osm tags. 5. Once we set this up, we'll need to give a fair bit of guidance on what sorts of things to look for. One thing that is quite tricky is importing coastline. One has to make sure that you delete all the previously existing coastline in the area and to connect the new coastline at the edges of its extent. Given the poor resolution of the PGS and the general disagreement between it and USGS on whether tidal riverbanks are coastline or not this can be quite a bit of work. Anyway, good work so far! If you have any technical questions, feel free to email me offlist. James ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
Looks decent. I'm surprised they don't have MassGIS's hydrography data ... this NHD is quite low res and offset compared to what they have. The conversion seems OK though. On Thu, Apr 28, 2011 at 7:11 PM, Ben Supnik wrote: > Hi Y'all, > > This is one sub-basin (01090001 - go Sox :-), isolated from the latest NHD > data, then converted via the latest rule set off of the wiki (with shapefile > key capitalizations fixed). > > http://dev.x-plane.com/download/01090001.zip > > If anyone has done NHD imports before, please take a quick look and let me > know if this looks useful. If so, I can batch the rest of the HUC8s. > > This particular archive contains every layer from the source in both shp > and osm format for comparison. > > cheers > Ben > -- > Scenery Home Page: http://scenery.x-plane.com/ > Scenery blog: http://www.x-plane.com/blog/ > Plugin SDK: http://www.xsquawkbox.net/xpsdk/ > X-Plane Wiki: http://wiki.x-plane.com/ > Scenery mailing list: x-plane-scen...@yahoogroups.com > Developer mailing list: x-plane-...@yahoogroups.com > > ___ > Talk-us mailing list > Talk-us@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk-us > ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
