Re: [Talk-us] NHD data extract
On Tue, Apr 26, 2011 at 7:49 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Y'all, The USGS was kind enough to send me the latest shapefile extract for the NHD yesterday. So my question is: is there some way I can process and repost this data that would make it more suitable for importers? There already is a process on the NHD wiki page. I read the wiki pages on NHD imports a few weeks ago, but found the USGS download process to be difficult - there wasn't a simple click here and get the shapefile you need to feed into shp2osm type interface. The process is to download a subbasin, convert to OSM, and import (checking for existing waterways). I could rehost the NHD files on my company's servers for a while, perhaps even in .osm format. But...the data extract is 'the whole database' split only by layer and file size. For example, the water body layer (which contains area lakes and ponds) comes as 5 shapefiles of between 500 and 800 MB each; there isn't any spatial partitioning, so you would need to load the entire data set to pull out lakes of interest for a given bounding box. Converting these into OSM probably won't be very helpful at this point. I could break the data up in a few ways... - It looks like the reach codes contain sub basin IDs as their first 8 digits. - I could break the data into square tiles. - I suppose it could be split by state. - Not all of the data types contain enough information to split like this (last I checked the linear features were the only ones with reach code. Polygonal features don't contain subbasin information) - Square tiles won't line up with existing NHD imports - States wouldn't line up either but may be easier to merge. For the second two cases, the data is not naturally split along those boundaries, so I'd have to either induce a cut (not preferable) or have a few features be not quite where they'd be expected. Anyway, I'm open to ideas...my end goal is to do enough of the GIS processing type work that mappers who are enthusiastic and want to improve their local water areas (but aren't comfortable with command line tools and shapefiles) can grab a pre-converted chunk of data, inspect it, and import some of it. (Side note: I tried to do a test of a importing a multipolygon as a vector background in Potlatch 2, and it failed for both .osm and .shp format data, https://trac.openstreetmap.org/ticket/3701 - has anyone successfully imported water data this way, or should I expect that users will have to use JOSM?) Yes, I wrote my shp-to-osm Java code specifically to handle the multipolygons in NHD and it seems to work fine. I even uploaded some of them in Wisconsin. I stopped because the server was periodically rejecting my imports and would leave thousands of orphaned nodes. This is when I stopped importing NHD. cheers Ben PS If anyone wants the raw data, let me know...I have enough broadband that I could push it to a handful of users if interested. We have NHD from circa last year on the OSMF US servers. It may be useful to update that data. I'll be in contact when we have the space in a week or so. -Ian ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data extract
On Tue, Apr 26, 2011 at 8:20 AM, James Umbanhowar jumba...@unc.edu wrote: On Tuesday 26 April 2011 09:09:21 Ian Dees wrote: On Tue, Apr 26, 2011 at 7:49 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Snip I could rehost the NHD files on my company's servers for a while, perhaps even in .osm format. Snip We have NHD from circa last year on the OSMF US servers. It may be useful to update that data. I'll be in contact when we have the space in a week or so. -Ian Ian, Is it possible to host the .osm files for different subbasins on the OSMF US servers? While the conversion process is reasonably easy, some mappers might be more comfortable taking their local subbasin in osm format and then doing data quality control on that file. Yes, definitely. I've been meaning to do that for quite some time. Can anyone that has uploaded NHD update the wiki pages so I know which files need converting? ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data extract
On Tue, Apr 26, 2011 at 9:09 AM, Ian Dees ian.d...@gmail.com wrote: On Tue, Apr 26, 2011 at 7:49 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Y'all, The USGS was kind enough to send me the latest shapefile extract for the NHD yesterday. So my question is: is there some way I can process and repost this data that would make it more suitable for importers? There already is a process on the NHD wiki page. Also, there are current and archived discussions of imports, etc. on the imports@ list. Do have a look there for some great background. How about this as an alternative or side project? Why not load the complete NHD in a stand-alone rendering system, and render it as a transparent overly tile set? That would then be available for: - testing new NHD rendering styles - comparing NHD to existing rendering - as a helper layer for mappers to trace - etc. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data extract
Hi Y'all, From what I can tell: - Every water body does get a reach ID. I've seen nulls in this file but haven't yet figured out what they are...in local sample areas, all water bodies have reach IDs. - If there is linkeage between areas and flow lines (which I _thought_ there was when reading the docs this morning) I don't see it when browsing the real data. - I can confirm that areas cross sub-basin boundaries. (I downloaded the sub-basin shapefiles to check.) I'm not sure where this leaves us...sub basins are small and manageable but don't partition the data particularly well (in that it looks like we'd need an expensive spatial check for rivers). Users also don't necessarily know their sub-basin code unless they can find an online source to browse or view shapefiles. Richard's idea of building an NHD tile map for tracing seems very do-able, but it wouldn't save a ton of time - every water feature would have to be hand-traced, even though we do have them in vector form already. But I'm not sure that anything other than tile maps provide the level of user simplicity I was hoping for, e.g. being able to find just the part of the data you want to import in a form that's ready for import. cheers Ben On 4/26/11 9:58 AM, James Umbanhowar wrote: On Tuesday 26 April 2011 09:35:00 Ian Dees wrote: On Tue, Apr 26, 2011 at 8:30 AM, Ben Supnikbsup...@xsquawkbox.net wrote: Hi Ian, Where on the OSM servers is the current NHD data, and in what format? I didn't see it referenced in the wiki work-flow. They're currently only used to render TopOSM and aren't available via HTTP or whatnot. I had plans to convert it all to OSM but ran into the problem you're seeing below: - Not all of the data types contain enough information to split like this (last I checked the linear features were the only ones with reach code. Polygonal features don't contain subbasin information) I am looking at this now. Some of the polygonal lakes definitely do contain reach codes, others contain NULL. From what I can tell, there are fcodes that indicate that there is no reach code (or any meta data). For the polygonal area data (which includes large rivers) there are indeed no reach codes. :-( If I remember correctly the area data has some sort of ID which links it to the waterway (which has a reach code). The area data is what wraps a waterway when it has a width greater than some standard too wide for a single line width. Some of the areas, namely large river areas, cross subbasin boundaries so this might be why they cannot be localized to one subbasin. This would be an issue if there were to be one file per subbasin. Either the entire riverbank would need to be assigned to one subbasin or it would need to be split at the subbasin boundaries. This might also be the case for waterbodies. I recall when I imported the coastal NC data there were many wetlands that crossed subbasin boundaries and were duplicated. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data extract
On Tuesday 26 April 2011 12:50:41 Ben Supnik wrote: Hi Y'all, From what I can tell: - Every water body does get a reach ID. I've seen nulls in this file but haven't yet figured out what they are...in local sample areas, all water bodies have reach IDs. - If there is linkeage between areas and flow lines (which I _thought_ there was when reading the docs this morning) I don't see it when browsing the real data. I don't know if there is a database linkage, but the flowlines that are inside an area have a different FCode to indicate that they are connectors. - I can confirm that areas cross sub-basin boundaries. (I downloaded the sub-basin shapefiles to check.) I'm not sure where this leaves us...sub basins are small and manageable but don't partition the data particularly well (in that it looks like we'd need an expensive spatial check for rivers). Users also don't necessarily know their sub-basin code unless they can find an online source to browse or view shapefiles. I actually think that the spatial check for rivers is not that expensive. If someone is going to do a subbasin import well, they need to do a fairly extensive check anyway to look for previously mapped features such as rivers and waterbodies. Double checking to make sure that the riverbanks have not been imported is not difficult. There will typically be only one large riverbank relation per subbasin, maximum. I just think we would need to agree on some process to dole these out to the subbasin files sanely. Per the subbasin code, I think the NHD has a boundary file that could be used to make a layer that would allow mappers to figure out their local code if they couldn't do it via the National Map web interface. Richard's idea of building an NHD tile map for tracing seems very do-able, but it wouldn't save a ton of time - every water feature would have to be hand-traced, even though we do have them in vector form already. But I'm not sure that anything other than tile maps provide the level of user simplicity I was hoping for, e.g. being able to find just the part of the data you want to import in a form that's ready for import. cheers Ben ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data extract
On 4/26/2011 12:50 PM, Ben Supnik wrote: Richard's idea of building an NHD tile map for tracing seems very do-able, but it wouldn't save a ton of time - every water feature would have to be hand-traced, even though we do have them in vector form already. I'm not at all comfortable with the concept of tracing data. (Tracing aerial imagery makes sense) How is tracing data thought to be any better than direct import? That somehow slowing the process down will add Soul to the process? Should we all plan to run around with Transits, and digging up survey markers so that we don't have to rely on those Soulless GPSs? And that way we can confirm administrative boundaries while we're at it. !? I don't follow everything being explored in the NHD import, but Ben and others are asking the exact right questions and heading in the right direction in my opinion. The end result should be chunks of data small enough to allow local mappers to import and confirm a small area at a time. An expensive spatial check may be necessary, not only to be able to properly divide the import into regions, but to properly interface with data that has already been uploaded without duplication. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
