Re: [Talk-us] NHD imports
Hi Y'all, First, I'm sorry that I didn't jump into this thread earlier. I am the graphics lead on X-Plane, and for our latest version we switched from TIGER+VMAP0+swbd to OSM for our vector water body and road data. In the case of the US, this is perceived as a step backward by our users because TIGER, while not as accurate as a lot of the OSM water data, is reasonably complete. Now that we use OSM, we are missing lakes that were present in older sim versions. At the time the consensus on various lists about importing was that individual authors should import relatively small areas (e.g. single water sheds) and check the import against existing data, rather than simply blasting in the entire data set. So I wanted to solve two problems to make things easier for users who had time and motivation but not data processing skills: 1. Getting NHD data. At the time the NHD data was not available in shape-file format; some very nice person at the USGS basically burned me DVDs of the whole data set and mailed them to me, because the data was not online. 2. The conversion required running a non-GUI tool, which meant having command line skills, etc. So I ran the import script over the entire data set and uploaded the resulting .osm files so individuals could use them. There was a third step I was hoping could be solved: if you new to OSM, importing data isn't trivial. Richard Fairhurst had a cool feature in Potlatch 2 where PL2 could be told of the presence of a vector layer and import it visually. But I was never able to get the data to work with PL2. I had to put the whole project on hold as X-Plane's ship date loomed closer and things went from crazy to really-freaking-crazy at work. So to answer a few of Paul's questions: - They're of 2011 data, not the latest NHD data Yep -- at the time I started the project it was a rip of 'latest', but that was a while ago. :-) - I wasn't able to find a rules file from bsupnik for the conversion, so they can't be rerun with latest data. He hasn't been active on the lists since 2011 and hasn't replied I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java nerd, sorry) and the latest rules at the time from the Wiki. If it is useful, I can send anyone who wants them a rip of the rules file I was using at the time and/or the JAR, bt... - The NHD data model changed since 2011 Yeah, I was something like that a few months after I put this down and went oh hell I haven't had time to look at what happened; is the data model a major revision to the point where everything I did makes no sense, or is it just renaming of fields? Other issues about the import: - I wouldn't say it is split from multiple files - rather I would say it is not joined - that is, the splits are the sub-basin splits that I think persist in the NHD. Actually, I take that back - I think the tool chain intentionally avoids really huge OSM entities to avoid imports that can fail. - There should be OSM codes like waterway=bla on at least -some- of the data. The import program rules from the time did include FCODE and FTYPE and had no facility to drop entities that had only NHD but no OSM tags. At the time this was also considered not-so-good, but I wasn't in a position to rebuild the tool. Finally, just my 0.02: I am not an OSM expert and not an import expert buuut in my uninformed opinion, having original 'foreign' keys on an imported database isn't very useful. The import guideline is to sanely merge the import with existing data, and the goal is to create something 'osm native' when done, hence the importance of OSM tags and the notion of not importing data with no OSM meaning. I think OSM really needs to be treated as one giant layer with heterogenious tags. Imagine that a lake is missing from NHD. A user opens potlatch and draws it, puts water=lake, and walks away, no FCODE or GNIS_ID or anything like that. How can a consumer of OSM data handle this case? Only by looking at semantic data like water=lake, and by analyzing the union of all OSM data that has 'interesting' tags. Now we go to re-import NHD - is this ever going to happen? (First: given how hard it's been to get NHD in even once, I am skeptical. :-) Every watershed has to be hand re-merged again. Where I'm going with this is: I think imports should not be viewed as 'layers' into OSM; if you ever want to look at NHD with a layer, export OSM into your favorite GIS, import NHD into your GIS too and now you have layers. OSM imports are destructive merges and need to be carefully managed as such, with the target data format being OSM-centric. So at the time I was annoyed that we couldn't just import all of NHD at once and 'get lakes everywhere' but while that would get a lot of waterbodies into empty places, it's probably not good for OSM or the US mapping community. Okay, that's the end of my rant, which is probably
Re: [Talk-us] NHD imports
Hi everybody, as an active mapper focused on the topology consistence of the map, I don't think that any automatic re-import would make any good. The initial bulk imports left the map in the terrible state and I can't even imagine how badly it would be damaged if all the NHD data is re-imported. I think it might be nice to have a simple, even text-like (Russians mappers have something similarhttp://translate.google.com/translate?hl=ensl=rutl=enu=http%3A%2F%2Fvwo.osm.rambler.ru%2F), web interface to NHD data so mappers would be able to check water objects manually and make a decision on re-importing and validating them. Best regards, Ivan. On Mon, Oct 29, 2012 at 9:04 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Y'all, First, I'm sorry that I didn't jump into this thread earlier. I am the graphics lead on X-Plane, and for our latest version we switched from TIGER+VMAP0+swbd to OSM for our vector water body and road data. In the case of the US, this is perceived as a step backward by our users because TIGER, while not as accurate as a lot of the OSM water data, is reasonably complete. Now that we use OSM, we are missing lakes that were present in older sim versions. At the time the consensus on various lists about importing was that individual authors should import relatively small areas (e.g. single water sheds) and check the import against existing data, rather than simply blasting in the entire data set. So I wanted to solve two problems to make things easier for users who had time and motivation but not data processing skills: 1. Getting NHD data. At the time the NHD data was not available in shape-file format; some very nice person at the USGS basically burned me DVDs of the whole data set and mailed them to me, because the data was not online. 2. The conversion required running a non-GUI tool, which meant having command line skills, etc. So I ran the import script over the entire data set and uploaded the resulting .osm files so individuals could use them. There was a third step I was hoping could be solved: if you new to OSM, importing data isn't trivial. Richard Fairhurst had a cool feature in Potlatch 2 where PL2 could be told of the presence of a vector layer and import it visually. But I was never able to get the data to work with PL2. I had to put the whole project on hold as X-Plane's ship date loomed closer and things went from crazy to really-freaking-crazy at work. So to answer a few of Paul's questions: - They're of 2011 data, not the latest NHD data Yep -- at the time I started the project it was a rip of 'latest', but that was a while ago. :-) - I wasn't able to find a rules file from bsupnik for the conversion, so they can't be rerun with latest data. He hasn't been active on the lists since 2011 and hasn't replied I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java nerd, sorry) and the latest rules at the time from the Wiki. If it is useful, I can send anyone who wants them a rip of the rules file I was using at the time and/or the JAR, bt... - The NHD data model changed since 2011 Yeah, I was something like that a few months after I put this down and went oh hell I haven't had time to look at what happened; is the data model a major revision to the point where everything I did makes no sense, or is it just renaming of fields? Other issues about the import: - I wouldn't say it is split from multiple files - rather I would say it is not joined - that is, the splits are the sub-basin splits that I think persist in the NHD. Actually, I take that back - I think the tool chain intentionally avoids really huge OSM entities to avoid imports that can fail. - There should be OSM codes like waterway=bla on at least -some- of the data. The import program rules from the time did include FCODE and FTYPE and had no facility to drop entities that had only NHD but no OSM tags. At the time this was also considered not-so-good, but I wasn't in a position to rebuild the tool. Finally, just my 0.02: I am not an OSM expert and not an import expert buuut in my uninformed opinion, having original 'foreign' keys on an imported database isn't very useful. The import guideline is to sanely merge the import with existing data, and the goal is to create something 'osm native' when done, hence the importance of OSM tags and the notion of not importing data with no OSM meaning. I think OSM really needs to be treated as one giant layer with heterogenious tags. Imagine that a lake is missing from NHD. A user opens potlatch and draws it, puts water=lake, and walks away, no FCODE or GNIS_ID or anything like that. How can a consumer of OSM data handle this case? Only by looking at semantic data like water=lake, and by analyzing the union of all OSM data that has 'interesting' tags. Now we go to re-import NHD - is this ever going to happen? (First: given how hard it's been to
Re: [Talk-us] NHD imports
This is only a partial reply - I should have more detail this afternoon when I have more time. From: Ben Supnik [mailto:bsup...@xsquawkbox.net] Subject: Re: [Talk-us] NHD imports 2. The conversion required running a non-GUI tool, which meant having command line skills, etc. This is actually a plus for me - handling all of NHD without command line scripts would be difficult. There's a *lot* of data. So I ran the import script over the entire data set and uploaded the resulting .osm files so individuals could use them. There was a third step I was hoping could be solved: if you new to OSM, importing data isn't trivial. Richard Fairhurst had a cool feature in Potlatch 2 where PL2 could be told of the presence of a vector layer and import it visually. But I was never able to get the data to work with PL2. This is actually mostly solved now - http://www.cyclestreets.net/edit/locate/ has a PL2 instance which has a vector layer background for all of the UK. There are a few issues like loading many GB of data into the server supplying the vector layer and making it available without deploying your own PL2 instance, but these are fairly minor. I used shp-to-osm-0.8.2, jar'd by someone else (I am not a big java nerd, sorry) and the latest rules at the time from the Wiki. If it is useful, I can send anyone who wants them a rip of the rules file I was using at the time and/or the JAR, bt... Ah, good to know. - The NHD data model changed since 2011 Yeah, I was something like that a few months after I put this down and went oh hell I haven't had time to look at what happened; is the data model a major revision to the point where everything I did makes no sense, or is it just renaming of fields? The common FCodes have remained the same but some of the less common ones have changed. Other issues about the import: - I wouldn't say it is split from multiple files - rather I would say it is not joined - that is, the splits are the sub-basin splits that I think persist in the NHD. Actually, I take that back - I think the tool chain intentionally avoids really huge OSM entities to avoid imports that can fail. The splits I'm talking about are being split into layers. You have lakes on one layer and rivers on another, etc. The splits by area remain similar although I believe the pre-staged files cover a larger area (HUC4 basins) than what you were given did. - There should be OSM codes like waterway=bla on at least -some- of the data. The import program rules from the time did include FCODE and FTYPE and had no facility to drop entities that had only NHD but no OSM tags. At the time this was also considered not-so-good, but I wasn't in a position to rebuild the tool. Maybe I picked an odd region or layer. There are some areas in the NHD that are quite different from all the others and use FCodes in different ways. I should know the dangers of concluding something by looking only at one part of NHD :) So at the time I was annoyed that we couldn't just import all of NHD at once and 'get lakes everywhere' but while that would get a lot of waterbodies into empty places, it's probably not good for OSM or the US mapping community. That's my view too. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD imports
Hi Paul, On 10/29/12 11:47 AM, Paul Norman wrote: 2. The conversion required running a non-GUI tool, which meant having command line skills, etc. This is actually a plus for me - handling all of NHD without command line scripts would be difficult. There's a *lot* of data. Right - for anyone working on the _entire_ NHD data set, you really need automation, command-line, and to be comfortable with bulk processing. My original goal was to do this on the behalf of new mappers, while leaving the 'edit and review' process to everyone else, since it has to be done by hand. This is actually mostly solved now - http://www.cyclestreets.net/edit/locate/ has a PL2 instance which has a vector layer background for all of the UK. There are a few issues like loading many GB of data into the server supplying the vector layer and making it available without deploying your own PL2 instance, but these are fairly minor. Cool -- my thinking was that the closer we get the NHD data to being 'click ready' the more people can help to get it reviewed and into OSM. In particular, I was looking to find an answer to my user's question how can I help get the lakes back near where I live, one that didn't require import skills. If I can say load OSM on this potlatch and import this pre-existing layer if the lake is missing with some editorial guidelines, that's good for a general audience. The splits I'm talking about are being split into layers. You have lakes on one layer and rivers on another, etc. The splits by area remain similar although I believe the pre-staged files cover a larger area (HUC4 basins) than what you were given did. Ah - yeah, that's all 'what I got' - HUC8 split by a bunch of layers based on topology. Since I requested shapefiles, that may have been inevitable. :-) Maybe I picked an odd region or layer. There are some areas in the NHD that are quite different from all the others and use FCodes in different ways. I should know the dangers of concluding something by looking only at one part of NHD :) If you were looking at an area that should have had sane tagging and it did not contain sane OSM codes, let me know the HUC8 code or something and I can at least look at the original shape files in QGIS and see if they were majorly borked. I did look at 2 or 3 HUC8s to make sure they were sane and they seemed plausibly full of tags... cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD imports
In June, user Bsupnik converted the entire NHD dataset into OSM format. The files are available at http://bsupnik.dev.openstreetmap.org/NHD. He mentioned that he was working on creating better quality files. The files that he created look good. They are missing the names though. It looks like they just need a couple minor tweaks because the files generally looked good. Just wondering if any progress has been made. These would be ideal once the bugs are worked out so people could review and upload the data in their area. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD imports
From: Nathan Mixter [mailto:nmix...@gmail.com] Sent: Sunday, October 28, 2012 1:39 PM To: talk-us Subject: Re: [Talk-us] NHD imports In June, user Bsupnik converted the entire NHD dataset into OSM format. The files are available at http://bsupnik.dev.openstreetmap.org/NHD. He mentioned that he was working on creating better quality files. The files that he created look good. They are missing the names though. It looks like they just need a couple minor tweaks because the files generally looked good. Just wondering if any progress has been made. These would be ideal once the bugs are worked out so people could review and upload the data in their area. I downloaded a random area (17010104) and looked at these. There were no real OSM tags on the ways, only gnis:fcode, gnis:ftype, nhd:com_id and source. I'd question anyone proposing an import with both fcode and ftype because ftype can be inferred from fcode. Also, it suffers from an inherent limitation by splitting it up among multiple files. This method loses the connections between the layers, resulting in duplicate nodes and the topology not being connected. I looked at a separate-layer approach when I first started looking at NHD and with how NHD is structured with many interdependent layers it doesn't make sense to do it that way. You have to merge all the layers together as a multi-layer file and then convert to OSM. I'm not sure if shp-to-osm can handle multi-layer sources. A couple of other specific issues with these conversions: - They're of 2011 data, not the latest NHD data - I wasn't able to find a rules file from bsupnik for the conversion, so they can't be rerun with latest data. He hasn't been active on the lists since 2011 and hasn't replied - The NHD data model changed since 2011 If anyone is aware of where the file with the rules bsupnik used resides, please let me know. It would really help what I'm working on. Also, needless to say, someone who wanted to import a basin would need to read and follow http://wiki.openstreetmap.org/wiki/Import/Guidelines ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
Past NHD imports have made vast multipolygons which can be difficult to interpret without a view of the whole thing. This made particular problems for tiles@home/osmarender, which tried to render the multipolygons without loading their out-of-area members, leading to water-land inversion in a lot of places. Editing these multipolygons may be error-prone too, though need for such editing should be rare. If you can somehow limit the size of the multipolygons, this issue can probably be mitigated. On Oct 27, 2012 7:29 PM, Clifford Snow cliff...@snowandsnow.us wrote: I saw bsupnik's wiki page, http://wiki.openstreetmap.org/wiki/User:Bsupnik, on importing of NHD into OSM. I'm working on National Parks in Washington State. After spending countless hours tracing in streams and rivers into OSM, I've finally decided that importing makes more sense. I'm wondering what the consensus was on using NHD in OSM? The data can be downloaded as shapefiles from the USGS website. With Paul Norman's ogr2osm script and translation tables, the data looks like to could be imported into OSM. In the areas I'm focusing on, it would need to be done almost stream by stream since some of the data already exists. And doing it in small batches allows for easy alignment with a Bing image. (These are areas that a survey is nearly impossible due to the large amount of difficult terrain involved.) One of the advantages to using the NHD data over topo maps is getting the correct names on the streams. The existing topo maps make it difficult to determine named streams from un-named ones. Before I bring this up on the imports list, I thought I'd ask the US community about their opinion about importing the data. -- Clifford ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
On 10/27/2012 7:28 PM, Clifford Snow wrote: Before I bring this up on the imports list, I thought I'd ask the US community about their opinion about importing the data. I agree that it is a good idea to import NHD. It is an asset when doing surveys and updates: you can review it for any changes and make corrections. - Use the latest tools and techniques to create the NHD dataset. - Review tag assignments for the area of interest. - Merge it into the existing ways - preserve existing hydro work when possible. For water features that you created yourself, you can decide whether to replace with NHD data. On the other hand, I believe NHD should not be imported into US OSM Edit deserts. It's best to wait until there is a community who wants an NHD import. In the meantime, the NHD for that area may be updated and when it is finally imported, it will be as fresh as possible. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
On Sat, Oct 27, 2012 at 5:13 PM, David ``Smith'' vidthe...@gmail.comwrote: Past NHD imports have made vast multipolygons which can be difficult to interpret without a view of the whole thing. This made particular problems for tiles@home/osmarender, which tried to render the multipolygons without loading their out-of-area members, leading to water-land inversion in a lot of places. Editing these multipolygons may be error-prone too, though need for such editing should be rare. If you can somehow limit the size of the multipolygons, this issue can probably be mitigated. Right now I'm only looking to import streams (creeks and rivers) and possibly small lakes into OSM. I did a small sample of lakes and found them to be only useful for 1) identifying a lake and 2) getting the name. The area I'm working with has hundreds of small un-named lakes and a few small named lakes. It's easy to miss a lake when looking at a bing image so the data identifies the water. However, the polygon shape from NHD is nothing like the actual lake on the bing image. I guess the quick answer is I don't plan to import multipolygons. -- Clifford ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
On Sat, Oct 27, 2012 at 5:15 PM, Mike N nice...@att.net wrote: On 10/27/2012 7:28 PM, Clifford Snow wrote: Before I bring this up on the imports list, I thought I'd ask the US community about their opinion about importing the data. I agree that it is a good idea to import NHD. It is an asset when doing surveys and updates: you can review it for any changes and make corrections. Most of the area's I'm mapping are too remote and not likely to ever be surveyed in person. - Use the latest tools and techniques to create the NHD dataset. - Review tag assignments for the area of interest. - Merge it into the existing ways - preserve existing hydro work when possible. For water features that you created yourself, you can decide whether to replace with NHD data. Agreed. Usually if there is existing data, it's data someone has manually added and is often better than NHD. That's way a bulk import isn't practical. On the other hand, I believe NHD should not be imported into US OSM Edit deserts. It's best to wait until there is a community who wants an NHD import. In the meantime, the NHD for that area may be updated and when it is finally imported, it will be as fresh as possible. Please help me understand what you mean by OSM Edit deserts. The areas I'm looking at are mostly untouched except for the original TIGER and GNIS imports. Are you suggesting not importing into those areas? -- Clifford ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
On 10/27/2012 8:35 PM, Clifford Snow wrote: On the other hand, I believe NHD should not be imported into US OSM Edit deserts. It's best to wait until there is a community who wants an NHD import. In the meantime, the NHD for that area may be updated and when it is finally imported, it will be as fresh as possible. Please help me understand what you mean by OSM Edit deserts. The areas I'm looking at are mostly untouched except for the original TIGER and GNIS imports. Are you suggesting not importing into those areas? Another way I would express it is not to mass import for a whole state or multi-state area where there is no active community. I know of several such areas that span multiple counties and many thousands of square miles. The only objects that have been touched are Interstate and US highways. For your case, since you are comparing it against existing data and Bing imagery and possible consultation with Topo maps, it is entirely appropriate to use NHD data. In effect, because of your interest, you are the active mapper in the area, even though it is not all ground surveys. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
Mike N writes: For your case, since you are comparing it against existing data and Bing imagery and possible consultation with Topo maps, it is entirely appropriate to use NHD data. In effect, because of your interest, you are the active mapper in the area, even though it is not all ground surveys. I would like to have every stream in the U.S. available as a .osm file. So, say, I am running along some road or railroad, and I see a stream that doesn't exist in OSM yet. Rather than doing the clicky-click myself, it would be great if I could load up the NHD version of that stream. I could compare it against the aerial photos, and the data already in OSM, and decide for myself if I want to use that or do the clicky click. -- --my blog is athttp://blog.russnelson.com Crynwr supports open source software 521 Pleasant Valley Rd. | +1 315-600-8815 Potsdam, NY 13676-3213 | Sheepdog ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
On Sat, Oct 27, 2012 at 6:19 PM, Russ Nelson nel...@crynwr.com wrote: Mike N writes: For your case, since you are comparing it against existing data and Bing imagery and possible consultation with Topo maps, it is entirely appropriate to use NHD data. In effect, because of your interest, you are the active mapper in the area, even though it is not all ground surveys. I would like to have every stream in the U.S. available as a .osm file. So, say, I am running along some road or railroad, and I see a stream that doesn't exist in OSM yet. Rather than doing the clicky-click myself, it would be great if I could load up the NHD version of that stream. I could compare it against the aerial photos, and the data already in OSM, and decide for myself if I want to use that or do the clicky click. Having the NHD as a layer like the TIGER data would be nice. -- Clifford ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD Imports
From: Russ Nelson [mailto:nel...@crynwr.com] Sent: Saturday, October 27, 2012 6:19 PM To: OSM US Talk List Subject: Re: [Talk-us] NHD Imports I would like to have every stream in the U.S. available as a .osm file. So, say, I am running along some road or railroad, and I see a stream that doesn't exist in OSM yet. Rather than doing the clicky-click myself, it would be great if I could load up the NHD version of that stream. I could compare it against the aerial photos, and the data already in OSM, and decide for myself if I want to use that or do the clicky click. For what it's worth, I'm working on this, but it's hard work and takes time to do it right. I'd be open to coordinating with someone else, but they'd need some basic python knowledge, the ability to compile gdal to read .mdb, the ability to use git to work with others, and the experience required to develop translations. Also, some of the other software pieces needed aren't there yet. NHD is massive and I have no idea how some parts of the toolchain will work when I start throwing hundreds of gigs of data at them. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
