[Talk-us] NHD import

[Martijn van Exel]

Hi,

Remapping in CA, I come across some weird stuff.
Here's some NHD 'data':
http://www.openstreetmap.org/?lat=35.17764&lon=-119.12641&zoom=16
Either the aerial imagery is way off here, or this is just bad data.
If it is, I presume that there has been a review of this data before 
import, this is the exception, and the vast majority of imported NHD 
objects actually do represent reality. I hope.

--
Martijn van Exel

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Clifford Snow]

On Sun, Apr 8, 2012 at 10:39 PM, Martijn van Exel  wrote:

> Hi,
>
> Remapping in CA, I come across some weird stuff.
> Here's some NHD 'data':
> http://www.openstreetmap.org/?**lat=35.17764&lon=-119.12641&**zoom=16
> Either the aerial imagery is way off here, or this is just bad data.
> If it is, I presume that there has been a review of this data before
> import, this is the exception, and the vast majority of imported NHD
> objects actually do represent reality. I hope.
> --
> Martijn van Exel
>

Are you talking about the water - lakes and ponds?  From reading  nmixter's
diary, he/she has posted comments about mapping farms.  One comment
suggested taking the import to the talk-us mailing list. BTW - I did just
drive through some farm land in Western Washington.  Farmers had dug
temporary canals to help drain (or so I assumed) the water from the field
so they could plant.  I probably wouldn't map it unless they were
permanent.

-- 
Clifford
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Martijn van Exel]

On 4/9/2012 12:00 AM, Clifford Snow wrote:



On Sun, Apr 8, 2012 at 10:39 PM, Martijn van Exel mailto:mve...@gmail.com>> wrote:

Hi,

Remapping in CA, I come across some weird stuff.
Here's some NHD 'data':
http://www.openstreetmap.org/?__lat=35.17764&lon=-119.12641&__zoom=16 

Either the aerial imagery is way off here, or this is just bad data.
If it is, I presume that there has been a review of this data before
import, this is the exception, and the vast majority of imported NHD
objects actually do represent reality. I hope.
--
Martijn van Exel


Are you talking about the water - lakes and ponds?  From reading
nmixter's diary, he/she has posted comments about mapping farms.  One
comment suggested taking the import to the talk-us mailing list. BTW - I
did just drive through some farm land in Western Washington.  Farmers
had dug temporary canals to help drain (or so I assumed) the water from
the field so they could plant.  I probably wouldn't map it unless they
were permanent.



Yes, I see a lot of water features that are just not corroborated by the 
aerial imagery, which could mean one of at least three things:

1) The aerial imagery is out of date
2) The NHD data is out of date
3) The NHD data represents something I don't understand (the future, a 
temporary situation (which should not be in OSM), something underground?)


--
Martijn van Exel

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Gregory Arenius]

Yes, I see a lot of water features that are just not corroborated by the
> aerial imagery, which could mean one of at least three things:
> 1) The aerial imagery is out of date
> 2) The NHD data is out of date
> 3) The NHD data represents something I don't understand (the future, a
> temporary situation (which should not be in OSM), something underground?)
>
>
Some of the water features in NHD are also seasonal, although that is
usually tagged in the data.   Also irrigation canals are often just ditches
and can be hard to identify from aerial imagery especially the smaller ones
as they aren't always in use.

The tags on a lot of those features have some gnis:type tags that say
ditch-canal or something like that but the OSM tag is just canal which
doesn't really do a great job describing the situation exactly.

I agree, though, the data you pointed out looks pretty odd, especially the
square shapes.

I'm surprised that NHD has data that includes irrigation ditches as small
as some of the ones noted above.  Anybody know how they gathered all of
that data?

Greg
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Toby Murray]

On Mon, Apr 9, 2012 at 1:25 AM, Gregory Arenius  wrote:
>
>
>
>> Yes, I see a lot of water features that are just not corroborated by the
>> aerial imagery, which could mean one of at least three things:
>> 1) The aerial imagery is out of date
>> 2) The NHD data is out of date
>> 3) The NHD data represents something I don't understand (the future, a
>> temporary situation (which should not be in OSM), something underground?)
>>
>
> Some of the water features in NHD are also seasonal, although that is
> usually tagged in the data.   Also irrigation canals are often just ditches
> and can be hard to identify from aerial imagery especially the smaller ones
> as they aren't always in use.
>
> The tags on a lot of those features have some gnis:type tags that say
> ditch-canal or something like that but the OSM tag is just canal which
> doesn't really do a great job describing the situation exactly.
>
> I agree, though, the data you pointed out looks pretty odd, especially the
> square shapes.
>
> I'm surprised that NHD has data that includes irrigation ditches as small as
> some of the ones noted above.  Anybody know how they gathered all of that
> data?
>
> Greg

I saw some odd water features along the coast in California as well
when I was remapping coastlines by importing some NHD data. I don't
have a link handy but I think it came from a California specific data
source and some of the ways didn't have any OSM renderable tags.

Yay for imports?

Toby

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Mike N]

On 4/9/2012 1:39 AM, Martijn van Exel wrote:

Here's some NHD 'data':


  Perhaps rice fields, based on a Google Street View, but I can't see 
for sure.


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Randal Hale]

I had to teach a class on Friday and it involved NHD Data.

NHD data is supposed to be an ever evolving dataset. The beginning's of 
it are 1:24k USGS Topographic Maps. As time goes on and Lidar 
becomes more prevalent the dataset 
will improve. Tennessee is slated to get a high resolution dataset of 
NHD data collected form photogrametrically acquired data in the next few 
years.


Because NHD data is based of 1:24k quad sheets (and in come cases USFWS 
Wetland Maps) it's dated - in Chattanooga it's probably 40 to 50 years 
old. Streams change. Ponds disappear. Things become channelized. If you 
compare it to the NAIP or Bing Aerial Imagery in some cases it's 
remarkably close and in some it so far off you wonder what happened.


There is also a second glitch with the data - since NHD is based off the 
1:24k topo maps it's not entirely accurate. The USGS changed their 
definitions of what consisted of a blue line stream from "it's a drain" 
to "it's got water in it". It didn't affect Lakes/Rivers so much but the 
blue line streams are questionable unless they are viewed with Aerial 
Photography (and in my opinion need to be viewed in Stereo or site visit 
to see what is occurring with it).


I say all of that - it's better than having nothing. At least here we 
can improve it.


Randy

Randal Hale, GISP
North River Geographic Systems, Inc
http://www.northrivergeographic.com
423.653.3611 rjh...@northrivergeographic.com
twitter:rjhale
http://about.me/rjhale


On 4/9/2012 2:11 AM, Martijn van Exel wrote:

On 4/9/2012 12:00 AM, Clifford Snow wrote:



On Sun, Apr 8, 2012 at 10:39 PM, Martijn van Exel mailto:mve...@gmail.com>> wrote:

Hi,

Remapping in CA, I come across some weird stuff.
Here's some NHD 'data':

http://www.openstreetmap.org/?__lat=35.17764&lon=-119.12641&__zoom=16 


Either the aerial imagery is way off here, or this is just bad data.
If it is, I presume that there has been a review of this data before
import, this is the exception, and the vast majority of imported NHD
objects actually do represent reality. I hope.
--
Martijn van Exel


Are you talking about the water - lakes and ponds?  From reading
nmixter's diary, he/she has posted comments about mapping farms.  One
comment suggested taking the import to the talk-us mailing list. BTW - I
did just drive through some farm land in Western Washington.  Farmers
had dug temporary canals to help drain (or so I assumed) the water from
the field so they could plant.  I probably wouldn't map it unless they
were permanent.



Yes, I see a lot of water features that are just not corroborated by 
the aerial imagery, which could mean one of at least three things:

1) The aerial imagery is out of date
2) The NHD data is out of date
3) The NHD data represents something I don't understand (the future, a 
temporary situation (which should not be in OSM), something underground?)


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[Brett Lord-Casitllo]

Most recent USGS topos show that area as water features.
NAIP 2012 shows water there; looks like wetlands restoration maybe?
--Brett
Brett Lord-Castillo
Information Systems Designer/GIS Programmer
St. Louis County Police
Office of Emergency Management
14847 Ladue Bluffs Crossing Drive
Chesterfield, MO 63017
Office: 314-628-5400    Fax: 314-628-5508
Direct: 314-628-5407    ___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import

[TC Haddad]

If it were coastal Oregon or SW Washington, there would be a good chance
that those were Cranberry farms (bogs).

Tanya

On Mon, Apr 9, 2012 at 6:28 AM, Brett Lord-Casitllo wrote:

Most recent USGS topos show that area as water features.
> NAIP 2012 shows water there; looks like wetlands restoration maybe?
> --Brett
> Brett Lord-Castillo
> Information Systems Designer/GIS Programmer
> St. Louis County Police
> Office of Emergency Management
> 14847 Ladue Bluffs Crossing Drive
> Chesterfield, MO 63017
> Office: 314-628-5400Fax: 314-628-5508
> Direct: 314-628-5407
>
>
> ___
> Talk-us mailing list
> Talk-us@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-us
>
>
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

[Talk-us] NHD Import tagging - ForeShore

[Mike N.]

I have begun a coastal area import.  I noticed that my data has a new FCODE 
type not contained in inland areas - ForeShore (36400).The Wiki suggests 
it be mapped to water=tidal

This type was not mapped in the conversion script  polyshp2osm-Area.py. 
Although it is still a proposed type, I think it makes sense to map it to a 
type rather than bring it in untagged and add something later.

http://wiki.openstreetmap.org/wiki/Proposed_features/Water_cover

   Let me know if there might be a better solution.

  Thanks,

   Mike
 


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

[Talk-us] NHD import and conversion - sample data

[Ben Supnik]

Hi Y'all,

This is one sub-basin (01090001 - go Sox :-), isolated from the latest 
NHD data, then converted via the latest rule set off of the wiki (with 
shapefile key capitalizations fixed).


http://dev.x-plane.com/download/01090001.zip

If anyone has done NHD imports before, please take a quick look and let 
me know if this looks useful.  If so, I can batch the rest of the HUC8s.


This particular archive contains every layer from the source in both shp 
and osm format for comparison.


cheers
Ben
--
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://www.x-plane.com/blog/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scen...@yahoogroups.com
Developer mailing list: x-plane-...@yahoogroups.com

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

[Talk-us] NHD import: what data quality is acceptable?

[Kevin Kenny]

A few months ago, I tried to get started on trying to resume the NHD
import in my area - and some of the places where I hike. I'm trying
to check results with both P2 and JOSM, and tripping over a lot of
things, which made me put the project back on hold for a while. (I
had some other things to be about.)

But now I'm starting to reconsider again, after hearing that there are
others out there thinking the NHD import is desirable for an "Open Trail
Map." I mentioned this in a message earlier today.

So, let me review some of the things that have me scratching my head.

(1) The mapping from NHD feature codes to OSM tags is incomplete (and
not quite consistent). That's fine; for all the FCodes in my area, I've
been able to find features that I'm familiar with that I can describe.
So I think I have that problem fairly well licked. (I may have to invent
a tag or two, like "waterway=rise" and "waterway=sink" to describe
streams that go underground and appear again in karst terrain.)

(2) One historic complaint I've seen about the NHD import is that it
clutters the map, and that the assignment of "river" and "stream" is
difficult.  What I'd propose is to tag as "river" anything that has
more than two nodes of its flowline as "Artificial Path", and "stream"
anything that's smaller.  In my area (eastern New York), for instance,
that would mark out the Schoharie Creek, the Mohawk, the Hudson, and
the lower reaches of the Catskill, the Kaaterskill, the Esopus, and
maybe a few others. Is this a reasonable approach?

(3) Is it necessary to tag shorelines with name=? It would be something
of an effort to identify what flowline belongs to a shoreline, and it's
somewhat ambiguous near confluences. I'm inclined not to do anything
about this issue unless there's an overwhelming consensus that it
needs to be addressed.

(4) What about cadastral lines (administrative boundaries, and land
use, leisure, etc.) that appear to follow flowlines? My inclination is
not to touch these at all, even if the flowline is being refined.
Oftentimes, when a stream changes course, the cadastre remains
unchanged. Only the recorder's office in the jurisdiction in question
would actually know the situation. I think I'd rather see slightly
inconsistent boundaries than mess up something like that.

(5) In the area I have in mind, there are very few ways that actually
would need to be conflated (a few major rivers). Is it likely to be
called vandalism if I confine myself to copying the OSM-specific tags
from the OSM ways onto the NHD ones? I trust a land survey rather
more than I do someone tracing a shoreline from Bing imagery: in
well-graded streams, the shorelines can be variable and ambiguous,
and NHD has pretty sound information about high water marks.

(6) When I try a limited import, I get a lot of JOSM warnings about
waterways crossing highways. Do people think that all of these have
to be fixed before importing? In that case, I'll have to confine myself
to the very small area that has highway-crossing-stream that I can
visit myself (to try to settle what the boundaries of the bridge,
dam or culvert are, and what type of waterwork it is). Is it considered
acceptable simply to leave the crossings unmarked? (They still
render correctly in Mapnik, for what it's worth.)

(7) In the event that I find a node tagged with something other than
a waterway (or landuse=reservoir, man_made=dam, etc.) that collides
with an NHD node, is the correct approach to conflate the nodes,
introduce a duplicate, or offset the new node? Does this question
even have a correct answer?

I'm trying seriously to "do no harm", and getting paralyzed by the
fact that there seem to be no firm guidelines on what is acceptable.
I'm trying to start with areas where I have firm local knowledge,
although clearly I cannot survey streams on private lands, so I
have to depend on NHD there. But the result is not going to be of
much use to me unless I can generalize what I learn to do better
NHD imports in areas that I know less well.
--
73 de ke9tv/2, Kevin

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[Ian Dees]

Looks decent. I'm surprised they don't have MassGIS's hydrography data ...
this NHD is quite low res and offset compared to what they have.

The conversion seems OK though.

On Thu, Apr 28, 2011 at 7:11 PM, Ben Supnik  wrote:

> Hi Y'all,
>
> This is one sub-basin (01090001 - go Sox :-), isolated from the latest NHD
> data, then converted via the latest rule set off of the wiki (with shapefile
> key capitalizations fixed).
>
> http://dev.x-plane.com/download/01090001.zip
>
> If anyone has done NHD imports before, please take a quick look and let me
> know if this looks useful.  If so, I can batch the rest of the HUC8s.
>
> This particular archive contains every layer from the source in both shp
> and osm format for comparison.
>
> cheers
> Ben
> --
> Scenery Home Page: http://scenery.x-plane.com/
> Scenery blog: http://www.x-plane.com/blog/
> Plugin SDK: http://www.xsquawkbox.net/xpsdk/
> X-Plane Wiki: http://wiki.x-plane.com/
> Scenery mailing list: x-plane-scen...@yahoogroups.com
> Developer mailing list: x-plane-...@yahoogroups.com
>
> ___
> Talk-us mailing list
> Talk-us@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-us
>
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[James Umbanhowar]

On Thursday 28 April 2011 20:11:39 Ben Supnik wrote:
> Hi Y'all,
> 
> This is one sub-basin (01090001 - go Sox :-), isolated from the latest
> NHD data, then converted via the latest rule set off of the wiki (with
> shapefile key capitalizations fixed).
> 
> http://dev.x-plane.com/download/01090001.zip
> 
> If anyone has done NHD imports before, please take a quick look and let
> me know if this looks useful.  If so, I can batch the rest of the HUC8s.
> 
> This particular archive contains every layer from the source in both shp
> and osm format for comparison.
> 
> cheers
> Ben


Hi Ben,

I've done a fair bit of importing of NHD so I'll have a fair bit of critique 
given my prior mistakes that I've had to correct.  Here are my opinions:

1.  The medium resolution is far too coarse.  Use high resolution.

2.  You should use one file per attribute.  If you are using Ian's shp-to-osm 
tool, crank up the changeset size to some very large value to accomplish this.

3.  Duplicate nodes need to be merged, especially in the flowline files.  The 
data currently have duplicate nodes a every point where one reach intersects 
another.  This creates many duplicate nodes.  The Corine data import folks 
wrote a java tool to remove these 
(http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import)
 
which I use to remove the duplicate nodes.

4.  The conversion rules you are using seem to have a lot of issues.  I see 
some of these on the default rules for shp-to-osm wiki page and I have 
corrected them where I see them, but others I can't identify their source:
a  I didn't see any names.  If there is a GNIS_name this should be the 
source for the name.
b.  The 55800 connectors should have a waterway tag.  I typically 
default 
them to waterway:stream and then change them to waterway:river if their name 
has a "River" in it.
c. 55600 should be tagged natural:coastline
d. 36400 is foreshore and should either be additionally tagged as 
water:tidal or ignored.  
e. 46000/46006  should be tagged as waterway:riverbank if it is a 
polygon
f.  It would probably be good to have a discussion of how much of the 
additional NHD/GNIS tags we want to import.  I think we should import the 
FCODE and
g.  In points and lines files there were many things that had no osm 
tags 
including gauging stations and nonearthen shores.  We should not import things 
that have no osm tags.

5.  Once we set this up, we'll need to give a fair bit of guidance on what 
sorts of things to look for.  One thing that is quite tricky is importing 
coastline.  One has to make sure that you delete all the previously existing 
coastline in the area and to connect the new coastline at the edges of its 
extent.  Given the poor resolution of the PGS and the general disagreement 
between it and USGS on whether tidal riverbanks are coastline or not this can 
be quite a bit of work.

Anyway, good work so far!  If you have any technical questions, feel free to 
email me offlist.

James

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[Ben Supnik]

Hi Y'all,

So first: a new extraact is attached with...

- Dupes removed.
- GNIS names fixed.
- One file per export.

http://dev.x-plane.com/download/01090001.zip

(Be sure to nuke caches to get the build from today's date...)

On 4/29/11 9:51 AM, James Umbanhowar wrote:


2.  You should use one file per attribute.  If you are using Ian's shp-to-osm
tool, crank up the changeset size to some very large value to accomplish this.


That seems to work so far...I had to boost heap size to cope with the 
flow line files.



3.  Duplicate nodes need to be merged, especially in the flowline files.  The
data currently have duplicate nodes a every point where one reach intersects
another.  This creates many duplicate nodes.  The Corine data import folks
wrote a java tool to remove these
(http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import)
which I use to remove the duplicate nodes.


With some modification, that seems to have worked.


4.  The conversion rules you are using seem to have a lot of issues.  I see
some of these on the default rules for shp-to-osm wiki page and I have
corrected them where I see them, but others I can't identify their source:


So first, those rules did come off the wiki.  For some reason, 
capitalization in the shapefiles is...a bit funky.  I tried to fix some, 
but may have missed them.


The GNIS names are just me being stupid - I fixed all capitalization 
issues except for that one.  The other issues (specific codes) are due 
to the limits of the rule set I took.


Since the posted sub-basin dump has the original .shp files, would it be 
simpler for you to simply correct the rules and then send me back the 
rule sheet you used for inclusion directly in my process?



f.  It would probably be good to have a discussion of how much of the
additional NHD/GNIS tags we want to import.  I think we should import the
FCODE and


That's beyond my pay grade...I'll trust whatever y'all come up with. :-)


g.  In points and lines files there were many things that had no osm 
tags
including gauging stations and nonearthen shores.  We should not import things
that have no osm tags.


Hrm...interesting.  The tools don't currently strip untagged data, do 
they?  Is there an existing tool that strips 'null' data?



5.  Once we set this up, we'll need to give a fair bit of guidance on what
sorts of things to look for.  One thing that is quite tricky is importing
coastline.  One has to make sure that you delete all the previously existing
coastline in the area and to connect the new coastline at the edges of its
extent.  Given the poor resolution of the PGS and the general disagreement
between it and USGS on whether tidal riverbanks are coastline or not this can
be quite a bit of work.


Right...I would go a step further and say that I'd be terrified of the 
results of users attempting to use these extracts to create coastline 
without a lot of experience.  In some cases, the cut-over where OSM 
switches from coastline edge marking to closed water bodies might not be 
in the same place as NHD.


cheers
ben




--
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://www.x-plane.com/blog/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scen...@yahoogroups.com
Developer mailing list: x-plane-...@yahoogroups.com

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[Ian Dees]

On Fri, Apr 29, 2011 at 10:49 AM, Ben Supnik  wrote:
>
>  3.  Duplicate nodes need to be merged, especially in the flowline files.
>>  The
>> data currently have duplicate nodes a every point where one reach
>> intersects
>> another.  This creates many duplicate nodes.  The Corine data import folks
>> wrote a java tool to remove these
>> (
>> http://wiki.openstreetmap.org/wiki/WikiProject_Corine_Land_Cover/Corine_Data_Import
>> )
>> which I use to remove the duplicate nodes.
>>
>
> With some modification, that seems to have worked.
>

I've been meaning to add a "node dedupe" step to the shp-to-osm suite. I'd
love to see what you had to modify to get it to work. I'd also like to add a
"way dedupe" at some point, too (to handle abutting areas like boundaries).


> g.  In points and lines files there were many things that had no
>> osm tags
>> including gauging stations and nonearthen shores.  We should not import
>> things
>> that have no osm tags.
>>
>
> Hrm...interesting.  The tools don't currently strip untagged data, do they?
>  Is there an existing tool that strips 'null' data?
>
>
There is an option (I think it's the "-t" flag?) to shp-to-osm to only
include OSM primitives that had tags applied.
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[Ben Supnik]

Hi Guys,

On 4/29/11 12:10 PM, Ian Dees wrote:


I've been meaning to add a "node dedupe" step to the shp-to-osm suite.
I'd love to see what you had to modify to get it to work. I'd also like
to add a "way dedupe" at some point, too (to handle abutting areas like
boundaries).


Well, that code for the CORINE imports is very one-off...I had to make 
two mods:


- Changed the tag delimetor from " to '
- Change the 'offset' in the node record for lat/lon from 3/5 to 5/7.

In other words, it's hard code to particular text output, it's not a 
general XML parser.



There is an option (I think it's the "-t" flag?) to shp-to-osm to only
include OSM primitives that had tags applied.


Oh _that's_ what that does. :-)  Hrm...but I'm not seeing untagged data 
in this latest import.  Is there an easy way to find such things in 
JOSM?  It looks to me that, because the ComID is imported as a tag, 
pretty much everything is going to get a tag, except for ways and nodes 
that are sub-parts of relations and ways, respectively.


cheers
ben

--
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://www.x-plane.com/blog/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scen...@yahoogroups.com
Developer mailing list: x-plane-...@yahoogroups.com

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import and conversion - sample data

[James Umbanhowar]

> Hi James,
> 
> > I did some more corrections on the rules files and I think that it
> > covers all the left over points I saw (including adding
> > waterway:stream to one of FCODE for Connectors that wasn't working).
> 
> Just to confirm, these changes are the ones I see on the wiki, right?
> 
Yes
> > In terms of untagged ways, if we don't import ComID's and the rest of
> > the additional NHD tags.
> 
> But ComIDs are still on the wiki.  I can add -t in case there are
> untagged nodes/ways, but I don't have a strong opinion re: keeping or
> nuking back-references like ComIDs.
My opinion is not strong, which is why I didn't nuke them there.  I had 
stopped using them, as the prospect of ever referencing back to the NHD seemed 
nearly infinitessimal.  Also, it would make conversion much easier ;).  
> 
> cheers
> ben

James

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Greg Troxel]

Kevin Kenny  writes:

> A few months ago, I tried to get started on trying to resume the NHD
> import in my area - and some of the places where I hike. I'm trying
> to check results with both P2 and JOSM, and tripping over a lot of
> things, which made me put the project back on hold for a while. (I
> had some other things to be about.)

Working on this in my area is on my Copious Spare Time list.

> (2) One historic complaint I've seen about the NHD import is that it
> clutters the map,

That seems like a bogus criticism; renderers can choose to render or
ignore.

If the criticism is cluttering the database, that's a harder call, but
water features are broadly important and it seems unreasonable to
exclude them on clutter grounds.  Sooner or later we're going to need a
layer concept in editors and probably the API.

> and that the assignment of "river" and "stream" is
> difficult.  What I'd propose is to tag as "river" anything that has
> more than two nodes of its flowline as "Artificial Path", and "stream"
> anything that's smaller.  In my area (eastern New York), for instance,
> that would mark out the Schoharie Creek, the Mohawk, the Hudson, and
> the lower reaches of the Catskill, the Kaaterskill, the Esopus, and
> maybe a few others. Is this a reasonable approach?

It seems like following the NHD norms is sensible.  For those who didn't
go read the wiki
   http://wiki.openstreetmap.org/wiki/NHD
"Artificial Path" is a NHD codepoint for river flowlines when NHD has
riverbank data.

Are you just talking about someone not quite being sure in edge cases?
Or are people upset about river/stream seeming off in OSM because it is
off in NHD?  Or something else?

> (4) What about cadastral lines (administrative boundaries, and land
> use, leisure, etc.) that appear to follow flowlines? My inclination is
> not to touch these at all, even if the flowline is being refined.
> Oftentimes, when a stream changes course, the cadastre remains
> unchanged. Only the recorder's office in the jurisdiction in question
> would actually know the situation. I think I'd rather see slightly
> inconsistent boundaries than mess up something like that.

That sounds like a good plan to me.

> (5) In the area I have in mind, there are very few ways that actually
> would need to be conflated (a few major rivers). Is it likely to be
> called vandalism if I confine myself to copying the OSM-specific tags
> from the OSM ways onto the NHD ones? I trust a land survey rather
> more than I do someone tracing a shoreline from Bing imagery: in
> well-graded streams, the shorelines can be variable and ambiguous,
> and NHD has pretty sound information about high water marks.

In my view, if you are acting in good faith and believe the post-edit
map to be better than the pre-edit map, it is not even close to
vandalism.

Often what I do when I want to change something and am a bit unsure is
to see who last edited it and message them.   Usually I don't get an
answer, but sometimes I do and it's been overwhelmingly positive (as in
"sure, that sounds like progress").

So you could write to people that added the big rivers and ask about
their data and whether they think it is better/worse than NHD.

> (6) When I try a limited import, I get a lot of JOSM warnings about
> waterways crossing highways. Do people think that all of these have
> to be fixed before importing? In that case, I'll have to confine myself
> to the very small area that has highway-crossing-stream that I can
> visit myself (to try to settle what the boundaries of the bridge,
> dam or culvert are, and what type of waterwork it is). Is it considered
> acceptable simply to leave the crossings unmarked? (They still
> render correctly in Mapnik, for what it's worth.)

waterways crossing highways is how it is.  They shouldn't be connected,
because you can't navigate from one to the other.   Having the waterway
improves the map, and the fact that it doesn't have detail about bridges
or aqueducts etc. should not be a bar to the improvement.  No where
else in OSM do we require quality standards to be met - just that the
map after the edit is better.

> (7) In the event that I find a node tagged with something other than
> a waterway (or landuse=reservoir, man_made=dam, etc.) that collides
> with an NHD node, is the correct approach to conflate the nodes,
> introduce a duplicate, or offset the new node? Does this question
> even have a correct answer?

I would merge the nodes.

I think a key point is that if you are looking at an existing map
feature and at NHD data for a single actual thing, and you are thinking
about that case, and maybe looking at imagery and deciding what's the
best thing to do, that's 100% fine, and isn't really "importing" as it
is railed against.

If you were going to write a program to bring in large amounts of NHD
stuff that you didn't look at and have it find matching nodes and merge
them without human review, that would be something else.

> I'm trying seriously t

Re: [Talk-us] NHD import: what data quality is acceptable?

[Paul Norman]

> From: Kevin Kenny [mailto:kken...@nycap.rr.com]
> Subject: [Talk-us] NHD import: what data quality is acceptable?
> 
> A few months ago, I tried to get started on trying to resume the NHD
> import in my area - and some of the places where I hike. I'm trying to
> check results with both P2 and JOSM, and tripping over a lot of things,
> which made me put the project back on hold for a while. (I had some
> other things to be about.)
> 
> But now I'm starting to reconsider again, after hearing that there are
> others out there thinking the NHD import is desirable for an "Open Trail
> Map." I mentioned this in a message earlier today.
> 
> So, let me review some of the things that have me scratching my head.
> 
> (1) The mapping from NHD feature codes to OSM tags is incomplete (and
> not quite consistent). That's fine; for all the FCodes in my area, I've
> been able to find features that I'm familiar with that I can describe.
> So I think I have that problem fairly well licked. (I may have to invent
> a tag or two, like "waterway=rise" and "waterway=sink" to describe
> streams that go underground and appear again in karst terrain.)

The mappings on the wiki are not only incomplete and inconsistent, they're
for an older NHD version and sometimes clearly wrong.

I posted a better one
(http://lists.openstreetmap.org/pipermail/talk-us/2012-July/008502.html)
earlier this month but it didn't attract any comments, and it's not complete
either. It handles most of the FCodes but still is missing a couple. It also
needs some post-processing to clean up over-noded ways and some other
matters.

The main weakness with NHD data that I find is that there is no way to
distinguish between an OSM waterway=stream and waterway=river


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[James Umbanhowar]

On Sun, 2012-07-22 at 18:33 -0700, Paul Norman wrote:

> 
> The main weakness with NHD data that I find is that there is no way to
> distinguish between an OSM waterway=stream and waterway=river
> 

Why not use the name?



___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Paul Norman]

> From: James Umbanhowar [mailto:jumba...@gmail.com]
> Sent: Sunday, July 22, 2012 6:43 PM
> To: talk-us@openstreetmap.org
> Subject: Re: [Talk-us] NHD import: what data quality is acceptable?
> 
> On Sun, 2012-07-22 at 18:33 -0700, Paul Norman wrote:
> 
> >
> > The main weakness with NHD data that I find is that there is no way to
> > distinguish between an OSM waterway=stream and waterway=river
> >
> 
> Why not use the name?

The name is the best way I've found and is in fact what I'm using
(https://github.com/pnorman/ogr2osm-translations/blob/69434ec55a48e9e4858ef3
5f2489949b2020e527/us_nhd.py#L234) but it's not 100%

No way is perhaps a bit strong, no fully reliable way would be more
accurate.


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Kevin Kenny]

On 07/22/2012 10:09 PM, Paul Norman wrote:

From: James Umbanhowar [mailto:jumba...@gmail.com]
Sent: Sunday, July 22, 2012 6:43 PM
To: talk-us@openstreetmap.org
Subject: Re: [Talk-us] NHD import: what data quality is acceptable?

On Sun, 2012-07-22 at 18:33 -0700, Paul Norman wrote:



The main weakness with NHD data that I find is that there is no way to
distinguish between an OSM waterway=stream and waterway=river



Why not use the name?


The name is the best way I've found and is in fact what I'm using
(https://github.com/pnorman/ogr2osm-translations/blob/69434ec55a48e9e4858ef3
5f2489949b2020e527/us_nhd.py#L234) but it's not 100%

No way is perhaps a bit strong, no fully reliable way would be more
accurate.


Using ArtificialPath comes awfully close around here. There are a lot
of things named 'creek', 'kill' or 'brook' around here that would be
surrounded by 'waterway=riverbank', and I would presume that anything
that rates having its banks drawn is already a 'river' according to
http://wiki.openstreetmap.org/wiki/Tag:waterway%3Driver . There are
also a number of tiny streams that would be tagged 'river' because
they are tradiitonally identified as rivers' headwaters. I say
"traditionally", because they often do not correctly identify the
longest reach of a given river. It would somewhat silly tagging
the "West Branch Neversink River" as a 'river' - it's quite a
minor stream. But by the same token, the Schoharie Creek, the
Catskill, the Kaaterskill, and the Esopus Creek are quite sizable
rivers whose flooding wiped out entire villages in Hurricane Irene.

I guess to some extent my local knowledge is coming into play, but
the features that the NHD curators have drawn areas around and
promoted to ArtificialPath are almost exactly the things that
I would label 'river.'

The reason for 'at least three nodes of ArtificialPath' is that it
eliminates most of the detritus where artificial paths are created to
join a tributary stream with a major river's centerline.

--
73 de ke9tv/2, Kevin

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Kevin Kenny]

On 07/22/2012 09:33 PM, Paul Norman wrote:

The mappings on the wiki are not only incomplete and inconsistent, they're
for an older NHD version and sometimes clearly wrong.

I posted a better one
(http://lists.openstreetmap.org/pipermail/talk-us/2012-July/008502.html)
earlier this month but it didn't attract any comments, and it's not complete
either. It handles most of the FCodes but still is missing a couple. It also
needs some post-processing to clean up over-noded ways and some other
matters.


Right. I downloaded and looked at your code, but I was already
pretty far along when you posted that message. I'd mostly been
working by diligently examining, each time I encountered an FCode that I
haven't seen before, what the feature actually is, from personal
knowledge. (I then often presume that other features having the same
FCode are the same general sort of thing.) Except for likely having to
invent some stuff for karst features, I think that I have a pretty
sound tag mapping. I'll go back at some point and check how it differs
from yours. At a quick glance, they're pretty similar.

I'm using a somewhat different workflow, doing a lot of the heavy
lifting in PostGIS. My general plan involves clipping of flowlines,
areas and waterbodies to HU12 basins so that I have bite-sized pieces
to process with minimal connections to make at the edges: ideally
a single connection, but sometimes the HU12 watershed lines are
slightly misdrawn and pull in tiny bits of streams that actually
belong to another basin. PostGIS also gives me a fairly easy way to do
collision checking and find candidates for conflation.

Oh, by the way, my plan is to include nhd:reach_code and
nhd:permanent_id tags, to facilitate conflation in the event that
another NHD version obsoletes the current one.
--
73 de ke9tv/2, Kevin

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Paul Norman]

> From: Kevin Kenny [mailto:kken...@nycap.rr.com]
> Sent: Monday, July 23, 2012 5:45 AM
> To: talk-us@openstreetmap.org
> Subject: Re: [Talk-us] NHD import: what data quality is acceptable?
> 
> On 07/22/2012 09:33 PM, Paul Norman wrote:
> > The mappings on the wiki are not only incomplete and inconsistent,
> > they're for an older NHD version and sometimes clearly wrong.
> >
> > I posted a better one
> > (http://lists.openstreetmap.org/pipermail/talk-us/2012-July/008502.htm
> > l) earlier this month but it didn't attract any comments, and it's not
> > complete either. It handles most of the FCodes but still is missing a
> > couple. It also needs some post-processing to clean up over-noded ways
> > and some other matters.
> 
> Right. I downloaded and looked at your code, but I was already pretty
> far along when you posted that message. I'd mostly been working by
> diligently examining, each time I encountered an FCode that I haven't
> seen before, what the feature actually is, from personal knowledge. (I
> then often presume that other features having the same FCode are the
> same general sort of thing.) 

This unfortunately falls short. I find that you need to check the FCode
across at least 3 different parts of the country to be sure. I've found
there are regional variations in how FCodes are used.

I hope to get back to my code in the next week. With the redaction it hasn't
been a high priority. Also, no one has proposed a NHD import lately.


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Kevin Kenny]

On 07/23/2012 10:49 AM, Paul Norman wrote:

This unfortunately falls short. I find that you need to check the FCode
across at least 3 different parts of the country to be sure. I've found
there are regional variations in how FCodes are used.


But I'm not *doing* 3 different parts of the country. I'm doing *my*
part of the country. I'm not touching areas where I have no knowledge
of the local geography.

___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Apollinaris Schoell]

On Jul 22, 2012, at 5:25 PM, Greg Troxel wrote:
>> (2) One historic complaint I've seen about the NHD import is that it
>> clutters the map,
> 
> That seems like a bogus criticism; renderers can choose to render or
> ignore.
> 
> If the criticism is cluttering the database, that's a harder call, but
> water features are broadly important and it seems unreasonable to
> exclude them on clutter grounds.  Sooner or later we're going to need a
> layer concept in editors and probably the API.
> 

If the imported features are tagged correct it's all good. But I have seen 
imports of "streams" where you will never find water except while a huge storm. 
This wasn't NHD and I am not sure if it  applies here.  As long as the data is 
verified against nature in some places it's perfectly fine to import.

> 
>> (5) In the area I have in mind, there are very few ways that actually
>> would need to be conflated (a few major rivers). Is it likely to be
>> called vandalism if I confine myself to copying the OSM-specific tags
>> from the OSM ways onto the NHD ones? I trust a land survey rather
>> more than I do someone tracing a shoreline from Bing imagery: in
>> well-graded streams, the shorelines can be variable and ambiguous,
>> and NHD has pretty sound information about high water marks.
> 
> In my view, if you are acting in good faith and believe the post-edit
> map to be better than the pre-edit map, it is not even close to
> vandalism.

+1 

> 
>> (6) When I try a limited import, I get a lot of JOSM warnings about
>> waterways crossing highways. Do people think that all of these have
>> to be fixed before importing? In that case, I'll have to confine myself
>> to the very small area that has highway-crossing-stream that I can
>> visit myself (to try to settle what the boundaries of the bridge,
>> dam or culvert are, and what type of waterwork it is). Is it considered
>> acceptable simply to leave the crossings unmarked? (They still
>> render correctly in Mapnik, for what it's worth.)
> 
> waterways crossing highways is how it is.  They shouldn't be connected,
> because you can't navigate from one to the other.   Having the waterway
> improves the map, and the fact that it doesn't have detail about bridges
> or aqueducts etc. should not be a bar to the improvement.  No where
> else in OSM do we require quality standards to be met - just that the
> map after the edit is better.

agreed, to me the validator check in JOSM is too strict. some mappers 
workaround by adding a layer -1 to all waterways. But that's mapping for the 
validator and doesn't add any value but clutter to the DB.

> 
>> I'm trying to start with areas where I have firm local knowledge,
>> although clearly I cannot survey streams on private lands, so I
>> have to depend on NHD there. But the result is not going to be of
>> much use to me unless I can generalize what I learn to do better
>> NHD imports in areas that I know less well.
> 
> I would say that if NHD says there is a stream, and there's no data
> about hydrography in OSM near it, then the map is better off with the
> NHD data.  My impression is that NHD data that I see on the map while
> driving appears quite in sync with reality.

Any data source is more or less accurate in different regions. If you verify it 
in local areas with your knowledge it's ok to extrapolate it to county or even 
state level. But if you verify in New York it's not a good idea to import NHD 
in Arizona and Alaska with the same rules. 

> ___
> Talk-us mailing list
> Talk-us@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-us


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Brian May]

On 7/23/2012 10:49 AM, Paul Norman wrote:
I hope to get back to my code in the next week. With the redaction it 
hasn't been a high priority. Also, no one has proposed a NHD import 
lately.

Paul,

Could you provide some NHD files in OSM format for a few parts of FL? 
I'm willing to do some manual importing / cleanup / merging with 
existing quality linework using JOSM. For me, just getting past the 
conversion to OSM format is the biggest barrier to working with NHD 
data. I'm currently interested in doing some hydro work for SW FL in the 
Punta Gorda area as well as filling out some hydro in SE FL, i.e. Palm 
Beach, Broward, Miami-Dade. In the past, I've manually imported some 
hydro polygons from the landcover SHPs provided by the water management 
districts in FL. Mainly used it to fill in lakes, but also updated some 
PGS coastlines with it. Also looking to see if the NHD data is a better 
source for hydro areas in FL vs. the landcover data.


Thanks,

Brian


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us

Re: [Talk-us] NHD import: what data quality is acceptable?

[Paul Norman]

> From: Kevin Kenny [mailto:kken...@nycap.rr.com]
> Subject: Re: [Talk-us] NHD import: what data quality is acceptable?
> 
> On 07/23/2012 10:49 AM, Paul Norman wrote:
> > This unfortunately falls short. I find that you need to check the
> > FCode across at least 3 different parts of the country to be sure.
> > I've found there are regional variations in how FCodes are used.
> 
> But I'm not *doing* 3 different parts of the country. I'm doing *my*
> part of the country. I'm not touching areas where I have no knowledge of
> the local geography.

Yes - but I'm trying to work on something that will work across the country.
Of course if you're working locally, only what they do locally matters and
the data set is more homogenous.

I'm hoping the end result of what I do will be something similar to CanVec
for a data source, but having learned from its mistakes.


___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us