Re: [Talk-us] NHD data import question
Hi Y'all, So let me see if I can summarize where we are so far... - I could cut the NHD data down to sub-basins. It would not be a perfect cutting (e.g. each sub-basin might need a little bit of data hanging off since the raw data isn't cut on sub-basin boundaries) but it would be pretty close, and we expect users to manually check their imports. I could also potentially convert the shapefiles to .osm format on a sub-basin basis. - It would be nice to make it easier for users to find their sub-basins. I'm afraid I don't know how to use the mapnik tool chain to create a slippy map though. (Left to my own, I would probably convert the sub-basin boundaries to KML...that makes me a heretic, right? :-) We could also just post the boundary shapefile, but some users may not be comfortable with shapefiles. - Importing on a sub-basin basis should be possible in JOSM, if not a bit of a mouthful. Importing via potlatch2 vector layers should be possible - others have seen this work although I have not. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
Yep: if you have the ability and time to cut the national dataset into subbasins I'll give you an SFTP to upload them to and I'll start converting them to OSM. From there we could add links to the pre-converted subbasin OSM bundles to the wiki pages. Bonus points for converting the subbasin boundary files to KML/GeoJSON/GML and overlaying them on a import status map that a user could click on to find their subbasin. I'd be happy to host all this. -Ian On Thu, Apr 28, 2011 at 7:37 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Y'all, So let me see if I can summarize where we are so far... - I could cut the NHD data down to sub-basins. It would not be a perfect cutting (e.g. each sub-basin might need a little bit of data hanging off since the raw data isn't cut on sub-basin boundaries) but it would be pretty close, and we expect users to manually check their imports. I could also potentially convert the shapefiles to .osm format on a sub-basin basis. - It would be nice to make it easier for users to find their sub-basins. I'm afraid I don't know how to use the mapnik tool chain to create a slippy map though. (Left to my own, I would probably convert the sub-basin boundaries to KML...that makes me a heretic, right? :-) We could also just post the boundary shapefile, but some users may not be comfortable with shapefiles. - Importing on a sub-basin basis should be possible in JOSM, if not a bit of a mouthful. Importing via potlatch2 vector layers should be possible - others have seen this work although I have not. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
Hi Ian, I should be able to convert the NHD data sometime in the next week or so. Also, if/when you want the original data, let me know. For the conversion, do we want medium or high resolution? Re: the import status map, I don't know how to make an actual map from themshort of simply providing a KML file and saying go look on Google Earth, what formats are most useful for making an overlay status map? (I could do that conversion and hand it off to someone.) cheers ben On 4/28/11 8:48 AM, Ian Dees wrote: Yep: if you have the ability and time to cut the national dataset into subbasins I'll give you an SFTP to upload them to and I'll start converting them to OSM. From there we could add links to the pre-converted subbasin OSM bundles to the wiki pages. Bonus points for converting the subbasin boundary files to KML/GeoJSON/GML and overlaying them on a import status map that a user could click on to find their subbasin. I'd be happy to host all this. -Ian On Thu, Apr 28, 2011 at 7:37 AM, Ben Supnik bsup...@xsquawkbox.net mailto:bsup...@xsquawkbox.net wrote: Hi Y'all, So let me see if I can summarize where we are so far... - I could cut the NHD data down to sub-basins. It would not be a perfect cutting (e.g. each sub-basin might need a little bit of data hanging off since the raw data isn't cut on sub-basin boundaries) but it would be pretty close, and we expect users to manually check their imports. I could also potentially convert the shapefiles to .osm format on a sub-basin basis. - It would be nice to make it easier for users to find their sub-basins. I'm afraid I don't know how to use the mapnik tool chain to create a slippy map though. (Left to my own, I would probably convert the sub-basin boundaries to KML...that makes me a heretic, right? :-) We could also just post the boundary shapefile, but some users may not be comfortable with shapefiles. - Importing on a sub-basin basis should be possible in JOSM, if not a bit of a mouthful. Importing via potlatch2 vector layers should be possible - others have seen this work although I have not. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com mailto:x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com mailto:x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org mailto:Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
On Thu, Apr 28, 2011 at 7:53 AM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Ian, I should be able to convert the NHD data sometime in the next week or so. Also, if/when you want the original data, let me know. For the conversion, do we want medium or high resolution? We'll want high resolution. Re: the import status map, I don't know how to make an actual map from themshort of simply providing a KML file and saying go look on Google Earth, what formats are most useful for making an overlay status map? (I could do that conversion and hand it off to someone.) If you want to send me the shapefile I can work on making that map. cheers ben On 4/28/11 8:48 AM, Ian Dees wrote: Yep: if you have the ability and time to cut the national dataset into subbasins I'll give you an SFTP to upload them to and I'll start converting them to OSM. From there we could add links to the pre-converted subbasin OSM bundles to the wiki pages. Bonus points for converting the subbasin boundary files to KML/GeoJSON/GML and overlaying them on a import status map that a user could click on to find their subbasin. I'd be happy to host all this. -Ian On Thu, Apr 28, 2011 at 7:37 AM, Ben Supnik bsup...@xsquawkbox.net mailto:bsup...@xsquawkbox.net wrote: Hi Y'all, So let me see if I can summarize where we are so far... - I could cut the NHD data down to sub-basins. It would not be a perfect cutting (e.g. each sub-basin might need a little bit of data hanging off since the raw data isn't cut on sub-basin boundaries) but it would be pretty close, and we expect users to manually check their imports. I could also potentially convert the shapefiles to .osm format on a sub-basin basis. - It would be nice to make it easier for users to find their sub-basins. I'm afraid I don't know how to use the mapnik tool chain to create a slippy map though. (Left to my own, I would probably convert the sub-basin boundaries to KML...that makes me a heretic, right? :-) We could also just post the boundary shapefile, but some users may not be comfortable with shapefiles. - Importing on a sub-basin basis should be possible in JOSM, if not a bit of a mouthful. Importing via potlatch2 vector layers should be possible - others have seen this work although I have not. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com mailto:x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com mailto:x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org mailto:Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
On Thursday 28 April 2011 08:53:56 Ben Supnik wrote: Hi Ian, I should be able to convert the NHD data sometime in the next week or so. Also, if/when you want the original data, let me know. For the conversion, do we want medium or high resolution? Re: the import status map, I don't know how to make an actual map from themshort of simply providing a KML file and saying go look on Google Earth, what formats are most useful for making an overlay status map? (I could do that conversion and hand it off to someone.) cheers ben I've been checking some of the imported data and my general feel is that it is overdigitized. I don't know if medium versus high reflects just the quality or the amount of digitization. That would be something to check. One could also just run some sort of simplification algorithm on all the data. Also, are the plans to make one file for each subbasin or one file per attribute (flowline, waterbody etc...). With one file, one can remove all the duplicate nodes before uploading. Maybe it would be useful to do a test subbasin with whatever toolchain you're going to use and then post it so that mappers can check to see how the conversion meshes with our general wants. On 4/28/11 8:48 AM, Ian Dees wrote: Yep: if you have the ability and time to cut the national dataset into subbasins I'll give you an SFTP to upload them to and I'll start converting them to OSM. From there we could add links to the pre-converted subbasin OSM bundles to the wiki pages. Bonus points for converting the subbasin boundary files to KML/GeoJSON/GML and overlaying them on a import status map that a user could click on to find their subbasin. I'd be happy to host all this. -Ian On Thu, Apr 28, 2011 at 7:37 AM, Ben Supnik bsup...@xsquawkbox.net mailto:bsup...@xsquawkbox.net wrote: Hi Y'all, So let me see if I can summarize where we are so far... - I could cut the NHD data down to sub-basins. It would not be a perfect cutting (e.g. each sub-basin might need a little bit of data hanging off since the raw data isn't cut on sub-basin boundaries) but it would be pretty close, and we expect users to manually check their imports. I could also potentially convert the shapefiles to .osm format on a sub-basin basis. - It would be nice to make it easier for users to find their sub-basins. I'm afraid I don't know how to use the mapnik tool chain to create a slippy map though. (Left to my own, I would probably convert the sub-basin boundaries to KML...that makes me a heretic, right? :-) We could also just post the boundary shapefile, but some users may not be comfortable with shapefiles. - Importing on a sub-basin basis should be possible in JOSM, if not a bit of a mouthful. Importing via potlatch2 vector layers should be possible - others have seen this work although I have not. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com mailto:x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com mailto:x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org mailto:Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
On Thu, Apr 28, 2011 at 8:08 AM, James Umbanhowar jumba...@gmail.comwrote: On Thursday 28 April 2011 08:53:56 Ben Supnik wrote: Hi Ian, I should be able to convert the NHD data sometime in the next week or so. Also, if/when you want the original data, let me know. For the conversion, do we want medium or high resolution? Re: the import status map, I don't know how to make an actual map from themshort of simply providing a KML file and saying go look on Google Earth, what formats are most useful for making an overlay status map? (I could do that conversion and hand it off to someone.) cheers ben I've been checking some of the imported data and my general feel is that it is overdigitized. I don't know if medium versus high reflects just the quality or the amount of digitization. That would be something to check. One could also just run some sort of simplification algorithm on all the data. I haven't seen any excess digitization, but I have seen incorrect data (stream networks completely missing from aerial imagery and vice versa). I suppose we could do some light simplification on the more complicated waterbodies. Also, are the plans to make one file for each subbasin or one file per attribute (flowline, waterbody etc...). With one file, one can remove all the duplicate nodes before uploading. In the past it was file per datatype because all of the tools work with one shapefile at a time. Theoretically we could post-process the OSM data to remove duplicate nodes between the data types. (and if we do that we might as well download the area from the API and do a check to see if the data already exists). Maybe it would be useful to do a test subbasin with whatever toolchain you're going to use and then post it so that mappers can check to see how the conversion meshes with our general wants. There have been several tests already and data has already been imported. I'm happy to have people try out a subbasin as I'm running the conversion scripts (I imagine it will take a while) so we can restart the conversion if a bug is found. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
Hi James, I've been checking some of the imported data and my general feel is that it is overdigitized. I don't know if medium versus high reflects just the quality or the amount of digitization. That would be something to check. One could also just run some sort of simplification algorithm on all the data. There are a lot of nodes. :-) Also, are the plans to make one file for each subbasin or one file per attribute (flowline, waterbody etc...). With one file, one can remove all the duplicate nodes before uploading. My plan was to create one .osm file per sub-basin x layer (the data comes in layers of areas, waterbodies, points, lines, and flow-lines). So we'd have five layers in separate files per sub-basin. I _think_ merging that in .osm would be relatively straight forward, but it might be easier for users to cope with layered. (Do we even...want flow lines? The data is enormous, most of it isn't going to lead to map-visualizable things, and the 'value' of the data set, namely hydrological analysis, might not last long once the data is in OSM and the water network's hydrological properties aren't being enforced during edits. Plus if only part of the flow lines are imported...) Maybe it would be useful to do a test subbasin with whatever toolchain you're going to use and then post it so that mappers can check to see how the conversion meshes with our general wants. That's a good idea and should be straight forward. I can post the medium and high-res sub-basins and let y'all compare. cheers ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD data import question
I've been checking some of the imported data and my general feel is that it is overdigitized. I don't know if medium versus high reflects just the quality or the amount of digitization. That would be something to check. One could also just run some sort of simplification algorithm on all the data. I definitely found this to be the case for some parts of the area I did (Washington County, Oregon). I particularly remember it on areas like ponds and lakes; not so much on rivers and streams. Most of them were reasonable, but a fair number of them had way too many nodes. I used the JOSM simplify way quite a bit, which worked perfectly in my opinion. It may be analogous to TIGER, in that NHD may have a composite dataset that got created in multiple ways. It may show up more in some places than others. - Alan ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
[Talk-us] NHD import and conversion - sample data
Hi Y'all, This is one sub-basin (01090001 - go Sox :-), isolated from the latest NHD data, then converted via the latest rule set off of the wiki (with shapefile key capitalizations fixed). http://dev.x-plane.com/download/01090001.zip If anyone has done NHD imports before, please take a quick look and let me know if this looks useful. If so, I can batch the rest of the HUC8s. This particular archive contains every layer from the source in both shp and osm format for comparison. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] NHD import and conversion - sample data
Looks decent. I'm surprised they don't have MassGIS's hydrography data ... this NHD is quite low res and offset compared to what they have. The conversion seems OK though. On Thu, Apr 28, 2011 at 7:11 PM, Ben Supnik bsup...@xsquawkbox.net wrote: Hi Y'all, This is one sub-basin (01090001 - go Sox :-), isolated from the latest NHD data, then converted via the latest rule set off of the wiki (with shapefile key capitalizations fixed). http://dev.x-plane.com/download/01090001.zip If anyone has done NHD imports before, please take a quick look and let me know if this looks useful. If so, I can batch the rest of the HUC8s. This particular archive contains every layer from the source in both shp and osm format for comparison. cheers Ben -- Scenery Home Page: http://scenery.x-plane.com/ Scenery blog: http://www.x-plane.com/blog/ Plugin SDK: http://www.xsquawkbox.net/xpsdk/ X-Plane Wiki: http://wiki.x-plane.com/ Scenery mailing list: x-plane-scen...@yahoogroups.com Developer mailing list: x-plane-...@yahoogroups.com ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
