[OSM-talk] OSM TIGER-based map quality
When overlaying OSM with the recently available TIGER 2007 shapefile data for Census Blocks in Alameda County (California), I'm encountering both an offset and difference in relative position of the linework. In short, OSM's data looks a lot more accurate and consistent -- streets that should be straight actually look straight in OSM, but often zig-zag in the TIGER 2007 edges and tabblock shapefiles. For a visual, visit: http://www.giswebsite.com/demos/tiger_overlays.html Any observations or ideas about where the misalignment might come from? Though it smells partially like a datum problem, that path hasn't yielded any solutions yet. Thanks, Jonathan ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, Apr 04, 2008 at 08:29:56AM +, Jonathan W. Lowe wrote: When overlaying OSM with the recently available TIGER 2007 shapefile data for Census Blocks in Alameda County (California), I'm encountering both an offset and difference in relative position of the linework. In short, OSM's data looks a lot more accurate and consistent -- streets that should be straight actually look straight in OSM, but often zig-zag in the TIGER 2007 edges and tabblock shapefiles. For a visual, visit: http://www.giswebsite.com/demos/tiger_overlays.html Um, isn't this the whole point of OSM? The TIGER data was imported so that it could be improved manually by users. This doesn't include just the geometry either: attributes have been changed as well. (See http://crschmidt.net/osm/history.html?type=wayid=21456366) Any observations or ideas about where the misalignment might come from? This misalignment is common in TIGER: It's designed for 1:10 accuracy. Anything more than that (you're at about 1:7500 there) is not going to be accurate. (Or at least likely to not be.) Though it smells partially like a datum problem, that path hasn't yielded any solutions yet. Doubtful. You'd have a more significant shift. This is the real power of OSM at work. See it, and marvel in its glory. :) Regards, -- Christopher Schmidt MetaCarta ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, 2008-04-04 at 07:05 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 08:29:56AM +, Jonathan W. Lowe wrote: When overlaying OSM with the recently available TIGER 2007 shapefile data for Census Blocks in Alameda County (California), I'm encountering both an offset and difference in relative position of the linework. In short, OSM's data looks a lot more accurate and consistent -- streets that should be straight actually look straight in OSM, but often zig-zag in the TIGER 2007 edges and tabblock shapefiles. For a visual, visit: http://www.giswebsite.com/demos/tiger_overlays.html Um, isn't this the whole point of OSM? The TIGER data was imported so that it could be improved manually by users. This doesn't include just the geometry either: attributes have been changed as well. (See http://crschmidt.net/osm/history.html?type=wayid=21456366) Thanks, Chris -- are you suggesting that the TIGER 2007 reissue from the US Census is not lining up with OSM in some (rather extensive) areas because the geometry in those areas has been completely improved manually since the upload of TIGER completed earlier this year. Yes, obviously, such a major improvement in accuracy from manual attention is one of the (many) wonderful benefits of OSM -- I guess I'm just surprised that so much manual editing could have been accomplished for such an extensive area given the short time TIGER has been available for editing in OSM. Is there any existing interface for visualizing which features have changed in OSM for a given area? If that is indeed the case, since the Census Blocks come from the same topologic source as the streets do, I wonder if there's any existing routine that would enable me to run the same edits on the Census Blocks geometries, or use the OSM streets to derive Census Block boundaries and then conflate the Block identifiers. Thinking out loud here... Any observations or ideas about where the misalignment might come from? This misalignment is common in TIGER: It's designed for 1:10 accuracy. Anything more than that (you're at about 1:7500 there) is not going to be accurate. (Or at least likely to not be.) I may not be communicating the scenario clearly enough -- TIGER misalignment is common with other non-TIGER data certainly, but TIGER data is not misaligned with other TIGER data of the same issue. I'm encountering what I thought was the latter case. TIGER data models both visible physical features such as streets, but also invisible census data collection boundaries such as blocks and tracts. The block and tract boundaries and the street centerlines are derived from the same single topologic source, so when viewed together, line up perfectly unless one or the other has been altered independently since original release. The 2007 shapefile issue confirms this -- linestrings in the edges shapefile perfectly match the polygon boundaries in the tabblock shapefile. But neither match the OSM geometry, at least using the conversion and display methods I've used within Berkeley's geographic extents. Since all the data has the same parentage, I initially thought there would be a tighter match between TIGER 2007 and OSM TIGER, but not if the editing to OSM TIGER has been as extensive as you seem to be suggesting. Though it smells partially like a datum problem, that path hasn't yielded any solutions yet. Doubtful. You'd have a more significant shift. Experimenting with NAD27 to NAD83 conversions actually produces a similar shift, depending on the region. But I agree -- it seems to be something else. Hm. This is the real power of OSM at work. See it, and marvel in its glory. :) Yes, marvelous it most certainly is. I currently use OSM (via an OpenLayers client) with multiple existing customers and have supported the whole ethos of open-source in published articles, starting with one on PostGIS, since 2001, so am no stranger to the glory. (You and I met, Chris, at FOSS4G2006 in Switzerland, where you introduced yourself as Boy Genius.) Regards, ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, Apr 04, 2008 at 01:53:24PM +0100, Jonathan W. Lowe wrote: On Fri, 2008-04-04 at 07:05 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 08:29:56AM +, Jonathan W. Lowe wrote: When overlaying OSM with the recently available TIGER 2007 shapefile data for Census Blocks in Alameda County (California), I'm encountering both an offset and difference in relative position of the linework. In short, OSM's data looks a lot more accurate and consistent -- streets that should be straight actually look straight in OSM, but often zig-zag in the TIGER 2007 edges and tabblock shapefiles. For a visual, visit: http://www.giswebsite.com/demos/tiger_overlays.html Um, isn't this the whole point of OSM? The TIGER data was imported so that it could be improved manually by users. This doesn't include just the geometry either: attributes have been changed as well. (See http://crschmidt.net/osm/history.html?type=wayid=21456366) Thanks, Chris -- are you suggesting that the TIGER 2007 reissue from the US Census is not lining up with OSM in some (rather extensive) areas because the geometry in those areas has been completely improved manually since the upload of TIGER completed earlier this year. Yes, obviously, such a major improvement in accuracy from manual attention is one of the (many) wonderful benefits of OSM -- I guess I'm just surprised that so much manual editing could have been accomplished for such an extensive area given the short time TIGER has been available for editing in OSM. Is there any existing interface for visualizing which features have changed in OSM for a given area? That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) Any observations or ideas about where the misalignment might come from? This misalignment is common in TIGER: It's designed for 1:10 accuracy. Anything more than that (you're at about 1:7500 there) is not going to be accurate. (Or at least likely to not be.) I may not be communicating the scenario clearly enough -- TIGER misalignment is common with other non-TIGER data certainly, but TIGER data is not misaligned with other TIGER data of the same issue. I'm encountering what I thought was the latter case. No: what you're encountering is TIGER misalignment with reality. OSM is much closer to reality in this instance than TIGER. TIGER data models both visible physical features such as streets, but also invisible census data collection boundaries such as blocks and tracts. The block and tract boundaries and the street centerlines are derived from the same single topologic source, so when viewed together, line up perfectly unless one or the other has been altered independently since original release. The 2007 shapefile issue confirms this -- linestrings in the edges shapefile perfectly match the polygon boundaries in the tabblock shapefile. But neither match the OSM geometry, at least using the conversion and display methods I've used within Berkeley's geographic extents. Right. Because TIGER is wrong :) Since all the data has the same parentage, I initially thought there would be a tighter match between TIGER 2007 and OSM TIGER, but not if the editing to OSM TIGER has been as extensive as you seem to be suggesting. That is correct. Though it smells partially like a datum problem, that path hasn't yielded any solutions yet. Doubtful. You'd have a more significant shift. Experimenting with NAD27 to NAD83 conversions actually produces a similar shift, depending on the region. But I agree -- it seems to be something else. Hm. I would have expected it to be more significant by a factor of about two, from observation, but I could be mis-guessing the scale here: I'm used to working in Cambridge, rather than SF. In any case, the misalignment is just a standard misalignment of TIGER data to reality: OSM is much closer to reality, despite the initial import being from TIGER. This is the real power of OSM at work. See it, and marvel in its glory. :) Yes, marvelous it most certainly is. I currently use OSM (via an OpenLayers client) with multiple existing customers and have supported the whole ethos of open-source in published articles, starting with one on PostGIS, since 2001, so am no stranger to the glory. (You and I met, Chris, at FOSS4G2006 in Switzerland, where you introduced yourself as Boy Genius.) I'm now Meta Ninja, since I have to manage other ninjas, and Leslie Hawthorn (Summer of Code organizer) told me I was too old, at 23, to be a Boy Genius anymore. :) Regards, -- Christopher Schmidt
Re: [OSM-talk] OSM TIGER-based map quality
Dave, From what I've seen, yes. Or at least for Santa Clara County, California. To bolster this, the main edge files do not contain street address ranges in the 2007 data, but the US Census Bureau gives as one of the two possible options of adding this information to the file is taking the address ranges from the 2006 TIGER/Line data by matching on the tlid field. A pretty good indication that the tlid fields match. Dan On Fri, 2008-04-04 at 10:31 -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 09:12 -0400, Christopher Schmidt wrote: That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) If this is true, then we can probably go back and fix it. We have all of the TIGER tlid (unique ids) for all of the uploaded data. Have those remained the same in the new TIGER data? -- Dave ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk -- Dan Putler Sauder School of Business University of British Columbia ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, Apr 04, 2008 at 10:31:11AM -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 09:12 -0400, Christopher Schmidt wrote: That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) If this is true, then we can probably go back and fix it. We have all of the TIGER tlid (unique ids) for all of the uploaded data. Have those remained the same in the new TIGER data? Fix... what? OSM is correct here... perhaps you mean We can improve TIGER 2007 with OSM data? Regards, -- Christopher Schmidt MetaCarta ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, 2008-04-04 at 13:51 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 10:31:11AM -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 09:12 -0400, Christopher Schmidt wrote: That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) If this is true, then we can probably go back and fix it. We have all of the TIGER tlid (unique ids) for all of the uploaded data. Have those remained the same in the new TIGER data? Fix... what? OSM is correct here... perhaps you mean We can improve TIGER 2007 with OSM data? Did I misunderstand what's going on? I assumed that the TIGER '07 data is better than the TIGER '05 (or so) data that we populated OSM with. If it is better, we can update OSM from TIGER '07. -- Dave ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
Dave, Not all counties have been improved through the MAF/TIGER Accuracy Improvement Project (MTAIP). Here is a link to the counties that have not been improved in the 2007 TIGER data: http://www.census.gov/geo/www/tiger/tgrshp2007/tgrshp07nomtaip.txt Dan On Fri, 2008-04-04 at 10:53 -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 13:51 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 10:31:11AM -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 09:12 -0400, Christopher Schmidt wrote: That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) If this is true, then we can probably go back and fix it. We have all of the TIGER tlid (unique ids) for all of the uploaded data. Have those remained the same in the new TIGER data? Fix... what? OSM is correct here... perhaps you mean We can improve TIGER 2007 with OSM data? Did I misunderstand what's going on? I assumed that the TIGER '07 data is better than the TIGER '05 (or so) data that we populated OSM with. If it is better, we can update OSM from TIGER '07. -- Dave ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk -- Dan Putler Sauder School of Business University of British Columbia ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, 2008-04-04 at 11:04 -0700, Dan Putler wrote: Not all counties have been improved through the MAF/TIGER Accuracy Improvement Project (MTAIP). Here is a link to the counties that have not been improved in the 2007 TIGER data: http://www.census.gov/geo/www/tiger/tgrshp2007/tgrshp07nomtaip.txt Cool, thanks for the pointer. That looks like ~1100 of the ~3500 total counties in the US. -- Dave ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] OSM TIGER-based map quality
On Fri, 2008-04-04 at 14:08 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 10:53:43AM -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 13:51 -0400, Christopher Schmidt wrote: On Fri, Apr 04, 2008 at 10:31:11AM -0700, Dave Hansen wrote: On Fri, 2008-04-04 at 09:12 -0400, Christopher Schmidt wrote: That is exactly what I am suggesting, based on an examination of the objects in the area you're looking at: all of them have been updated several times (as the history URL above demonstrates) by a single user in what seems to clearly be a cleanup effort. (This was based on using an OpenLayers map to select 10 different streets based on the lonlat I observed in your JOSM instance and view the history for each.) If this is true, then we can probably go back and fix it. We have all of the TIGER tlid (unique ids) for all of the uploaded data. Have those remained the same in the new TIGER data? Fix... what? OSM is correct here... perhaps you mean We can improve TIGER 2007 with OSM data? Did I misunderstand what's going on? I assumed that the TIGER '07 data is better than the TIGER '05 (or so) data that we populated OSM with. If it is better, we can update OSM from TIGER '07. The images that were shared in this thread demonstrate that OSM is *way* better than TIGER '05 or '07. ...that's true of Berkeley, Chris, but in any US geography where nobody has yet edited the original TIGER upload, OSM is equivalent to TIGER '05 in all but storage format, and therefore may not be as accurate as TIGER '07. Regards, ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
