Re: [Talk-us] Tiger 2007 Data
On Tue, Oct 28, 2008 at 7:12 AM, Nick Hocking [EMAIL PROTECTED]wrote: Well, #2 would be nice but it would be tricky to detect a collision with an existing way. Frankly, because the first TIGER import was done, the number of completely new ways that would be added in a new import would be small, and the number of those ways that conflict with ways added manually by editors would be even smaller. So, I think it's a small sacrifice to have to remove a few duplicated roads in exchange for county-wide improved accuracy. Karl, I agree #2 could be tricky but I believe it is essential. I don't think you can corrupt someones edits and then say to them sorry, we decided to sacrifice your hard work because we determined that it was for the greater good. I don't see it as corrupting. It's not mangling the mapper's work in any way. If they don't like the new overlapping road, then just delete the TIGER one. It hope that is not the OSM way. My sense is that the OSM way is do it your damn self. Also I despute your statement of a few but notwithstanding that, this is not a numbers game.. I have to admit I don't have hard numbers to back up my statement, but intuitively, how many roads would have been added in 2 years? I'm sure there are some statistics somewhere stating the number of roads in TIGER 2005 (2004?) and in TIGER 2007. That would give a magnitude to the issue. gee officer - I only killed a few people - there are hundreds left is not going to get you very far. Really? You're going to compare overlapping ways with a capital crime? Still, I will say that if the numbers of duplicated way are so few then I think that the person who creates the duplicated ways should also fix them up. Then I'd have no problem with the uploads so long as rules 1 and 3 are also kept. That goes back to *detecting* the overlapping ways. Can be done by JOSM's validator, so maybe there's hope. Maybe it could be done like Dave handled the last import--if anyone is concerned about conflicts, they can handle their county themselves. Karl ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
I do not think there is anything to gain from the counties that are listed to not be improved. My county (orange county fl.) is one that was NOT improved yet, and in QGIS it looks to be unmodified from the original tiger import. There are not any new tags in the data that would add anything either. Orange county Fl. does not have dual carriage ways, but Lake county Fl. which was updated does have them. I have looked at the current county files, and the same is true. So I'm not sure it is the tiger improvement making it dual carriage ways, or the fact that the source county has them as dual to start with. Dale On Tue, Oct 28, 2008 at 6:46 PM, Dave Hansen [EMAIL PROTECTED] wrote: On Tue, 2008-10-28 at 20:39 +1100, Nick Hocking wrote: On a more infalmatory note, I checked with yahoo and google and it seems that in a LOT or areas more than half of the tiger roads don't actually exist. The mappers must have been paid by the road :-) Is tiger 2007 any better in this regard and does it yet have dual carriageways for all the interstates? I don't know. That's why I'm writing the conversion scripts, so I can look at it in my existing OSM tools and see how it renders. If anyone knows, or wants to look into it, please stick whatever you learn on the wiki. I'll make sure to keep checking it. -- Dave ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us -- Dale Puch ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 data
On Sat, 2008-10-25 at 10:37 +1100, Nick Hocking wrote: I'm firmly convinced that automatic uploads should only go into areas where there are NO user edited nodes or ways. Other updates need to be done manully to avoid data corruption. You have absolutely shown a number of cases where there was no merging and the TIGER data was simply splatted over existing data. This is certainly one of the downsides of the approach that was tried before. I uploaded Oregon first because I had already talked to all the mappers in my own state. I then proceeded to upload all the states that were completely empty. After that, I used this map: http://ted.mielczarek.org/code/osm/counties/ and uploaded only counties that were virtually empty. Interestingly, people started contacting me pretty quickly saying you missed my county! That's because the areas with the most data were also the places with the most active mappers! Almost all the prolific mappers knew that they could never compete with the pure amount of TIGER data and went to the heroic effort of merging their existing work with it. In the end, I think there was only a single county (out of ~3000) in the US that didn't get TIGER data in one way shape or form, and I gave people plenty of time to decline. So, I completely disagree that the merging can and should be done manually. There's simply too much data. It's also not feasible to blacklist every county that has *ever* had a single edit. But, I don't want to corrupt anything. Just as before, I'm going to let all the decisions be made by the local mappers. If you're concerned, please stay on this list because I'll always announce things here (at least). -- Dave ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
Ahh but if the tiger data was deleted or modified a comparison to the original would show that, and that way it can be skipped. So either it isn't fixed now, or already has been. Either way I think the new import should default to not disturb the existing edits. Granted even more checking and smarts for the import scripts are needed to detect where someone has mapped what tiger did not have before. In those cases selectively adding tiger tags to the existing data may help, or it may benefit things if it was overlayed and reviewed once anyhow. I guess I would ask, how much work you think it is to fix or manually map the areas that the tiger data was already corrected? VS. fixing potential problems with the new tiger update import. To me the risks are pretty low if something like the above is done. The other problems that will be run into is that roads are always changing. So is the GIS data that is avaliable to import. So the review process is continual even without imports. But with imports, we at least take advantage of some of the corrections nad improvments done for us. The sooner we figure our better ways to import and integrate new data the better. The less people fear imports messing up the contributions, the more they will be willing to do. It absolutely is an important issue. -- Dale Puch On Sat, Oct 25, 2008 at 2:00 AM, Nick Hocking [EMAIL PROTECTED]wrote: Dale, I don't think it's reasonable to expect mappers to continually go over the areas they have contributed to in order to remove obosolete ways bulk-loaded over their own. I think it best to hold off more editing untill the last tiger import is complete. Heck - the 2015 bulk upload may even have dual carriageways for the interstates. I kid you not the Tiger interstates are woefull from a routing perspective. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 data
On Sat, Oct 25, 2008 at 12:37 AM, Nick Hocking [EMAIL PROTECTED]wrote: Again, let's calm down a little bit. Were you around for the last import? Did you see how I handled data conflicts in that one? Was there a problem there that needs fixing this time around? Yes I think there were problems. There have been some diary entries bemoaning the fact that their edits had been blown away by the Tiger data upload. From my own experience I noticed some problems as well. My travels took me through my second most favourite place on earth - Death Valley. When I went to edit in the tracks I noticed that on Badwater road there were two roads (and still are). One edited in by 80n and another uploaded Tiger one and both intertwined. Since then another mapper has made some improvements to the Tiger one although he has not yet removed the reviewed tag. I've also been slack in this respect but am in the process of fixing this. More serious is the situation of Golden Canyon Road. I walked up this washway a small way (as you can see from my gps tracks) but didn't want to repeat the fatal decision of a German Tourist so I quickly went back to the car. This was in mid summer. Google Earth readily shows the correct nature of the car park area and washway status. 80n has this tagged correctly as footway but Tiger declares it highway, residential, a most incorrect tagging, and dangerous as well. Needless to say these two ways are intertwined as well. I haven't looked up the edit dates but I'll take a bet that 80n didn't edit in this footway and not realise that there was an incorrect residential way already mapped. I had already fixed up the other end of this track (at Zabriski Point) but had put the whole area in the too hard basket and decided that when I finally get my tracks all edited in, I would attempt to fix up Death Valley. My work in Death Valley pre-dates the TIGER import by a long way. Golden Canyon is really really not a residential road. We didn't have the highway=road tag back when the original TIGER import was done, but I see many roads in the US tagged as residential that are anything from a freeway to a dirt track and most things inbetween. Perhaps, if any tiger data is re-imported a lot of the roads tagged as highway=residential should be replaced by highway=road, particularly outside large urban areas. This would at least make it easier to see what does or doesn't get touched afterwards. I had also noticed problems in San Francisco, where there were two Lombard Streets on top of each other, one with the squiggly bit straight. Fortunately someone has corrected this. On a side note, I believe that the correct thing to do is to just delete the Tiger reviewed tag rather than turn it to yes. Is this correct? I'm firmly convinced that automatic uploads should only go into areas where there are NO user edited nodes or ways. Other updates need to be done manully to avoid data corruption. Cheers Nick ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
If by not disturbing existing edits you include the overlaying of other data on top of existing ones, then I completely agree. Also there are countless non-existant roads crossing the interstates. These have had to be deleted and I agree, must never come back. And yes I agree that any altered tiger data should be left just as is and no ways added. I don't agree that new tiger data should ever be overlaid. It should replace old unmodified tiger data and should not be added near any other data. This would just repeat the cycle where good data is degraded by the overlay of older info and must then be re-edited. My observation is that OSM mappers produce more accurate and uptodate data than any one else. Therefore if the bulk uploading interferes with OSM data then the quality of the OSM data will be degraded. PS - if you see this line then this post hasn't been truncated ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
Nick Hocking writes: Can you confirm that any bulk upload of Tiger 2007 date will not erase or be overlaid over/under/alongside any existing user edits. I'm not Dave, but I'm quite sure that 1) he won't be smashing any user edits, and 2) I don't support the smashing of my edits either. I've been careful to set true the reviewed attribute on my edits. Have you? Has everyone? If so, then the path is clear: don't overwrite anything that's been reviewed. -- --my blog is athttp://blog.russnelson.com | Unregulation is a slippery Crynwr sells support for free software | PGPok | slope to prosperity and 521 Pleasant Valley Rd. | +1 315-323-1241 | freedom. Potsdam, NY 13676-3213 | Sheepdog | ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
On Fri, Oct 24, 2008 at 9:46 AM, Russ Nelson [EMAIL PROTECTED] wrote: Nick Hocking writes: Can you confirm that any bulk upload of Tiger 2007 date will not erase or be overlaid over/under/alongside any existing user edits. I'm not Dave, but I'm quite sure that 1) he won't be smashing any user edits, and 2) I don't support the smashing of my edits either. I've been careful to set true the reviewed attribute on my edits. Have you? Has everyone? If so, then the path is clear: don't overwrite anything that's been reviewed. I don't agree. If the reviewed flag was changed to yes without any change to the imported data, then that means the new TIGER data will be better than the old (reviewed) data. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
On Fri, 2008-10-24 at 22:09 +1100, Nick Hocking wrote: Can you confirm that any bulk upload of Tiger 2007 date will not erase or be overlaid over/under/alongside any existing user edits. On my last US trip I've got about 6000 miles of gps tracks. I've only edited in a few hundred miles of them so far but am reluctant to do any more work if the Tiger 2007 data will destroy what I've done so far. Dude. Chill. :) I don't know what's going to happen. I don't even really have working data to look at, yet. Let's say there's a county with a single node set to reviewed=yes. the rest of the 4 million nodes in the county aren't set as reviewed. That county has much improved TIGER data. Should we leave it alone? What about 10 nodes? 100? We need to answer those questions at some point. Personally I don't really care where we draw the line. Destroying (or corrupting) existing edits would nullify the current efforts by Cloudmade to enhance a US community (IMHO). Again, let's calm down a little bit. Were you around for the last import? Did you see how I handled data conflicts in that one? Was there a problem there that needs fixing this time around? -- Dave ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 data
Again, let's calm down a little bit. Were you around for the last import? Did you see how I handled data conflicts in that one? Was there a problem there that needs fixing this time around? Yes I think there were problems. There have been some diary entries bemoaning the fact that their edits had been blown away by the Tiger data upload. From my own experience I noticed some problems as well. My travels took me through my second most favourite place on earth - Death Valley. When I went to edit in the tracks I noticed that on Badwater road there were two roads (and still are). One edited in by 80n and another uploaded Tiger one and both intertwined. Since then another mapper has made some improvements to the Tiger one although he has not yet removed the reviewed tag. I've also been slack in this respect but am in the process of fixing this. More serious is the situation of Golden Canyon Road. I walked up this washway a small way (as you can see from my gps tracks) but didn't want to repeat the fatal decision of a German Tourist so I quickly went back to the car. This was in mid summer. Google Earth readily shows the correct nature of the car park area and washway status. 80n has this tagged correctly as footway but Tiger declares it highway, residential, a most incorrect tagging, and dangerous as well. Needless to say these two ways are intertwined as well. I haven't looked up the edit dates but I'll take a bet that 80n didn't edit in this footway and not realise that there was an incorrect residential way already mapped. I had already fixed up the other end of this track (at Zabriski Point) but had put the whole area in the too hard basket and decided that when I finally get my tracks all edited in, I would attempt to fix up Death Valley. I had also noticed problems in San Francisco, where there were two Lombard Streets on top of each other, one with the squiggly bit straight. Fortunately someone has corrected this. On a side note, I believe that the correct thing to do is to just delete the Tiger reviewed tag rather than turn it to yes. Is this correct? I'm firmly convinced that automatic uploads should only go into areas where there are NO user edited nodes or ways. Other updates need to be done manully to avoid data corruption. Cheers Nick ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
I'll try again - last post appreas to have been truncated. Again, let's calm down a little bit. Were you around for the last import? Did you see how I handled data conflicts in that one? Was there a problem there that needs fixing this time around? Yes I think there were problems. There have been some diary entries bemoaning the fact that their edits had been blown away by the Tiger data upload. From my own experience I noticed some problems as well. My travels took me through my second most favourite place on earth - Death Valley. When I went to edit in the tracks I noticed that on Badwater road there were two roads (and still are). One edited in by 80n and another uploaded Tiger one and both intertwined. Since then another mapper has made some improvements to the Tiger one although he has not yet removed the reviewed tag. I've also been slack in this respect but am in the process of fixing this. More serious is the situation of Golden Canyon Road. I walked up this washway a small way (as you can see from my gps tracks) but didn't want to repeat the fatal decision of a German Tourist so I quickly went back to the car. This was in mid summer. Google Earth readily shows the correct nature of the car park area and washway status. 80n has this tagged correctly as footway but Tiger declares it highway, residential, a most incorrect tagging, and dangerous as well. Needless to say these two ways are intertwined as well. I haven't looked up the edit dates but I'll take a bet that 80n didn't edit in this footway and not realise that there was an incorrect residential way already mapped. I had already fixed up the other end of this track (at Zabriski Point) but had put the whole area in the too hard basket and decided that when I finally get my tracks all edited in, I would attempt to fix up Death Valley. I had also noticed problems in San Francisco, where there were two Lombard Streets on top of each other, one with the squiggly bit straight. Fortunately someone has corrected this. On a side note, I believe that the correct thing to do is to just delete the Tiger reviewed tag rather than turn it to yes. Is this correct? I'm firmly convinced that automatic uploads should only go into areas where there are NO user edited nodes or ways. Other updates need to be done manully to avoid data corruption. Cheers Nick ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
Dave Hansen [EMAIL PROTECTED] writes: On Sun, 2008-10-19 at 11:33 -0700, Alan Brown wrote: Then, decide how if/when it is appropriate to write over the old TIGER stuff with new. Or, to merge it somehow. Be very, very careful here. Conflation is a difficult thing. I used to work at Tele Atlas, and there was a major project to conflate Tele Atlas North American data and GDT data (a company they just acquired). They had at least a hundred people committed full time to completing the task in a year (I don't know the exact number), with tens of millions in funding - and they failed in a big way. The head of the North American division got axed as a result. It's not a difficult thing if nothing has been changed since the last import. My goal for now is simply to overwrite the things that haven't been updated by people since the last time I touched them. A good start would certainly be if the data was converted to OSM format and made available somwhere for people to check and compare with existing OSM data. Matthias ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
[Talk-us] Tiger 2007 Data
Has anyone looked at importing the TIGER 2007 data yet? I was going to start coding up the conversion utilities to get started. It appears that this shapefile format may have existing OSM converters out there. Anyone want to admit to having one? ;) -- Dave ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
On Sun, Oct 19, 2008 at 11:14 AM, Dave Hansen [EMAIL PROTECTED] wrote: On Sun, 2008-10-19 at 11:10 -0700, Karl Newman wrote: On Sun, Oct 19, 2008 at 10:54 AM, Dave Hansen [EMAIL PROTECTED] wrote: Has anyone looked at importing the TIGER 2007 data yet? I was going to start coding up the conversion utilities to get started. It appears that this shapefile format may have existing OSM converters out there. Anyone want to admit to having one? ;) -- Dave What were you looking to import? I thought your TIGER import was a one-shot deal. I *wish* it was a 1-shot deal. That darn Steve C. seems to think that maps should be both accurate *and* up to date. The nerve! Anyway, it seems that the new data is substantially better than the old in some places. What I'd like to do is twofold: 1. regenerate .osm files for all the new TIGER data 2. Compare new .osm files to old TIGER data plus stuff edited in OSM Then, decide how if/when it is appropriate to write over the old TIGER stuff with new. Or, to merge it somehow. -- Dave Ah. Quite a trick... I seem to recall that somebody had wanted to do that earlier, but I think they got discouraged by the difficulty. Karl ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
Then, decide how if/when it is appropriate to write over the old TIGER stuff with new. Or, to merge it somehow. Be very, very careful here. Conflation is a difficult thing. I used to work at Tele Atlas, and there was a major project to conflate Tele Atlas North American data and GDT data (a company they just acquired). They had at least a hundred people committed full time to completing the task in a year (I don't know the exact number), with tens of millions in funding - and they failed in a big way. The head of the North American division got axed as a result. If there's a particular layer that has useful info, that's not already present - that could be imported successfully. Otherwise, the most useful thing would be some sort of set-up where someone could view changes to the new TIGER data simultaneously with the OSM data, and choose what to import. Or - if you can determine if a whole region was untouched - replace previously imported data with new data. What you don't want to mess up is the careful work people have done to improve the quality of data in a region manually. If you try to merge data automatically without careful oversight, you could destroy a lot of hard work. A one-time import - to create a starting point for people to edit - makes all the sense in the world. Subsequent updates are a much harder thing. -Alan ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Tiger 2007 Data
On Sun, 2008-10-19 at 11:33 -0700, Alan Brown wrote: Then, decide how if/when it is appropriate to write over the old TIGER stuff with new. Or, to merge it somehow. Be very, very careful here. Conflation is a difficult thing. I used to work at Tele Atlas, and there was a major project to conflate Tele Atlas North American data and GDT data (a company they just acquired). They had at least a hundred people committed full time to completing the task in a year (I don't know the exact number), with tens of millions in funding - and they failed in a big way. The head of the North American division got axed as a result. It's not a difficult thing if nothing has been changed since the last import. My goal for now is simply to overwrite the things that haven't been updated by people since the last time I touched them. -- Dave ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
