Re: [Talk-us] Road Expansions and Seattle Imports (if you're the Seattle importer, please read!)
On Sat, May 19, 2012 at 1:31 PM, Mike N nice...@att.net wrote: On 5/19/2012 12:52 PM, Serge Wroclawski wrote: Ultimately, the only way to get a 100% successful upload is to query the changeset on failure to determine what was actually uploaded, then resume from that point. While true, that method in isolatiuon is a bit complex and should be unnecessary. The OSC upload method is transactional in nature. If you get a positive response from it, you can know the upload succeeded. The issue is that there can be a network interruption between the upload completion and the return response. During that time the transaction has completed successfully but the client won't know. If there is a 1:1 correlation between OSC and changeset, that's fine, but generally uploads are often done with large chagesets. There are a few ways to handle this (including async upload methods in the API code, which has been proposed), but right now, AFAIK, there's no public code for making reliable mass uploads to OSM. Assistance requested! I'll post something when it's more ready, but if you're a python programmer, I'd like your help. - Serge ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Road Expansions and Seattle Imports (if you're the Seattle importer, please read!)
From: Serge Wroclawski [mailto:emac...@gmail.com] Seattle importer, please read!) Getting back to the current TIGER expansion, I decided it would be a mistake to try a large scale import without having some way of ensuring that the uploads were being done correctly, and so I've been working on a queue based uploader. It will take these issues into account, handle retries, etc. Not to discourage you from the queue based uploader which would be generally useful, is it necessary in this case? When you have an interrupted import you don't know what you have to still upload without some work. In the name expansion case, couldn't you generate a new extract and run the code again? Many bulk-modification and bulk-deletion process can be run safely multiple times and I imagine this one could too. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Road Expansions and Seattle Imports (if you're the Seattle importer, please read!)
On Sat, May 19, 2012 at 4:26 PM, Paul Norman penor...@mac.com wrote: From: Serge Wroclawski [mailto:emac...@gmail.com] Seattle importer, please read!) Getting back to the current TIGER expansion, I decided it would be a mistake to try a large scale import without having some way of ensuring that the uploads were being done correctly, and so I've been working on a queue based uploader. It will take these issues into account, handle retries, etc. Not to discourage you from the queue based uploader which would be generally useful, is it necessary in this case? If not now, then when? There's no upload process that I know of which runs reliably and safely. That's what I'm trying to build. If one exists, please tell me about it; in the meantime, I'll continue work on this one. - Serge ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Road Expansions and Seattle Imports (if you're the Seattle importer, please read!)
On 19 May 2012 21:02, Serge Wroclawski emac...@gmail.com wrote: On Sat, May 19, 2012 at 1:31 PM, Mike N nice...@att.net wrote: On 5/19/2012 12:52 PM, Serge Wroclawski wrote: Ultimately, the only way to get a 100% successful upload is to query the changeset on failure to determine what was actually uploaded, then resume from that point. While true, that method in isolatiuon is a bit complex and should be unnecessary. The OSC upload method is transactional in nature. If you get a positive response from it, you can know the upload succeeded. The issue is that there can be a network interruption between the upload completion and the return response. During that time the transaction has completed successfully but the client won't know. If there is a 1:1 correlation between OSC and changeset, that's fine, but generally uploads are often done with large chagesets. There are a few ways to handle this (including async upload methods in the API code, which has been proposed), but right now, AFAIK, there's no public code for making reliable mass uploads to OSM. As mentioned on irc, I trust the tool set I did the original TIGER expansion with. I've used it for many other uploads too and I never had to revert a single upload, i.e. it was always possible to resume the process in case of a network error or conflict. I've heard of two or three more people who used it successfully. It also (tries to) solve the problem of nodes being uploaded in bulk before ways get uploaded, before relations, so this minimizes the probability of someone spotting orphan nodes in the middle of an import and removing them. Unfortunately it's not very user friendly and requires manual intervention sometimes. I'd try improving it instead of starting from the beginning though. It's documented here: http://wiki.openstreetmap.org/wiki/Upload.py It is also limited to xml files that fit in memory because it uses the python ElementTree parser right now. Cheers ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us