Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
I'd also like to add that people get more excited about OpenStreetMap when they see their changes instantly added. I've trained people in both Potlatch, Potlatch2 and JOSM. I pick the tool depending on specific class. Areas with bad/no internet access we use JOSM and changes are immediately seen (people never seem that excited). -Kate On Jun 6, 2011, at 12:41 AM, Serge Wroclawski wrote: On Sun, Jun 5, 2011 at 6:34 PM, Stephan Knauss o...@stephans-server.de wrote: On 05.06.2011 22:18, Serge Wroclawski wrote: For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. When I've done this kind of training, it's been for a disaster, and we need the real data, and the real api. In what way does the dev API differ from the real one that is affecting your training? Or is it just that there is not enough data available? You can upload an extract of the area you are using into dev in advance to have data to play with. Maybe my comment wasn't clear. When I've done training for OSM, it's been in the context of a crisis event when we had 20-40 people using areal imagery to examine an area and map it. They would classify roads, detect destroyed buildings, identify tent cities, etc. The point wasn't just to train them in some abstract way, but to make real change. - Serge ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
2011/6/5 Serge Wroclawski emac...@gmail.com: On Sun, Jun 5, 2011 at 6:34 PM, Stephan Knauss o...@stephans-server.de wrote: On 05.06.2011 22:18, Serge Wroclawski wrote: For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. When I've done this kind of training, it's been for a disaster, and we need the real data, and the real api. In what way does the dev API differ from the real one that is affecting your training? Or is it just that there is not enough data available? You can upload an extract of the area you are using into dev in advance to have data to play with. Maybe my comment wasn't clear. When I've done training for OSM, it's been in the context of a crisis event when we had 20-40 people using areal imagery to examine an area and map it. They would classify roads, detect destroyed buildings, identify tent cities, etc. The point wasn't just to train them in some abstract way, but to make real change. Same for me, just not in real crisis situation. It is big difference if people are just drawing points and lines or do real mapping. It is also about being a bit viral - students can show then the updated map to their friends: look, I did it myself. We have tried with dev things, but for most motivated people it is needed just for the first 10 minutes. Also we have always started with P2, JOSM is too scary for the first introduction. So offline OSM files is not an option. -- Jaak ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 6 June 2011 17:55, Jaak Laineste jaak.laine...@gmail.com wrote: Also we have always started with P2, JOSM is too scary for the first introduction. So offline OSM files is not an option. I keep hearing this, but I must be weird, because I had the opposite reaction both when I first started and when I show somebody how to use it. I took one look at Potlatch and thought I want something that works offline to test with, with a save button when I'm happy. I'm much happier teaching with JOSM than P2. I didn't use Potlatch for at least a year after I started. Stephen ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
Stephen Hope writes: I keep hearing this, but I must be weird, because I had the opposite reaction both when I first started and when I show somebody how to use it. I took one look at Potlatch and thought I want something that works offline to test with, with a save button when I'm happy. I'm much happier teaching with JOSM than P2. I didn't use Potlatch for at least a year after I started. I always started people off with JOSM, back when it was my job. It's faster and more productive teaching people when you don't have to wait for uncertain bandwidth to download your data. And I don't like how everything is bound to the left button. That said, JOSM has many usability issues. I keep threatening to rewrite it in a modern progamming language like Python, but it's got HUGE functionality behind those usability problems. -- --my blog is athttp://blog.russnelson.com Crynwr supports open source software 521 Pleasant Valley Rd. | +1 315-600-8815 Potsdam, NY 13676-3213 | Sheepdog ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
Hi, this reminded me an issue what I've had twice - I had made two sessions of hands-on trainings for OSM, with about 30-40 computers in a classroom. In both cases when they started to do actual data download/editing, then last ones were rejected from API calls. Perhaps just as we were under same IP address. You can imagine that having this technical issue in front of the all the leading geography teachers of our country can be quite embarrassing. Is there a procedure or contact to register an IP for temporary higher load for such cases? Jaak 2011/6/2 Grant Slater openstreet...@firefishy.com: On 2 June 2011 03:42, David Murn da...@incanberra.com.au wrote: On Wed, 2011-06-01 at 22:18 +0100, Tom Hughes wrote: Yes, you got blocked on 16th May by the looks of it. I don't think I did it so it was probably one of my colleagues. Is there not some sort of audit trail or changelog for when users get blocked? Yes, there is an internal sysadmin audit trail. I think it would be useful if one could find out if theyre blocked or if there is some other server problem, without having to ask on the mailing list. API... HTTP Error 403 - Forbidden connection have been blocked. This is used on mass downloaders who degrade the API for mappers. [*] HTTP Error 509 - Bandwidth Limit Exceeded requested excessive amounts of data in a short period. Automatically removed after dropped below threshold. API + WWW served an average of 2529965 requests per day for May 2011. Tiles downloads are auto-magically slowed down after reaching a high threshold. Number of IP addresses currently being slowed down: http://munin.openstreetmap.org/openstreetmap/konqi.openstreetmap/squid_delay_pools.html Tile downloads are normally in excess of 1000 per second. Suggestions, work-arounds? Use the full history dump, as somebody has already suggested. Also, you can use the daily/minutely diffs if you want to keep up with 'real-time' map edits. Yes. The replication diffs contain all the changes applied for the period. *: There is also an ancient release of JOSM blocked which flips lat/lon coordinates. Regards Grant ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Jaak Laineste ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 05.06.2011 20:46, Jaak Laineste wrote: this reminded me an issue what I've had twice - I had made two sessions of hands-on trainings for OSM, with about 30-40 computers in [...] Is there a procedure or contact to register an IP for temporary higher load for such cases? You could use offline data and load this into JSOM. To rely on online ressources for a training is always risky. For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. Stephan ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On Sun, Jun 5, 2011 at 3:31 PM, Stephan Knauss o...@stephans-server.de wrote: On 05.06.2011 20:46, Jaak Laineste wrote: Is there a procedure or contact to register an IP for temporary higher load for such cases? You could use offline data and load this into JSOM. To rely on online ressources for a training is always risky. In my experience, using Potlatch is easier. People can use their own computers and they already have Flash, so we can begin immediately.[1] For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. When I've done this kind of training, it's been for a disaster, and we need the real data, and the real api. - Serge [1] For the few times I encountered a Free Software person who wouldn't use Flash, I told them the PL source was free, and directed them to use Josm. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 05.06.2011 22:18, Serge Wroclawski wrote: For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. When I've done this kind of training, it's been for a disaster, and we need the real data, and the real api. In what way does the dev API differ from the real one that is affecting your training? Or is it just that there is not enough data available? You can upload an extract of the area you are using into dev in advance to have data to play with. Stephan ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On Sun, Jun 5, 2011 at 6:34 PM, Stephan Knauss o...@stephans-server.de wrote: On 05.06.2011 22:18, Serge Wroclawski wrote: For doing test edits: Why not use the dev api? Then you won't have to worry about uploads breaking something. When I've done this kind of training, it's been for a disaster, and we need the real data, and the real api. In what way does the dev API differ from the real one that is affecting your training? Or is it just that there is not enough data available? You can upload an extract of the area you are using into dev in advance to have data to play with. Maybe my comment wasn't clear. When I've done training for OSM, it's been in the context of a crisis event when we had 20-40 people using areal imagery to examine an area and map it. They would classify roads, detect destroyed buildings, identify tent cities, etc. The point wasn't just to train them in some abstract way, but to make real change. - Serge ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 2 June 2011 03:42, David Murn da...@incanberra.com.au wrote: On Wed, 2011-06-01 at 22:18 +0100, Tom Hughes wrote: Yes, you got blocked on 16th May by the looks of it. I don't think I did it so it was probably one of my colleagues. Is there not some sort of audit trail or changelog for when users get blocked? Yes, there is an internal sysadmin audit trail. I think it would be useful if one could find out if theyre blocked or if there is some other server problem, without having to ask on the mailing list. API... HTTP Error 403 - Forbidden connection have been blocked. This is used on mass downloaders who degrade the API for mappers. [*] HTTP Error 509 - Bandwidth Limit Exceeded requested excessive amounts of data in a short period. Automatically removed after dropped below threshold. API + WWW served an average of 2529965 requests per day for May 2011. Tiles downloads are auto-magically slowed down after reaching a high threshold. Number of IP addresses currently being slowed down: http://munin.openstreetmap.org/openstreetmap/konqi.openstreetmap/squid_delay_pools.html Tile downloads are normally in excess of 1000 per second. Suggestions, work-arounds? Use the full history dump, as somebody has already suggested. Also, you can use the daily/minutely diffs if you want to keep up with 'real-time' map edits. Yes. The replication diffs contain all the changes applied for the period. *: There is also an ancient release of JOSM blocked which flips lat/lon coordinates. Regards Grant ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
Hi all, Quick question regarding the OSM API. I have a server setup for my OSM research - it has a very fast Internet link. My personal, home, broadband is much slower and limited to 5Gb per month. Anyway during April I downloaded the history files (in OSM-XML) for a couple of thousand ways. I just dumped wget commands into a bash script and let it chug away. The total size of all the XML files is about 1200 Mb. Anyway wanted to download an addition 2,000 ways (their full history) from the same machine today - example wget http://www.openstreetmap.org/api/0.6/way/2048/history -O ./world-xml/history-2048.xml I keep getting an ERROR 303 .. The error message doesn't make sense to me. However when I try this command from my machine at home - it downloads no problem. Have I found myself on a banned list? It would not be my intention to hog resources on anyone else - but I didn't think the requests were excessive. It took a couple of days to download the history files - i put an hour pause into the bash script after every 500 files downloaded. Suggestions, work-arounds? Thanks, Peter ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On Wed, Jun 1, 2011 at 4:11 PM, Peter Mooney petermoone...@gmail.com wrote: Hi all, Quick question regarding the OSM API. [ ... ] Have I found myself on a banned list? It would not be my intention to hog resources on anyone else - but I didn't think the requests were excessive. Perhaps you have been blocked / throttled. You should consider using the complete history planet file. It is served from another machine without loading the editing API. http://planet.openstreetmap.org/full-experimental/ You'll have to download more initially, but if you find you want to experiment with more data, you already have it. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 11-06-01 10:11 PM, Peter Mooney wrote: Hi all, Quick question regarding the OSM API. I have a server setup for my OSM research - it has a very fast Internet link. My personal, home, broadband is much slower and limited to 5Gb per month. Anyway during April I downloaded the history files (in OSM-XML) for a couple of thousand ways. I just dumped wget commands into a bash script and let it chug away. The total size of all the XML files is about 1200 Mb. Anyway wanted to download an addition 2,000 ways (their full history) from the same machine today - example wget http://www.openstreetmap.org/api/0.6/way/2048/history -O ./world-xml/history-2048.xml I keep getting an ERROR 303 .. The error message doesn't make sense to me. However when I try this command from my machine at home - it downloads no problem. Have I found myself on a banned list? It would not be my intention to hog resources on anyone else - but I didn't think the requests were excessive. It took a couple of days to download the history files - i put an hour pause into the bash script after every 500 files downloaded. Suggestions, work-arounds? Thanks, Peter Hi Peter, When you've been downloading too much data, you'll get a 509 error. See here: http://wiki.openstreetmap.org/wiki/API_v0.6#Error_codes The download limit is unknown, but it's far, far more than the average user. A while back I got this error while I was downloading many sections of urbanized areas while importing data. Using a dump would not be possible, since I would miss my (and other's) previous uploads, made about an hour before. I even noticed that using the data feature on the main OSM site was not working anymore. I was still able to upload data though. For your intention, the suggestion proposed by Richard is the way to go. The API is not particularly well suited to obtain the history from. You'll often get errors (something like: it took the API too long to retrieve the history data) when getting histories of large objects, and/or objects with long histories. Frank ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 01/06/11 21:11, Peter Mooney wrote: wanted to download an addition 2,000 ways (their full history) from the same machine today - example wget http://www.openstreetmap.org/api/0.6/way/2048/history -O ./world-xml/history-2048.xml I keep getting an ERROR 303 .. The error message doesn't make sense to me. Actually you are getting 403 Forbidden, at least for some of your requests. Many of them will timeout. Have I found myself on a banned list? It would not be my intention to hog resources on anyone else - but I didn't think the requests were excessive. It took a couple of days to download the history files - i put an hour pause into the bash script after every 500 files downloaded. Yes, you got blocked on 16th May by the looks of it. I don't think I did it so it was probably one of my colleagues. Scraping bulk history data on the sort of scale you are talking about certainly is viewed as excessive by us - the API is designed to support editors not bulk exporting of data. Suggestions, work-arounds? Use the full history dump, as somebody has already suggested. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On 01/06/11 21:48, Frank Steggink wrote: When you've been downloading too much data, you'll get a 509 error. See here: http://wiki.openstreetmap.org/wiki/API_v0.6#Error_codes That's only for the automated limits, and they currently only apply to the map call. In this case the user was making history calls and got manually blocked from the whole site. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Bandwidth limit/IP blocking - Error 303 on the OSM API?
On Wed, 2011-06-01 at 22:18 +0100, Tom Hughes wrote: Yes, you got blocked on 16th May by the looks of it. I don't think I did it so it was probably one of my colleagues. Is there not some sort of audit trail or changelog for when users get blocked? I think it would be useful if one could find out if theyre blocked or if there is some other server problem, without having to ask on the mailing list. Suggestions, work-arounds? Use the full history dump, as somebody has already suggested. Also, you can use the daily/minutely diffs if you want to keep up with 'real-time' map edits. David ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
