Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
Hi Am 07.10.2013 18:32, schrieb Roland Olbricht: > Dear Bryce, > > I've cross-checked the Rambler instance. I'm sorry it indeed doesn't work on > that instance. It looks like an element on the network, most likely Nginx, > disconnects any connection if no data is sent for ten minutes. > > A similar thing happened on overpass-api.de after 20 to 30 minutes. In that > case, that Apache log indicates that the client has aborted the connection, > but the client still waits. Looks like again something on the network did > break the inactive connection. Hum, thet bweggs for some pull API where you can submit your query with an email adresse supplied and the API will send you an email, when you're query has finished. It could send you a 24h valid link to your data with it. Regards, Peter ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
brycenesbitt wrote > On Sun, Oct 6, 2013 at 11:41 AM, Paul Norman < > penorman@ > > wrote: > >> http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water]took >> about 3 minutes total. 14.5MB result. >> > > Thanks, however no luck here: > > # wget > http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water] > -O > drinking_water_yes_d.osm > --2013-10-06 16:54:28-- > http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water] > Resolving jxapi.openstreetmap.org... 128.40.168.100 > Connecting to jxapi.openstreetmap.org|128.40.168.100|:80... connected. > HTTP request sent, awaiting response... 500 INTERNAL SERVER ERROR > 2013-10-06 16:57:28 ERROR 500: INTERNAL SERVER ERROR. > > Also my followup queries required Overpass, I'd rather draw from one > source. Looks like the 500 there was because the query timed out. I've got a fairly aggressive timeout (5 minutes) because people tend to send some pretty crazy queries to jxapi.osm.org. I can bump that up though if I write some scraper-defense code. What followup queries do you have that require Overpass? -- View this message in context: http://gis.19327.n5.nabble.com/Overpass-or-XAPI-API-servers-for-60-000-node-queries-tp5780374p5780564.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
On Mon, Oct 7, 2013 at 9:32 AM, Roland Olbricht wrote: > Dear Bryce, > > I've cross-checked the Rambler instance. I'm sorry it indeed doesn't work > on > that instance. It looks like an element on the network, most likely Nginx, > disconnects any connection if no data is sent for ten minutes. And if were not Nginx, it is possible something else in the chain would close the dang connection. Perhaps I could get the client to send some useless kepalive packets (e.g. spurious newlines). ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
Dear Bryce, I've cross-checked the Rambler instance. I'm sorry it indeed doesn't work on that instance. It looks like an element on the network, most likely Nginx, disconnects any connection if no data is sent for ten minutes. A similar thing happened on overpass-api.de after 20 to 30 minutes. In that case, that Apache log indicates that the client has aborted the connection, but the client still waits. Looks like again something on the network did break the inactive connection. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
On Sun, Oct 6, 2013 at 11:15 AM, Roland Olbricht wrote: > > Please do not use a cancel-restart strategy. This may double the load > because > not in all cases the Apache server cancels the abandoned query. > Sadly, it's all that I can get to work. I suggest the following strategy: Please prefer the Rambler instance. It is > for this use case probably the fastest. Then, feel free to use a much > larger > timeout... On the other hand, > please omit the element-limit... I've tried all that, and get the same result. The usual case is a hung wget with no timeout evident. If the server is timing out, the http connection never gets the message. The rambler instance has returned results in the smallest number of cases tested. Here's an example query (and note the simple version without the does the same thing). A small bounding box version of this query works fine on all servers. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
On Sun, Oct 6, 2013 at 11:41 AM, Paul Norman wrote: > http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water]took > about 3 minutes total. 14.5MB result. > Thanks, however no luck here: # wget http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water] -O drinking_water_yes_d.osm --2013-10-06 16:54:28-- http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water] Resolving jxapi.openstreetmap.org... 128.40.168.100 Connecting to jxapi.openstreetmap.org|128.40.168.100|:80... connected. HTTP request sent, awaiting response... 500 INTERNAL SERVER ERROR 2013-10-06 16:57:28 ERROR 500: INTERNAL SERVER ERROR. Also my followup queries required Overpass, I'd rather draw from one source. - Separately, there seem to be a number of slightly different lists of public XAPI/Overpass servers: https://wiki.openstreetmap.org/wiki/Jxapi#Overpass_API https://wiki.openstreetmap.org/wiki/Overpass_API http://wiki.openstreetmap.org/wiki/Overpass_API/status http://wiki.openstreetmap.org/wiki/Pt-br:Xapi Where would a good place be to make a single consolidated (and up to date) server status list, if it's not already out there? ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
http://jxapi.openstreetmap.org/xapi/api/0.6/node[amenity=drinking_water] took about 3 minutes total. 14.5MB result. From: Bryce Nesbitt [mailto:bry...@obviously.com] Sent: Sunday, October 06, 2013 9:53 AM To: talk@openstreetmap.org Subject: [OSM-talk] Overpass or XAPI API servers for 60,000 node queries? In the past I ran this same query via XAPI on mapquest, but that server has bit-rotted into uselessness. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
> In the past I could generally get the query to complete by running it once, > cancelling it, then running it a second time (maybe the 2nd time more data > was cached). Now I get either infinite timeout or a server rejection. > > The query is: > > > > > > > I'm glad that you bring up the whole subject. It's a challenging query for Overpass API. Malcolm has already sketched the long-term solution: operating from a SSD will largely solve the performance limitations, but currently I'm not aware of any cost-effective hosting options with large SSDs (more than 300 GB). Please do not use a cancel-restart strategy. This may double the load because not in all cases the Apache server cancels the abandoned query. It is possible that this strategy causes the rejection because the server rejects concurrent requests from the same IP. In fact, if a query is already running a second query from the same IP then the new request is first postponed for up to 15 seconds. It is immediately started if the former request completes during this time. If the former request doesn't complete within 15 seconds then the newer request is finally rejected. I suggest the following strategy: Please prefer the Rambler instance. It is for this use case probably the fastest. Then, feel free to use a much larger timeout. On a test run, the query had taken 12 minutes, so with a safety margin factor, please try 3600 seconds instead of 990. On the other hand, please omit the element-limit (and thus use the default of 512M instead). That's much more than necessary and the server is rather picky on RAM consuming requests. If the request is still rejected, please retry every few minutes. Peaks most often only last for less than an hour but aren't that regular. And trying a request doesn't hurt anybody. Cheers, Roland ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
Per the suggestion, I dropped the query time to 200 seconds, and same result from the public servers. The query is still running after several minutes, despite the timeout value of 200 seconds. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
I ran your query on my server and it completed in 90 seconds, even though another big query was already running. True, I do use an SSD for the database, so it is very fast. However, it proved that there is nothing in your query that gives Overpass indigestion. The public servers do run a fairness algorithm to prevent big queries from shutting out others. You might try reducing your timeout value as asking for such a long time will likely get the request bounced. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
