Translated: "URLFetch is rock-solid, except because it uses a shared IP pool it will erratically fail if you use it to fetch from almost any third-party service that pays attention to load." Which really isn't very rock-solid at all.
The shared IP pool is a significant problem with URLFetch, and you really need to be careful when using it. The standard workaround is to set up your own proxy servers elsewhere on the net - a PITA but not optional for many services. Here's the issue to star to hopefully get Google to do something about the issue: http://code.google.com/p/googleappengine/issues/detail?id=6644 FWIW, I've also found that URLFetch is occasionally less than snappy. But there are a lot of moving parts involved so it's hard to figure out exactly where to lay blame. One thing to watch out for is that the default URLFetch timeout is fairly short. I usually find it necessary to increase the timeout, especially with services with erratic performance (eg Facebook). Jeff On Thu, Oct 25, 2012 at 1:29 PM, Vinny P <vinny...@gmail.com> wrote: > In my experience, the reliability of URLFetch is rock-solid. The problem is > the external server that you're connecting to. The external server can be > relatively fast to respond (many web APIs, such as Google's goo.gl shortener > and so forth) or relatively slow and error-prone (the Reddit API in > particular is just absolutely terrible to access from AppEngine; since all > GAE urlfetches come from the same pool of IPs, the Reddit servers > deliberately throttle requests because they think all of the requests are > coming from a single poorly-behaved app, not multiple apps. And no, the > reddit api does not offer oauth or similar authentication). > > There's a couple of ways to mitigate this; you can use task queues to keep > retrying a urlfetch, backends to continuously urlfetch and cache the > results, find a different 3rd party service, etc. > > -Vinny P > > > On Thursday, October 25, 2012 11:58:26 AM UTC-5, Joshua Smith wrote: >> >> I use the python version, and get a couple failures a day. The easy answer >> is to treat it just like mail: always use a task, so that if it fails, it >> will retry. >> >> On Oct 25, 2012, at 12:12 PM, Deepak Singh <deepaks...@gmail.com> wrote: >> >> Hi Alll, >> >> I want to discuss here your experience about GAE Java URLFetchService. >> >> We are using async feature of this service to retrieve data from 3rd party >> servers and our business mainly depends on the data received from their >> servers. >> I observe that UrlFetch fails many times with java.io exception and thus >> we lose our business. >> >> So i would like to know your experience about its reliability, >> DeadlineExceededException cases, ways to handle it and all. >> >> Let us know how reliable is URLFetchService(GAE Java) ? >> >> Regards >> Deepak Singh >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-a...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengi...@googlegroups.com. >> >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/RgAEOStwEtMJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
