Hey Jeff, Timeout for queries is not the same as the timeout for the URLFetch. Since your call is made from within the BQ client lib, it's possible the normal error is mangled up there.
Now it may be a bit cumbersome to write, but maybe using URLFetch directly to call the BQ urls and setting a longer timeout there might solve the issue? At the very least, since the call will be a purely URLFetch one, the error message might contain more information than it currently does. Cheers On Wednesday, September 2, 2015 at 3:59:25 AM UTC-4, Jeff Schnitzer wrote: > > I also expect that it is a URLFetch issue and not a BQ issue. If it is a > timeout, it is not a typical timeout - the error is not > SocketTimeoutException. I'm using the standard Google BQ library with the > UrlFetchTransport. There does not appear to be a general way of setting a > timeout in this library, although I do set a timeout for queries of 30,000 > milliseconds and still see this error. I have not yet tried to debug into > the Google BQ client library to verify whether it actually sets the timeout > value. > > I don't expect that it's a timeout issue - I've seen those, and they are > more obvious than "Could not fetch URL". > > Jeff > > On Tue, Sep 1, 2015 at 6:52 AM, Patrice (Cloud Platform Support) < > pvoutsi...@google.com> wrote: > >> Hi again Jeff, >> >> So, looking back into the issue and your code, the only thing I can think >> of right now is that this may not be a BQ issue, but a URLfetch one. Are >> you sure your URLFetch has a proper timeout setup? This may not be simply >> because your timeout is too short, and sometimes the server takes a little >> bit longer than expected to respond, and that throws everything out. >> >> I don't see in your samples how you setup your timeout for URLFetch. Has >> it been set elsewhere, or do you use the default one? The default being >> only 5 seconds, its very possible this is your issue here. >> >> Let me know if setting your deadline OR your default deadline helps :). >> >> Cheers! >> >> >> On Thursday, August 27, 2015 at 10:29:53 AM UTC-4, Patrice (Cloud >> Platform Support) wrote: >>> >>> Hey Jeff! >>> >>> At least you're not losing data because of this. >>> >>> Normally you can do the same for your queries though. We always suggest >>> to code defensively, just in case there is an intermittent issue, or a >>> network issue, or maintenance on your instance, or any number of things. >>> >>> I'm still trying to understand if what you're seeing is expected or >>> higher than the norm. As soon as I get anything I will let you know. >>> >>> Cheers >>> >>> On Wednesday, August 26, 2015 at 6:17:27 PM UTC-4, Jeff Schnitzer wrote: >>>> >>>> The inserts are all running from a task queue with the default backoff >>>> behavior, so I should be ok there. All the inserts seem to work on retry - >>>> I'm not losing data - I just get errors in my logs and a nastygram from >>>> Rollbar. >>>> >>>> Failing queries are a bigger issue because users see that. >>>> >>>> Thanks, >>>> Jeff >>>> >>>> On Wed, Aug 26, 2015 at 8:25 AM, Patrice (Cloud Platform Support) < >>>> pvoutsi...@google.com> wrote: >>>> >>>>> Hey Jeff, >>>>> >>>>> Thank you for that gist, it definitely contains everything we can >>>>> possibly be needing :). >>>>> >>>>> Just quickly looking at your "Insert All", I think there might be a >>>>> way to completely get out of the "URL not find" bind. I can't help but >>>>> notice you try the insert once, with no fallback. Is there some kind of >>>>> exponential backoff in another layer of your application? Because using >>>>> an >>>>> exponential backoff to that insertAll (and probably to all the other >>>>> calls >>>>> that sometimes throw similar errors) should take you out of this. >>>>> >>>>> I will still investigate to try and see what I can gather as to the >>>>> root cause of this. As soon as I find anything, I'll update the thread >>>>> here. >>>>> >>>>> Cheers! >>>>> >>>>> On Tuesday, August 25, 2015 at 3:52:28 PM UTC-4, Jeff Schnitzer wrote: >>>>>> >>>>>> I put the query stacktrace and code in there, plus a few more >>>>>> timestamps of occurrences that happened today/yesterday. >>>>>> >>>>>> Thanks, >>>>>> Jeff >>>>>> >>>>>> On Tue, Aug 25, 2015 at 12:21 PM, Jeff Schnitzer <j...@infohazard.org >>>>>> > wrote: >>>>>> >>>>>>> gearlaunch-hub >>>>>>> >>>>>>> I'll add some more to the gist. >>>>>>> >>>>>>> Thanks, >>>>>>> Jeff >>>>>>> >>>>>>> On Tue, Aug 25, 2015 at 11:11 AM, Ryan (Cloud Platform Support) < >>>>>>> rbruy...@google.com> wrote: >>>>>>> >>>>>>>> More the better. What is your app id? >>>>>>>> >>>>>>>> On Tuesday, August 25, 2015 at 1:51:43 PM UTC-4, Jeff Schnitzer >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Thanks for looking into this. Here's a gist with some full >>>>>>>>> stacktraces and timestamps, plus the relevant code: >>>>>>>>> >>>>>>>>> https://gist.github.com/stickfigure/03149e24854936de1e1c >>>>>>>>> >>>>>>>>> One is a deferred task, one is a cron, but I also have failures >>>>>>>>> with normal user-facing query requests. I can dig some of those up >>>>>>>>> too if >>>>>>>>> it helps. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jeff >>>>>>>>> >>>>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Ryan (Cloud Platform Support) < >>>>>>>>> rbruy...@google.com> wrote: >>>>>>>>> >>>>>>>>>> Salutations Jeff, >>>>>>>>>> >>>>>>>>>> Can you send me the code you are using and full stack traces with >>>>>>>>>> time stamps (more recent ones the better and multiple ones for >>>>>>>>>> comparison). >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> On Thursday, August 20, 2015 at 12:11:51 PM UTC-4, Jeff Schnitzer >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I'm getting a lot of "Could not fetch URL" errors from BigQuery. >>>>>>>>>>> I'm not driving significant amounts of traffic to it (yet), but I >>>>>>>>>>> get one >>>>>>>>>>> of these maybe every 50 or so queries: >>>>>>>>>>> >>>>>>>>>>> Could not fetch URL: >>>>>>>>>>> https://www.googleapis.com/bigquery/v2/projects/gearlaunch-hub/queries >>>>>>>>>>> >>>>>>>>>>> I'm using the standard google-published bigquery client library >>>>>>>>>>> (Java). Any idea what is wrong? I don't have problems fetching from >>>>>>>>>>> any >>>>>>>>>>> other service. >>>>>>>>>>> >>>>>>>>>>> I'm ramping up traffic next week and this is going to start >>>>>>>>>>> happening a lot.... >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jeff >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Google App Engine" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to google-appengine+unsubscr...@googlegroups.com. >>>>>>>>>> To post to this group, send email to >>>>>>>>>> google-appengine@googlegroups.com. >>>>>>>>>> Visit this group at >>>>>>>>>> http://groups.google.com/group/google-appengine. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/a15cec2d-a66d-496b-9f99-91b5b3829322%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/a15cec2d-a66d-496b-9f99-91b5b3829322%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Google App Engine" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to google-appengine+unsubscr...@googlegroups.com. >>>>>>>> To post to this group, send email to >>>>>>>> google-appengine@googlegroups.com. >>>>>>>> Visit this group at http://groups.google.com/group/google-appengine >>>>>>>> . >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/google-appengine/7ff2bc22-d8bf-4795-9a08-44b7e83a0c56%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/google-appengine/7ff2bc22-d8bf-4795-9a08-44b7e83a0c56%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to google-appengine+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to google-appengine@googlegroups.com >>>>> . >>>>> Visit this group at http://groups.google.com/group/google-appengine. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/google-appengine/403de718-0190-425a-8d5c-cca90325e4bc%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/google-appengine/403de718-0190-425a-8d5c-cca90325e4bc%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f222d461-226e-49fb-9602-44b775f70eae%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
