[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

FWIW should note that my app is master/slave.  The pressure to move to 
HRD/Python 2.7 is very real, but right now I have too many concerns with 
replication delays and reading others' migration headaches with data 
volumes similar to mine, so I have no short-term plans to migrate.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/FddU5krbHIsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

I've been able to nearly solve the delay problem by setting countdown=1 in 
the Task constructor. This reduces the delay from 20 to about 1.5. Not sure 
why it's not closer to 1.0 but this will be fine. The time to serve the 
simple request is unaffected.

Still a strange bug (?).

Also, some other things lead me to believe the GAE just wants to "rest" for 
20 seconds. I started to wonder if in my previous tests I was just running 
things through too fast, even though that seemed unlikely since I was 
seeing the same delays in production when tasks take 1+ minutes to 
complete. But I put a time.sleep(10) in my handler just to make sure. 
Strangely, what this did was reduce the delay from 20 seconds down to 10. 
So the sleep gives 10 seconds of rest and then GAE still wants 10 more? 
Something like that. When I replaced time.sleep(10) with a tight loop 
lasting 10 seconds, my higher delays returned although they were more 
erratic (i.e. not consistently near 20 seconds).

Also, I tried creating a new app containing only the code I had written for 
my test, and got somewhat different results. I got the 
almost-exactly-20-seconds delay, but only about once every 4-5 requests. 
The other times it was near zero. Maybe this is because nothing else is 
going on in the app, so GAE has less need for "rest?" Almost as if it were 
a quota thing -- except the fact that I demonstrated earlier that I can 
eliminate the delay entirely by running tasks on a frontend, or using a 
5-instance backend, or (as now demonstrated) setting countdown=1 seems to 
eliminate that possibility.

All very strange.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/wYRmOXSR5-wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[pdknsk]

I had the same problem, with often quite exact 20 seconds delays. More
details in this thread.

http://groups.google.com/group/google-appengine/browse_thread/thread/e5588268dff9b97a

In another configuration, the delay increased to several minutes.
Please star this bug.

http://code.google.com/p/googleappengine/issues/detail?id=6022

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

Interesting. I saw your thread but wasn't entirely sure if it was the
issue. I think the thing that threw me off was that your delays were
being reflected in the request ms in the logs, while in my case they
mostly aren't.

Does setting the task countdown work for you? Or is ~1 second delay
still too much in your case?

On Jan 29, 10:24 am, pdknsk  wrote:
> I had the same problem, with often quite exact 20 seconds delays. More
> details in this thread.
>
> http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
> In another configuration, the delay increased to several minutes.
> Please star this bug.
>
> http://code.google.com/p/googleappengine/issues/detail?id=6022

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[pdknsk]

I haven't tried setting countdown. It's too slow for the mail queue
anyway. It might work on the other backend (mentioned in the bug).
I've moved most code away from backends though.
> *- After first minute of the batch run, Task Queue Details page
> consistently shows 6-8 tasks run in last minute even though logs show only
> 2-3 requests served per minute (no errors or anything abnormal shown in the
> logs either). I know for certain the only tasks running in the queue in
> question are from my tests. Why is the number overstated, and why does it
> only happen in the single-instance configuration?*

I've noticed sth. else, with might be related. When you stop a
backend, and put tasks on it, the tasks will obviously not run. In the
task queue overview in the dashboard however, tasks are reported to
have run in the last minute.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

And here is backends.yaml:

backends:
- name: overnight-external-data
  class: B1
  options: dynamic
  instances: 1

and queue.yaml:

queue:
- name: overnight-tasks
  rate: 50/s
  bucket_size: 50
  retry_parameters:
max_backoff_seconds: 1800

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/nIP2SeQDMUsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

Finally, it's probably an important clue that when I explicitly set 
countdown=1 when creating the task, the delay in executing the task is 
always almost exactly 1.5 seconds (not sure why it's not 1.0). If I don't 
set a countdown value, it's almost as if I had set countdown=20. Except 
that the ETA in the Task Queue Details page in the admin console reflects 
the approximate time the task was created, rather than adding 20 seconds to 
that.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/sbd5Clqozg8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Carter]

We regularly but erratically see 10-20 minute delays in running push
queue tasks.
The tasks sit in the queue with ETA as high as 20 minutes *ago*
without any errors or retries.

(the problem seems unrelated to queue settings since our Maximum Rate,
Enorced Rate and Maximum Concurrent all far exceed the queue's
throughput at the time of the delays)

Any tips or clues on how to prevent this while still using push queues
without backends?


On Feb 1, 9:03 pm, Robert Kluin  wrote:
> Hey Dave,
>   Hopefully Nick will be able to offer some insight into the cause of
> your issues.  I'd guess it is something related to having very few
> tasks (one) in thequeue, and it not getting scheduled rapidly.
>
>   In your case, you could use pull queues to immediately fetch the
> nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> and do the work in parallel.  Basically you'd have a backend that ran
> a loop (possibly initiated via a pushtask) that would lease atask,
> or tasks, from the pullqueue, do the work, delete those tasks, then
> repeat from the lease stage.  The cool thing is that if you're, for
> example, using URL Fetch to pull data  this might let you do it in
> parallel without increasing your costs much (if any).
>
> Robert
>
>
>
>
>
>
>
> On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> > Here are logs from three consecutivetaskexecutions over the past weekend,
> > with only identifying information removed. You'll see that eachtask
> > completes in a few milliseconds, but are 20 seconds apart (remember: I've
> > already checked myqueueconfigurations, nothing else is running on this
> > backend, and I later solved the problem by setting countdown=1 when adding
> > thetask).  I don't see any pending latency mentioned.
>
> > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> > task_name=15804554889304913211 instance=0
> > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0
> > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
> > instance=0
> > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0
> > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
> > instance=0
>
> > The 20 seconds seems to happen regardless of length oftask. Even though my
> > tasks mostly complete in a couple minutes, I do have cases where they take
> > several minutes, and I don't see a difference. Of course, when atasktakes
> > 5-10 minutes to complete, I'm going to notice and care about a 20-second
> >delaymuch less than when I'm trying to spin through a few tasks in a minute
> > (which is a real-world need for me as well).
>
> > When reading up on pull queues a while back, I was a little confused about
> > where I would use them with my own backends. I definitely could see an
> > application for offloading work to an AWS Linux instance. But in either
> > case, could you explain why it might help?
>
> > I saw you mention in a separate thread how M/S can perform differently from
> > HRD even in cases where one wouldn't expect to see a difference. When I get
> > around to it I'm going to create a tiny HRD app and run the same tests
> > through that.
>
> > I also wonder if M/S could be responsible for frequent latencies in my admin
> > console. Those have gotten more frequent and annoying the past couple of
> > months ...
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

In my case, since I was getting the 20-second delay almost 100% of the
time, setting countdown=1 was the answer.  If you only see it happen
every 20 or more request then of course it won't help.

In my case I also run all tasks on the backend. They're slightly more
expensive per hour than frontends (due merely to the lower number of
free hours) but in my case I more than make up for it with the fact
that I have full control on the number of requests that will spin up,
and I need to be able to control that number separately for tasks vs.
users hitting my site.

On Feb 5, 11:24 am, Carter  wrote:
> We regularly but erratically see 10-20 minute delays in running push
> queue tasks.
> The tasks sit in the queue with ETA as high as 20 minutes *ago*
> without any errors or retries.
>
> (the problem seems unrelated to queue settings since our Maximum Rate,
> Enorced Rate and Maximum Concurrent all far exceed the queue's
> throughput at the time of the delays)
>
> Any tips or clues on how to prevent this while still using push queues
> without backends?
>
> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
>
>
>
>
>
>
> > Hey Dave,
> >   Hopefully Nick will be able to offer some insight into the cause of
> > your issues.  I'd guess it is something related to having very few
> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>
> >   In your case, you could use pull queues to immediately fetch the
> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> > and do the work in parallel.  Basically you'd have a backend that ran
> > a loop (possibly initiated via a pushtask) that would lease atask,
> > or tasks, from the pullqueue, do the work, delete those tasks, then
> > repeat from the lease stage.  The cool thing is that if you're, for
> > example, using URL Fetch to pull data  this might let you do it in
> > parallel without increasing your costs much (if any).
>
> > Robert
>
> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> > > Here are logs from three consecutivetaskexecutions over the past weekend,
> > > with only identifying information removed. You'll see that eachtask
> > > completes in a few milliseconds, but are 20 seconds apart (remember: I've
> > > already checked myqueueconfigurations, nothing else is running on this
> > > backend, and I later solved the problem by setting countdown=1 when adding
> > > thetask).  I don't see any pending latency mentioned.
>
> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> > > task_name=15804554889304913211 instance=0
> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
> > > api_cpu_ms=0
> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
> > > instance=0
> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
> > > api_cpu_ms=0
> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
> > > instance=0
>
> > > The 20 seconds seems to happen regardless of length oftask. Even though my
> > > tasks mostly complete in a couple minutes, I do have cases where they take
> > > several minutes, and I don't see a difference. Of course, when atasktakes
> > > 5-10 minutes to complete, I'm going to notice and care about a 20-second
> > >delaymuch less than when I'm trying to spin through a few tasks in a minute
> > > (which is a real-world need for me as well).
>
> > > When reading up on pull queues a while back, I was a little confused about
> > > where I would use them with my own backends. I definitely could see an
> > > application for offloading work to an AWS Linux instance. But in either
> > > case, could you explain why it might help?
>
> > > I saw you mention in a separate thread how M/S can perform differently 
> > > from
> > > HRD even in cases where one wouldn't expect to see a difference. When I 
> > > get
> > > around to it I'm going to create a tiny HRD app and run the same tests
> > > through that.
>
> > > I also wonder if M/S could be responsible for frequent latencies in my 
> > > admin
> > > console. Those have gotten more frequent and annoying the past couple of
> > > months ...
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine" group.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> > > To post to this group, send email to google-appengine@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
F

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

> that I have full control on the number of requests that will spin up,
err, number of instances that will spin up, rather ...

On Feb 5, 11:30 am, Dave Loomer  wrote:
> In my case, since I was getting the 20-second delay almost 100% of the
> time, setting countdown=1 was the answer.  If you only see it happen
> every 20 or more request then of course it won't help.
>
> In my case I also run all tasks on the backend. They're slightly more
> expensive per hour than frontends (due merely to the lower number of
> free hours) but in my case I more than make up for it with the fact
> that I have full control on the number of requests that will spin up,
> and I need to be able to control that number separately for tasks vs.
> users hitting my site.
>
> On Feb 5, 11:24 am, Carter  wrote:
>
>
>
>
>
>
>
> > We regularly but erratically see 10-20 minute delays in running push
> > queue tasks.
> > The tasks sit in the queue with ETA as high as 20 minutes *ago*
> > without any errors or retries.
>
> > (the problem seems unrelated to queue settings since our Maximum Rate,
> > Enorced Rate and Maximum Concurrent all far exceed the queue's
> > throughput at the time of the delays)
>
> > Any tips or clues on how to prevent this while still using push queues
> > without backends?
>
> > On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
> > > Hey Dave,
> > >   Hopefully Nick will be able to offer some insight into the cause of
> > > your issues.  I'd guess it is something related to having very few
> > > tasks (one) in thequeue, and it not getting scheduled rapidly.
>
> > >   In your case, you could use pull queues to immediately fetch the
> > > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> > > and do the work in parallel.  Basically you'd have a backend that ran
> > > a loop (possibly initiated via a pushtask) that would lease atask,
> > > or tasks, from the pullqueue, do the work, delete those tasks, then
> > > repeat from the lease stage.  The cool thing is that if you're, for
> > > example, using URL Fetch to pull data  this might let you do it in
> > > parallel without increasing your costs much (if any).
>
> > > Robert
>
> > > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> > > > Here are logs from three consecutivetaskexecutions over the past 
> > > > weekend,
> > > > with only identifying information removed. You'll see that eachtask
> > > > completes in a few milliseconds, but are 20 seconds apart (remember: 
> > > > I've
> > > > already checked myqueueconfigurations, nothing else is running on this
> > > > backend, and I later solved the problem by setting countdown=1 when 
> > > > adding
> > > > thetask).  I don't see any pending latency mentioned.
>
> > > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> > > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> > > > task_name=15804554889304913211 instance=0
> > > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
> > > > api_cpu_ms=0
> > > > cpm_usd=0.60 queue_name=overnight-tasks 
> > > > task_name=15804554889304912461
> > > > instance=0
> > > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
> > > > api_cpu_ms=0
> > > > cpm_usd=0.60 queue_name=overnight-tasks 
> > > > task_name=4499136807998063691
> > > > instance=0
>
> > > > The 20 seconds seems to happen regardless of length oftask. Even though 
> > > > my
> > > > tasks mostly complete in a couple minutes, I do have cases where they 
> > > > take
> > > > several minutes, and I don't see a difference. Of course, when 
> > > > atasktakes
> > > > 5-10 minutes to complete, I'm going to notice and care about a 20-second
> > > >delaymuch less than when I'm trying to spin through a few tasks in a 
> > > >minute
> > > > (which is a real-world need for me as well).
>
> > > > When reading up on pull queues a while back, I was a little confused 
> > > > about
> > > > where I would use them with my own backends. I definitely could see an
> > > > application for offloading work to an AWS Linux instance. But in either
> > > > case, could you explain why it might help?
>
> > > > I saw you mention in a separate thread how M/S can perform differently 
> > > > from
> > > > HRD even in cases where one wouldn't expect to see a difference. When I 
> > > > get
> > > > around to it I'm going to create a tiny HRD app and run the same tests
> > > > through that.
>
> > > > I also wonder if M/S could be responsible for frequent latencies in my 
> > > > admin
> > > > console. Those have gotten more frequent and annoying the past couple of
> > > > months ...
>
> > > > --
> > > > You received this message because you are subscribed to the Google 
> > > > Groups
> > > > "Google App Engine" group.
> > > > To view this discussion on the web visit
> > > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> > > > To post to this group, send email to google-appengine@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > >

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[stevep]

Carter wrote: We regularly but erratically see 10-20 minute delays in
running push queue tasks.

Been a burr under the saddle forever. What I really don't understand
-- assuming GAE engineers never see the benefit of providing at least
one priority/reliability queue -- is why the heck there is never any
explanation about how tasks get scheduled, and why these weird delays
happen. It is either: 1) If we told you we would have to shoot you, or
2) We can't see the benefit of you understanding this.

-stevep


On Feb 5, 9:24 am, Carter  wrote:
> We regularly but erratically see 10-20 minute delays in running push
> queue tasks.
> The tasks sit in the queue with ETA as high as 20 minutes *ago*
> without any errors or retries.
>
> (the problem seems unrelated to queue settings since our Maximum Rate,
> Enorced Rate and Maximum Concurrent all far exceed the queue's
> throughput at the time of the delays)
>
> Any tips or clues on how to prevent this while still using push queues
> without backends?
>
> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
>
>
>
>
>
>
> > Hey Dave,
> >   Hopefully Nick will be able to offer some insight into the cause of
> > your issues.  I'd guess it is something related to having very few
> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>
> >   In your case, you could use pull queues to immediately fetch the
> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> > and do the work in parallel.  Basically you'd have a backend that ran
> > a loop (possibly initiated via a pushtask) that would lease atask,
> > or tasks, from the pullqueue, do the work, delete those tasks, then
> > repeat from the lease stage.  The cool thing is that if you're, for
> > example, using URL Fetch to pull data  this might let you do it in
> > parallel without increasing your costs much (if any).
>
> > Robert
>
> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> > > Here are logs from three consecutivetaskexecutions over the past weekend,
> > > with only identifying information removed. You'll see that eachtask
> > > completes in a few milliseconds, but are 20 seconds apart (remember: I've
> > > already checked myqueueconfigurations, nothing else is running on this
> > > backend, and I later solved the problem by setting countdown=1 when adding
> > > thetask).  I don't see any pending latency mentioned.
>
> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> > > task_name=15804554889304913211 instance=0
> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
> > > api_cpu_ms=0
> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
> > > instance=0
> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
> > > api_cpu_ms=0
> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
> > > instance=0
>
> > > The 20 seconds seems to happen regardless of length oftask. Even though my
> > > tasks mostly complete in a couple minutes, I do have cases where they take
> > > several minutes, and I don't see a difference. Of course, when atasktakes
> > > 5-10 minutes to complete, I'm going to notice and care about a 20-second
> > >delaymuch less than when I'm trying to spin through a few tasks in a minute
> > > (which is a real-world need for me as well).
>
> > > When reading up on pull queues a while back, I was a little confused about
> > > where I would use them with my own backends. I definitely could see an
> > > application for offloading work to an AWS Linux instance. But in either
> > > case, could you explain why it might help?
>
> > > I saw you mention in a separate thread how M/S can perform differently 
> > > from
> > > HRD even in cases where one wouldn't expect to see a difference. When I 
> > > get
> > > around to it I'm going to create a tiny HRD app and run the same tests
> > > through that.
>
> > > I also wonder if M/S could be responsible for frequent latencies in my 
> > > admin
> > > console. Those have gotten more frequent and annoying the past couple of
> > > months ...
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine" group.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> > > To post to this group, send email to google-appengine@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.goog

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

As the OP you may be interested in my app ID as well: mn-live.  I
provided some logs a few posts back and some exhaustive details at the
beginning.

However, you won't see this issue popping up anymore on my app since I
"solved" it by setting countdown=1 a week ago. Since then, tasks start
very reliably after a 1.5 second delay.  If I remove the countdown
parameter, then it returns to 20 seconds (+/- .01) pretty reliably.

On Feb 5, 5:59 pm, Nicholas Verne  wrote:
> We would have no need to shoot anyone.
>
> However, the explanations quickly become obsolete. They enter the
> folklore in the form that was current at the time and become
> entrenched as incorrect information when the implementations have
> changed.
>
> Task Queues use best effort scheduling. They're not real time all the
> time, although when our best efforts are running smoothly they can
> appear real time. For scheduling, the task eta marks the earliest time
> at which the task can run. We can't guarantee that a task WILL run at
> that time.
>
> Steve, we're interested to know about the 10-20 minute delays you've
> seen. Can you tell us the app id, queue, and whether the tasks were
> added transactionally? An example from your logs would be very
> helpful.
>
> Nick Verne
>
>
>
>
>
>
>
> On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
> > Carter wrote: We regularly but erratically see 10-20 minute delays in
> > running push queue tasks.
>
> > Been a burr under the saddle forever. What I really don't understand
> > -- assuming GAE engineers never see the benefit of providing at least
> > one priority/reliability queue -- is why the heck there is never any
> > explanation about how tasks get scheduled, and why these weird delays
> > happen. It is either: 1) If we told you we would have to shoot you, or
> > 2) We can't see the benefit of you understanding this.
>
> > -stevep
>
> > On Feb 5, 9:24 am, Carter  wrote:
> >> We regularly but erratically see 10-20 minute delays in running push
> >> queue tasks.
> >> The tasks sit in the queue with ETA as high as 20 minutes *ago*
> >> without any errors or retries.
>
> >> (the problem seems unrelated to queue settings since our Maximum Rate,
> >> Enorced Rate and Maximum Concurrent all far exceed the queue's
> >> throughput at the time of the delays)
>
> >> Any tips or clues on how to prevent this while still using push queues
> >> without backends?
>
> >> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
> >> > Hey Dave,
> >> >   Hopefully Nick will be able to offer some insight into the cause of
> >> > your issues.  I'd guess it is something related to having very few
> >> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>
> >> >   In your case, you could use pull queues to immediately fetch the
> >> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> >> > and do the work in parallel.  Basically you'd have a backend that ran
> >> > a loop (possibly initiated via a pushtask) that would lease atask,
> >> > or tasks, from the pullqueue, do the work, delete those tasks, then
> >> > repeat from the lease stage.  The cool thing is that if you're, for
> >> > example, using URL Fetch to pull data  this might let you do it in
> >> > parallel without increasing your costs much (if any).
>
> >> > Robert
>
> >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> >> > > Here are logs from three consecutivetaskexecutions over the past 
> >> > > weekend,
> >> > > with only identifying information removed. You'll see that eachtask
> >> > > completes in a few milliseconds, but are 20 seconds apart (remember: 
> >> > > I've
> >> > > already checked myqueueconfigurations, nothing else is running on this
> >> > > backend, and I later solved the problem by setting countdown=1 when 
> >> > > adding
> >> > > thetask).  I don't see any pending latency mentioned.
>
> >> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> >> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> >> > > task_name=15804554889304913211 instance=0
> >> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
> >> > > api_cpu_ms=0
> >> > > cpm_usd=0.60 queue_name=overnight-tasks 
> >> > > task_name=15804554889304912461
> >> > > instance=0
> >> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
> >> > > api_cpu_ms=0
> >> > > cpm_usd=0.60 queue_name=overnight-tasks 
> >> > > task_name=4499136807998063691
> >> > > instance=0
>
> >> > > The 20 seconds seems to happen regardless of length oftask. Even 
> >> > > though my
> >> > > tasks mostly complete in a couple minutes, I do have cases where they 
> >> > > take
> >> > > several minutes, and I don't see a difference. Of course, when 
> >> > > atasktakes
> >> > > 5-10 minutes to complete, I'm going to notice and care about a 
> >> > > 20-second
> >> > >delaymuch less than when I'm trying to spin through a few tasks in a 
> >> > >minute
> >> > > (which is a real-world need for me as well).
>
> >> > > When readin

[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[stevep]

Thanks Nick. Been on the forums for years, and that's the first I
remember having a GAE engineer delve into TQs at this level. (BTW
Carter replied already about his 20 minute starts).

Maybe over time TQ scheduling might expand to enhance multi-instance
optimization, and provide a means for a priority queue (with
constraints such as volume capacity).

-stevep

On Feb 5, 3:59 pm, Nicholas Verne  wrote:
> We would have no need to shoot anyone.
>
> However, the explanations quickly become obsolete. They enter the
> folklore in the form that was current at the time and become
> entrenched as incorrect information when the implementations have
> changed.
>
> Task Queues use best effort scheduling. They're not real time all the
> time, although when our best efforts are running smoothly they can
> appear real time. For scheduling, the task eta marks the earliest time
> at which the task can run. We can't guarantee that a task WILL run at
> that time.
>
> Steve, we're interested to know about the 10-20 minute delays you've
> seen. Can you tell us the app id, queue, and whether the tasks were
> added transactionally? An example from your logs would be very
> helpful.
>
> Nick Verne
>
>
>
>
>
>
>
> On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
> > Carter wrote: We regularly but erratically see 10-20 minute delays in
> > running push queue tasks.
>
> > Been a burr under the saddle forever. What I really don't understand
> > -- assuming GAE engineers never see the benefit of providing at least
> > one priority/reliability queue -- is why the heck there is never any
> > explanation about how tasks get scheduled, and why these weird delays
> > happen. It is either: 1) If we told you we would have to shoot you, or
> > 2) We can't see the benefit of you understanding this.
>
> > -stevep
>
> > On Feb 5, 9:24 am, Carter  wrote:
> >> We regularly but erratically see 10-20 minute delays in running push
> >> queue tasks.
> >> The tasks sit in the queue with ETA as high as 20 minutes *ago*
> >> without any errors or retries.
>
> >> (the problem seems unrelated to queue settings since our Maximum Rate,
> >> Enorced Rate and Maximum Concurrent all far exceed the queue's
> >> throughput at the time of the delays)
>
> >> Any tips or clues on how to prevent this while still using push queues
> >> without backends?
>
> >> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
> >> > Hey Dave,
> >> >   Hopefully Nick will be able to offer some insight into the cause of
> >> > your issues.  I'd guess it is something related to having very few
> >> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>
> >> >   In your case, you could use pull queues to immediately fetch the
> >> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
> >> > and do the work in parallel.  Basically you'd have a backend that ran
> >> > a loop (possibly initiated via a pushtask) that would lease atask,
> >> > or tasks, from the pullqueue, do the work, delete those tasks, then
> >> > repeat from the lease stage.  The cool thing is that if you're, for
> >> > example, using URL Fetch to pull data  this might let you do it in
> >> > parallel without increasing your costs much (if any).
>
> >> > Robert
>
> >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> >> > > Here are logs from three consecutivetaskexecutions over the past 
> >> > > weekend,
> >> > > with only identifying information removed. You'll see that eachtask
> >> > > completes in a few milliseconds, but are 20 seconds apart (remember: 
> >> > > I've
> >> > > already checked myqueueconfigurations, nothing else is running on this
> >> > > backend, and I later solved the problem by setting countdown=1 when 
> >> > > adding
> >> > > thetask).  I don't see any pending latency mentioned.
>
> >> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> >> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> >> > > task_name=15804554889304913211 instance=0
> >> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
> >> > > api_cpu_ms=0
> >> > > cpm_usd=0.60 queue_name=overnight-tasks 
> >> > > task_name=15804554889304912461
> >> > > instance=0
> >> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
> >> > > api_cpu_ms=0
> >> > > cpm_usd=0.60 queue_name=overnight-tasks 
> >> > > task_name=4499136807998063691
> >> > > instance=0
>
> >> > > The 20 seconds seems to happen regardless of length oftask. Even 
> >> > > though my
> >> > > tasks mostly complete in a couple minutes, I do have cases where they 
> >> > > take
> >> > > several minutes, and I don't see a difference. Of course, when 
> >> > > atasktakes
> >> > > 5-10 minutes to complete, I'm going to notice and care about a 
> >> > > 20-second
> >> > >delaymuch less than when I'm trying to spin through a few tasks in a 
> >> > >minute
> >> > > (which is a real-world need for me as well).
>
> >> > > When reading up on pull queues a while back, I was a little confused 
> >> 

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Robert Kluin]

Hi,
  You've got a lot of info there.  I tried to skim it, but might have
missed some stuff.

  In your experiments with multiple backends, they are all the same
"backend" right?  You're not targeting specific instances are you?  In
other words, is everything targeted at "thebackend," "1.thebackend,"
or multiple different backends?

  Do you see pending latencies in the request log headers?  That's
where the failfast header should help.  Rather than dispatch the task,
then wait for an instance to handle it, it will immediately get
returned to the queue.  I don't think it is intended to prevent
instances from being spun up.

  What happens if instead of backends you use front ends plus a
max_concurrent_requests equal to the number of backends you were
wanting?  Does it change the behavior you're seeing?

  You does the latency of the request itself impact what you're
seeing?  If you do a lot more work in each request does the situation
improve?


Robert



On Sun, Jan 29, 2012 at 11:56, pdknsk  wrote:
>
> I haven't tried setting countdown. It's too slow for the mail queue
> anyway. It might work on the other backend (mentioned in the bug).
> I've moved most code away from backends though.
> > *- After first minute of the batch run, Task Queue Details page
> > consistently shows 6-8 tasks run in last minute even though logs show only
> > 2-3 requests served per minute (no errors or anything abnormal shown in the
> > logs either). I know for certain the only tasks running in the queue in
> > question are from my tests. Why is the number overstated, and why does it
> > only happen in the single-instance configuration?*
>
> I've noticed sth. else, with might be related. When you stop a
> backend, and put tasks on it, the tasks will obviously not run. In the
> task queue overview in the dashboard however, tasks are reported to
> have run in the last minute.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

Hi Robert,

When targeting backend "thebackend", this was done merely by specifying 
target=thebackend when creating the Task object. And then in backends.yaml 
I modify the instances parameter for the backend in question for each of my 
various tests.

I'm not really too concerned about failfast, so I'll skip that one for the 
moment ...

I haven't played with max concurrent requests, but in the test where I used 
a frontend, I did not see any additional instances spin up (and my site is 
very low-use, so there was probably a max of 1-2 instances to start with).

I see the same length of delay (~20 seconds) whether I'm testing with my 
rapid-fire test handler, or running in production with tasks that spend 1+ 
minutes doing actual work (lots of RPCs- datastore, URLfetch, memcache).

Hope this helps.
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xF79RpgjpAYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Robert Kluin]

Hey Dave,
  So do you see pending latencies in the request log headers?

  What if you further increase the work done within each task so that
it takes, for example, 5 minutes per task?

  Have you considered using pull-queues since you're running all this
on backends?


Robert




On Tue, Jan 31, 2012 at 16:58, Dave Loomer  wrote:
> Hi Robert,
>
> When targeting backend "thebackend", this was done merely by specifying
> target=thebackend when creating the Task object. And then in backends.yaml I
> modify the instances parameter for the backend in question for each of my
> various tests.
>
> I'm not really too concerned about failfast, so I'll skip that one for the
> moment ...
>
> I haven't played with max concurrent requests, but in the test where I used
> a frontend, I did not see any additional instances spin up (and my site is
> very low-use, so there was probably a max of 1-2 instances to start with).
>
> I see the same length of delay (~20 seconds) whether I'm testing with my
> rapid-fire test handler, or running in production with tasks that spend 1+
> minutes doing actual work (lots of RPCs- datastore, URLfetch, memcache).
>
> Hope this helps.
> Dave
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/xF79RpgjpAYJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

Here are logs from three consecutive task executions over the past weekend, 
with only identifying information removed. You'll see that each task 
completes in a few milliseconds, but are 20 seconds apart (remember: I've 
already checked my queue configurations, nothing else is running on this 
backend, and I later solved the problem by setting countdown=1 when adding 
the task).  I don't see any pending latency mentioned.

0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304913211 instance=0
0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=15804554889304912461 instance=0
0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks 
task_name=4499136807998063691 instance=0


The 20 seconds seems to happen regardless of length of task. Even though my 
tasks mostly complete in a couple minutes, I do have cases where they take 
several minutes, and I don't see a difference. Of course, when a task takes 
5-10 minutes to complete, I'm going to notice and care about a 20-second 
delay much less than when I'm trying to spin through a few tasks in a 
minute (which is a real-world need for me as well).

When reading up on pull queues a while back, I was a little confused about 
where I would use them with my own backends. I definitely could see an 
application for offloading work to an AWS Linux instance. But in either 
case, could you explain why it might help?

I saw you mention in a separate thread how M/S can perform differently from 
HRD even in cases where one wouldn't expect to see a difference. When I get 
around to it I'm going to create a tiny HRD app and run the same tests 
through that.

I also wonder if M/S could be responsible for frequent latencies in my 
admin console. Those have gotten more frequent and annoying the past couple 
of months ...

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Nicholas Verne]

Dave,

Two questions will help us clarify what you're observing.

Are your tasks added transactionally?

You mention the pattern of executing a task then enqueuing another. In
the one backend case, is there typically no more than one task in the
queue?

Nick Verne

On Thu, Feb 2, 2012 at 6:25 AM, Dave Loomer  wrote:
> Here are logs from three consecutive task executions over the past weekend,
> with only identifying information removed. You'll see that each task
> completes in a few milliseconds, but are 20 seconds apart (remember: I've
> already checked my queue configurations, nothing else is running on this
> backend, and I later solved the problem by setting countdown=1 when adding
> the task).  I don't see any pending latency mentioned.
>
> 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> task_name=15804554889304913211 instance=0
> 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0
> cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
> instance=0
> 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0
> cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
> instance=0
>
>
> The 20 seconds seems to happen regardless of length of task. Even though my
> tasks mostly complete in a couple minutes, I do have cases where they take
> several minutes, and I don't see a difference. Of course, when a task takes
> 5-10 minutes to complete, I'm going to notice and care about a 20-second
> delay much less than when I'm trying to spin through a few tasks in a minute
> (which is a real-world need for me as well).
>
> When reading up on pull queues a while back, I was a little confused about
> where I would use them with my own backends. I definitely could see an
> application for offloading work to an AWS Linux instance. But in either
> case, could you explain why it might help?
>
> I saw you mention in a separate thread how M/S can perform differently from
> HRD even in cases where one wouldn't expect to see a difference. When I get
> around to it I'm going to create a tiny HRD app and run the same tests
> through that.
>
> I also wonder if M/S could be responsible for frequent latencies in my admin
> console. Those have gotten more frequent and annoying the past couple of
> months ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

The tasks are not run transactionally, and in my testing the task is the 
only one in queue. In fact, I also ran the tests *somewhat* successfully on 
a separate app where this was the only code running. I say somewhat 
because, as I stated in my original post, the 20-second delays didn't 
happen every time; more like once every four requests. But when they did 
occur, it was almost precisely 20 seconds.

Below is the entirety of my handler:

class AdhocHandler2(webapp.RequestHandler):
def get(self):
task_create_time = datetime.datetime.now()
import string
task_create_time_s = task_create_time.strftime("%Y-%m-%d %H:%M:%S") + "." + 
string.zfill(task_create_time.microsecond, 6)
t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': 
task_create_time_s}, target='overnight-external-data',
method='GET')
t.add(queue_name = 'overnight-tasks')

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/orTdYjTm7SkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Dave Loomer]

To do one better, here is the entirety of the Python code:

#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import wsgiref.handlers
import datetime, time

from google.appengine.ext import webapp, db
from google.appengine.ext.webapp.util import run_wsgi_app 
from google.appengine.api import taskqueue

class AdhocHandler2(webapp.RequestHandler):
def get(self):
task_create_time = datetime.datetime.now()
import string
task_create_time_s = task_create_time.strftime("%Y-%m-%d %H:%M:%S") + "." + 
string.zfill(task_create_time.microsecond, 6)
t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': 
task_create_time_s}, target='overnight-external-data',
method='GET')
t.add(queue_name = 'overnight-tasks')

def main():
application = webapp.WSGIApplication([ \
('/batch/adhoc2', AdhocHandler2), \
],
 debug=True)

run_wsgi_app(application)

if __name__ == '__main__':
main()

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ELiNYLg4B9cJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Robert Kluin]

Hey Dave,
  Hopefully Nick will be able to offer some insight into the cause of
your issues.  I'd guess it is something related to having very few
tasks (one) in the queue, and it not getting scheduled rapidly.

  In your case, you could use pull queues to immediately fetch the
next task when finished with a task.  Or even to fetch multiple tasks
and do the work in parallel.  Basically you'd have a backend that ran
a loop (possibly initiated via a push task) that would lease a task,
or tasks, from the pull queue, do the work, delete those tasks, then
repeat from the lease stage.  The cool thing is that if you're, for
example, using URL Fetch to pull data  this might let you do it in
parallel without increasing your costs much (if any).

Robert




On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
> Here are logs from three consecutive task executions over the past weekend,
> with only identifying information removed. You'll see that each task
> completes in a few milliseconds, but are 20 seconds apart (remember: I've
> already checked my queue configurations, nothing else is running on this
> backend, and I later solved the problem by setting countdown=1 when adding
> the task).  I don't see any pending latency mentioned.
>
> 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
> api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
> task_name=15804554889304913211 instance=0
> 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0
> cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461
> instance=0
> 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0
> cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
> instance=0
>
>
> The 20 seconds seems to happen regardless of length of task. Even though my
> tasks mostly complete in a couple minutes, I do have cases where they take
> several minutes, and I don't see a difference. Of course, when a task takes
> 5-10 minutes to complete, I'm going to notice and care about a 20-second
> delay much less than when I'm trying to spin through a few tasks in a minute
> (which is a real-world need for me as well).
>
> When reading up on pull queues a while back, I was a little confused about
> where I would use them with my own backends. I definitely could see an
> application for offloading work to an AWS Linux instance. But in either
> case, could you explain why it might help?
>
> I saw you mention in a separate thread how M/S can perform differently from
> HRD even in cases where one wouldn't expect to see a difference. When I get
> around to it I'm going to create a tiny HRD app and run the same tests
> through that.
>
> I also wonder if M/S could be responsible for frequent latencies in my admin
> console. Those have gotten more frequent and annoying the past couple of
> months ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Nicholas Verne]

We would have no need to shoot anyone.

However, the explanations quickly become obsolete. They enter the
folklore in the form that was current at the time and become
entrenched as incorrect information when the implementations have
changed.

Task Queues use best effort scheduling. They're not real time all the
time, although when our best efforts are running smoothly they can
appear real time. For scheduling, the task eta marks the earliest time
at which the task can run. We can't guarantee that a task WILL run at
that time.

Steve, we're interested to know about the 10-20 minute delays you've
seen. Can you tell us the app id, queue, and whether the tasks were
added transactionally? An example from your logs would be very
helpful.

Nick Verne

On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
> Carter wrote: We regularly but erratically see 10-20 minute delays in
> running push queue tasks.
>
> Been a burr under the saddle forever. What I really don't understand
> -- assuming GAE engineers never see the benefit of providing at least
> one priority/reliability queue -- is why the heck there is never any
> explanation about how tasks get scheduled, and why these weird delays
> happen. It is either: 1) If we told you we would have to shoot you, or
> 2) We can't see the benefit of you understanding this.
>
> -stevep
>
>
> On Feb 5, 9:24 am, Carter  wrote:
>> We regularly but erratically see 10-20 minute delays in running push
>> queue tasks.
>> The tasks sit in the queue with ETA as high as 20 minutes *ago*
>> without any errors or retries.
>>
>> (the problem seems unrelated to queue settings since our Maximum Rate,
>> Enorced Rate and Maximum Concurrent all far exceed the queue's
>> throughput at the time of the delays)
>>
>> Any tips or clues on how to prevent this while still using push queues
>> without backends?
>>
>> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>>
>>
>>
>>
>>
>>
>>
>> > Hey Dave,
>> >   Hopefully Nick will be able to offer some insight into the cause of
>> > your issues.  I'd guess it is something related to having very few
>> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>>
>> >   In your case, you could use pull queues to immediately fetch the
>> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
>> > and do the work in parallel.  Basically you'd have a backend that ran
>> > a loop (possibly initiated via a pushtask) that would lease atask,
>> > or tasks, from the pullqueue, do the work, delete those tasks, then
>> > repeat from the lease stage.  The cool thing is that if you're, for
>> > example, using URL Fetch to pull data  this might let you do it in
>> > parallel without increasing your costs much (if any).
>>
>> > Robert
>>
>> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer  wrote:
>> > > Here are logs from three consecutivetaskexecutions over the past weekend,
>> > > with only identifying information removed. You'll see that eachtask
>> > > completes in a few milliseconds, but are 20 seconds apart (remember: I've
>> > > already checked myqueueconfigurations, nothing else is running on this
>> > > backend, and I later solved the problem by setting countdown=1 when 
>> > > adding
>> > > thetask).  I don't see any pending latency mentioned.
>>
>> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47
>> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks
>> > > task_name=15804554889304913211 instance=0
>> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 
>> > > api_cpu_ms=0
>> > > cpm_usd=0.60 queue_name=overnight-tasks 
>> > > task_name=15804554889304912461
>> > > instance=0
>> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 
>> > > api_cpu_ms=0
>> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691
>> > > instance=0
>>
>> > > The 20 seconds seems to happen regardless of length oftask. Even though 
>> > > my
>> > > tasks mostly complete in a couple minutes, I do have cases where they 
>> > > take
>> > > several minutes, and I don't see a difference. Of course, when atasktakes
>> > > 5-10 minutes to complete, I'm going to notice and care about a 20-second
>> > >delaymuch less than when I'm trying to spin through a few tasks in a 
>> > >minute
>> > > (which is a real-world need for me as well).
>>
>> > > When reading up on pull queues a while back, I was a little confused 
>> > > about
>> > > where I would use them with my own backends. I definitely could see an
>> > > application for offloading work to an AWS Linux instance. But in either
>> > > case, could you explain why it might help?
>>
>> > > I saw you mention in a separate thread how M/S can perform differently 
>> > > from
>> > > HRD even in cases where one wouldn't expect to see a difference. When I 
>> > > get
>> > > around to it I'm going to create a tiny HRD app and run the same tests
>> > > through that.
>>
>> > > I also wonder if M/S could be responsible for frequent latencies in my 
>> > > admin
>> > >

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Robert Kluin]

That's interesting.  Did the queue sit there for a long time not
running anything, or running tasks very slowly?  Are the tasks in that
queue generally long-running?

I _very_ infrequently bump into that type of issue, but I periodically
will see one queue slow down for a while.  It *seems* to happen far
more often in queues with slower tasks, but I don't have any recent
empirical evidence of that.  And I *think* I've been told that should
not be the case.


Robert



On Sun, Feb 5, 2012 at 19:27, Carter Maslan  wrote:
> Nicholas -
>
> For our examples of the 10-20 minute delay:
> app_id=s~camiologger
> queue=image-label
> (but several other queues experience the same long delays sometimes:
> content-process, counter-update, etc...)
>
> The tasks were not added with transactions; just this code:
> Queue queueP =
> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH);
> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL)
>
> .param("key", contentKeyString)
>
> .method(TaskOptions.Method.GET));
>
>
> Let me know if you need more info.  We noticed this in the last few weeks.
> Carter
>
>
>
> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer  wrote:
>>
>> As the OP you may be interested in my app ID as well: mn-live.  I
>> provided some logs a few posts back and some exhaustive details at the
>> beginning.
>>
>> However, you won't see this issue popping up anymore on my app since I
>> "solved" it by setting countdown=1 a week ago. Since then, tasks start
>> very reliably after a 1.5 second delay.  If I remove the countdown
>> parameter, then it returns to 20 seconds (+/- .01) pretty reliably.
>>
>> On Feb 5, 5:59 pm, Nicholas Verne  wrote:
>> > We would have no need to shoot anyone.
>> >
>> > However, the explanations quickly become obsolete. They enter the
>> > folklore in the form that was current at the time and become
>> > entrenched as incorrect information when the implementations have
>> > changed.
>> >
>> > Task Queues use best effort scheduling. They're not real time all the
>> > time, although when our best efforts are running smoothly they can
>> > appear real time. For scheduling, the task eta marks the earliest time
>> > at which the task can run. We can't guarantee that a task WILL run at
>> > that time.
>> >
>> > Steve, we're interested to know about the 10-20 minute delays you've
>> > seen. Can you tell us the app id, queue, and whether the tasks were
>> > added transactionally? An example from your logs would be very
>> > helpful.
>> >
>> > Nick Verne
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
>> > > Carter wrote: We regularly but erratically see 10-20 minute delays in
>> > > running push queue tasks.
>> >
>> > > Been a burr under the saddle forever. What I really don't understand
>> > > -- assuming GAE engineers never see the benefit of providing at least
>> > > one priority/reliability queue -- is why the heck there is never any
>> > > explanation about how tasks get scheduled, and why these weird delays
>> > > happen. It is either: 1) If we told you we would have to shoot you, or
>> > > 2) We can't see the benefit of you understanding this.
>> >
>> > > -stevep
>> >
>> > > On Feb 5, 9:24 am, Carter  wrote:
>> > >> We regularly but erratically see 10-20 minute delays in running push
>> > >> queue tasks.
>> > >> The tasks sit in the queue with ETA as high as 20 minutes *ago*
>> > >> without any errors or retries.
>> >
>> > >> (the problem seems unrelated to queue settings since our Maximum
>> > >> Rate,
>> > >> Enorced Rate and Maximum Concurrent all far exceed the queue's
>> > >> throughput at the time of the delays)
>> >
>> > >> Any tips or clues on how to prevent this while still using push
>> > >> queues
>> > >> without backends?
>> >
>> > >> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>> >
>> > >> > Hey Dave,
>> > >> >   Hopefully Nick will be able to offer some insight into the cause
>> > >> > of
>> > >> > your issues.  I'd guess it is something related to having very few
>> > >> > tasks (one) in thequeue, and it not getting scheduled rapidly.
>> >
>> > >> >   In your case, you could use pull queues to immediately fetch the
>> > >> > nexttaskwhen finished with atask.  Or even to fetch multiple tasks
>> > >> > and do the work in parallel.  Basically you'd have a backend that
>> > >> > ran
>> > >> > a loop (possibly initiated via a pushtask) that would lease atask,
>> > >> > or tasks, from the pullqueue, do the work, delete those tasks, then
>> > >> > repeat from the lease stage.  The cool thing is that if you're, for
>> > >> > example, using URL Fetch to pull data  this might let you do it in
>> > >> > parallel without increasing your costs much (if any).
>> >
>> > >> > Robert
>> >
>> > >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer 
>> > >> > wrote:
>> > >> > > Here are logs from three consecutivetaskexecutions over the past
>> > >> > > weekend,
>> > >> > > with only identifying information removed. You'll see that
>> > >> >

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Carter Maslan]

I Just looked at the last 80 that ran.
That queue's tasks are running in between 19ms and 2486ms with most of them 
running around 28ms.  The variability relates to the number of quadtree 
searches needed, but other queues that experience similar delays have running 
time without much variation(e.g. predictable counter updates)
When the delays happen, there just aren't many tasks in the queue at all.
It appears that the delayed tasks are just sitting in the queue idle.



On Feb 5, 2012, at 9:17 PM, Robert Kluin  wrote:

> That's interesting.  Did the queue sit there for a long time not
> running anything, or running tasks very slowly?  Are the tasks in that
> queue generally long-running?
> 
> I _very_ infrequently bump into that type of issue, but I periodically
> will see one queue slow down for a while.  It *seems* to happen far
> more often in queues with slower tasks, but I don't have any recent
> empirical evidence of that.  And I *think* I've been told that should
> not be the case.
> 
> 
> Robert
> 
> 
> 
> On Sun, Feb 5, 2012 at 19:27, Carter Maslan  wrote:
>> Nicholas -
>> 
>> For our examples of the 10-20 minute delay:
>> app_id=s~camiologger
>> queue=image-label
>> (but several other queues experience the same long delays sometimes:
>> content-process, counter-update, etc...)
>> 
>> The tasks were not added with transactions; just this code:
>> Queue queueP =
>> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH);
>> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL)
>> 
>> .param("key", contentKeyString)
>> 
>> .method(TaskOptions.Method.GET));
>> 
>> 
>> Let me know if you need more info.  We noticed this in the last few weeks.
>> Carter
>> 
>> 
>> 
>> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer  wrote:
>>> 
>>> As the OP you may be interested in my app ID as well: mn-live.  I
>>> provided some logs a few posts back and some exhaustive details at the
>>> beginning.
>>> 
>>> However, you won't see this issue popping up anymore on my app since I
>>> "solved" it by setting countdown=1 a week ago. Since then, tasks start
>>> very reliably after a 1.5 second delay.  If I remove the countdown
>>> parameter, then it returns to 20 seconds (+/- .01) pretty reliably.
>>> 
>>> On Feb 5, 5:59 pm, Nicholas Verne  wrote:
 We would have no need to shoot anyone.
 
 However, the explanations quickly become obsolete. They enter the
 folklore in the form that was current at the time and become
 entrenched as incorrect information when the implementations have
 changed.
 
 Task Queues use best effort scheduling. They're not real time all the
 time, although when our best efforts are running smoothly they can
 appear real time. For scheduling, the task eta marks the earliest time
 at which the task can run. We can't guarantee that a task WILL run at
 that time.
 
 Steve, we're interested to know about the 10-20 minute delays you've
 seen. Can you tell us the app id, queue, and whether the tasks were
 added transactionally? An example from your logs would be very
 helpful.
 
 Nick Verne
 
 
 
 
 
 
 
 On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
> Carter wrote: We regularly but erratically see 10-20 minute delays in
> running push queue tasks.
 
> Been a burr under the saddle forever. What I really don't understand
> -- assuming GAE engineers never see the benefit of providing at least
> one priority/reliability queue -- is why the heck there is never any
> explanation about how tasks get scheduled, and why these weird delays
> happen. It is either: 1) If we told you we would have to shoot you, or
> 2) We can't see the benefit of you understanding this.
 
> -stevep
 
> On Feb 5, 9:24 am, Carter  wrote:
>> We regularly but erratically see 10-20 minute delays in running push
>> queue tasks.
>> The tasks sit in the queue with ETA as high as 20 minutes *ago*
>> without any errors or retries.
 
>> (the problem seems unrelated to queue settings since our Maximum
>> Rate,
>> Enorced Rate and Maximum Concurrent all far exceed the queue's
>> throughput at the time of the delays)
 
>> Any tips or clues on how to prevent this while still using push
>> queues
>> without backends?
 
>> On Feb 1, 9:03 pm, Robert Kluin  wrote:
 
>>> Hey Dave,
>>>   Hopefully Nick will be able to offer some insight into the cause
>>> of
>>> your issues.  I'd guess it is something related to having very few
>>> tasks (one) in thequeue, and it not getting scheduled rapidly.
 
>>>   In your case, you could use pull queues to immediately fetch the
>>> nexttaskwhen finished with atask.  Or even to fetch multiple tasks
>>> and do the work in parallel.  Basically you'd have a backend that
>>> ran
>>> a loop (possibly initiated via a pushtask) that would le

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Robert Kluin]

Does the app get a lot of front-end traffic or is it sitting idle when
the delays occur?




On Mon, Feb 6, 2012 at 01:38, Carter Maslan  wrote:
> I Just looked at the last 80 that ran.
> That queue's tasks are running in between 19ms and 2486ms with most of them 
> running around 28ms.  The variability relates to the number of quadtree 
> searches needed, but other queues that experience similar delays have running 
> time without much variation(e.g. predictable counter updates)
> When the delays happen, there just aren't many tasks in the queue at all.
> It appears that the delayed tasks are just sitting in the queue idle.
>
>
>
> On Feb 5, 2012, at 9:17 PM, Robert Kluin  wrote:
>
>> That's interesting.  Did the queue sit there for a long time not
>> running anything, or running tasks very slowly?  Are the tasks in that
>> queue generally long-running?
>>
>> I _very_ infrequently bump into that type of issue, but I periodically
>> will see one queue slow down for a while.  It *seems* to happen far
>> more often in queues with slower tasks, but I don't have any recent
>> empirical evidence of that.  And I *think* I've been told that should
>> not be the case.
>>
>>
>> Robert
>>
>>
>>
>> On Sun, Feb 5, 2012 at 19:27, Carter Maslan  wrote:
>>> Nicholas -
>>>
>>> For our examples of the 10-20 minute delay:
>>> app_id=s~camiologger
>>> queue=image-label
>>> (but several other queues experience the same long delays sometimes:
>>> content-process, counter-update, etc...)
>>>
>>> The tasks were not added with transactions; just this code:
>>> Queue queueP =
>>> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH);
>>> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL)
>>>
>>> .param("key", contentKeyString)
>>>
>>> .method(TaskOptions.Method.GET));
>>>
>>>
>>> Let me know if you need more info.  We noticed this in the last few weeks.
>>> Carter
>>>
>>>
>>>
>>> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer  wrote:

 As the OP you may be interested in my app ID as well: mn-live.  I
 provided some logs a few posts back and some exhaustive details at the
 beginning.

 However, you won't see this issue popping up anymore on my app since I
 "solved" it by setting countdown=1 a week ago. Since then, tasks start
 very reliably after a 1.5 second delay.  If I remove the countdown
 parameter, then it returns to 20 seconds (+/- .01) pretty reliably.

 On Feb 5, 5:59 pm, Nicholas Verne  wrote:
> We would have no need to shoot anyone.
>
> However, the explanations quickly become obsolete. They enter the
> folklore in the form that was current at the time and become
> entrenched as incorrect information when the implementations have
> changed.
>
> Task Queues use best effort scheduling. They're not real time all the
> time, although when our best efforts are running smoothly they can
> appear real time. For scheduling, the task eta marks the earliest time
> at which the task can run. We can't guarantee that a task WILL run at
> that time.
>
> Steve, we're interested to know about the 10-20 minute delays you've
> seen. Can you tell us the app id, queue, and whether the tasks were
> added transactionally? An example from your logs would be very
> helpful.
>
> Nick Verne
>
>
>
>
>
>
>
> On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
>> Carter wrote: We regularly but erratically see 10-20 minute delays in
>> running push queue tasks.
>
>> Been a burr under the saddle forever. What I really don't understand
>> -- assuming GAE engineers never see the benefit of providing at least
>> one priority/reliability queue -- is why the heck there is never any
>> explanation about how tasks get scheduled, and why these weird delays
>> happen. It is either: 1) If we told you we would have to shoot you, or
>> 2) We can't see the benefit of you understanding this.
>
>> -stevep
>
>> On Feb 5, 9:24 am, Carter  wrote:
>>> We regularly but erratically see 10-20 minute delays in running push
>>> queue tasks.
>>> The tasks sit in the queue with ETA as high as 20 minutes *ago*
>>> without any errors or retries.
>
>>> (the problem seems unrelated to queue settings since our Maximum
>>> Rate,
>>> Enorced Rate and Maximum Concurrent all far exceed the queue's
>>> throughput at the time of the delays)
>
>>> Any tips or clues on how to prevent this while still using push
>>> queues
>>> without backends?
>
>>> On Feb 1, 9:03 pm, Robert Kluin  wrote:
>
 Hey Dave,
   Hopefully Nick will be able to offer some insight into the cause
 of
 your issues.  I'd guess it is something related to having very few
 tasks (one) in thequeue, and it not getting scheduled rapidly.
>
   In your case, you could use pull queues to immed

Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks

[Carter Maslan]

The app always has front end traffic and I have noticed the delay at times
with high traffic.

On Feb 5, 2012 10:57 PM, "Robert Kluin"  wrote:

> Does the app get a lot of front-end traffic or is it sitting idle when
> the delays occur?
>
>
>
>
> On Mon, Feb 6, 2012 at 01:38, Carter Maslan  wrote:
> > I Just looked at the last 80 that ran.
> > That queue's tasks are running in between 19ms and 2486ms with most of
> them running around 28ms.  The variability relates to the number of
> quadtree searches needed, but other queues that experience similar delays
> have running time without much variation(e.g. predictable counter updates)
> > When the delays happen, there just aren't many tasks in the queue at all.
> > It appears that the delayed tasks are just sitting in the queue idle.
> >
> >
> >
> > On Feb 5, 2012, at 9:17 PM, Robert Kluin  wrote:
> >
> >> That's interesting.  Did the queue sit there for a long time not
> >> running anything, or running tasks very slowly?  Are the tasks in that
> >> queue generally long-running?
> >>
> >> I _very_ infrequently bump into that type of issue, but I periodically
> >> will see one queue slow down for a while.  It *seems* to happen far
> >> more often in queues with slower tasks, but I don't have any recent
> >> empirical evidence of that.  And I *think* I've been told that should
> >> not be the case.
> >>
> >>
> >> Robert
> >>
> >>
> >>
> >> On Sun, Feb 5, 2012 at 19:27, Carter Maslan  wrote:
> >>> Nicholas -
> >>>
> >>> For our examples of the 10-20 minute delay:
> >>> app_id=s~camiologger
> >>> queue=image-label
> >>> (but several other queues experience the same long delays sometimes:
> >>> content-process, counter-update, etc...)
> >>>
> >>> The tasks were not added with transactions; just this code:
> >>> Queue queueP =
> >>> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH);
> >>> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL)
> >>>
> >>> .param("key", contentKeyString)
> >>>
> >>> .method(TaskOptions.Method.GET));
> >>>
> >>>
> >>> Let me know if you need more info.  We noticed this in the last few
> weeks.
> >>> Carter
> >>>
> >>>
> >>>
> >>> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer  wrote:
> 
>  As the OP you may be interested in my app ID as well: mn-live.  I
>  provided some logs a few posts back and some exhaustive details at the
>  beginning.
> 
>  However, you won't see this issue popping up anymore on my app since I
>  "solved" it by setting countdown=1 a week ago. Since then, tasks start
>  very reliably after a 1.5 second delay.  If I remove the countdown
>  parameter, then it returns to 20 seconds (+/- .01) pretty reliably.
> 
>  On Feb 5, 5:59 pm, Nicholas Verne  wrote:
> > We would have no need to shoot anyone.
> >
> > However, the explanations quickly become obsolete. They enter the
> > folklore in the form that was current at the time and become
> > entrenched as incorrect information when the implementations have
> > changed.
> >
> > Task Queues use best effort scheduling. They're not real time all the
> > time, although when our best efforts are running smoothly they can
> > appear real time. For scheduling, the task eta marks the earliest
> time
> > at which the task can run. We can't guarantee that a task WILL run at
> > that time.
> >
> > Steve, we're interested to know about the 10-20 minute delays you've
> > seen. Can you tell us the app id, queue, and whether the tasks were
> > added transactionally? An example from your logs would be very
> > helpful.
> >
> > Nick Verne
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Feb 6, 2012 at 9:27 AM, stevep  wrote:
> >> Carter wrote: We regularly but erratically see 10-20 minute delays
> in
> >> running push queue tasks.
> >
> >> Been a burr under the saddle forever. What I really don't understand
> >> -- assuming GAE engineers never see the benefit of providing at
> least
> >> one priority/reliability queue -- is why the heck there is never any
> >> explanation about how tasks get scheduled, and why these weird
> delays
> >> happen. It is either: 1) If we told you we would have to shoot you,
> or
> >> 2) We can't see the benefit of you understanding this.
> >
> >> -stevep
> >
> >> On Feb 5, 9:24 am, Carter  wrote:
> >>> We regularly but erratically see 10-20 minute delays in running
> push
> >>> queue tasks.
> >>> The tasks sit in the queue with ETA as high as 20 minutes *ago*
> >>> without any errors or retries.
> >
> >>> (the problem seems unrelated to queue settings since our Maximum
> >>> Rate,
> >>> Enorced Rate and Maximum Concurrent all far exceed the queue's
> >>> throughput at the time of the delays)
> >
> >>> Any tips or clues on how to prevent this while still using push
> >>> queues
> >>> without