[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
FWIW should note that my app is master/slave. The pressure to move to HRD/Python 2.7 is very real, but right now I have too many concerns with replication delays and reading others' migration headaches with data volumes similar to mine, so I have no short-term plans to migrate. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FddU5krbHIsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
I've been able to nearly solve the delay problem by setting countdown=1 in the Task constructor. This reduces the delay from 20 to about 1.5. Not sure why it's not closer to 1.0 but this will be fine. The time to serve the simple request is unaffected. Still a strange bug (?). Also, some other things lead me to believe the GAE just wants to "rest" for 20 seconds. I started to wonder if in my previous tests I was just running things through too fast, even though that seemed unlikely since I was seeing the same delays in production when tasks take 1+ minutes to complete. But I put a time.sleep(10) in my handler just to make sure. Strangely, what this did was reduce the delay from 20 seconds down to 10. So the sleep gives 10 seconds of rest and then GAE still wants 10 more? Something like that. When I replaced time.sleep(10) with a tight loop lasting 10 seconds, my higher delays returned although they were more erratic (i.e. not consistently near 20 seconds). Also, I tried creating a new app containing only the code I had written for my test, and got somewhat different results. I got the almost-exactly-20-seconds delay, but only about once every 4-5 requests. The other times it was near zero. Maybe this is because nothing else is going on in the app, so GAE has less need for "rest?" Almost as if it were a quota thing -- except the fact that I demonstrated earlier that I can eliminate the delay entirely by running tasks on a frontend, or using a 5-instance backend, or (as now demonstrated) setting countdown=1 seems to eliminate that possibility. All very strange. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/wYRmOXSR5-wJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
I had the same problem, with often quite exact 20 seconds delays. More details in this thread. http://groups.google.com/group/google-appengine/browse_thread/thread/e5588268dff9b97a In another configuration, the delay increased to several minutes. Please star this bug. http://code.google.com/p/googleappengine/issues/detail?id=6022 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Interesting. I saw your thread but wasn't entirely sure if it was the issue. I think the thing that threw me off was that your delays were being reflected in the request ms in the logs, while in my case they mostly aren't. Does setting the task countdown work for you? Or is ~1 second delay still too much in your case? On Jan 29, 10:24 am, pdknsk wrote: > I had the same problem, with often quite exact 20 seconds delays. More > details in this thread. > > http://groups.google.com/group/google-appengine/browse_thread/thread/... > > In another configuration, the delay increased to several minutes. > Please star this bug. > > http://code.google.com/p/googleappengine/issues/detail?id=6022 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
I haven't tried setting countdown. It's too slow for the mail queue anyway. It might work on the other backend (mentioned in the bug). I've moved most code away from backends though. > *- After first minute of the batch run, Task Queue Details page > consistently shows 6-8 tasks run in last minute even though logs show only > 2-3 requests served per minute (no errors or anything abnormal shown in the > logs either). I know for certain the only tasks running in the queue in > question are from my tests. Why is the number overstated, and why does it > only happen in the single-instance configuration?* I've noticed sth. else, with might be related. When you stop a backend, and put tasks on it, the tasks will obviously not run. In the task queue overview in the dashboard however, tasks are reported to have run in the last minute. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
And here is backends.yaml: backends: - name: overnight-external-data class: B1 options: dynamic instances: 1 and queue.yaml: queue: - name: overnight-tasks rate: 50/s bucket_size: 50 retry_parameters: max_backoff_seconds: 1800 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/nIP2SeQDMUsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Finally, it's probably an important clue that when I explicitly set countdown=1 when creating the task, the delay in executing the task is always almost exactly 1.5 seconds (not sure why it's not 1.0). If I don't set a countdown value, it's almost as if I had set countdown=20. Except that the ETA in the Task Queue Details page in the admin console reflects the approximate time the task was created, rather than adding 20 seconds to that. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/sbd5Clqozg8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin wrote: > Hey Dave, > Hopefully Nick will be able to offer some insight into the cause of > your issues. I'd guess it is something related to having very few > tasks (one) in thequeue, and it not getting scheduled rapidly. > > In your case, you could use pull queues to immediately fetch the > nexttaskwhen finished with atask. Or even to fetch multiple tasks > and do the work in parallel. Basically you'd have a backend that ran > a loop (possibly initiated via a pushtask) that would lease atask, > or tasks, from the pullqueue, do the work, delete those tasks, then > repeat from the lease stage. The cool thing is that if you're, for > example, using URL Fetch to pull data this might let you do it in > parallel without increasing your costs much (if any). > > Robert > > > > > > > > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > > Here are logs from three consecutivetaskexecutions over the past weekend, > > with only identifying information removed. You'll see that eachtask > > completes in a few milliseconds, but are 20 seconds apart (remember: I've > > already checked myqueueconfigurations, nothing else is running on this > > backend, and I later solved the problem by setting countdown=1 when adding > > thetask). I don't see any pending latency mentioned. > > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > > task_name=15804554889304913211 instance=0 > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 > > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 > > instance=0 > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 > > instance=0 > > > The 20 seconds seems to happen regardless of length oftask. Even though my > > tasks mostly complete in a couple minutes, I do have cases where they take > > several minutes, and I don't see a difference. Of course, when atasktakes > > 5-10 minutes to complete, I'm going to notice and care about a 20-second > >delaymuch less than when I'm trying to spin through a few tasks in a minute > > (which is a real-world need for me as well). > > > When reading up on pull queues a while back, I was a little confused about > > where I would use them with my own backends. I definitely could see an > > application for offloading work to an AWS Linux instance. But in either > > case, could you explain why it might help? > > > I saw you mention in a separate thread how M/S can perform differently from > > HRD even in cases where one wouldn't expect to see a difference. When I get > > around to it I'm going to create a tiny HRD app and run the same tests > > through that. > > > I also wonder if M/S could be responsible for frequent latencies in my admin > > console. Those have gotten more frequent and annoying the past couple of > > months ... > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
In my case, since I was getting the 20-second delay almost 100% of the time, setting countdown=1 was the answer. If you only see it happen every 20 or more request then of course it won't help. In my case I also run all tasks on the backend. They're slightly more expensive per hour than frontends (due merely to the lower number of free hours) but in my case I more than make up for it with the fact that I have full control on the number of requests that will spin up, and I need to be able to control that number separately for tasks vs. users hitting my site. On Feb 5, 11:24 am, Carter wrote: > We regularly but erratically see 10-20 minute delays in running push > queue tasks. > The tasks sit in the queue with ETA as high as 20 minutes *ago* > without any errors or retries. > > (the problem seems unrelated to queue settings since our Maximum Rate, > Enorced Rate and Maximum Concurrent all far exceed the queue's > throughput at the time of the delays) > > Any tips or clues on how to prevent this while still using push queues > without backends? > > On Feb 1, 9:03 pm, Robert Kluin wrote: > > > > > > > > > Hey Dave, > > Hopefully Nick will be able to offer some insight into the cause of > > your issues. I'd guess it is something related to having very few > > tasks (one) in thequeue, and it not getting scheduled rapidly. > > > In your case, you could use pull queues to immediately fetch the > > nexttaskwhen finished with atask. Or even to fetch multiple tasks > > and do the work in parallel. Basically you'd have a backend that ran > > a loop (possibly initiated via a pushtask) that would lease atask, > > or tasks, from the pullqueue, do the work, delete those tasks, then > > repeat from the lease stage. The cool thing is that if you're, for > > example, using URL Fetch to pull data this might let you do it in > > parallel without increasing your costs much (if any). > > > Robert > > > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > > > Here are logs from three consecutivetaskexecutions over the past weekend, > > > with only identifying information removed. You'll see that eachtask > > > completes in a few milliseconds, but are 20 seconds apart (remember: I've > > > already checked myqueueconfigurations, nothing else is running on this > > > backend, and I later solved the problem by setting countdown=1 when adding > > > thetask). I don't see any pending latency mentioned. > > > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > > > task_name=15804554889304913211 instance=0 > > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 > > > api_cpu_ms=0 > > > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 > > > instance=0 > > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 > > > api_cpu_ms=0 > > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 > > > instance=0 > > > > The 20 seconds seems to happen regardless of length oftask. Even though my > > > tasks mostly complete in a couple minutes, I do have cases where they take > > > several minutes, and I don't see a difference. Of course, when atasktakes > > > 5-10 minutes to complete, I'm going to notice and care about a 20-second > > >delaymuch less than when I'm trying to spin through a few tasks in a minute > > > (which is a real-world need for me as well). > > > > When reading up on pull queues a while back, I was a little confused about > > > where I would use them with my own backends. I definitely could see an > > > application for offloading work to an AWS Linux instance. But in either > > > case, could you explain why it might help? > > > > I saw you mention in a separate thread how M/S can perform differently > > > from > > > HRD even in cases where one wouldn't expect to see a difference. When I > > > get > > > around to it I'm going to create a tiny HRD app and run the same tests > > > through that. > > > > I also wonder if M/S could be responsible for frequent latencies in my > > > admin > > > console. Those have gotten more frequent and annoying the past couple of > > > months ... > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To view this discussion on the web visit > > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > > > To post to this group, send email to google-appengine@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. F
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
> that I have full control on the number of requests that will spin up, err, number of instances that will spin up, rather ... On Feb 5, 11:30 am, Dave Loomer wrote: > In my case, since I was getting the 20-second delay almost 100% of the > time, setting countdown=1 was the answer. If you only see it happen > every 20 or more request then of course it won't help. > > In my case I also run all tasks on the backend. They're slightly more > expensive per hour than frontends (due merely to the lower number of > free hours) but in my case I more than make up for it with the fact > that I have full control on the number of requests that will spin up, > and I need to be able to control that number separately for tasks vs. > users hitting my site. > > On Feb 5, 11:24 am, Carter wrote: > > > > > > > > > We regularly but erratically see 10-20 minute delays in running push > > queue tasks. > > The tasks sit in the queue with ETA as high as 20 minutes *ago* > > without any errors or retries. > > > (the problem seems unrelated to queue settings since our Maximum Rate, > > Enorced Rate and Maximum Concurrent all far exceed the queue's > > throughput at the time of the delays) > > > Any tips or clues on how to prevent this while still using push queues > > without backends? > > > On Feb 1, 9:03 pm, Robert Kluin wrote: > > > > Hey Dave, > > > Hopefully Nick will be able to offer some insight into the cause of > > > your issues. I'd guess it is something related to having very few > > > tasks (one) in thequeue, and it not getting scheduled rapidly. > > > > In your case, you could use pull queues to immediately fetch the > > > nexttaskwhen finished with atask. Or even to fetch multiple tasks > > > and do the work in parallel. Basically you'd have a backend that ran > > > a loop (possibly initiated via a pushtask) that would lease atask, > > > or tasks, from the pullqueue, do the work, delete those tasks, then > > > repeat from the lease stage. The cool thing is that if you're, for > > > example, using URL Fetch to pull data this might let you do it in > > > parallel without increasing your costs much (if any). > > > > Robert > > > > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > > > > Here are logs from three consecutivetaskexecutions over the past > > > > weekend, > > > > with only identifying information removed. You'll see that eachtask > > > > completes in a few milliseconds, but are 20 seconds apart (remember: > > > > I've > > > > already checked myqueueconfigurations, nothing else is running on this > > > > backend, and I later solved the problem by setting countdown=1 when > > > > adding > > > > thetask). I don't see any pending latency mentioned. > > > > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > > > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > > > > task_name=15804554889304913211 instance=0 > > > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 > > > > api_cpu_ms=0 > > > > cpm_usd=0.60 queue_name=overnight-tasks > > > > task_name=15804554889304912461 > > > > instance=0 > > > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 > > > > api_cpu_ms=0 > > > > cpm_usd=0.60 queue_name=overnight-tasks > > > > task_name=4499136807998063691 > > > > instance=0 > > > > > The 20 seconds seems to happen regardless of length oftask. Even though > > > > my > > > > tasks mostly complete in a couple minutes, I do have cases where they > > > > take > > > > several minutes, and I don't see a difference. Of course, when > > > > atasktakes > > > > 5-10 minutes to complete, I'm going to notice and care about a 20-second > > > >delaymuch less than when I'm trying to spin through a few tasks in a > > > >minute > > > > (which is a real-world need for me as well). > > > > > When reading up on pull queues a while back, I was a little confused > > > > about > > > > where I would use them with my own backends. I definitely could see an > > > > application for offloading work to an AWS Linux instance. But in either > > > > case, could you explain why it might help? > > > > > I saw you mention in a separate thread how M/S can perform differently > > > > from > > > > HRD even in cases where one wouldn't expect to see a difference. When I > > > > get > > > > around to it I'm going to create a tiny HRD app and run the same tests > > > > through that. > > > > > I also wonder if M/S could be responsible for frequent latencies in my > > > > admin > > > > console. Those have gotten more frequent and annoying the past couple of > > > > months ... > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "Google App Engine" group. > > > > To view this discussion on the web visit > > > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > > > > To post to this group, send email to google-appengine@googlegroups.com. > > > > To unsubscribe from this group, send email to > > >
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter wrote: > We regularly but erratically see 10-20 minute delays in running push > queue tasks. > The tasks sit in the queue with ETA as high as 20 minutes *ago* > without any errors or retries. > > (the problem seems unrelated to queue settings since our Maximum Rate, > Enorced Rate and Maximum Concurrent all far exceed the queue's > throughput at the time of the delays) > > Any tips or clues on how to prevent this while still using push queues > without backends? > > On Feb 1, 9:03 pm, Robert Kluin wrote: > > > > > > > > > Hey Dave, > > Hopefully Nick will be able to offer some insight into the cause of > > your issues. I'd guess it is something related to having very few > > tasks (one) in thequeue, and it not getting scheduled rapidly. > > > In your case, you could use pull queues to immediately fetch the > > nexttaskwhen finished with atask. Or even to fetch multiple tasks > > and do the work in parallel. Basically you'd have a backend that ran > > a loop (possibly initiated via a pushtask) that would lease atask, > > or tasks, from the pullqueue, do the work, delete those tasks, then > > repeat from the lease stage. The cool thing is that if you're, for > > example, using URL Fetch to pull data this might let you do it in > > parallel without increasing your costs much (if any). > > > Robert > > > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > > > Here are logs from three consecutivetaskexecutions over the past weekend, > > > with only identifying information removed. You'll see that eachtask > > > completes in a few milliseconds, but are 20 seconds apart (remember: I've > > > already checked myqueueconfigurations, nothing else is running on this > > > backend, and I later solved the problem by setting countdown=1 when adding > > > thetask). I don't see any pending latency mentioned. > > > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > > > task_name=15804554889304913211 instance=0 > > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 > > > api_cpu_ms=0 > > > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 > > > instance=0 > > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 > > > api_cpu_ms=0 > > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 > > > instance=0 > > > > The 20 seconds seems to happen regardless of length oftask. Even though my > > > tasks mostly complete in a couple minutes, I do have cases where they take > > > several minutes, and I don't see a difference. Of course, when atasktakes > > > 5-10 minutes to complete, I'm going to notice and care about a 20-second > > >delaymuch less than when I'm trying to spin through a few tasks in a minute > > > (which is a real-world need for me as well). > > > > When reading up on pull queues a while back, I was a little confused about > > > where I would use them with my own backends. I definitely could see an > > > application for offloading work to an AWS Linux instance. But in either > > > case, could you explain why it might help? > > > > I saw you mention in a separate thread how M/S can perform differently > > > from > > > HRD even in cases where one wouldn't expect to see a difference. When I > > > get > > > around to it I'm going to create a tiny HRD app and run the same tests > > > through that. > > > > I also wonder if M/S could be responsible for frequent latencies in my > > > admin > > > console. Those have gotten more frequent and annoying the past couple of > > > months ... > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To view this discussion on the web visit > > >https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > > > To post to this group, send email to google-appengine@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.goog
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I "solved" it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne wrote: > We would have no need to shoot anyone. > > However, the explanations quickly become obsolete. They enter the > folklore in the form that was current at the time and become > entrenched as incorrect information when the implementations have > changed. > > Task Queues use best effort scheduling. They're not real time all the > time, although when our best efforts are running smoothly they can > appear real time. For scheduling, the task eta marks the earliest time > at which the task can run. We can't guarantee that a task WILL run at > that time. > > Steve, we're interested to know about the 10-20 minute delays you've > seen. Can you tell us the app id, queue, and whether the tasks were > added transactionally? An example from your logs would be very > helpful. > > Nick Verne > > > > > > > > On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: > > Carter wrote: We regularly but erratically see 10-20 minute delays in > > running push queue tasks. > > > Been a burr under the saddle forever. What I really don't understand > > -- assuming GAE engineers never see the benefit of providing at least > > one priority/reliability queue -- is why the heck there is never any > > explanation about how tasks get scheduled, and why these weird delays > > happen. It is either: 1) If we told you we would have to shoot you, or > > 2) We can't see the benefit of you understanding this. > > > -stevep > > > On Feb 5, 9:24 am, Carter wrote: > >> We regularly but erratically see 10-20 minute delays in running push > >> queue tasks. > >> The tasks sit in the queue with ETA as high as 20 minutes *ago* > >> without any errors or retries. > > >> (the problem seems unrelated to queue settings since our Maximum Rate, > >> Enorced Rate and Maximum Concurrent all far exceed the queue's > >> throughput at the time of the delays) > > >> Any tips or clues on how to prevent this while still using push queues > >> without backends? > > >> On Feb 1, 9:03 pm, Robert Kluin wrote: > > >> > Hey Dave, > >> > Hopefully Nick will be able to offer some insight into the cause of > >> > your issues. I'd guess it is something related to having very few > >> > tasks (one) in thequeue, and it not getting scheduled rapidly. > > >> > In your case, you could use pull queues to immediately fetch the > >> > nexttaskwhen finished with atask. Or even to fetch multiple tasks > >> > and do the work in parallel. Basically you'd have a backend that ran > >> > a loop (possibly initiated via a pushtask) that would lease atask, > >> > or tasks, from the pullqueue, do the work, delete those tasks, then > >> > repeat from the lease stage. The cool thing is that if you're, for > >> > example, using URL Fetch to pull data this might let you do it in > >> > parallel without increasing your costs much (if any). > > >> > Robert > > >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > >> > > Here are logs from three consecutivetaskexecutions over the past > >> > > weekend, > >> > > with only identifying information removed. You'll see that eachtask > >> > > completes in a few milliseconds, but are 20 seconds apart (remember: > >> > > I've > >> > > already checked myqueueconfigurations, nothing else is running on this > >> > > backend, and I later solved the problem by setting countdown=1 when > >> > > adding > >> > > thetask). I don't see any pending latency mentioned. > > >> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > >> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=15804554889304913211 instance=0 > >> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 > >> > > api_cpu_ms=0 > >> > > cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=15804554889304912461 > >> > > instance=0 > >> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 > >> > > api_cpu_ms=0 > >> > > cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=4499136807998063691 > >> > > instance=0 > > >> > > The 20 seconds seems to happen regardless of length oftask. Even > >> > > though my > >> > > tasks mostly complete in a couple minutes, I do have cases where they > >> > > take > >> > > several minutes, and I don't see a difference. Of course, when > >> > > atasktakes > >> > > 5-10 minutes to complete, I'm going to notice and care about a > >> > > 20-second > >> > >delaymuch less than when I'm trying to spin through a few tasks in a > >> > >minute > >> > > (which is a real-world need for me as well). > > >> > > When readin
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Thanks Nick. Been on the forums for years, and that's the first I remember having a GAE engineer delve into TQs at this level. (BTW Carter replied already about his 20 minute starts). Maybe over time TQ scheduling might expand to enhance multi-instance optimization, and provide a means for a priority queue (with constraints such as volume capacity). -stevep On Feb 5, 3:59 pm, Nicholas Verne wrote: > We would have no need to shoot anyone. > > However, the explanations quickly become obsolete. They enter the > folklore in the form that was current at the time and become > entrenched as incorrect information when the implementations have > changed. > > Task Queues use best effort scheduling. They're not real time all the > time, although when our best efforts are running smoothly they can > appear real time. For scheduling, the task eta marks the earliest time > at which the task can run. We can't guarantee that a task WILL run at > that time. > > Steve, we're interested to know about the 10-20 minute delays you've > seen. Can you tell us the app id, queue, and whether the tasks were > added transactionally? An example from your logs would be very > helpful. > > Nick Verne > > > > > > > > On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: > > Carter wrote: We regularly but erratically see 10-20 minute delays in > > running push queue tasks. > > > Been a burr under the saddle forever. What I really don't understand > > -- assuming GAE engineers never see the benefit of providing at least > > one priority/reliability queue -- is why the heck there is never any > > explanation about how tasks get scheduled, and why these weird delays > > happen. It is either: 1) If we told you we would have to shoot you, or > > 2) We can't see the benefit of you understanding this. > > > -stevep > > > On Feb 5, 9:24 am, Carter wrote: > >> We regularly but erratically see 10-20 minute delays in running push > >> queue tasks. > >> The tasks sit in the queue with ETA as high as 20 minutes *ago* > >> without any errors or retries. > > >> (the problem seems unrelated to queue settings since our Maximum Rate, > >> Enorced Rate and Maximum Concurrent all far exceed the queue's > >> throughput at the time of the delays) > > >> Any tips or clues on how to prevent this while still using push queues > >> without backends? > > >> On Feb 1, 9:03 pm, Robert Kluin wrote: > > >> > Hey Dave, > >> > Hopefully Nick will be able to offer some insight into the cause of > >> > your issues. I'd guess it is something related to having very few > >> > tasks (one) in thequeue, and it not getting scheduled rapidly. > > >> > In your case, you could use pull queues to immediately fetch the > >> > nexttaskwhen finished with atask. Or even to fetch multiple tasks > >> > and do the work in parallel. Basically you'd have a backend that ran > >> > a loop (possibly initiated via a pushtask) that would lease atask, > >> > or tasks, from the pullqueue, do the work, delete those tasks, then > >> > repeat from the lease stage. The cool thing is that if you're, for > >> > example, using URL Fetch to pull data this might let you do it in > >> > parallel without increasing your costs much (if any). > > >> > Robert > > >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > >> > > Here are logs from three consecutivetaskexecutions over the past > >> > > weekend, > >> > > with only identifying information removed. You'll see that eachtask > >> > > completes in a few milliseconds, but are 20 seconds apart (remember: > >> > > I've > >> > > already checked myqueueconfigurations, nothing else is running on this > >> > > backend, and I later solved the problem by setting countdown=1 when > >> > > adding > >> > > thetask). I don't see any pending latency mentioned. > > >> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > >> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=15804554889304913211 instance=0 > >> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 > >> > > api_cpu_ms=0 > >> > > cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=15804554889304912461 > >> > > instance=0 > >> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 > >> > > api_cpu_ms=0 > >> > > cpm_usd=0.60 queue_name=overnight-tasks > >> > > task_name=4499136807998063691 > >> > > instance=0 > > >> > > The 20 seconds seems to happen regardless of length oftask. Even > >> > > though my > >> > > tasks mostly complete in a couple minutes, I do have cases where they > >> > > take > >> > > several minutes, and I don't see a difference. Of course, when > >> > > atasktakes > >> > > 5-10 minutes to complete, I'm going to notice and care about a > >> > > 20-second > >> > >delaymuch less than when I'm trying to spin through a few tasks in a > >> > >minute > >> > > (which is a real-world need for me as well). > > >> > > When reading up on pull queues a while back, I was a little confused > >>
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Hi, You've got a lot of info there. I tried to skim it, but might have missed some stuff. In your experiments with multiple backends, they are all the same "backend" right? You're not targeting specific instances are you? In other words, is everything targeted at "thebackend," "1.thebackend," or multiple different backends? Do you see pending latencies in the request log headers? That's where the failfast header should help. Rather than dispatch the task, then wait for an instance to handle it, it will immediately get returned to the queue. I don't think it is intended to prevent instances from being spun up. What happens if instead of backends you use front ends plus a max_concurrent_requests equal to the number of backends you were wanting? Does it change the behavior you're seeing? You does the latency of the request itself impact what you're seeing? If you do a lot more work in each request does the situation improve? Robert On Sun, Jan 29, 2012 at 11:56, pdknsk wrote: > > I haven't tried setting countdown. It's too slow for the mail queue > anyway. It might work on the other backend (mentioned in the bug). > I've moved most code away from backends though. > > *- After first minute of the batch run, Task Queue Details page > > consistently shows 6-8 tasks run in last minute even though logs show only > > 2-3 requests served per minute (no errors or anything abnormal shown in the > > logs either). I know for certain the only tasks running in the queue in > > question are from my tests. Why is the number overstated, and why does it > > only happen in the single-instance configuration?* > > I've noticed sth. else, with might be related. When you stop a > backend, and put tasks on it, the tasks will obviously not run. In the > task queue overview in the dashboard however, tasks are reported to > have run in the last minute. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Hi Robert, When targeting backend "thebackend", this was done merely by specifying target=thebackend when creating the Task object. And then in backends.yaml I modify the instances parameter for the backend in question for each of my various tests. I'm not really too concerned about failfast, so I'll skip that one for the moment ... I haven't played with max concurrent requests, but in the test where I used a frontend, I did not see any additional instances spin up (and my site is very low-use, so there was probably a max of 1-2 instances to start with). I see the same length of delay (~20 seconds) whether I'm testing with my rapid-fire test handler, or running in production with tasks that spend 1+ minutes doing actual work (lots of RPCs- datastore, URLfetch, memcache). Hope this helps. Dave -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xF79RpgjpAYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Hey Dave, So do you see pending latencies in the request log headers? What if you further increase the work done within each task so that it takes, for example, 5 minutes per task? Have you considered using pull-queues since you're running all this on backends? Robert On Tue, Jan 31, 2012 at 16:58, Dave Loomer wrote: > Hi Robert, > > When targeting backend "thebackend", this was done merely by specifying > target=thebackend when creating the Task object. And then in backends.yaml I > modify the instances parameter for the backend in question for each of my > various tests. > > I'm not really too concerned about failfast, so I'll skip that one for the > moment ... > > I haven't played with max concurrent requests, but in the test where I used > a frontend, I did not see any additional instances spin up (and my site is > very low-use, so there was probably a max of 1-2 instances to start with). > > I see the same length of delay (~20 seconds) whether I'm testing with my > rapid-fire test handler, or running in production with tasks that spend 1+ > minutes doing actual work (lots of RPCs- datastore, URLfetch, memcache). > > Hope this helps. > Dave > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/xF79RpgjpAYJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Here are logs from three consecutive task executions over the past weekend, with only identifying information removed. You'll see that each task completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked my queue configurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding the task). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length of task. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when a task takes 5-10 minutes to complete, I'm going to notice and care about a 20-second delay much less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Dave, Two questions will help us clarify what you're observing. Are your tasks added transactionally? You mention the pattern of executing a task then enqueuing another. In the one backend case, is there typically no more than one task in the queue? Nick Verne On Thu, Feb 2, 2012 at 6:25 AM, Dave Loomer wrote: > Here are logs from three consecutive task executions over the past weekend, > with only identifying information removed. You'll see that each task > completes in a few milliseconds, but are 20 seconds apart (remember: I've > already checked my queue configurations, nothing else is running on this > backend, and I later solved the problem by setting countdown=1 when adding > the task). I don't see any pending latency mentioned. > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > task_name=15804554889304913211 instance=0 > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 > instance=0 > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 > instance=0 > > > The 20 seconds seems to happen regardless of length of task. Even though my > tasks mostly complete in a couple minutes, I do have cases where they take > several minutes, and I don't see a difference. Of course, when a task takes > 5-10 minutes to complete, I'm going to notice and care about a 20-second > delay much less than when I'm trying to spin through a few tasks in a minute > (which is a real-world need for me as well). > > When reading up on pull queues a while back, I was a little confused about > where I would use them with my own backends. I definitely could see an > application for offloading work to an AWS Linux instance. But in either > case, could you explain why it might help? > > I saw you mention in a separate thread how M/S can perform differently from > HRD even in cases where one wouldn't expect to see a difference. When I get > around to it I'm going to create a tiny HRD app and run the same tests > through that. > > I also wonder if M/S could be responsible for frequent latencies in my admin > console. Those have gotten more frequent and annoying the past couple of > months ... > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
The tasks are not run transactionally, and in my testing the task is the only one in queue. In fact, I also ran the tests *somewhat* successfully on a separate app where this was the only code running. I say somewhat because, as I stated in my original post, the 20-second delays didn't happen every time; more like once every four requests. But when they did occur, it was almost precisely 20 seconds. Below is the entirety of my handler: class AdhocHandler2(webapp.RequestHandler): def get(self): task_create_time = datetime.datetime.now() import string task_create_time_s = task_create_time.strftime("%Y-%m-%d %H:%M:%S") + "." + string.zfill(task_create_time.microsecond, 6) t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': task_create_time_s}, target='overnight-external-data', method='GET') t.add(queue_name = 'overnight-tasks') -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/orTdYjTm7SkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
To do one better, here is the entirety of the Python code: #!/usr/bin/env python # # Copyright 2007 Google Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # import wsgiref.handlers import datetime, time from google.appengine.ext import webapp, db from google.appengine.ext.webapp.util import run_wsgi_app from google.appengine.api import taskqueue class AdhocHandler2(webapp.RequestHandler): def get(self): task_create_time = datetime.datetime.now() import string task_create_time_s = task_create_time.strftime("%Y-%m-%d %H:%M:%S") + "." + string.zfill(task_create_time.microsecond, 6) t = taskqueue.Task(url='/batch/adhoc', params={'task_create_time': task_create_time_s}, target='overnight-external-data', method='GET') t.add(queue_name = 'overnight-tasks') def main(): application = webapp.WSGIApplication([ \ ('/batch/adhoc2', AdhocHandler2), \ ], debug=True) run_wsgi_app(application) if __name__ == '__main__': main() -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ELiNYLg4B9cJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in the queue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the next task when finished with a task. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a push task) that would lease a task, or tasks, from the pull queue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: > Here are logs from three consecutive task executions over the past weekend, > with only identifying information removed. You'll see that each task > completes in a few milliseconds, but are 20 seconds apart (remember: I've > already checked my queue configurations, nothing else is running on this > backend, and I later solved the problem by setting countdown=1 when adding > the task). I don't see any pending latency mentioned. > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks > task_name=15804554889304913211 instance=0 > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 > cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 > instance=0 > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 > instance=0 > > > The 20 seconds seems to happen regardless of length of task. Even though my > tasks mostly complete in a couple minutes, I do have cases where they take > several minutes, and I don't see a difference. Of course, when a task takes > 5-10 minutes to complete, I'm going to notice and care about a 20-second > delay much less than when I'm trying to spin through a few tasks in a minute > (which is a real-world need for me as well). > > When reading up on pull queues a while back, I was a little confused about > where I would use them with my own backends. I definitely could see an > application for offloading work to an AWS Linux instance. But in either > case, could you explain why it might help? > > I saw you mention in a separate thread how M/S can perform differently from > HRD even in cases where one wouldn't expect to see a difference. When I get > around to it I'm going to create a tiny HRD app and run the same tests > through that. > > I also wonder if M/S could be responsible for frequent latencies in my admin > console. Those have gotten more frequent and annoying the past couple of > months ... > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: > Carter wrote: We regularly but erratically see 10-20 minute delays in > running push queue tasks. > > Been a burr under the saddle forever. What I really don't understand > -- assuming GAE engineers never see the benefit of providing at least > one priority/reliability queue -- is why the heck there is never any > explanation about how tasks get scheduled, and why these weird delays > happen. It is either: 1) If we told you we would have to shoot you, or > 2) We can't see the benefit of you understanding this. > > -stevep > > > On Feb 5, 9:24 am, Carter wrote: >> We regularly but erratically see 10-20 minute delays in running push >> queue tasks. >> The tasks sit in the queue with ETA as high as 20 minutes *ago* >> without any errors or retries. >> >> (the problem seems unrelated to queue settings since our Maximum Rate, >> Enorced Rate and Maximum Concurrent all far exceed the queue's >> throughput at the time of the delays) >> >> Any tips or clues on how to prevent this while still using push queues >> without backends? >> >> On Feb 1, 9:03 pm, Robert Kluin wrote: >> >> >> >> >> >> >> >> > Hey Dave, >> > Hopefully Nick will be able to offer some insight into the cause of >> > your issues. I'd guess it is something related to having very few >> > tasks (one) in thequeue, and it not getting scheduled rapidly. >> >> > In your case, you could use pull queues to immediately fetch the >> > nexttaskwhen finished with atask. Or even to fetch multiple tasks >> > and do the work in parallel. Basically you'd have a backend that ran >> > a loop (possibly initiated via a pushtask) that would lease atask, >> > or tasks, from the pullqueue, do the work, delete those tasks, then >> > repeat from the lease stage. The cool thing is that if you're, for >> > example, using URL Fetch to pull data this might let you do it in >> > parallel without increasing your costs much (if any). >> >> > Robert >> >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer wrote: >> > > Here are logs from three consecutivetaskexecutions over the past weekend, >> > > with only identifying information removed. You'll see that eachtask >> > > completes in a few milliseconds, but are 20 seconds apart (remember: I've >> > > already checked myqueueconfigurations, nothing else is running on this >> > > backend, and I later solved the problem by setting countdown=1 when >> > > adding >> > > thetask). I don't see any pending latency mentioned. >> >> > > 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 >> > > api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks >> > > task_name=15804554889304913211 instance=0 >> > > 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 >> > > api_cpu_ms=0 >> > > cpm_usd=0.60 queue_name=overnight-tasks >> > > task_name=15804554889304912461 >> > > instance=0 >> > > 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 >> > > api_cpu_ms=0 >> > > cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 >> > > instance=0 >> >> > > The 20 seconds seems to happen regardless of length oftask. Even though >> > > my >> > > tasks mostly complete in a couple minutes, I do have cases where they >> > > take >> > > several minutes, and I don't see a difference. Of course, when atasktakes >> > > 5-10 minutes to complete, I'm going to notice and care about a 20-second >> > >delaymuch less than when I'm trying to spin through a few tasks in a >> > >minute >> > > (which is a real-world need for me as well). >> >> > > When reading up on pull queues a while back, I was a little confused >> > > about >> > > where I would use them with my own backends. I definitely could see an >> > > application for offloading work to an AWS Linux instance. But in either >> > > case, could you explain why it might help? >> >> > > I saw you mention in a separate thread how M/S can perform differently >> > > from >> > > HRD even in cases where one wouldn't expect to see a difference. When I >> > > get >> > > around to it I'm going to create a tiny HRD app and run the same tests >> > > through that. >> >> > > I also wonder if M/S could be responsible for frequent latencies in my >> > > admin >> > >
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
That's interesting. Did the queue sit there for a long time not running anything, or running tasks very slowly? Are the tasks in that queue generally long-running? I _very_ infrequently bump into that type of issue, but I periodically will see one queue slow down for a while. It *seems* to happen far more often in queues with slower tasks, but I don't have any recent empirical evidence of that. And I *think* I've been told that should not be the case. Robert On Sun, Feb 5, 2012 at 19:27, Carter Maslan wrote: > Nicholas - > > For our examples of the 10-20 minute delay: > app_id=s~camiologger > queue=image-label > (but several other queues experience the same long delays sometimes: > content-process, counter-update, etc...) > > The tasks were not added with transactions; just this code: > Queue queueP = > QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); > TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) > > .param("key", contentKeyString) > > .method(TaskOptions.Method.GET)); > > > Let me know if you need more info. We noticed this in the last few weeks. > Carter > > > > On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer wrote: >> >> As the OP you may be interested in my app ID as well: mn-live. I >> provided some logs a few posts back and some exhaustive details at the >> beginning. >> >> However, you won't see this issue popping up anymore on my app since I >> "solved" it by setting countdown=1 a week ago. Since then, tasks start >> very reliably after a 1.5 second delay. If I remove the countdown >> parameter, then it returns to 20 seconds (+/- .01) pretty reliably. >> >> On Feb 5, 5:59 pm, Nicholas Verne wrote: >> > We would have no need to shoot anyone. >> > >> > However, the explanations quickly become obsolete. They enter the >> > folklore in the form that was current at the time and become >> > entrenched as incorrect information when the implementations have >> > changed. >> > >> > Task Queues use best effort scheduling. They're not real time all the >> > time, although when our best efforts are running smoothly they can >> > appear real time. For scheduling, the task eta marks the earliest time >> > at which the task can run. We can't guarantee that a task WILL run at >> > that time. >> > >> > Steve, we're interested to know about the 10-20 minute delays you've >> > seen. Can you tell us the app id, queue, and whether the tasks were >> > added transactionally? An example from your logs would be very >> > helpful. >> > >> > Nick Verne >> > >> > >> > >> > >> > >> > >> > >> > On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: >> > > Carter wrote: We regularly but erratically see 10-20 minute delays in >> > > running push queue tasks. >> > >> > > Been a burr under the saddle forever. What I really don't understand >> > > -- assuming GAE engineers never see the benefit of providing at least >> > > one priority/reliability queue -- is why the heck there is never any >> > > explanation about how tasks get scheduled, and why these weird delays >> > > happen. It is either: 1) If we told you we would have to shoot you, or >> > > 2) We can't see the benefit of you understanding this. >> > >> > > -stevep >> > >> > > On Feb 5, 9:24 am, Carter wrote: >> > >> We regularly but erratically see 10-20 minute delays in running push >> > >> queue tasks. >> > >> The tasks sit in the queue with ETA as high as 20 minutes *ago* >> > >> without any errors or retries. >> > >> > >> (the problem seems unrelated to queue settings since our Maximum >> > >> Rate, >> > >> Enorced Rate and Maximum Concurrent all far exceed the queue's >> > >> throughput at the time of the delays) >> > >> > >> Any tips or clues on how to prevent this while still using push >> > >> queues >> > >> without backends? >> > >> > >> On Feb 1, 9:03 pm, Robert Kluin wrote: >> > >> > >> > Hey Dave, >> > >> > Hopefully Nick will be able to offer some insight into the cause >> > >> > of >> > >> > your issues. I'd guess it is something related to having very few >> > >> > tasks (one) in thequeue, and it not getting scheduled rapidly. >> > >> > >> > In your case, you could use pull queues to immediately fetch the >> > >> > nexttaskwhen finished with atask. Or even to fetch multiple tasks >> > >> > and do the work in parallel. Basically you'd have a backend that >> > >> > ran >> > >> > a loop (possibly initiated via a pushtask) that would lease atask, >> > >> > or tasks, from the pullqueue, do the work, delete those tasks, then >> > >> > repeat from the lease stage. The cool thing is that if you're, for >> > >> > example, using URL Fetch to pull data this might let you do it in >> > >> > parallel without increasing your costs much (if any). >> > >> > >> > Robert >> > >> > >> > On Wed, Feb 1, 2012 at 14:25, Dave Loomer >> > >> > wrote: >> > >> > > Here are logs from three consecutivetaskexecutions over the past >> > >> > > weekend, >> > >> > > with only identifying information removed. You'll see that >> > >> >
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
I Just looked at the last 80 that ran. That queue's tasks are running in between 19ms and 2486ms with most of them running around 28ms. The variability relates to the number of quadtree searches needed, but other queues that experience similar delays have running time without much variation(e.g. predictable counter updates) When the delays happen, there just aren't many tasks in the queue at all. It appears that the delayed tasks are just sitting in the queue idle. On Feb 5, 2012, at 9:17 PM, Robert Kluin wrote: > That's interesting. Did the queue sit there for a long time not > running anything, or running tasks very slowly? Are the tasks in that > queue generally long-running? > > I _very_ infrequently bump into that type of issue, but I periodically > will see one queue slow down for a while. It *seems* to happen far > more often in queues with slower tasks, but I don't have any recent > empirical evidence of that. And I *think* I've been told that should > not be the case. > > > Robert > > > > On Sun, Feb 5, 2012 at 19:27, Carter Maslan wrote: >> Nicholas - >> >> For our examples of the 10-20 minute delay: >> app_id=s~camiologger >> queue=image-label >> (but several other queues experience the same long delays sometimes: >> content-process, counter-update, etc...) >> >> The tasks were not added with transactions; just this code: >> Queue queueP = >> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); >> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) >> >> .param("key", contentKeyString) >> >> .method(TaskOptions.Method.GET)); >> >> >> Let me know if you need more info. We noticed this in the last few weeks. >> Carter >> >> >> >> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer wrote: >>> >>> As the OP you may be interested in my app ID as well: mn-live. I >>> provided some logs a few posts back and some exhaustive details at the >>> beginning. >>> >>> However, you won't see this issue popping up anymore on my app since I >>> "solved" it by setting countdown=1 a week ago. Since then, tasks start >>> very reliably after a 1.5 second delay. If I remove the countdown >>> parameter, then it returns to 20 seconds (+/- .01) pretty reliably. >>> >>> On Feb 5, 5:59 pm, Nicholas Verne wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: > Carter wrote: We regularly but erratically see 10-20 minute delays in > running push queue tasks. > Been a burr under the saddle forever. What I really don't understand > -- assuming GAE engineers never see the benefit of providing at least > one priority/reliability queue -- is why the heck there is never any > explanation about how tasks get scheduled, and why these weird delays > happen. It is either: 1) If we told you we would have to shoot you, or > 2) We can't see the benefit of you understanding this. > -stevep > On Feb 5, 9:24 am, Carter wrote: >> We regularly but erratically see 10-20 minute delays in running push >> queue tasks. >> The tasks sit in the queue with ETA as high as 20 minutes *ago* >> without any errors or retries. >> (the problem seems unrelated to queue settings since our Maximum >> Rate, >> Enorced Rate and Maximum Concurrent all far exceed the queue's >> throughput at the time of the delays) >> Any tips or clues on how to prevent this while still using push >> queues >> without backends? >> On Feb 1, 9:03 pm, Robert Kluin wrote: >>> Hey Dave, >>> Hopefully Nick will be able to offer some insight into the cause >>> of >>> your issues. I'd guess it is something related to having very few >>> tasks (one) in thequeue, and it not getting scheduled rapidly. >>> In your case, you could use pull queues to immediately fetch the >>> nexttaskwhen finished with atask. Or even to fetch multiple tasks >>> and do the work in parallel. Basically you'd have a backend that >>> ran >>> a loop (possibly initiated via a pushtask) that would le
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Does the app get a lot of front-end traffic or is it sitting idle when the delays occur? On Mon, Feb 6, 2012 at 01:38, Carter Maslan wrote: > I Just looked at the last 80 that ran. > That queue's tasks are running in between 19ms and 2486ms with most of them > running around 28ms. The variability relates to the number of quadtree > searches needed, but other queues that experience similar delays have running > time without much variation(e.g. predictable counter updates) > When the delays happen, there just aren't many tasks in the queue at all. > It appears that the delayed tasks are just sitting in the queue idle. > > > > On Feb 5, 2012, at 9:17 PM, Robert Kluin wrote: > >> That's interesting. Did the queue sit there for a long time not >> running anything, or running tasks very slowly? Are the tasks in that >> queue generally long-running? >> >> I _very_ infrequently bump into that type of issue, but I periodically >> will see one queue slow down for a while. It *seems* to happen far >> more often in queues with slower tasks, but I don't have any recent >> empirical evidence of that. And I *think* I've been told that should >> not be the case. >> >> >> Robert >> >> >> >> On Sun, Feb 5, 2012 at 19:27, Carter Maslan wrote: >>> Nicholas - >>> >>> For our examples of the 10-20 minute delay: >>> app_id=s~camiologger >>> queue=image-label >>> (but several other queues experience the same long delays sometimes: >>> content-process, counter-update, etc...) >>> >>> The tasks were not added with transactions; just this code: >>> Queue queueP = >>> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); >>> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) >>> >>> .param("key", contentKeyString) >>> >>> .method(TaskOptions.Method.GET)); >>> >>> >>> Let me know if you need more info. We noticed this in the last few weeks. >>> Carter >>> >>> >>> >>> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer wrote: As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I "solved" it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne wrote: > We would have no need to shoot anyone. > > However, the explanations quickly become obsolete. They enter the > folklore in the form that was current at the time and become > entrenched as incorrect information when the implementations have > changed. > > Task Queues use best effort scheduling. They're not real time all the > time, although when our best efforts are running smoothly they can > appear real time. For scheduling, the task eta marks the earliest time > at which the task can run. We can't guarantee that a task WILL run at > that time. > > Steve, we're interested to know about the 10-20 minute delays you've > seen. Can you tell us the app id, queue, and whether the tasks were > added transactionally? An example from your logs would be very > helpful. > > Nick Verne > > > > > > > > On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: >> Carter wrote: We regularly but erratically see 10-20 minute delays in >> running push queue tasks. > >> Been a burr under the saddle forever. What I really don't understand >> -- assuming GAE engineers never see the benefit of providing at least >> one priority/reliability queue -- is why the heck there is never any >> explanation about how tasks get scheduled, and why these weird delays >> happen. It is either: 1) If we told you we would have to shoot you, or >> 2) We can't see the benefit of you understanding this. > >> -stevep > >> On Feb 5, 9:24 am, Carter wrote: >>> We regularly but erratically see 10-20 minute delays in running push >>> queue tasks. >>> The tasks sit in the queue with ETA as high as 20 minutes *ago* >>> without any errors or retries. > >>> (the problem seems unrelated to queue settings since our Maximum >>> Rate, >>> Enorced Rate and Maximum Concurrent all far exceed the queue's >>> throughput at the time of the delays) > >>> Any tips or clues on how to prevent this while still using push >>> queues >>> without backends? > >>> On Feb 1, 9:03 pm, Robert Kluin wrote: > Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. > In your case, you could use pull queues to immed
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
The app always has front end traffic and I have noticed the delay at times with high traffic. On Feb 5, 2012 10:57 PM, "Robert Kluin" wrote: > Does the app get a lot of front-end traffic or is it sitting idle when > the delays occur? > > > > > On Mon, Feb 6, 2012 at 01:38, Carter Maslan wrote: > > I Just looked at the last 80 that ran. > > That queue's tasks are running in between 19ms and 2486ms with most of > them running around 28ms. The variability relates to the number of > quadtree searches needed, but other queues that experience similar delays > have running time without much variation(e.g. predictable counter updates) > > When the delays happen, there just aren't many tasks in the queue at all. > > It appears that the delayed tasks are just sitting in the queue idle. > > > > > > > > On Feb 5, 2012, at 9:17 PM, Robert Kluin wrote: > > > >> That's interesting. Did the queue sit there for a long time not > >> running anything, or running tasks very slowly? Are the tasks in that > >> queue generally long-running? > >> > >> I _very_ infrequently bump into that type of issue, but I periodically > >> will see one queue slow down for a while. It *seems* to happen far > >> more often in queues with slower tasks, but I don't have any recent > >> empirical evidence of that. And I *think* I've been told that should > >> not be the case. > >> > >> > >> Robert > >> > >> > >> > >> On Sun, Feb 5, 2012 at 19:27, Carter Maslan wrote: > >>> Nicholas - > >>> > >>> For our examples of the 10-20 minute delay: > >>> app_id=s~camiologger > >>> queue=image-label > >>> (but several other queues experience the same long delays sometimes: > >>> content-process, counter-update, etc...) > >>> > >>> The tasks were not added with transactions; just this code: > >>> Queue queueP = > >>> QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); > >>> TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) > >>> > >>> .param("key", contentKeyString) > >>> > >>> .method(TaskOptions.Method.GET)); > >>> > >>> > >>> Let me know if you need more info. We noticed this in the last few > weeks. > >>> Carter > >>> > >>> > >>> > >>> On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer wrote: > > As the OP you may be interested in my app ID as well: mn-live. I > provided some logs a few posts back and some exhaustive details at the > beginning. > > However, you won't see this issue popping up anymore on my app since I > "solved" it by setting countdown=1 a week ago. Since then, tasks start > very reliably after a 1.5 second delay. If I remove the countdown > parameter, then it returns to 20 seconds (+/- .01) pretty reliably. > > On Feb 5, 5:59 pm, Nicholas Verne wrote: > > We would have no need to shoot anyone. > > > > However, the explanations quickly become obsolete. They enter the > > folklore in the form that was current at the time and become > > entrenched as incorrect information when the implementations have > > changed. > > > > Task Queues use best effort scheduling. They're not real time all the > > time, although when our best efforts are running smoothly they can > > appear real time. For scheduling, the task eta marks the earliest > time > > at which the task can run. We can't guarantee that a task WILL run at > > that time. > > > > Steve, we're interested to know about the 10-20 minute delays you've > > seen. Can you tell us the app id, queue, and whether the tasks were > > added transactionally? An example from your logs would be very > > helpful. > > > > Nick Verne > > > > > > > > > > > > > > > > On Mon, Feb 6, 2012 at 9:27 AM, stevep wrote: > >> Carter wrote: We regularly but erratically see 10-20 minute delays > in > >> running push queue tasks. > > > >> Been a burr under the saddle forever. What I really don't understand > >> -- assuming GAE engineers never see the benefit of providing at > least > >> one priority/reliability queue -- is why the heck there is never any > >> explanation about how tasks get scheduled, and why these weird > delays > >> happen. It is either: 1) If we told you we would have to shoot you, > or > >> 2) We can't see the benefit of you understanding this. > > > >> -stevep > > > >> On Feb 5, 9:24 am, Carter wrote: > >>> We regularly but erratically see 10-20 minute delays in running > push > >>> queue tasks. > >>> The tasks sit in the queue with ETA as high as 20 minutes *ago* > >>> without any errors or retries. > > > >>> (the problem seems unrelated to queue settings since our Maximum > >>> Rate, > >>> Enorced Rate and Maximum Concurrent all far exceed the queue's > >>> throughput at the time of the delays) > > > >>> Any tips or clues on how to prevent this while still using push > >>> queues > >>> without