[google-appengine] Re: Runaway Tasks
I did not - i'm definitely going to add that, thank Dale. I re- enabled the queue the other day and it's been behaving as expected, so it was definitely some external factor that made it spin out of control. My project is a cloud based data historian on www.nimbits.com where users attach arduino devices to the services and record second by second data, so when they delete a data point I need to chew through the obsolete values that were recorded to that point. I ran out of quota on a day where a user deleted a point with millions of recorded values on it. It was supposed to just be a small background task. On Mar 6, 11:44 am, Dale dalehu...@gmail.com wrote: Did you set max_concurrent_requests for that queue? Perhaps there was some latency on the datastore and your queue was adding 5 tasks each second but all the tasks were still running until the 10 s datastore timeout and then retrying. In that case you would get 50 tasks executing in parallel all querying 1000 records. Could explain the high CPU usage. Dale On Mar 5, 5:33 pm, Benjamin bsaut...@gmail.com wrote: Hi Nick, The app id is nimbits1 (nimbits1.appspot.com) The spike occurred around 12pm EST on March 3rd. 95% of the cpu being used at the time was Task Queues deletedata which is set to 5/sec with a bucket size of 5. Thanks! On Mar 4, 4:21 pm, Nicholas Verne nve...@google.com wrote: Could you give us your app id and the approximate time of the gobbling of CPU so we can investigate? Thanks, Nick On Sat, Mar 5, 2011 at 5:49 AM, Benjamin bsaut...@gmail.com wrote: I have a Task that's purpose is to chew away at data that needs to be deleted. I have millions of records that need to go, so the task uses the low level api and key only queries to grab 1000 delete them and restart the task until the remaining count is zero. The task is set to 5/sec with a bucket size of 5. The other day it skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds / second, i think the lights in New York dimmed). Am i missing some setting here? How do i keep a task queue chugging along at a fixed rate without a burst like that, regardless of how many tasks are in the queue? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Runaway Tasks
This is a template for what I hope is how to best handle a low priory task with very little impact on the system if it fails. It just cleans up expired data and should not build up. Any other feedback is appreciated. queue namedeletedata/name rate20/s/rate bucket-size40/bucket-size max-concurrent-requests50/max-concurrent-requests task-retry-limit1/task-retry-limit /queue On Mar 7, 7:40 am, Benjamin bsaut...@gmail.com wrote: I did not - i'm definitely going to add that, thank Dale. I re- enabled the queue the other day and it's been behaving as expected, so it was definitely some external factor that made it spin out of control. My project is a cloud based data historian onwww.nimbits.comwhere users attach arduino devices to the services and record second by second data, so when they delete a data point I need to chew through the obsolete values that were recorded to that point. I ran out of quota on a day where a user deleted a point with millions of recorded values on it. It was supposed to just be a small background task. On Mar 6, 11:44 am, Dale dalehu...@gmail.com wrote: Did you set max_concurrent_requests for that queue? Perhaps there was some latency on the datastore and your queue was adding 5 tasks each second but all the tasks were still running until the 10 s datastore timeout and then retrying. In that case you would get 50 tasks executing in parallel all querying 1000 records. Could explain the high CPU usage. Dale On Mar 5, 5:33 pm, Benjamin bsaut...@gmail.com wrote: Hi Nick, The app id is nimbits1 (nimbits1.appspot.com) The spike occurred around 12pm EST on March 3rd. 95% of the cpu being used at the time was Task Queues deletedata which is set to 5/sec with a bucket size of 5. Thanks! On Mar 4, 4:21 pm, Nicholas Verne nve...@google.com wrote: Could you give us your app id and the approximate time of the gobbling of CPU so we can investigate? Thanks, Nick On Sat, Mar 5, 2011 at 5:49 AM, Benjamin bsaut...@gmail.com wrote: I have a Task that's purpose is to chew away at data that needs to be deleted. I have millions of records that need to go, so the task uses the low level api and key only queries to grab 1000 delete them and restart the task until the remaining count is zero. The task is set to 5/sec with a bucket size of 5. The other day it skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds / second, i think the lights in New York dimmed). Am i missing some setting here? How do i keep a task queue chugging along at a fixed rate without a burst like that, regardless of how many tasks are in the queue? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Runaway Tasks
Hi Benjamin. Just a thought: for a background task that's not critical 20/s and 50 concurrent looks a bit high. Maybe set it even lower (5/s, bucket 5, 5 concurrent) and then monitor the queue. If it often gets 'too long' (whatever that means for your app) you can turn it up. I may be preaching to the choir but remember tasks also compete with incoming requests for your apps' resources. Also many long running tasks could push your average latency above 1000ms and encourage AppEngine to stop spinning up new instances. Dale On Mar 7, 2:47 pm, Benjamin bsaut...@gmail.com wrote: This is a template for what I hope is how to best handle a low priory task with very little impact on the system if it fails. It just cleans up expired data and should not build up. Any other feedback is appreciated. queue namedeletedata/name rate20/s/rate bucket-size40/bucket-size max-concurrent-requests50/max-concurrent-requests task-retry-limit1/task-retry-limit /queue On Mar 7, 7:40 am, Benjamin bsaut...@gmail.com wrote: I did not - i'm definitely going to add that, thank Dale. I re- enabled the queue the other day and it's been behaving as expected, so it was definitely some external factor that made it spin out of control. My project is a cloud based data historian onwww.nimbits.comwhere users attach arduino devices to the services and record second by second data, so when they delete a data point I need to chew through the obsolete values that were recorded to that point. I ran out of quota on a day where a user deleted a point with millions of recorded values on it. It was supposed to just be a small background task. On Mar 6, 11:44 am, Dale dalehu...@gmail.com wrote: Did you set max_concurrent_requests for that queue? Perhaps there was some latency on the datastore and your queue was adding 5 tasks each second but all the tasks were still running until the 10 s datastore timeout and then retrying. In that case you would get 50 tasks executing in parallel all querying 1000 records. Could explain the high CPU usage. Dale On Mar 5, 5:33 pm, Benjamin bsaut...@gmail.com wrote: Hi Nick, The app id is nimbits1 (nimbits1.appspot.com) The spike occurred around 12pm EST on March 3rd. 95% of the cpu being used at the time was Task Queues deletedata which is set to 5/sec with a bucket size of 5. Thanks! On Mar 4, 4:21 pm, Nicholas Verne nve...@google.com wrote: Could you give us your app id and the approximate time of the gobbling of CPU so we can investigate? Thanks, Nick On Sat, Mar 5, 2011 at 5:49 AM, Benjamin bsaut...@gmail.com wrote: I have a Task that's purpose is to chew away at data that needs to be deleted. I have millions of records that need to go, so the task uses the low level api and key only queries to grab 1000 delete them and restart the task until the remaining count is zero. The task is set to 5/sec with a bucket size of 5. The other day it skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds / second, i think the lights in New York dimmed). Am i missing some setting here? How do i keep a task queue chugging along at a fixed rate without a burst like that, regardless of how many tasks are in the queue? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Runaway Tasks
Did you set max_concurrent_requests for that queue? Perhaps there was some latency on the datastore and your queue was adding 5 tasks each second but all the tasks were still running until the 10 s datastore timeout and then retrying. In that case you would get 50 tasks executing in parallel all querying 1000 records. Could explain the high CPU usage. Dale On Mar 5, 5:33 pm, Benjamin bsaut...@gmail.com wrote: Hi Nick, The app id is nimbits1 (nimbits1.appspot.com) The spike occurred around 12pm EST on March 3rd. 95% of the cpu being used at the time was Task Queues deletedata which is set to 5/sec with a bucket size of 5. Thanks! On Mar 4, 4:21 pm, Nicholas Verne nve...@google.com wrote: Could you give us your app id and the approximate time of the gobbling of CPU so we can investigate? Thanks, Nick On Sat, Mar 5, 2011 at 5:49 AM, Benjamin bsaut...@gmail.com wrote: I have a Task that's purpose is to chew away at data that needs to be deleted. I have millions of records that need to go, so the task uses the low level api and key only queries to grab 1000 delete them and restart the task until the remaining count is zero. The task is set to 5/sec with a bucket size of 5. The other day it skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds / second, i think the lights in New York dimmed). Am i missing some setting here? How do i keep a task queue chugging along at a fixed rate without a burst like that, regardless of how many tasks are in the queue? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Runaway Tasks
Hi Nick, The app id is nimbits1 (nimbits1.appspot.com) The spike occurred around 12pm EST on March 3rd. 95% of the cpu being used at the time was Task Queues deletedata which is set to 5/sec with a bucket size of 5. Thanks! On Mar 4, 4:21 pm, Nicholas Verne nve...@google.com wrote: Could you give us your app id and the approximate time of the gobbling of CPU so we can investigate? Thanks, Nick On Sat, Mar 5, 2011 at 5:49 AM, Benjamin bsaut...@gmail.com wrote: I have a Task that's purpose is to chew away at data that needs to be deleted. I have millions of records that need to go, so the task uses the low level api and key only queries to grab 1000 delete them and restart the task until the remaining count is zero. The task is set to 5/sec with a bucket size of 5. The other day it skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds / second, i think the lights in New York dimmed). Am i missing some setting here? How do i keep a task queue chugging along at a fixed rate without a burst like that, regardless of how many tasks are in the queue? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
