Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
The more I think about this, the more I think the fundamental problem is the aggressive retry schedule on the task queue. This is really easy to fix in your own app. With the exception of the task queue, it seems like every GAE expense grows proportionally with user traffic. However, one coding error can easily cause the task queue to DOS your app with retries. For most tasks, if two tries don't work then you might as well wait an hour before trying again - chances are some third-party service is down or there may be a code issue that requires developer intervention. Maybe you catch the problem after a couple hours, so your bill is 4X what it normally would be... but this is better than the two alternatives: 100X bill, or your app going down due to quota exhaustion. I'm going to change my retry schedule now, and I recommend everyone else do the same. Jeff On Fri, Jun 7, 2013 at 5:32 AM, Marcel Manz marcel.m...@gmail.com wrote: I guess open an issue for these suggestions and star them would be a very good way for GAE Team to hear us. Added, please star: https://code.google.com/p/googleappengine/issues/detail?id=9455 Marcel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
I guess open an issue for these suggestions and star them would be a very good way for GAE Team to hear us. Added, please star: https://code.google.com/p/googleappengine/issues/detail?id=9455 Marcel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
I guess open an issue for these suggestions and star them would be a very good way for GAE Team to hear us. Saludos. Moisés Belchín. 2013/5/28 Bryce Cutt pandas...@gmail.com +1 for adding an optional daily limit for premier accounts. Obviously I work to remove this sort of issue but bugs happen and I don't see an easy way for me to see my monthly bill amount before I am actually billed. I kind of miss daily usage reports with dollar values On Tuesday, May 28, 2013 1:51:59 AM UTC-7, Marcel Manz wrote: For starting would it be too much to ask for a notification (e-mail, text message etc) at certain thresholds of the daily budget, e.g. when 75% is exceeded? Once you're a premier client you no longer have the daily budget option and can basically spend whatever money per day (unless there's some internal limit set by Google). I like your notification warning idea, but that should work for both non-premier (credit cards) as well as premier apps (which are getting invoiced). That said it would be good if Google would also introduce a daily limit for premier apps for protective reasons. If our code would do wrong and create such loops we could end up paying $100'000 per day or more. So as a premier client you have to be really careful not to create such loop holes. Marcel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
For starting would it be too much to ask for a notification (e-mail, text message etc) at certain thresholds of the daily budget, e.g. when 75% is exceeded? Once you're a premier client you no longer have the daily budget option and can basically spend whatever money per day (unless there's some internal limit set by Google). I like your notification warning idea, but that should work for both non-premier (credit cards) as well as premier apps (which are getting invoiced). That said it would be good if Google would also introduce a daily limit for premier apps for protective reasons. If our code would do wrong and create such loops we could end up paying $100'000 per day or more. So as a premier client you have to be really careful not to create such loop holes. Marcel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
+1 for adding an optional daily limit for premier accounts. Obviously I work to remove this sort of issue but bugs happen and I don't see an easy way for me to see my monthly bill amount before I am actually billed. I kind of miss daily usage reports with dollar values On Tuesday, May 28, 2013 1:51:59 AM UTC-7, Marcel Manz wrote: For starting would it be too much to ask for a notification (e-mail, text message etc) at certain thresholds of the daily budget, e.g. when 75% is exceeded? Once you're a premier client you no longer have the daily budget option and can basically spend whatever money per day (unless there's some internal limit set by Google). I like your notification warning idea, but that should work for both non-premier (credit cards) as well as premier apps (which are getting invoiced). That said it would be good if Google would also introduce a daily limit for premier apps for protective reasons. If our code would do wrong and create such loops we could end up paying $100'000 per day or more. So as a premier client you have to be really careful not to create such loop holes. Marcel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
It's always fun to read other appengine success stories, when I saw $1000 a day, I was pretty excited, turns out it was a different kind of explosion The cause is a bit unclear, however it seems that the cost was $1000 because appengine did it's job well The scenario you described probably happened to each of us at a scale, I don't know what happens when you consume your daily quota, is it easily replenished etc? But certainly it would be helpful to have a safeguard etc like you mentioned, but it would be really hard to define/implement, it would also be confusing, could produce false positives and side-effects etc. Generally it's a good idea to not repeat tasks at all, if it's possible it's a great idea to design a system that will retry them automatically. For example a periodical data store query based mapreduce that would re-do the task if the element is not marked etc. my .02 On Monday, May 27, 2013 6:46:37 AM UTC+3, Mike wrote: Hi guys Today our app went offline for 30 minutes because we hit our daily budget of $1,000. The cost was primarily in database reads. Normally we'd be expecting $100 - $200. There were some database timeouts just before we went offline. One initial theory is that the database failing caused many tasks to continually retry. Unfortunately I think, at the time, were were doing a full backup and also syncing our customer records with MailChimp. Did anyone else notice another similar? Anyone got any ideas about how to limit costs blowouts like this when there is some problem with the underlying infrastructure and you have a lot of failing tasks basically DoS'ing your own site? The only solution I can think of is to have a really really high daily budget, and try to periodically detect abnormal usage... That's quite risky though if you don't detect it in time. It's hard to know who should fix this. Is it AppEngine's job because the infrastructure (potentially) failed? Or is it our fault for not trying to detect that? That's really hard when you're running a really big MapReduce, for example, where you expect a few errors, but you don't want errors to continue to fester and then take your site offline. Maybe there could be a setting to automatically pause a task queue and email the administrators if X% of tasks fail within a certain timeframe? Cheers Mike -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Huge increase in datastore reads - hit $1,000 a day
For starting would it be too much to ask for a notification (e-mail, text message etc) at certain thresholds of the daily budget, e.g. when 75% is exceeded? PK http://www.gae123.com On May 27, 2013, at 12:44 PM, Kaan Soral kaanso...@gmail.com wrote: It's always fun to read other appengine success stories, when I saw $1000 a day, I was pretty excited, turns out it was a different kind of explosion The cause is a bit unclear, however it seems that the cost was $1000 because appengine did it's job well The scenario you described probably happened to each of us at a scale, I don't know what happens when you consume your daily quota, is it easily replenished etc? But certainly it would be helpful to have a safeguard etc like you mentioned, but it would be really hard to define/implement, it would also be confusing, could produce false positives and side-effects etc. Generally it's a good idea to not repeat tasks at all, if it's possible it's a great idea to design a system that will retry them automatically. For example a periodical data store query based mapreduce that would re-do the task if the element is not marked etc. my .02 On Monday, May 27, 2013 6:46:37 AM UTC+3, Mike wrote: Hi guys Today our app went offline for 30 minutes because we hit our daily budget of $1,000. The cost was primarily in database reads. Normally we'd be expecting $100 - $200. There were some database timeouts just before we went offline. One initial theory is that the database failing caused many tasks to continually retry. Unfortunately I think, at the time, were were doing a full backup and also syncing our customer records with MailChimp. Did anyone else notice another similar? Anyone got any ideas about how to limit costs blowouts like this when there is some problem with the underlying infrastructure and you have a lot of failing tasks basically DoS'ing your own site? The only solution I can think of is to have a really really high daily budget, and try to periodically detect abnormal usage... That's quite risky though if you don't detect it in time. It's hard to know who should fix this. Is it AppEngine's job because the infrastructure (potentially) failed? Or is it our fault for not trying to detect that? That's really hard when you're running a really big MapReduce, for example, where you expect a few errors, but you don't want errors to continue to fester and then take your site offline. Maybe there could be a setting to automatically pause a task queue and email the administrators if X% of tasks fail within a certain timeframe? Cheers Mike -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
