For starting would it be too much to ask for a notification (e-mail, text message etc) at certain thresholds of the daily budget, e.g. when 75% is exceeded?
PK http://www.gae123.com On May 27, 2013, at 12:44 PM, Kaan Soral <kaanso...@gmail.com> wrote: > It's always fun to read other appengine success stories, when I saw $1000 a > day, I was pretty excited, turns out it was a different kind of explosion > > The cause is a bit unclear, however it seems that the cost was $1000 because > appengine did it's job well > > The scenario you described probably happened to each of us at a scale, I > don't know what happens when you consume your daily quota, is it easily > replenished etc? > > But certainly it would be helpful to have a safeguard etc like you mentioned, > but it would be really hard to define/implement, it would also be confusing, > could produce false positives and side-effects etc. > > Generally it's a good idea to not repeat tasks at all, if it's possible it's > a great idea to design a system that will retry them automatically. For > example a periodical data store query based mapreduce that would re-do the > task if the element is not marked etc. > > my .02 > > On Monday, May 27, 2013 6:46:37 AM UTC+3, Mike wrote: >> >> Hi guys >> >> Today our app went offline for 30 minutes because we hit our daily budget of >> $1,000. The cost was primarily in database reads. Normally we'd be expecting >> $100 - $200. >> >> There were some database timeouts just before we went offline. One initial >> theory is that the database failing caused many tasks to continually retry. >> Unfortunately I think, at the time, were were doing a full backup and also >> syncing our customer records with MailChimp. >> >> Did anyone else notice another similar? Anyone got any ideas about how to >> limit costs blowouts like this when there is some problem with the >> underlying infrastructure and you have a lot of failing tasks basically >> DoS'ing your own site? The only "solution" I can think of is to have a >> really really high daily budget, and try to periodically detect abnormal >> usage... That's quite risky though if you don't detect it in time. >> >> It's hard to know who should fix this. Is it AppEngine's job because the >> infrastructure (potentially) failed? Or is it our fault for not trying to >> detect that? That's really hard when you're running a really big MapReduce, >> for example, where you expect a few errors, but you don't want errors to >> continue to fester and then take your site offline. >> >> Maybe there could be a setting to automatically pause a task queue and email >> the administrators if X% of tasks fail within a certain timeframe? >> >> Cheers >> Mike > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
