[google-appengine] Re: Huge number of datastore reads?

[Dave Peck]

The ID is get-cloak-live.

No remote queries; everything would have to have been generated by
handling requests (some of which were our cron jobs.)

Thanks,
Dave

On Mar 6, 4:53 pm, Alfred Fuller 
wrote:
> What is your app id?
>
> Did you perform a lot of queries using remote api?
>
>
>
>
>
>
>
> On Tue, Mar 6, 2012 at 4:40 PM, Dave Peck  wrote:
> > My production application went over its daily budget today. Let's just
> > say that my daily budget is roughly 3x what I've ever actually needed
> > on my busiest day. Today was not my busiest day by a long shot.
>
> > It appears that I serviced a small number of requests today. However,
> > the dashboard claims that I performed roughly 10,000x the number of
> > datastore read ops as requests, at a cost of
> > $LOTS_OF_MONEY_FOR_ONE_DAY
>
> > This seems like nonsense. My code hasn't changed in a while. I've
> > never reached a quota quite like this. And there's no way my average
> > request requires 10,000 read ops. Just no.
>
> > Google team -- is there someone I can speak with? I'd like to
> > understand in detail what happened and how to prevent it going
> > forward.
>
> > -Dave
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Greg]

I've noticed a couple of blips in memcache today - if you cache using
memcache, maybe you're having to revert to datastore more often than
usual?

Cheers
Greg.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Dave Peck]

To be clear: I lost real customers and real money today.

The problem was compounded by apparent issues with Google Checkout on
the iPhone. I got a downtime notification and quickly determined the
root cause. I was on the road, so I pulled over and immediately logged
in with my iPhone to update the billing information. I more-than-
doubled our daily budget and submitted it. It looked like everything
worked. (That is to say: the form let me type in a new budget and
submit it.) Billing was "frozen" for 30 minutes. I kept refreshing,
only to discover that for whatever reason _it hadn't worked_. 30
minutes later, I tried again. This doomed us to another 30 minutes of
downtime. It wasn't until I raced home to my laptop that I was able to
successfully update the billing. This needs to be fixed.

The worst of it: we looked like rank amateurs in several ways, but
perhaps none greater than our 500 page. App Engine decided to display
a generic Google-logo'd "Over Quota" 500 page instead of our custom
500 error page. What customer in their right mind, after seeing such
an embarrassment, would think that we're serious about our business?

-Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Dave Peck]

Although I was hoping for useful, non-judgmental replies too.

-Dave

On Mar 6, 10:08 pm, "Brandon Wirtz"  wrote:
> > The worst of it: we looked like rank amateurs in several ways, but perhaps
>
> I'd say "no offense" but I'd be lying.
>
> "More than doubled"
>
> My Daily budget sits at 50x my biggest recorded day on almost every app I
> have.  If your model is to make money, and you are pretty sure you don't
> have a bug that is going to cost you millions of dollars. There is no reason
> to not set your budget sky high. If you have only 30 minutes to get your app
> back with in quota, you are rank amateurs. Your Scale is only limited by
> your budget. If you aren't going to set you budget to as large as you
> imagine you would want to scale you are doing it wrong.
>
> If you are going to run with the training wheels on, don't be surprised when
> you hit a pot hole and your back tire just spins with you going nowhere.
>
> -Brandon

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Dave Peck]

 > Should be possible:http://code.google.com/appengine/docs/java/
config/appconfig.html#Cust...

Ah, nice feature. Thanks; curious when this was added.

Cheers,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Dave Peck]

Hi Chris,

Thanks!

> Sorry that you had issues with Checkout. We continue to work with them on
> improvements that they should make to the service (and I've forward this
> thread on to them).

Good to hear. If I can provide further details, just let me know. In
the final analysis, it seemed to us that the billing update issued
from my iPhone worked... but it didn't. More accurate feedback would
have been helpful.

> Did you figure out why the DS reads increased? Is this still an issue?

The operating theory at the moment is that a change to how our back-
end services report back to App Engine triggered a latent (and really
nasty) performance bug. We're still tracking down the details now;
Alfred has been immensely helpful offline!

> Setting your budget higher is probably not a bad idea, but totally up to
> you. I think the pros and cons have been outlined (in one way or another)
> here :-)

Indeed they were. ;-)

Cheers,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Alexander Botov]

Experiencing same problem, 5 minutes normal usage of the application
adds up about 10K datastore reads, is it related to 1.6.3 upgrade?
remote_api is not used ...

Regards,
--alex

On Mar 7, 2:53 am, Alfred Fuller 
wrote:
> What is your app id?
>
> Did you perform a lot of queries using remote api?
>
>
>
>
>
>
>
> On Tue, Mar 6, 2012 at 4:40 PM, Dave Peck  wrote:
> > My production application went over its daily budget today. Let's just
> > say that my daily budget is roughly 3x what I've ever actually needed
> > on my busiest day. Today was not my busiest day by a long shot.
>
> > It appears that I serviced a small number of requests today. However,
> > the dashboard claims that I performed roughly 10,000x the number of
> > datastore read ops as requests, at a cost of
> > $LOTS_OF_MONEY_FOR_ONE_DAY
>
> > This seems like nonsense. My code hasn't changed in a while. I've
> > never reached a quota quite like this. And there's no way my average
> > request requires 10,000 read ops. Just no.
>
> > Google team -- is there someone I can speak with? I'd like to
> > understand in detail what happened and how to prevent it going
> > forward.
>
> > -Dave
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Huge number of datastore reads?

[Brian]

Dave, did you get any resolution to this issue? I seem to be having a similar 
issue--
 5 minutes of read ops on a few objects via remote api consumes my free 
daily quota of 50K. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/BRM-dNM3p6oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Huge number of datastore reads?

[Brandon Wirtz]

> The worst of it: we looked like rank amateurs in several ways, but perhaps

I'd say "no offense" but I'd be lying.

"More than doubled"

My Daily budget sits at 50x my biggest recorded day on almost every app I
have.  If your model is to make money, and you are pretty sure you don't
have a bug that is going to cost you millions of dollars. There is no reason
to not set your budget sky high. If you have only 30 minutes to get your app
back with in quota, you are rank amateurs. Your Scale is only limited by
your budget. If you aren't going to set you budget to as large as you
imagine you would want to scale you are doing it wrong.

If you are going to run with the training wheels on, don't be surprised when
you hit a pot hole and your back tire just spins with you going nowhere.

-Brandon


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Huge number of datastore reads?

[Jeff Schnitzer]

On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz  wrote:

> > The worst of it: we looked like rank amateurs in several ways, but
> perhaps
>
> If you are going to run with the training wheels on, don't be surprised
> when
> you hit a pot hole and your back tire just spins with you going nowhere.
>

Sorry Brandon, you're way off-base here.  There's no excuse for buried land
mines like this to exist.

Nobody realizes that you get frozen out of the system for 30+ minutes at a
time until you actually get screwed by the system.  And no matter what you
think you've set your budget to, something can come along and blow it.

For example, every night I process a significant volume of CSV data
downloaded from a technically unsavvy organization.  They seem to delight
in finding new and interesting ways to break my (popular third-party) CSV
parser.  The latest was a name field containing "Blah Blah, Jr" - you can
guess what that did.  After a few rounds of this I have hacked my parser
enough to consume just about any garbage they can feed me.  Nevertheless,
there were quite a few nights early on where this error caused a large
task-queue-driven data processing operation to fail.  With the default
retry schedule, it quickly ate through my
(seemed-to-be-generous-at-the-time) billing quota.

Fortunately my technically unsavvy organization is sufficiently clumsy that
these problems came up early.  If they were only slightly more clever,
these issues might not surface until long into production, when it really
mattered.

As those of us in Javaland were reminded today, GAE fails us every now and
then.  It needs to fail gracefully.  The default over-quota message is WAY
more embarrassing than "XYZ is experiencing technical difficulties" - how
can we override it?

If...you are pretty sure you don't have a bug that is going to cost you
> millions of dollars...


...then you are delusional :-)

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Huge number of datastore reads?

[Mos]

Hi Jeff,

+1  on all your statements.

> The default over-quota message is WAY more embarrassing than "XYZ is
experiencing technical difficulties" - how can we override it?

Should be possible:
http://code.google.com/appengine/docs/java/config/appconfig.html#Custom_Error_Responses

My application had not any overquota yet, thus can't tell if it works.

Cheers
Mos

---
http://www.mosbase.com


On Wed, Mar 7, 2012 at 9:49 AM, Jeff Schnitzer  wrote:

> On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz  wrote:
>
>> > The worst of it: we looked like rank amateurs in several ways, but
>> perhaps
>>
>> If you are going to run with the training wheels on, don't be surprised
>> when
>> you hit a pot hole and your back tire just spins with you going nowhere.
>>
>
> Sorry Brandon, you're way off-base here.  There's no excuse for buried
> land mines like this to exist.
>
> Nobody realizes that you get frozen out of the system for 30+ minutes at a
> time until you actually get screwed by the system.  And no matter what you
> think you've set your budget to, something can come along and blow it.
>
> For example, every night I process a significant volume of CSV data
> downloaded from a technically unsavvy organization.  They seem to delight
> in finding new and interesting ways to break my (popular third-party) CSV
> parser.  The latest was a name field containing "Blah Blah, Jr" - you can
> guess what that did.  After a few rounds of this I have hacked my parser
> enough to consume just about any garbage they can feed me.  Nevertheless,
> there were quite a few nights early on where this error caused a large
> task-queue-driven data processing operation to fail.  With the default
> retry schedule, it quickly ate through my
> (seemed-to-be-generous-at-the-time) billing quota.
>
> Fortunately my technically unsavvy organization is sufficiently clumsy
> that these problems came up early.  If they were only slightly more clever,
> these issues might not surface until long into production, when it really
> mattered.
>
> As those of us in Javaland were reminded today, GAE fails us every now and
> then.  It needs to fail gracefully.  The default over-quota message is WAY
> more embarrassing than "XYZ is experiencing technical difficulties" - how
> can we override it?
>
> If...you are pretty sure you don't have a bug that is going to cost you
>> millions of dollars...
>
>
> ...then you are delusional :-)
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Huge number of datastore reads?

[Jeff Schnitzer]

On Wed, Mar 7, 2012 at 4:09 AM, Mos  wrote:

> > The default over-quota message is WAY more embarrassing than "XYZ is
> experiencing technical difficulties" - how can we override it?
>
> Should be possible:
>
> http://code.google.com/appengine/docs/java/config/appconfig.html#Custom_Error_Responses
>
> My application had not any overquota yet, thus can't tell if it works.
>
>
That's great!  I guess I need to check for doc updates more often... I
don't remember that being there.

Thanks,
Jeff

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: Huge number of datastore reads?

[Brandon Wirtz]

I don't disagree that the Payment Page is horribly broken.  And Google
Wallet is horrible, and a "notify me if quota is estimated to be reached in
6 hours" wouldn't be an awesome feature.

But like your CSV input. We'd put that through a validator before it went
"live" and that validator might only be another app that says "Hey did this
error" but nothing changes in a Live deployment without humans. For small
things. we'd hope that 50x would still do us.

 

If you don't want to be down have a high budget. If you don't want to wake
up broke one morning because of a bug in your code, set the budget low.  

Assume you can only change your budget between 9 and 5 EST on week days.
Which sucks, but I make the same assumption with a number of my clouds If
for no other reason than every so often the limiting factor is I need to
call a Credit card company to say "Hey I really do want to give Amazon more
than $3000 today. Yes I realize I gave them $2500 every day this week, but
today I want to give them $6000."





 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Jeff Schnitzer
Sent: Wednesday, March 07, 2012 1:49 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: Huge number of datastore reads?

 

On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz  wrote:

> The worst of it: we looked like rank amateurs in several ways, but perhaps

If you are going to run with the training wheels on, don't be surprised when
you hit a pot hole and your back tire just spins with you going nowhere.

 

Sorry Brandon, you're way off-base here.  There's no excuse for buried land
mines like this to exist.

 

Nobody realizes that you get frozen out of the system for 30+ minutes at a
time until you actually get screwed by the system.  And no matter what you
think you've set your budget to, something can come along and blow it.

 

For example, every night I process a significant volume of CSV data
downloaded from a technically unsavvy organization.  They seem to delight in
finding new and interesting ways to break my (popular third-party) CSV
parser.  The latest was a name field containing "Blah Blah, Jr" - you can
guess what that did.  After a few rounds of this I have hacked my parser
enough to consume just about any garbage they can feed me.  Nevertheless,
there were quite a few nights early on where this error caused a large
task-queue-driven data processing operation to fail.  With the default retry
schedule, it quickly ate through my (seemed-to-be-generous-at-the-time)
billing quota.

 

Fortunately my technically unsavvy organization is sufficiently clumsy that
these problems came up early.  If they were only slightly more clever, these
issues might not surface until long into production, when it really
mattered.

 

As those of us in Javaland were reminded today, GAE fails us every now and
then.  It needs to fail gracefully.  The default over-quota message is WAY
more embarrassing than "XYZ is experiencing technical difficulties" - how
can we override it?

 

If...you are pretty sure you don't have a bug that is going to cost you
millions of dollars...

 

...then you are delusional :-)

 

Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Huge number of datastore reads?

[Chris Ramsdale]

Dave,

Sorry that you had issues with Checkout. We continue to work with them on
improvements that they should make to the service (and I've forward this
thread on to them).

Setting your budget higher is probably not a bad idea, but totally up to
you. I think the pros and cons have been outlined (in one way or another)
here :-)

Did you figure out why the DS reads increased? Is this still an issue?

-- Chris

Product Manager, Google App Engine

On Wed, Mar 7, 2012 at 6:54 AM, Brandon Wirtz  wrote:

> I don’t disagree that the Payment Page is horribly broken.  And Google
> Wallet is horrible, and a “notify me if quota is estimated to be reached in
> 6 hours” wouldn’t be an awesome feature.
>
> But like your CSV input. We’d put that through a validator before it went
> “live” and that validator might only be another app that says “Hey did this
> error” but nothing changes in a Live deployment without humans. For small
> things… we’d hope that 50x would still do us.
>
> ** **
>
> If you don’t want to be down have a high budget. If you don’t want to wake
> up broke one morning because of a bug in your code, set the budget low.
>
> Assume you can only change your budget between 9 and 5 EST on week days.
> Which sucks, but I make the same assumption with a number of my clouds If
> for no other reason than every so often the limiting factor is I need to
> call a Credit card company to say “Hey I really do want to give Amazon more
> than $3000 today. Yes I realize I gave them $2500 every day this week, but
> today I want to give them $6000.”
>
>
>
> 
>
> ** **
>
> *From:* google-appengine@googlegroups.com [mailto:
> google-appengine@googlegroups.com] *On Behalf Of *Jeff Schnitzer
> *Sent:* Wednesday, March 07, 2012 1:49 AM
> *To:* google-appengine@googlegroups.com
> *Subject:* Re: [google-appengine] Re: Huge number of datastore reads?
>
> ** **
>
> On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz  wrote:
> 
>
> > The worst of it: we looked like rank amateurs in several ways, but
> perhaps
>
> If you are going to run with the training wheels on, don't be surprised
> when
> you hit a pot hole and your back tire just spins with you going nowhere.**
> **
>
> ** **
>
> Sorry Brandon, you're way off-base here.  There's no excuse for buried
> land mines like this to exist.
>
> ** **
>
> Nobody realizes that you get frozen out of the system for 30+ minutes at a
> time until you actually get screwed by the system.  And no matter what you
> think you've set your budget to, something can come along and blow it.
>
> ** **
>
> For example, every night I process a significant volume of CSV data
> downloaded from a technically unsavvy organization.  They seem to delight
> in finding new and interesting ways to break my (popular third-party) CSV
> parser.  The latest was a name field containing "Blah Blah, Jr" - you can
> guess what that did.  After a few rounds of this I have hacked my parser
> enough to consume just about any garbage they can feed me.  Nevertheless,
> there were quite a few nights early on where this error caused a large
> task-queue-driven data processing operation to fail.  With the default
> retry schedule, it quickly ate through my
> (seemed-to-be-generous-at-the-time) billing quota.
>
> ** **
>
> Fortunately my technically unsavvy organization is sufficiently clumsy
> that these problems came up early.  If they were only slightly more clever,
> these issues might not surface until long into production, when it really
> mattered.
>
> ** **
>
> As those of us in Javaland were reminded today, GAE fails us every now and
> then.  It needs to fail gracefully.  The default over-quota message is WAY
> more embarrassing than "XYZ is experiencing technical difficulties" - how
> can we override it?
>
> ** **
>
> If...you are pretty sure you don't have a bug that is going to cost you
> millions of dollars...
>
> ** **
>
> ...then you are delusional :-)
>
> ** **
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group

Re: [google-appengine] Re: Huge number of datastore reads?

[Alexander Botov]

Hi Chris, 

I would like to add to this thread that we definitely experience increase 
of DS reads from couple of days without any significant change in the code. 
We just upgraded to HRD and we still have the issue. App id is spendgatel. 

Best,
--alex

On Thursday, March 8, 2012 1:16:02 AM UTC+2, Chris Ramsdale wrote:
>
> Dave,
>
> Sorry that you had issues with Checkout. We continue to work with them on 
> improvements that they should make to the service (and I've forward this 
> thread on to them).
>
> Setting your budget higher is probably not a bad idea, but totally up to 
> you. I think the pros and cons have been outlined (in one way or another) 
> here :-)
>
> Did you figure out why the DS reads increased? Is this still an issue?
>
> -- Chris
>
> Product Manager, Google App Engine
>
> On Wed, Mar 7, 2012 at 6:54 AM, Brandon Wirtz  wrote:
>
>> I don’t disagree that the Payment Page is horribly broken.  And Google 
>> Wallet is horrible, and a “notify me if quota is estimated to be reached in 
>> 6 hours” wouldn’t be an awesome feature.
>>
>> But like your CSV input. We’d put that through a validator before it went 
>> “live” and that validator might only be another app that says “Hey did this 
>> error” but nothing changes in a Live deployment without humans. For small 
>> things… we’d hope that 50x would still do us.
>>
>> ** **
>>
>> If you don’t want to be down have a high budget. If you don’t want to 
>> wake up broke one morning because of a bug in your code, set the budget 
>> low.  
>>
>> Assume you can only change your budget between 9 and 5 EST on week days.  
>> Which sucks, but I make the same assumption with a number of my clouds If 
>> for no other reason than every so often the limiting factor is I need to 
>> call a Credit card company to say “Hey I really do want to give Amazon more 
>> than $3000 today. Yes I realize I gave them $2500 every day this week, but 
>> today I want to give them $6000.”
>>
>>
>>
>> 
>>
>> ** **
>>
>> *From:* google-appengine@googlegroups.com [mailto:
>> google-appengine@googlegroups.com] *On Behalf Of *Jeff Schnitzer
>> *Sent:* Wednesday, March 07, 2012 1:49 AM
>> *To:* google-appengine@googlegroups.com
>> *Subject:* Re: [google-appengine] Re: Huge number of datastore reads?
>>
>> ** **
>>
>> On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz  
>> wrote:
>>
>> > The worst of it: we looked like rank amateurs in several ways, but 
>> perhaps
>>
>> If you are going to run with the training wheels on, don't be surprised 
>> when
>> you hit a pot hole and your back tire just spins with you going nowhere.*
>> ***
>>
>> ** **
>>
>> Sorry Brandon, you're way off-base here.  There's no excuse for buried 
>> land mines like this to exist.
>>
>> ** **
>>
>> Nobody realizes that you get frozen out of the system for 30+ minutes at 
>> a time until you actually get screwed by the system.  And no matter what 
>> you think you've set your budget to, something can come along and blow it.
>> 
>>
>> ** **
>>
>> For example, every night I process a significant volume of CSV data 
>> downloaded from a technically unsavvy organization.  They seem to delight 
>> in finding new and interesting ways to break my (popular third-party) CSV 
>> parser.  The latest was a name field containing "Blah Blah, Jr" - you can 
>> guess what that did.  After a few rounds of this I have hacked my parser 
>> enough to consume just about any garbage they can feed me.  Nevertheless, 
>> there were quite a few nights early on where this error caused a large 
>> task-queue-driven data processing operation to fail.  With the default 
>> retry schedule, it quickly ate through my 
>> (seemed-to-be-generous-at-the-time) billing quota.
>>
>> ** **
>>
>> Fortunately my technically unsavvy organization is sufficiently clumsy 
>> that these problems came up early.  If they were only slightly more clever, 
>> these issues might not surface until long into production, when it really 
>> mattered.
>>
>> ** **
>>
>> As those of us in Javaland were reminded today, GAE fails us every now 
>> and then.  It needs to fail gracefully.  The default over-quota message is 
>> WAY more embarrassing than "XYZ is experiencing technical difficulties" - 
>> how can we override it?
>>
>> ** **
>>
>> If...you are pretty sure you don't have a bug that is