[google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Mike Lawrence]

*I hope Google is not counting on everyone enabling multithreading to make 
this new model work.*
*If so, I'm afraid I'll reluctantly have to move from GAE. *

I volume tested my GAE web app using jmeter with and without GAE 
multithreading enabled.
Hope others will confirm/refute my findings:

1000 user threads, each doing 100 web page requests that issue a couple of 
backend datastore reads with sessions enabled:
- without multithreading
- 30-40 nodes
   - average 3 sec response time
   - only a few 500 errors
- with multi threading
   - 3 nodes 
   - average 4 sec response time
   - many 500 errors

*Summary: for my app (Java, Stripes, Slim3), multi-threading is slower and 
less stable under heavy load, so I turned it off.*
*
*
 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Mike Lawrence]

Is this math correct for 3 reserved instances?

*Current plan:*
  * $9/month*

*New plan *($9/month platform fee + ($0.05 / hour for reserved instance * 24 
hrs/day * 30 days /month  x 3 instances) 
   *$117 /month*

That can't be right? 13 times more? Please tell me I'm wrong.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: RE: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Mike Lawrence]

You cant use a 3rd party library to add threading.
The API to create a new thread is not in the white list so it's not allowed


Sincerely,  Mike Lawrence


On Wed, May 11, 2011 at 2:32 PM, Kyle Finley  wrote:

> What if you were to re-wright using datastore plus?
> http://code.google.com/p/appengine-ndb-experiment/
>
> Would this be comparable to threading?
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Mike Lawrence]

Just retested my app with the new 1.5.0 GAE SDK. 
Results are completely different. Could be SDK or infrastructure changes. 
Previously, the multithreaded mode
would only spawn 3 instances. Now it spawned 8.
 

1000 user threads, each doing 100 web page requests that issue a couple of 
backend datastore reads with sessions enabled:
- without multithreading
   - 30 nodes
   - average 2.4 sec response time
   - no errors
- with multi threading
   - 8 nodes 
   - average 1.2 sec response time
   - no errors

*Summary: for my app (Java, Stripes, Slim3), multi-threading is twice as 
fast and requires about 1/4th as many instances.*
*
*
*
*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] GAE Starts dynamic instances for no reason... causing unnecessary cold start delays

[Mike Lawrence]

I purchased 3 Always-On instances.
My site is under construction with no traffic.
When I hit my site, GAE fires up a new dynamic instance to service the
request when there are 3 idle instances!
My app starts in 2.0 seconds (using stripes)
But GAE takes 9.4 seconds to reply (why?).
Really annoying.
Why pay for Always-On when you get the crappy response time of dynamic
instances?

App Id: WordPong

EST:
2011-07-27 17:37:04.859 /game/Game.wp?_eventName=questionList 200
9481ms 2063cpu_ms 103api_cpu_ms 1kb Mozilla/5
2011-07-27 12:37:51 Completed update of a new default version
version=22.2011-07-27T19:37:17Z

Here's the screen shot of my instances at the time of the request:
http://dl.dropbox.com/u/473572/Untitled.jpg

Why are there any dynamic instances running at all when there are 3
idle always-on instances available?
Looks like a serious bug where GAE is wasting resources and providing
poor response times for no reason.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: GAE Starts dynamic instances for no reason... causing unnecessary cold start delays

[Mike Lawrence]

Wow that's awesome. Thanks Jon for the very thoughtful analysis.
I understand what's going on now.

I recommend you change the scheduler to favor instances that have been
warmed up
over N milliseconds where N is the average start-up time for the app.

You should never send a request to a new instance when there is an
idle instance (dynamic or reserved) that is hot.
Without this change, new requests will be delayed unnecessarily which
is unacceptable for a user-facing app.

Thanks again for the detailed analysis and recommendations.
Love the log searching tricks.

On Jul 27, 7:12 pm, Jon McAlister  wrote:
> So, there's a couple of things going on here. I'll see if I can help explain.
>
> The first is that with 1.5.2 we changed how resident instances work,
> so that if a dynamic instance existed, the scheduler would prefer an
> idle dynamic instance to an idle resident instance. Further, if a
> resident instance was busy, we would use this as a cue to warmup a new
> dynamic instance. The point of these changes is to turn the resident
> instances into the reserve for the app. They are generally idle, but
> if the app gets more traffic than it can handle with non-reserved
> instances, then it will use some of the reserved instances (and this
> will in turn spawn a new dynamic instance).
>
> Generally, Always On is going away with the new billing plan, and
> being replaced by Min Idle Instances, which is how the reserved
> instances have been changed to behave with 1.5.2. We're continuing to
> evaluate all aspects here, both how well these reserve instances are
> working, what we should be doing, what we should change about the
> scheduler and the billing plan, and so on.
>
> In terms of this specific example, the slow request was caused by
> general bigtable slowness during that time interval. This can be seen
> somewhat 
> here:http://code.google.com/status/appengine/detail/datastore/2011/07/27#a...
>
> This can also be investigated somewhat using our logs viewer. For
> example, we can see all loading requests for an app 
> with:https://appengine.google.com/logs?app_id=wordpong&severity_level_over
> Note how the only loading requests this app has received have been
> /_ah/warmup
>
> Also we can see all requests sent to a specific instance. Here's the
> one with the log line you listed 
> above:https://appengine.google.com/logs?app_id=wordpong&severity_level_over
> Note how the first request the instance served was /_ah/warmup,
> followed by a pause of 4 seconds, followed by the /game/Game.wp
> request which ran for 9 seconds.
>
> There are a couple of things that can be done now to get different
> behaviors. One is to set Max Idle Instances to three, which will kill
> off the dynamic instances for your app, and leave the app with just
> the resident instances. The other is to use Backends, which will give
> you direct control over how many instances run for your app and their
> properties:http://code.google.com/appengine/docs/java/backends/overview.html
>
> Hopefully that helps. There is also a lengthy discussion going on 
> at:http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
>
>
>
>
>
>
> On Wed, Jul 27, 2011 at 2:59 PM, Mike Lawrence  wrote:
> > I purchased 3 Always-On instances.
> > My site is under construction with no traffic.
> > When I hit my site, GAE fires up a new dynamic instance to service the
> > request when there are 3 idle instances!
> > My app starts in 2.0 seconds (using stripes)
> > But GAE takes 9.4 seconds to reply (why?).
> > Really annoying.
> > Why pay for Always-On when you get the crappy response time of dynamic
> > instances?
>
> > App Id: WordPong
>
> > EST:
> > 2011-07-27 17:37:04.859 /game/Game.wp?_eventName=questionList 200
> > 9481ms 2063cpu_ms 103api_cpu_ms 1kb Mozilla/5
> > 2011-07-27 12:37:51     Completed update of a new default version
> > version=22.2011-07-27T19:37:17Z
>
> > Here's the screen shot of my instances at the time of the request:
> >http://dl.dropbox.com/u/473572/Untitled.jpg
>
> > Why are there any dynamic instances running at all when there are 3
> > idle always-on instances available?
> > Looks like a serious bug where GAE is wasting resources and providing
> > poor response times for no reason.
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> 

[google-appengine] Re: GAE Starts dynamic instances for no reason... causing unnecessary cold start delays

[Mike Lawrence]

Just ran a volume test on my server.
The three reserved instances are not getting any traffic.
That can't be normal.
Instead GAE spawned 3 dynamic instances to handle all the load.
I have set Max Idle instances to 3.
Why would you ever want instances to sit idle under a load test?
If you have capacity (idle instances), why not use them when you're
under heavy load?
http://dl.dropbox.com/u/473572/Untitled2.jpg
Wouldn't I be charged for 6 instance under the new pricing model when
I'm only really using 3?


On Jul 27, 7:12 pm, Jon McAlister  wrote:
> So, there's a couple of things going on here. I'll see if I can help explain.
>
> The first is that with 1.5.2 we changed how resident instances work,
> so that if a dynamic instance existed, the scheduler would prefer an
> idle dynamic instance to an idle resident instance. Further, if a
> resident instance was busy, we would use this as a cue to warmup a new
> dynamic instance. The point of these changes is to turn the resident
> instances into the reserve for the app. They are generally idle, but
> if the app gets more traffic than it can handle with non-reserved
> instances, then it will use some of the reserved instances (and this
> will in turn spawn a new dynamic instance).
>
> Generally, Always On is going away with the new billing plan, and
> being replaced by Min Idle Instances, which is how the reserved
> instances have been changed to behave with 1.5.2. We're continuing to
> evaluate all aspects here, both how well these reserve instances are
> working, what we should be doing, what we should change about the
> scheduler and the billing plan, and so on.
>
> In terms of this specific example, the slow request was caused by
> general bigtable slowness during that time interval. This can be seen
> somewhat 
> here:http://code.google.com/status/appengine/detail/datastore/2011/07/27#a...
>
> This can also be investigated somewhat using our logs viewer. For
> example, we can see all loading requests for an app 
> with:https://appengine.google.com/logs?app_id=wordpong&severity_level_over
> Note how the only loading requests this app has received have been
> /_ah/warmup
>
> Also we can see all requests sent to a specific instance. Here's the
> one with the log line you listed 
> above:https://appengine.google.com/logs?app_id=wordpong&severity_level_over
> Note how the first request the instance served was /_ah/warmup,
> followed by a pause of 4 seconds, followed by the /game/Game.wp
> request which ran for 9 seconds.
>
> There are a couple of things that can be done now to get different
> behaviors. One is to set Max Idle Instances to three, which will kill
> off the dynamic instances for your app, and leave the app with just
> the resident instances. The other is to use Backends, which will give
> you direct control over how many instances run for your app and their
> properties:http://code.google.com/appengine/docs/java/backends/overview.html
>
> Hopefully that helps. There is also a lengthy discussion going on 
> at:http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
>
>
>
>
>
>
> On Wed, Jul 27, 2011 at 2:59 PM, Mike Lawrence  wrote:
> > I purchased 3 Always-On instances.
> > My site is under construction with no traffic.
> > When I hit my site, GAE fires up a new dynamic instance to service the
> > request when there are 3 idle instances!
> > My app starts in 2.0 seconds (using stripes)
> > But GAE takes 9.4 seconds to reply (why?).
> > Really annoying.
> > Why pay for Always-On when you get the crappy response time of dynamic
> > instances?
>
> > App Id: WordPong
>
> > EST:
> > 2011-07-27 17:37:04.859 /game/Game.wp?_eventName=questionList 200
> > 9481ms 2063cpu_ms 103api_cpu_ms 1kb Mozilla/5
> > 2011-07-27 12:37:51     Completed update of a new default version
> > version=22.2011-07-27T19:37:17Z
>
> > Here's the screen shot of my instances at the time of the request:
> >http://dl.dropbox.com/u/473572/Untitled.jpg
>
> > Why are there any dynamic instances running at all when there are 3
> > idle always-on instances available?
> > Looks like a serious bug where GAE is wasting resources and providing
> > poor response times for no reason.
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: GAE Starts dynamic instances for no reason... causing unnecessary cold start delays

[Mike Lawrence]

thanks for clearing that up.
I'm sure a lot of people will be confused by this.
Maybe add a $ next to instances that we are being charged for so it's 
obvious?
Thanks for your excellent support

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/4x8OnZt-sSUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: My master/slave to high replication datastore migration experience

[Mike Lawrence]

if you use slim3 for your data store operations you can avoid all this 
parenting nonsense.
slim3 allows you to update two of the same kind in a single transaction!
why should we be required to place entities that are not logically related 
into a parenting relationship just to get transaction support?

I have a game where a user domain object has a list of friends (users). 
Try modeling that as a parent relationship. It's self-referential. A user is 
not a parent of another user.
Then try adding a list of games domain objects,
and add a new friend to a new game in a single transaction.
With stand-alone domain objects it's simple.

Back to the original post
I'm finding the latency on the master/slave datastore is too unpredictable 
for my game.
Hopefully migrating will eliminate some of the long latency, and dynamic 
instance start problems i'm seeing.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/1zQr36sv23sJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: My master/slave to high replication datastore migration experience

[Mike Lawrence]

tcp guarantees delivery built upon udp
which doesn't, so it's not too hard to believe you can build something more 
robust on top of a limited implementation 

http://lmgtfy.com/?q=slim3+global+transaction

it's open source so you can view the implementation for yourself to certify 
it's sound. I'd ask the slim3 user group. 
they're friendly and very responsive. 

good luck 

Mike Lawrence


On Aug 15, 2011, at 10:08 PM, Tim Hoffman  wrote:

> HI Mike.
> 
> You said "slim3 allows you to update two of the same kind in a single 
> transaction!"
> 
> How does slim3 achieve this as the underlying datastore doesn't "currently"  
> support multiple entity group transactions.
> 
> It would seem this would be a slim3 transaction (I know nothing about slim3 
> by the way) and not a datastore operation,
> which would suggest that the transaction isn't reliable.  
> 
> But would love to be enligntened on this topic.
> 
> Rgds
> 
> Tim 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/google-appengine/-/weML1S1dK6EJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

Just burned several days with Google Enterprise Support trying to 
troubleshoot intermittent long latencies; 
unnecessary dynamic instance starts; and failures during dynamic instance 
starts on a multi-threaded GAE Java app
running on Master/Slave (MS) data store.   

My typical app latencies on MS were under 1 second per request. Periodically 
I'd see 5-20 second delays with very little traffic.
My app starts in under 3 seconds (stripes/slim3), so when instances failed 
to start in 30 seconds, I suspected a GAE issue.

After many attempts to resolve the issue, we finally decided to to try 
switching to the High Replication (HR) data store.
All of the issues disappeared!!! The long latencies were all gone. My 
traffic was easily being served with a single instance.
No more phantom dynamic instances. No more instance start crashes.

I just volume tested my app. My median request went from 2.5 seconds on MS 
to under 1 second on HR with 1000 simultaneous users.

Previously I was paying for 3 always-on instances. With HR enabled, 
there was no need for any always-on instances.

So if High Replication is faster, and more stable, why should Google spend 
valuable resources supporting Master/Slave?
Is the cost savings of MS worth all the issues it creates? Not for me.

If you're experiencing similar problems on MS I highly recommend giving HR a 
shot.

FYI, migration to HR was trivial: 
- created a new app id, 
- deployed my app to the new id, 
- used my app to initialize the data (this will be problematic for some), 
- had google point the old app id to the new app id.
now serving from the original URL.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/XCUAKZ5E1LYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

For the one week off of Master/Slave and on HR:


   - up-time has been 100%
   - max response time has improved... down from 22+ seconds to about 2-6 
   seconds.
   - Average response time is still sub-second for the week.








-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/FZ0hSiyK6aAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

In general, my app does 3 db gets for every db query.

No modifications were necessary with my app. Using slim3 db for
transactions.

1000 simutaneous users were logged in using my own authentication (not
federated) with sessions enabled.
I only use session to record the fact that a user has authenticated
I save app state in hidden form html.

hope that helps

Sincerely,  Mike Lawrence


On Mon, Aug 22, 2011 at 10:54 AM, thstart  wrote:

> +Mike Lawrence
>
> As you post I have couple of questions to you.
>
> >Average response time is still sub-second for the week.
>
> Do you use queries to fetch data or you use a lists?
>
> Do you modified your app when moving to HR, or just transferred it?
>
> >1000 simultaneous users.
>
> Are you users logged in or just retrieving data anonymously?
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/rpGj8t4Gy-0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Time for a community-sourced Google App Status page?

[Mike Lawrence]

Hope this post doesn't get me in trouble with my Google friends, but 
I've about had it with the Google App Engine System Status page.
   http://code.google.com/status/appengine

Yesterday my app was unreachable four times with 500 error pages, 
but the System Status page above shows no issues.

*Google Enterprise Support:*
*"I can confirm that we had a transient infrastructure issue that caused a 
temporary increase of latency that affected your application instances. This 
should now be fixed as traffic has been moved to different datacenters."  
*
I'd prefer Google fix/enhance their App Status page so we have more 
visibility to what's really going on.
But, It's probably not in Googles best interest since Amazon will likely use 
it against them for marketing FUD.

So, how about we build our own App Status page that transparently collects 
and reports app status?

We could:
1. build and publish source to a simple custom servlet that Java developers 
could add to their apps eg:
   http://appid.appspot.com/appstatus
   that returns json data that we can aggregate into a community-sourced 
system status page.
2. build an app engine site 
a. for registering your app engine URL and how often you want it tested
b. viewing the current combined status of all sites
c. polling the registered sites for status

We'd probably want to add an entity ("eg dual") that we could hit for 
testing db operations.

Thoughts?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/jBAGlQh-VPcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.