[google-appengine] Re: Backend performance, compared

A couple quick notes... I can verify Brandon's post about high throughput apps having extreme performance, while low throughput, test apps don't scale up very well. There is a learning algorithm at play, which favors production apps over test apps. Go figure ;-} " I might expect better resu

Re: [google-appengine] Re: Backend performance, compared

While it is possible to imagine doing aggregation with memcache - maybe - it's very hard to imagine it being worth the effort. It's like using a hammer to punch staples - sure, it's possible, but you'll break a lot fewer fingers if you just use a stapler. How do you pick memcache keys for thousan

[google-appengine] Re: Backend performance, compared

On Aug 13, 4:43 pm, Hernan Liendo wrote: > MemCache.getAll(keys) method could work if you know the keys. Using sharded > keys you could reach them all. I wrote a simple test application for memcache earlier. I managed to collect 1k keys reliably for my simple test application, and I would bet tha

Re: [google-appengine] Re: Backend performance, compared

On Monday, August 13, 2012 3:10:17 AM UTC-3, Jeff Schnitzer wrote: > > Memcache doesn't work for this application. Completely aside from > reliability issues, there's no memcache instruction for "give me all > the data". At best you can fetch some number of individual keys, but > that brings

Re: [google-appengine] Re: Backend performance, compared

Memcache doesn't work for this application. Completely aside from reliability issues, there's no memcache instruction for "give me all the data". At best you can fetch some number of individual keys, but that brings up two problems: 1) Trying to do a batch fetch of 10k keys (or more) 2) How do

Re: [google-appengine] Re: Backend performance, compared

Good point. I had unconsciously assumed that a replicated memcached solution would be much more efficient than a replicated data store solution, but if the replication and not the disk I/O is what's expensive, then that assumption falls down. Generally speaking, I'm having difficulty architect

Re: [google-appengine] Re: Backend performance, compared

By *guaranteed*, do you mean that this should be kept synchronized across multiple data centers? I ask because in the case of a data center emergency, your app spins up at a different data center. The HR datastore is always in sync, the memcache is always local to the DC. Comparing the put-laten

Re: [google-appengine] Re: Backend performance, compared

That would work up until your data got evicted at the wrong time... Though this gets me thinking: it would be really, really nice for many applications to have a *guaranteed* memcached-style in-memory table that you could use to share data between instances (and between FE/BE). Unlike memcache

Re: [google-appengine] Re: Backend performance, compared

*This is the best thread during this month! On Racetown we've similar issues. Based on our experience having different services such as GAE and Rackspace in the same project (ie scores vs. the rest of the game) tend to complex things. After a while you will need to share data between both techn

[google-appengine] Re: Backend performance, compared

The reason I'm currently using them is to avoid insane instancing behavior on front ends. (Which is sad, because I'm basically paying extra to avoid a feature.) I just commented on your SO post, btw, asking clarification on whether your rules are only for backends, because I definitely *don't*

Re: [google-appengine] Re: Backend performance, compared

I generally am sympathetic to "GAE costs more because it offers more". But there's a finite limit to what that "more" is worth. 2X? 3X? Consider that those 10 B1s are barely keeping up with a load of 2k users. A single $11/mo Node instance handles more than 10k users in my tests. As we've bee

Re: [google-appengine] Re: Backend performance, compared

On Thu, Aug 9, 2012 at 3:40 PM, Jeff Schnitzer wrote: > I think I mentioned this before, but there is no I/O in the problem. > It just collects data in RAM (thousands of individual submissions) and > waits for the reaper come get the entire set in one fetch. This is > why I do not expect concurr

Re: [google-appengine] Re: Backend performance, compared

I think I mentioned this before, but there is no I/O in the problem. It just collects data in RAM (thousands of individual submissions) and waits for the reaper come get the entire set in one fetch. This is why I do not expect concurrency to help. Jeff On Thu, Aug 9, 2012 at 3:31 AM, Johan Euphr

Re: [google-appengine] Re: Backend performance, compared

I just answered a question on the same topic on Stack Overflow: http://stackoverflow.com/a/11882719/656408 On Thu, Aug 9, 2012 at 12:31 PM, Johan Euphrosine wrote: > > > > On Thu, Aug 9, 2012 at 7:16 AM, Jeff Schnitzer wrote: > >> On Wed, Aug 8, 2012 at 5:44 PM, Waleed Abdulla wrote: >> > Tha

Re: [google-appengine] Re: Backend performance, compared

On Thu, Aug 9, 2012 at 7:16 AM, Jeff Schnitzer wrote: > On Wed, Aug 8, 2012 at 5:44 PM, Waleed Abdulla wrote: > > Thanks, Jeff. Is it possible to repeat the test with qps < 10 to rule out > > the limit that Johan pointed out? In other words, how big is the > performance > > difference if you had

Re: [google-appengine] Re: Backend performance, compared

First of all, thank you for the detailed cost comparison and the honest feedback. We really appreciate it. The data itself is not very far from what I observed in my test. Certainly, we should try making the backends cheaper. The easiest fix will be this issue: http://code.google.com/p/googleappen

Re: [google-appengine] Re: Backend performance, compared

On Wed, Aug 8, 2012 at 11:01 PM, Kristopher Giesing wrote: > Do we know for sure that front ends are any faster? Their individual > throughput limits might just be masked by having more of them spin up. I expect frontends are about the same. But frontends can't aggregate scores so they aren't r

[google-appengine] Re: Backend performance, compared

On Wed, Aug 8, 2012 at 10:05 PM, vlad wrote: > Because I thought that all 3 methods that did not work involved storing data > in Datastore. If you used Backend RAM as storage, well, ok it is an > inventive way to use Backend, I guess. I think Backends are a wrong feature > all together. I just doe

Re: [google-appengine] Re: Backend performance, compared

Do we know for sure that front ends are any faster? Their individual throughput limits might just be masked by having more of them spin up. - Kris I really don't know why backends are slow. Maybe it has something to > do with the request queueing system? Throughput sucks even when > backend

Re: [google-appengine] Re: Backend performance, compared

On Wed, Aug 8, 2012 at 5:44 PM, Waleed Abdulla wrote: > Thanks, Jeff. Is it possible to repeat the test with qps < 10 to rule out > the limit that Johan pointed out? In other words, how big is the performance > difference if you had less requests that do more work? You must mean concurrency less

[google-appengine] Re: Backend performance, compared

Because I thought that all 3 methods that did not work involved storing data in Datastore. If you used Backend RAM as storage, well, ok it is an inventive way to use Backend, I guess. I think Backends are a wrong feature all together. I just does not with GAE concept as a scalable, no-configur

Re: [google-appengine] Re: Backend performance, compared

This is his evidence that higher-use apps have better performance. I'm not surprised that memcache hit rates would be higher (since you're more likely to evict your neighbors than vice versa) but the latency difference does surprise me. I wonder if Google's balancing algorithms tend to starve

[google-appengine] Re: Backend performance, compared

I have a Linode instance. How do I run the test? > BTW I picked Rackspace for this test because I happened to already > have a VPS there. I would love to see latency numbers for Linode, > EC2, and Google Compute Engine. Does anyone want to run the test? > I'm still waiting for an invite to

Re: [google-appengine] Re: Backend performance, compared

Why are you posting memcache stats in a thread about backends? Jeff On Wed, Aug 8, 2012 at 11:18 AM, Drake wrote: > *Number of Instances *-* > **Details > * > > *Average QPS* > > *Average Latency* > > *Average Memory* > > 3

Re: [google-appengine] Re: Backend performance, compared

Thanks, Jeff. Is it possible to repeat the test with qps < 10 to rule out the limit that Johan pointed out? In other words, how big is the performance difference if you had less requests that do more work? On Wed, Aug 8, 2012 at 10:48 AM, Jeff Schnitzer wrote: > On Wed, Aug 8, 2012 at 10:42 A

RE: [google-appengine] Re: Backend performance, compared

Number of Instances - Details Average QPS Average Latency Average Memory 3 total (1 Resident) 1.017 568.7 ms 61.5 MBytes Hit count: 237801 Miss count: 1940100 Hit ratio: 10% Item count: 43785 item(s) To

Re: [google-appengine] Re: Backend performance, compared

On Wed, Aug 8, 2012 at 10:42 AM, Drake wrote: > Jeff your Low volume is effecting your numbers. The more you do the more > consistent the numbers become. I don't know what black magic Google does, > but the more people I have the more my instances get faster, the faster my > datastore reads (and

RE: [google-appengine] Re: Backend performance, compared

Jeff your Low volume is effecting your numbers. The more you do the more consistent the numbers become. I don't know what black magic Google does, but the more people I have the more my instances get faster, the faster my datastore reads (and writes) become, and the better the hit ratio on my memc

[google-appengine] Re: Backend performance, compared

On Wed, Aug 8, 2012 at 9:24 AM, vlad wrote: > Jeff, what storage subsystem did you use on your dedicated host? If you kept > all data in RAM then it is just a sleuth of hand. Overall I am not surprised > about lame performance of backends and pull queues. I am a bit surprised on > the cost of doin

[google-appengine] Re: Backend performance, compared

Jeff, what storage subsystem did you use on your dedicated host? If you kept all data in RAM then it is just a sleuth of hand. Overall I am not surprised about lame performance of backends and pull queues. I am a bit surprised on the cost of doing it in Datastore. Good work! -- You received th

[google-appengine] Re: Backend performance, compared

But why are the B4 backends for your example this slow? They are much bigger (RAM) than 256MB rackspaces and have dedicated fast CPU cores. I have some CPU intensive tasks that are really OK on backends. Backends are crazy expensive, but slow?! 80 qps? Maybe its: * slow network access via Http

[google-appengine] Re: Backend performance, compared

Terrifying. So, I assume Richard's medium-term plan is to set up an "outboard" collector endpoint, and have a single GAE backend reap from that? - Kris On Tuesday, August 7, 2012 7:10:43 PM UTC-7, Jeff Schnitzer wrote: > > If you've been reading various threads on this list you know that > Ric

[google-appengine] Re: Backend performance, compared

$60/m or $600/m? On Wednesday, August 8, 2012 10:10:43 AM UTC+8, Jeff Schnitzer wrote: > > If you've been reading various threads on this list you know that > Richard has been having trouble getting his mobile game to run > smoothly on GAE. It's a little unusual because timing is coordinated >