Hi Nick,
Hmm, I was running tests on a billing enabled appspot today. 100
requests/test.
10 threads getting a URL with a 3 second sleep (to emulate
computation) on appspot, was the most I could get without getting 500
errors.
If I raised the thread pool beyond 10, I started getting errors??
That doesn't reconcile very well with this statement from the
appengine website.
"Requests
The total number of requests to the app. The per-minute quotas for
application with billing enabled allow for up to 500 requests per
second--more than one billion requests per month. If your application
requires even higher quotas than the "billing-enabled" values listed
below, you can request an increase in these limits here.
"
Is there some billing setting that affects this?
Cheers, Gary
PS. dead simple request handler.
import time
from django import http
def sit(req):
time.sleep(3)
return http.HttpResponse('foo')
errors are:
03-01 04:15PM 48.177 /sit/91 500 10019ms 0cpu_ms 0kb gzip(gfe)
153.90.236.210 - - [01/Mar/2010:16:15:58 -0800] "GET /sit/91 HTTP/1.1"
500 0 - "gzip(gfe)" ".appspot.com"
W 03-01 04:15PM 58.197
Request was aborted after waiting too long to attempt to service your
request. Most likely, this indicates that you have reached your
simultaneous dynamic request limit. This is almost always due to
excessively high latency in your app. Please see
http://code.google.com/appengine/docs/quotas.html for more details.
On Mar 1, 2:36 pm, Michael Wesner <mike.wes...@webfilings.com> wrote:
> Correction/addition to my last email.
>
> It turns out that our requests for this EC2 pull thing are actually much
> faster now. Gary and our other devs have reworked it. I need updated
> numbers, but they don't take 10s, probably more like 2s. We still have some
> heavy ~5s services though, so the same issue exists with the simul-req stuff,
> just to less extent. We don't actually hit this limit much now with the
> current beta that is in production, but it is low traffic at the moment. We
> are just getting ready to ramp up heavily.
>
> I asked Nick what we should do, well just today after my last email, I have
> made contact with a Developer Advocate and whatnot, which is fantastic. It
> looks like we, as a business, will be able to have better contact with the
> GAE team. We would very much like to continue working with you to figure out
> what actions we can take and what provisioning we can do to make our product
> successful and scale it as we grow in the near future. Gary Orser will be
> replying to this thread soon with more findings from both our real app code
> and a little test app we are using and which he will share with you.
>
> We plan on having a presence at Google I/O this year as we did at PyCon.
> Hopefully we can even get setup in the demonstration area at I/O.
>
> Thanks Nick for your help. Could we possibly setup a quick skype conf call
> at some point?
>
> -Mike Wesner
>
> On Mar 1, 2010, at 1:13 PM, Michael Wesner wrote:
>
>
>
> > Nick,
>
> > If we (I work with Gary) require fairly heavy requests which run for
> > multiple seconds then it is not possible to get anywhere near 400 QPS.
> > The math used on the docs page only applies to 75ms requests.
>
> > (1000 ms/second / 75 ms/request) * 30 = 400 requests/second
>
> > so lets say each request takes 10 seconds (and ours, pulling data to EC2
> > for a heavy operation that we can't do on GAE could take that much since we
> > have to process and update some XML before sending it)
>
> > (1000 ms/second / 10000 ms/request) * 30 = 3 requests/second
>
> > And that does not even take into account all the other traffic to our
> > application, nor the fact that many users could be doing this same heavy
> > operation at the same time. Our application will see spikes in this type
> > of activity also. The docs also mention that CPU heavy apps incur
> > penalties, which is vague and scary.
>
> > Great effort is put into doing things in the most efficient way possible,
> > but not everyones apps can do everything in 75ms. Most all of our service
> > calls are under 250ms. We do have a little overhead from our framework
> > which we are constantly working on improving. Our application is AMF
> > service/object based which is inherently heavy compared to simple web
> > requests. It limits the amount of memcache work we can do also, but we are
> > also working on improving our use of that.
>
> > We easily hit these boundaries during testing so I think we really need
> > much higher simultaneous dynamic request limits for not only our production
> > instance but our dev/qa instances so we can test and load them to some
> > degree. Our QA team could easily bust this limit 20 times over.
>
> > So, Nick Johnson... I ask your advice. We are running a company/product
> > on GAE. We are more than happy to pay for quota/service/extra assistance
> > in these matters. What do you suggest we do?
>
> > I should also mention that I spoke with Brett Slatkin at PyCon and he is
> > now at least semi-familiar with the scope of product we have developed. I
> > have exchanged contact info with him but have not heard anything back from
> > him yet. We would really appreciate contact or even a brief meeting at
> > some point (in person or otherwise).
>
> > Thanks,
>
> > -Mike Wesner
>
> > On Mar 1, 2010, at 7:40 AM, Nick Johnson (Google) wrote:
>
> >> Hi Gary,
>
> >> Practically speaking, for an app that hasn't been given elevated
> >> permissions, you should be able to have at least 30 concurrent requests -
> >> equating to around 400 QPS if your app is fairly efficient. What problems
> >> are you running into that lead you to conclude you're hitting a limit at 4
> >> QPS, and that the problem is at App Engine's end?
>
> >> -Nick Johnson
>
> >> On Sat, Feb 27, 2010 at 8:23 PM, Gary Orser <garyor...@gmail.com> wrote:
> >> Hi all,
>
> >> We were trying to create programmatic parallel access to our appengine
> >> application.
>
> >> From EC2, we were attempting (with threads) to run parallel access
> >> (url gets/posts) to
> >> our appid. There are some long running processes that we need to run
> >> on EC2, for which
> >> we would like to get a bunch of information (entities + processing on
> >> appspot) quickly.
>
> >> We seem to be running into a limit on the number of accesses that are
> >> allowed.
> >> (4 threads seems to be the effective limit)
>
> >> Is there some sort of denial of service limit imposed on multiple
> >> accesses from a single IP?
>
> >> Cheers, Gary
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appeng...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group
> >> athttp://groups.google.com/group/google-appengine?hl=en.
>
> >> --
> >> Nick Johnson, Developer Programs Engineer, App Engine
> >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> >> 368047
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appeng...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group
> >> athttp://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
