What I'm suggesting is.. You need to create a simple test setup that
recreates this dynamic request limit error.. (It definitely should not
take 8mb of code).
I will see if I can create a handler like the one you posted, deploy
it, and then run 30 seperate processes that keep getting from that
handler.. (I can write this up in less than 10kb or python code)...
My guess is this will work. Without seeing sample code.. I can't tell
where you may be going wrong (or where GAE may be breaking)
On 3/2/10, Gary Orser <garyor...@gmail.com> wrote:
>
> Actually, 4 threads was before we optimized server side, and set up
> the test environment.
>
> I have a tarball, about 8mb, with the test environment. (django and
> libraries, grrr)
> What is the best way to post this? I don't see any file attachments
> on groups.
>
> Cheers, Gary
>
> On Mar 2, 8:23 am, Eli Jones <eli.jo...@gmail.com> wrote:
>> Are these threads you're using (at this point, it really seems like you
>> should post some simplified code to illustrate the issue at hand) waiting
>> for their response before trying to get again?
>>
>> Posting some code to help recreate this issue will lead to a much faster
>> resolution.. as it stands.. I just know that someone on the internet has
>> "10
>> threads" that are hitting a dynamic request limit.
>>
>> I also know that in the initial e-mail, when the request took longer to
>> return.. these "threads" were hitting a lower dynamic request limit (only
>> 4
>> could run). This suggest that there is an important detail to how your
>> "threads" are doing their work.. and we would need that to provide useful
>> help.
>>
>> Thanks for info.
>>
>> On Tue, Mar 2, 2010 at 10:01 AM, Gary Orser <garyor...@gmail.com> wrote:
>>
>> > But that's the point. I can not reach 30 active requests.
>> > I can only reach 10 active requests without error.
>>
>> > Any ideas on how I can debug this?
>>
>> > Cheers, Gary.
>>
>> > On Mar 2, 7:05 am, "Nick Johnson (Google)" <nick.john...@google.com>
>> > wrote:
>> > > Hi,
>>
>> > > On Tue, Mar 2, 2010 at 1:54 PM, Wooble <geoffsp...@gmail.com> wrote:
>> > > > The 500 requests per second number relies on the
>> > > > probably-unreasonable
>> > > > assumption that each request can complete in ~75ms. Deliberately
>> > > > making your requests take a whole 3 seconds each is, obviously, not
>> > > > going to work. You can only have 10 instances active at a time by
>> > > > default; if the pages you're serving actually take 3 seconds to
>> > > > complete you'll need to optimize things a whole lot or be stuck with
>> > > > a
>> > > > 3.33 request/sec maximum.
>>
>> > > Actually, the default limit is 30 active requests.
>>
>> > > -Nick Johnson
>>
>> > > > On Mar 1, 11:33 pm, Gary Orser <garyor...@gmail.com> wrote:
>> > > > > Hi Nick,
>>
>> > > > > Hmm, I was running tests on a billing enabled appspot today. 100
>> > > > > requests/test.
>>
>> > > > > 10 threads getting a URL with a 3 second sleep (to emulate
>> > > > > computation) on appspot, was the most I could get without getting
>> > > > > 500
>> > > > > errors.
>> > > > > If I raised the thread pool beyond 10, I started getting errors??
>>
>> > > > > That doesn't reconcile very well with this statement from the
>> > > > > appengine website.
>> > > > > "Requests
>> > > > > The total number of requests to the app. The per-minute quotas
>> > for
>> > > > > application with billing enabled allow for up to 500 requests per
>> > > > > second--more than one billion requests per month. If your
>> > > > > application
>> > > > > requires even higher quotas than the "billing-enabled" values
>> > > > > listed
>> > > > > below, you can request an increase in these limits here.
>> > > > > "
>>
>> > > > > Is there some billing setting that affects this?
>>
>> > > > > Cheers, Gary
>>
>> > > > > PS. dead simple request handler.
>>
>> > > > > import time
>> > > > > from django import http
>> > > > > def sit(req):
>> > > > > time.sleep(3)
>> > > > > return http.HttpResponse('foo')
>>
>> > > > > errors are:
>>
>> > > > > 03-01 04:15PM 48.177 /sit/91 500 10019ms 0cpu_ms 0kb gzip(gfe)
>> > > > > 153.90.236.210 - - [01/Mar/2010:16:15:58 -0800] "GET /sit/91
>> > HTTP/1.1"
>> > > > > 500 0 - "gzip(gfe)" ".appspot.com"
>> > > > > W 03-01 04:15PM 58.197
>> > > > > Request was aborted after waiting too long to attempt to service
>> > > > > your
>> > > > > request. Most likely, this indicates that you have reached your
>> > > > > simultaneous dynamic request limit. This is almost always due to
>> > > > > excessively high latency in your app. Please seehttp://
>> > > > code.google.com/appengine/docs/quotas.htmlfor more details.
>>
>> > > > > On Mar 1, 2:36 pm, Michael Wesner <mike.wes...@webfilings.com>
>> > wrote:
>>
>> > > > > > Correction/addition to my last email.
>>
>> > > > > > It turns out that our requests for this EC2 pull thing are
>> > > > > > actually
>> > > > much faster now. Gary and our other devs have reworked it. I need
>> > updated
>> > > > numbers, but they don't take 10s, probably more like 2s. We still
>> > > > have
>> > some
>> > > > heavy ~5s services though, so the same issue exists with the
>> > > > simul-req
>> > > > stuff, just to less extent. We don't actually hit this limit much
>> > > > now
>> > with
>> > > > the current beta that is in production, but it is low traffic at the
>> > moment.
>> > > > We are just getting ready to ramp up heavily.
>>
>> > > > > > I asked Nick what we should do, well just today after my last
>> > email, I
>> > > > have made contact with a Developer Advocate and whatnot, which is
>> > fantastic.
>> > > > It looks like we, as a business, will be able to have better
>> > > > contact
>> > with
>> > > > the GAE team. We would very much like to continue working with you
>> > > > to
>> > figure
>> > > > out what actions we can take and what provisioning we can do to make
>> > our
>> > > > product successful and scale it as we grow in the near future. Gary
>> > Orser
>> > > > will be replying to this thread soon with more findings from both
>> > > > our
>> > real
>> > > > app code and a little test app we are using and which he will share
>> > with
>> > > > you.
>>
>> > > > > > We plan on having a presence at Google I/O this year as we did
>> > > > > > at
>> > > > PyCon. Hopefully we can even get setup in the demonstration area at
>> > I/O.
>>
>> > > > > > Thanks Nick for your help. Could we possibly setup a quick
>> > > > > > skype
>> > conf
>> > > > call at some point?
>>
>> > > > > > -Mike Wesner
>>
>> > > > > > On Mar 1, 2010, at 1:13 PM, Michael Wesner wrote:
>>
>> > > > > > > Nick,
>>
>> > > > > > > If we (I work with Gary) require fairly heavy requests which
>> > > > > > > run
>> > for
>> > > > multiple seconds then it is not possible to get anywhere near 400
>> > > > QPS.
>> > The
>> > > > math used on the docs page only applies to 75ms requests.
>>
>> > > > > > > (1000 ms/second / 75 ms/request) * 30 = 400 requests/second
>>
>> > > > > > > so lets say each request takes 10 seconds (and ours, pulling
>> > > > > > > data
>> > to
>> > > > EC2 for a heavy operation that we can't do on GAE could take that
>> > > > much
>> > since
>> > > > we have to process and update some XML before sending it)
>>
>> > > > > > > (1000 ms/second / 10000 ms/request) * 30 = 3 requests/second
>>
>> > > > > > > And that does not even take into account all the other traffic
>> > > > > > > to
>> > our
>> > > > application, nor the fact that many users could be doing this same
>> > heavy
>> > > > operation at the same time. Our application will see spikes in this
>> > type of
>> > > > activity also. The docs also mention that CPU heavy apps incur
>> > penalties,
>> > > > which is vague and scary.
>>
>> > > > > > > Great effort is put into doing things in the most efficient
>> > > > > > > way
>> > > > possible, but not everyones apps can do everything in 75ms. Most all
>> > > > of
>> > our
>> > > > service calls are under 250ms. We do have a little overhead from our
>> > > > framework which we are constantly working on improving. Our
>> > application is
>> > > > AMF service/object based which is inherently heavy compared to
>> > > > simple
>> > web
>> > > > requests. It limits the amount of memcache work we can do also, but
>> > > > we
>> > are
>> > > > also working on improving our use of that.
>>
>> > > > > > > We easily hit these boundaries during testing so I think we
>> > really
>> > > > need much higher simultaneous dynamic request limits for not only
>> > > > our
>> > > > production instance but our dev/qa instances so we can test and load
>> > them to
>> > > > some degree. Our QA team could easily bust this limit 20 times
>> > > > over.
>>
>> > > > > > > So, Nick Johnson... I ask your advice. We are running a
>> > > > company/product on GAE. We are more than happy to pay for
>> > > > quota/service/extra assistance in these matters. What do you suggest
>> > > > we
>> > do?
>>
>> > > > > > > I should also mention that I spoke with Brett Slatkin at PyCon
>> > and he
>> > > > is now at least semi-familiar with the scope of product we have
>> > developed.
>> > > > I have exchanged contact info with him but have not heard anything
>> > back
>> > > > from him yet. We would really appreciate contact or even a brief
>> > meeting at
>> > > > some point (in person or otherwise).
>>
>> > > > > > > Thanks,
>>
>> > > > > > > -Mike Wesner
>>
>> > > > > > > On Mar 1, 2010, at 7:40 AM, Nick Johnson (Google) wrote:
>>
>> > > > > > >> Hi Gary,
>>
>> > > > > > >> Practically speaking, for an app that hasn't been given
>> > > > > > >> elevated
>> > > > permissions, you should be able to have at least 30 concurrent
>> > > > requests
>> > -
>> > > > equating to around 400 QPS if your app is fairly efficient. What
>> > problems
>> > > > are you running into that lead you to conclude you're hitting a
>> > > > limit
>> > at 4
>> > > > QPS, and that the problem is at App Engine's end?
>>
>> > > > > > >> -Nick Johnson
>>
>> > > > > > >> On Sat, Feb 27, 2010 at 8:23 PM, Gary Orser <
>> > garyor...@gmail.com>
>> > > > wrote:
>> > > > > > >> Hi all,
>>
>> > > > > > >> We were trying to create programmatic parallel access to our
>> > > > appengine
>> > > > > > >> application.
>>
>> > > > > > >> From EC2, we were attempting (with threads) to run parallel
>> > access
>> > > > > > >> (url gets/posts) to
>> > > > > > >> our appid. There are some long running processes that we
>> > > > > > >> need
>> > to
>> > > > run
>> > > > > > >> on EC2, for which
>> > > > > > >> we would like to get a bunch of information (entities +
>> > processing
>> > > > on
>> > > > > > >> appspot) quickly.
>>
>> > > > > > >> We seem to be running into a limit on the number of accesses
>> > that
>> > > > are
>> > > > > > >> allowed.
>> > > > > > >> (4 threads seems to be the effective limit)
>>
>> > > > > > >> Is there some sort of denial of service limit imposed on
>> > multiple
>> > > > > > >> accesses from a single IP?
>>
>> > > > > > >> Cheers, Gary
>>
>> > > > > > >> --
>> > > > > > >> You received this message because you are subscribed to the
>> > Google
>> > > > Groups "Google App Engine" group.
>> > > > > > >> To post to this group, send email to
>> > > > google-appeng...@googlegroups.com.
>> > > > > > >> To unsubscribe from this group, send email to
>> > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>> > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>
>> > > > .
>> > > > > > >> For more options, visit this group athttp://
>> > > > groups.google.com/group/google-appengine?hl=en.
>>
>> > > > > > >> --
>> > > > > > >> Nick Johnson, Developer Programs Engineer, App Engine
>> > > > > > >> Google Ireland Ltd. :: Registered in Dublin,
>>
>> ...
>>
>> read more »
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Sent from my mobile device
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
