Actually, 4 threads was before we optimized server side, and set up
the test environment.
I have a tarball, about 8mb, with the test environment. (django and
libraries, grrr)
What is the best way to post this? I don't see any file attachments
on groups.
Cheers, Gary
On Mar 2, 8:23 am, Eli Jones <eli.jo...@gmail.com> wrote:
> Are these threads you're using (at this point, it really seems like you
> should post some simplified code to illustrate the issue at hand) waiting
> for their response before trying to get again?
>
> Posting some code to help recreate this issue will lead to a much faster
> resolution.. as it stands.. I just know that someone on the internet has "10
> threads" that are hitting a dynamic request limit.
>
> I also know that in the initial e-mail, when the request took longer to
> return.. these "threads" were hitting a lower dynamic request limit (only 4
> could run). This suggest that there is an important detail to how your
> "threads" are doing their work.. and we would need that to provide useful
> help.
>
> Thanks for info.
>
> On Tue, Mar 2, 2010 at 10:01 AM, Gary Orser <garyor...@gmail.com> wrote:
>
> > But that's the point. I can not reach 30 active requests.
> > I can only reach 10 active requests without error.
>
> > Any ideas on how I can debug this?
>
> > Cheers, Gary.
>
> > On Mar 2, 7:05 am, "Nick Johnson (Google)" <nick.john...@google.com>
> > wrote:
> > > Hi,
>
> > > On Tue, Mar 2, 2010 at 1:54 PM, Wooble <geoffsp...@gmail.com> wrote:
> > > > The 500 requests per second number relies on the probably-unreasonable
> > > > assumption that each request can complete in ~75ms. Deliberately
> > > > making your requests take a whole 3 seconds each is, obviously, not
> > > > going to work. You can only have 10 instances active at a time by
> > > > default; if the pages you're serving actually take 3 seconds to
> > > > complete you'll need to optimize things a whole lot or be stuck with a
> > > > 3.33 request/sec maximum.
>
> > > Actually, the default limit is 30 active requests.
>
> > > -Nick Johnson
>
> > > > On Mar 1, 11:33 pm, Gary Orser <garyor...@gmail.com> wrote:
> > > > > Hi Nick,
>
> > > > > Hmm, I was running tests on a billing enabled appspot today. 100
> > > > > requests/test.
>
> > > > > 10 threads getting a URL with a 3 second sleep (to emulate
> > > > > computation) on appspot, was the most I could get without getting 500
> > > > > errors.
> > > > > If I raised the thread pool beyond 10, I started getting errors??
>
> > > > > That doesn't reconcile very well with this statement from the
> > > > > appengine website.
> > > > > "Requests
> > > > > The total number of requests to the app. The per-minute quotas
> > for
> > > > > application with billing enabled allow for up to 500 requests per
> > > > > second--more than one billion requests per month. If your application
> > > > > requires even higher quotas than the "billing-enabled" values listed
> > > > > below, you can request an increase in these limits here.
> > > > > "
>
> > > > > Is there some billing setting that affects this?
>
> > > > > Cheers, Gary
>
> > > > > PS. dead simple request handler.
>
> > > > > import time
> > > > > from django import http
> > > > > def sit(req):
> > > > > time.sleep(3)
> > > > > return http.HttpResponse('foo')
>
> > > > > errors are:
>
> > > > > 03-01 04:15PM 48.177 /sit/91 500 10019ms 0cpu_ms 0kb gzip(gfe)
> > > > > 153.90.236.210 - - [01/Mar/2010:16:15:58 -0800] "GET /sit/91
> > HTTP/1.1"
> > > > > 500 0 - "gzip(gfe)" ".appspot.com"
> > > > > W 03-01 04:15PM 58.197
> > > > > Request was aborted after waiting too long to attempt to service your
> > > > > request. Most likely, this indicates that you have reached your
> > > > > simultaneous dynamic request limit. This is almost always due to
> > > > > excessively high latency in your app. Please seehttp://
> > > > code.google.com/appengine/docs/quotas.htmlfor more details.
>
> > > > > On Mar 1, 2:36 pm, Michael Wesner <mike.wes...@webfilings.com>
> > wrote:
>
> > > > > > Correction/addition to my last email.
>
> > > > > > It turns out that our requests for this EC2 pull thing are actually
> > > > much faster now. Gary and our other devs have reworked it. I need
> > updated
> > > > numbers, but they don't take 10s, probably more like 2s. We still have
> > some
> > > > heavy ~5s services though, so the same issue exists with the simul-req
> > > > stuff, just to less extent. We don't actually hit this limit much now
> > with
> > > > the current beta that is in production, but it is low traffic at the
> > moment.
> > > > We are just getting ready to ramp up heavily.
>
> > > > > > I asked Nick what we should do, well just today after my last
> > email, I
> > > > have made contact with a Developer Advocate and whatnot, which is
> > fantastic.
> > > > It looks like we, as a business, will be able to have better contact
> > with
> > > > the GAE team. We would very much like to continue working with you to
> > figure
> > > > out what actions we can take and what provisioning we can do to make
> > our
> > > > product successful and scale it as we grow in the near future. Gary
> > Orser
> > > > will be replying to this thread soon with more findings from both our
> > real
> > > > app code and a little test app we are using and which he will share
> > with
> > > > you.
>
> > > > > > We plan on having a presence at Google I/O this year as we did at
> > > > PyCon. Hopefully we can even get setup in the demonstration area at
> > I/O.
>
> > > > > > Thanks Nick for your help. Could we possibly setup a quick skype
> > conf
> > > > call at some point?
>
> > > > > > -Mike Wesner
>
> > > > > > On Mar 1, 2010, at 1:13 PM, Michael Wesner wrote:
>
> > > > > > > Nick,
>
> > > > > > > If we (I work with Gary) require fairly heavy requests which run
> > for
> > > > multiple seconds then it is not possible to get anywhere near 400 QPS.
> > The
> > > > math used on the docs page only applies to 75ms requests.
>
> > > > > > > (1000 ms/second / 75 ms/request) * 30 = 400 requests/second
>
> > > > > > > so lets say each request takes 10 seconds (and ours, pulling data
> > to
> > > > EC2 for a heavy operation that we can't do on GAE could take that much
> > since
> > > > we have to process and update some XML before sending it)
>
> > > > > > > (1000 ms/second / 10000 ms/request) * 30 = 3 requests/second
>
> > > > > > > And that does not even take into account all the other traffic to
> > our
> > > > application, nor the fact that many users could be doing this same
> > heavy
> > > > operation at the same time. Our application will see spikes in this
> > type of
> > > > activity also. The docs also mention that CPU heavy apps incur
> > penalties,
> > > > which is vague and scary.
>
> > > > > > > Great effort is put into doing things in the most efficient way
> > > > possible, but not everyones apps can do everything in 75ms. Most all of
> > our
> > > > service calls are under 250ms. We do have a little overhead from our
> > > > framework which we are constantly working on improving. Our
> > application is
> > > > AMF service/object based which is inherently heavy compared to simple
> > web
> > > > requests. It limits the amount of memcache work we can do also, but we
> > are
> > > > also working on improving our use of that.
>
> > > > > > > We easily hit these boundaries during testing so I think we
> > really
> > > > need much higher simultaneous dynamic request limits for not only our
> > > > production instance but our dev/qa instances so we can test and load
> > them to
> > > > some degree. Our QA team could easily bust this limit 20 times over.
>
> > > > > > > So, Nick Johnson... I ask your advice. We are running a
> > > > company/product on GAE. We are more than happy to pay for
> > > > quota/service/extra assistance in these matters. What do you suggest we
> > do?
>
> > > > > > > I should also mention that I spoke with Brett Slatkin at PyCon
> > and he
> > > > is now at least semi-familiar with the scope of product we have
> > developed.
> > > > I have exchanged contact info with him but have not heard anything
> > back
> > > > from him yet. We would really appreciate contact or even a brief
> > meeting at
> > > > some point (in person or otherwise).
>
> > > > > > > Thanks,
>
> > > > > > > -Mike Wesner
>
> > > > > > > On Mar 1, 2010, at 7:40 AM, Nick Johnson (Google) wrote:
>
> > > > > > >> Hi Gary,
>
> > > > > > >> Practically speaking, for an app that hasn't been given elevated
> > > > permissions, you should be able to have at least 30 concurrent requests
> > -
> > > > equating to around 400 QPS if your app is fairly efficient. What
> > problems
> > > > are you running into that lead you to conclude you're hitting a limit
> > at 4
> > > > QPS, and that the problem is at App Engine's end?
>
> > > > > > >> -Nick Johnson
>
> > > > > > >> On Sat, Feb 27, 2010 at 8:23 PM, Gary Orser <
> > garyor...@gmail.com>
> > > > wrote:
> > > > > > >> Hi all,
>
> > > > > > >> We were trying to create programmatic parallel access to our
> > > > appengine
> > > > > > >> application.
>
> > > > > > >> From EC2, we were attempting (with threads) to run parallel
> > access
> > > > > > >> (url gets/posts) to
> > > > > > >> our appid. There are some long running processes that we need
> > to
> > > > run
> > > > > > >> on EC2, for which
> > > > > > >> we would like to get a bunch of information (entities +
> > processing
> > > > on
> > > > > > >> appspot) quickly.
>
> > > > > > >> We seem to be running into a limit on the number of accesses
> > that
> > > > are
> > > > > > >> allowed.
> > > > > > >> (4 threads seems to be the effective limit)
>
> > > > > > >> Is there some sort of denial of service limit imposed on
> > multiple
> > > > > > >> accesses from a single IP?
>
> > > > > > >> Cheers, Gary
>
> > > > > > >> --
> > > > > > >> You received this message because you are subscribed to the
> > Google
> > > > Groups "Google App Engine" group.
> > > > > > >> To post to this group, send email to
> > > > google-appeng...@googlegroups.com.
> > > > > > >> To unsubscribe from this group, send email to
> > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
> > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>
> > > > .
> > > > > > >> For more options, visit this group athttp://
> > > > groups.google.com/group/google-appengine?hl=en.
>
> > > > > > >> --
> > > > > > >> Nick Johnson, Developer Programs Engineer, App Engine
> > > > > > >> Google Ireland Ltd. :: Registered in Dublin,
>
> ...
>
> read more »
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
