Eli,
Python as usual is sooo much more elegant than java.
I confirmed your test results on a billing enabled appspot.
Cheers, Gary
On Mar 2, 9:57 pm, Eli Jones <eli.jo...@gmail.com> wrote:
> I did my own testing in python. And, I definitely couldn't get 30
> simultaneous requests to work.
>
> This was done on a test app that does not have Billing Enabled.. so not sure
> if that affects the Dynamic Request Limit.
>
> Either way, it seems that for this test, the effective Limit was 16
> simultaneous accesses. As soon as I modified my code to try 17, it started
> throwing a few Dynamic Request Limit errors.. though, the Logs page
> AppEngine just refers to it as another Request (with code 500 and a time of
> 10 seconds).
>
> Here is the code for the handler.. after receiving a GET, it sleeps for 3
> seconds and the responds with a 'hello':
>
> from google.appengine.ext import webapp
> from google.appengine.ext.webapp.util import run_wsgi_app
> import time
>
> class meTest(webapp.RequestHandler):
> def get(self):
> time.sleep(3)
> meid = self.request.get('id')
> self.response.out.write('hi! foo%s'%meid)
>
> application = webapp.WSGIApplication([('/test/meTest',meTest)],
> debug = True)
>
> def main():
> run_wsgi_app(application)
>
> if __name__ == "__main__":
> main()
>
> And here is they python code I am running on my local machine to test. It
> fires off 17 threads. Each thread requests a GET from the handler and
> prints the response, three times in a row:
>
> import httplib
> import threading
>
> class meThread(threading.Thread):
> def run(self):
> for i in range(3):
> conn = httplib.HTTPConnection('datastoretester.appspot.com')
> conn.request("GET","/test/meTest?id=%s" % self.getName())
> response = conn.getresponse()
> print response.read()
> conn.close()
>
> for i in range(17):
> meThread(name=str(i+1)).start()
>
> As soon as I scale the range back to (16), no more Dynamic Request Error.
>
> Special notes: I am not sure if there is some sort of built in
> rate-limiting for requests from the same IP address.. but when sending 17
> threads at the handler.. it is not responding in 3 seconds to their GETs..
> It takes about 11 seconds.
>
> If you look in the Logs, for a run with 17 threads, you see this:
>
> 03-02 08:31PM 00.384 /test/meTest?id=16 200 12855ms 19cpu_ms 0kb gzip(gfe)
> 03-02 08:31PM 01.266 /test/meTest?id=7 200 11973ms 19cpu_ms 0kb gzip(gfe)
> 03-02 08:31PM 01.283 /test/meTest?id=15 500 10017ms 0cpu_ms 0kb gzip(gfe)
> 03-02 08:30PM 58.280 /test/meTest?id=4 200 11952ms 0cpu_ms 0kb gzip(gfe)
>
> (Notice the 500 error in there as well.. that's the one that mentions the
> Dynamic Request Limit).
>
> Now, if I run the test with only 4 threads, it looks nice and quick like
> this:
>
> 03-02 08:35PM 25.602 /test/meTest?id=1 200 3009ms 0cpu_ms 0kb gzip(gfe)
> 03-02 08:35PM 22.637 /test/meTest?id=2 200 3008ms 0cpu_ms 0kb gzip(gfe)
> 03-02 08:35PM 22.623 /test/meTest?id=3 200 3009ms 0cpu_ms 0kb gzip(gfe)
> 03-02 08:35PM 22.603 /test/meTest?id=4 200 3008ms 0cpu_ms 0kb gzip(gfe)
>
> Once I start testing with more than 4 threads, it starts to slow down in its
> response time...
>
> So, I'd guess something would need to be clarified.. is there some internal
> limiting going on per ip address? Does a "long" running process have a
> lower simultaneous request limit?
>
> On Tue, Mar 2, 2010 at 12:21 PM, Gary Orser <garyor...@gmail.com> wrote:
> > Eli,
>
> > You have the python request server.
> > Here is the java client:
> > You'll have to get the libraries yourself.
>
> > Cheers, Gary
>
> > import java.util.ArrayList;
> > import java.util.concurrent.Callable;
> > import java.util.concurrent.ExecutionException;
> > import java.util.concurrent.ExecutorService;
> > import java.util.concurrent.Executors;
> > import java.util.concurrent.Future;
>
> > import org.apache.commons.io.IOUtils;
> > import org.apache.http.HttpHost;
> > import org.apache.http.HttpResponse;
> > import org.apache.http.HttpVersion;
> > import org.apache.http.client.HttpClient;
> > import org.apache.http.client.methods.HttpGet;
> > import org.apache.http.conn.ClientConnectionManager;
> > import org.apache.http.conn.params.ConnManagerParams;
> > import org.apache.http.conn.params.ConnPerRouteBean;
> > import org.apache.http.conn.scheme.PlainSocketFactory;
> > import org.apache.http.conn.scheme.Scheme;
> > import org.apache.http.conn.scheme.SchemeRegistry;
> > import org.apache.http.conn.ssl.SSLSocketFactory;
> > import org.apache.http.impl.client.DefaultHttpClient;
> > import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager;
> > import org.apache.http.params.BasicHttpParams;
> > import org.apache.http.params.HttpParams;
> > import org.apache.http.params.HttpProtocolParams;
>
> > import org.apache.commons.logging.Log;
> > import org.apache.commons.logging.LogFactory;
>
> > public class Main
> > {
> > private Log log = LogFactory.getLog(Main.class);
> > // ADJUST: number of threads to make requests on
> > public static int NUM_PARALLEL_SECTION_REQUESTS = 20;
> > public static HttpParams httpParams = new BasicHttpParams();
> > {
> > httpParams.setBooleanParameter("http.protocol.expect-
> > continue", false);
> > // ADJUST: if this is included, will use 8888 as a
> > proxy port. Charles Proxy defaults to this port.
> > //httpParams.setParameter("http.route.default-proxy",
> > new HttpHost("localhost", 8888));
> > }
>
> > protected class GetSection implements Callable<String>
> > {
> > protected int index;
> > protected HttpClient client;
> > protected String URL;
> > public GetSection(int index, HttpClient client, String
> > URL)
> > {
> > this.index = index;
> > this.client = client;
> > this.URL = URL; }
> > public String call() throws Exception
> > {
> > HttpGet getSection = new
> > HttpGet(URL); HttpResponse respSection =
> > client.execute(getSection);
> > String foo =
> > IOUtils.toString(respSection.getEntity().getContent(), "UTF-8");
> > return foo;
> > } }
>
> > public static void main(String[] args) throws Exception
> > { new Main().maint(args);
> > }
>
> > public void maint(String[] args) throws Exception {
> > SchemeRegistry schemeRegistry = new SchemeRegistry();
> > schemeRegistry.register(new Scheme("http",
> > PlainSocketFactory.getSocketFactory(), 80));
> > schemeRegistry.register(new Scheme("https",
> > SSLSocketFactory.getSocketFactory(), 443));
> > HttpParams params = new BasicHttpParams();
> > ConnManagerParams.setMaxTotalConnections(params,
> > NUM_PARALLEL_SECTION_REQUESTS);
> > ConnManagerParams.setMaxConnectionsPerRoute(params,
> > new ConnPerRouteBean(NUM_PARALLEL_SECTION_REQUESTS));
> > HttpProtocolParams.setVersion(params,
> > HttpVersion.HTTP_1_1);
> > ClientConnectionManager cm = new
> > ThreadSafeClientConnManager(params, schemeRegistry);
> > HttpClient client = new DefaultHttpClient(cm,
> > httpParams);
>
> > ExecutorService es =
> > Executors.newFixedThreadPool(NUM_PARALLEL_SECTION_REQUESTS);
> > // ADJUST: total number of requests to make.
> > int numSections = 100;
> > ArrayList<Future<String>> futures = new
> > ArrayList<Future<String>>(numSections);
> > log.info("queuing requests");
> > for (int i = 0; i < numSections; i++)
> > {
> > // ADJUST: set a real hostname here
> > futures.add(es.submit(new GetSection(i,
> > client, "http://yourappid.appspot.com/sit/"; + Integer.toString(i))));
> > // ADJUST: stagger initial requests with this
> > sleep
> > //Thread.sleep(200);
> > }
>
> > es.shutdown();
>
> > log.info("waiting for thread pool to finish");
> > while (!es.isTerminated())
> > Thread.sleep(500);
>
> > log.info("all requests queued");
>
> > try
> > {
> > for (Future<String> future: futures)
> > future.get();
> > log.info("got all futures");
> > }
> > catch (ExecutionException e)
> > {
> > // TODO: not really sure what to do if cause
> > is Throwable but not Exception
> > if (e.getCause() instanceof Exception)
> > throw (Exception)e.getCause();
> > }
> > }
>
> > On Mar 2, 9:50 am, Eli Jones <eli.jo...@gmail.com> wrote:
> > > What I'm suggesting is.. You need to create a simple test setup that
> > > recreates this dynamic request limit error.. (It definitely should not
> > > take 8mb of code).
>
> > > I will see if I can create a handler like the one you posted, deploy
> > > it, and then run 30 seperate processes that keep getting from that
> > > handler.. (I can write this up in less than 10kb or python code)...
>
> > > My guess is this will work. Without seeing sample code.. I can't tell
> > > where you may be going wrong (or where GAE may be breaking)
>
> > > On 3/2/10, Gary Orser <garyor...@gmail.com> wrote:
>
> > > > Actually, 4 threads was before we optimized server side, and set up
> > > > the test environment.
>
> > > > I have a tarball, about 8mb, with the test environment. (django and
> > > > libraries, grrr)
> > > > What is the best way to post this? I don't see any file attachments
> > > > on groups.
>
> > > > Cheers, Gary
>
> > > > On Mar 2, 8:23 am, Eli Jones <eli.jo...@gmail.com> wrote:
> > > >> Are these threads you're using (at this point, it really seems like
> > you
> > > >> should post some simplified code to illustrate the issue at hand)
> > waiting
> > > >> for their response before trying to get again?
>
> > > >> Posting some code to help recreate this issue will lead to a much
> > faster
>
> ...
>
> read more »
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
