Eli, Python as usual is sooo much more elegant than java.
I confirmed your test results on a billing enabled appspot. Cheers, Gary On Mar 2, 9:57 pm, Eli Jones <eli.jo...@gmail.com> wrote: > I did my own testing in python. And, I definitely couldn't get 30 > simultaneous requests to work. > > This was done on a test app that does not have Billing Enabled.. so not sure > if that affects the Dynamic Request Limit. > > Either way, it seems that for this test, the effective Limit was 16 > simultaneous accesses. As soon as I modified my code to try 17, it started > throwing a few Dynamic Request Limit errors.. though, the Logs page > AppEngine just refers to it as another Request (with code 500 and a time of > 10 seconds). > > Here is the code for the handler.. after receiving a GET, it sleeps for 3 > seconds and the responds with a 'hello': > > from google.appengine.ext import webapp > from google.appengine.ext.webapp.util import run_wsgi_app > import time > > class meTest(webapp.RequestHandler): > def get(self): > time.sleep(3) > meid = self.request.get('id') > self.response.out.write('hi! foo%s'%meid) > > application = webapp.WSGIApplication([('/test/meTest',meTest)], > debug = True) > > def main(): > run_wsgi_app(application) > > if __name__ == "__main__": > main() > > And here is they python code I am running on my local machine to test. It > fires off 17 threads. Each thread requests a GET from the handler and > prints the response, three times in a row: > > import httplib > import threading > > class meThread(threading.Thread): > def run(self): > for i in range(3): > conn = httplib.HTTPConnection('datastoretester.appspot.com') > conn.request("GET","/test/meTest?id=%s" % self.getName()) > response = conn.getresponse() > print response.read() > conn.close() > > for i in range(17): > meThread(name=str(i+1)).start() > > As soon as I scale the range back to (16), no more Dynamic Request Error. > > Special notes: I am not sure if there is some sort of built in > rate-limiting for requests from the same IP address.. but when sending 17 > threads at the handler.. it is not responding in 3 seconds to their GETs.. > It takes about 11 seconds. > > If you look in the Logs, for a run with 17 threads, you see this: > > 03-02 08:31PM 00.384 /test/meTest?id=16 200 12855ms 19cpu_ms 0kb gzip(gfe) > 03-02 08:31PM 01.266 /test/meTest?id=7 200 11973ms 19cpu_ms 0kb gzip(gfe) > 03-02 08:31PM 01.283 /test/meTest?id=15 500 10017ms 0cpu_ms 0kb gzip(gfe) > 03-02 08:30PM 58.280 /test/meTest?id=4 200 11952ms 0cpu_ms 0kb gzip(gfe) > > (Notice the 500 error in there as well.. that's the one that mentions the > Dynamic Request Limit). > > Now, if I run the test with only 4 threads, it looks nice and quick like > this: > > 03-02 08:35PM 25.602 /test/meTest?id=1 200 3009ms 0cpu_ms 0kb gzip(gfe) > 03-02 08:35PM 22.637 /test/meTest?id=2 200 3008ms 0cpu_ms 0kb gzip(gfe) > 03-02 08:35PM 22.623 /test/meTest?id=3 200 3009ms 0cpu_ms 0kb gzip(gfe) > 03-02 08:35PM 22.603 /test/meTest?id=4 200 3008ms 0cpu_ms 0kb gzip(gfe) > > Once I start testing with more than 4 threads, it starts to slow down in its > response time... > > So, I'd guess something would need to be clarified.. is there some internal > limiting going on per ip address? Does a "long" running process have a > lower simultaneous request limit? > > On Tue, Mar 2, 2010 at 12:21 PM, Gary Orser <garyor...@gmail.com> wrote: > > Eli, > > > You have the python request server. > > Here is the java client: > > You'll have to get the libraries yourself. > > > Cheers, Gary > > > import java.util.ArrayList; > > import java.util.concurrent.Callable; > > import java.util.concurrent.ExecutionException; > > import java.util.concurrent.ExecutorService; > > import java.util.concurrent.Executors; > > import java.util.concurrent.Future; > > > import org.apache.commons.io.IOUtils; > > import org.apache.http.HttpHost; > > import org.apache.http.HttpResponse; > > import org.apache.http.HttpVersion; > > import org.apache.http.client.HttpClient; > > import org.apache.http.client.methods.HttpGet; > > import org.apache.http.conn.ClientConnectionManager; > > import org.apache.http.conn.params.ConnManagerParams; > > import org.apache.http.conn.params.ConnPerRouteBean; > > import org.apache.http.conn.scheme.PlainSocketFactory; > > import org.apache.http.conn.scheme.Scheme; > > import org.apache.http.conn.scheme.SchemeRegistry; > > import org.apache.http.conn.ssl.SSLSocketFactory; > > import org.apache.http.impl.client.DefaultHttpClient; > > import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager; > > import org.apache.http.params.BasicHttpParams; > > import org.apache.http.params.HttpParams; > > import org.apache.http.params.HttpProtocolParams; > > > import org.apache.commons.logging.Log; > > import org.apache.commons.logging.LogFactory; > > > public class Main > > { > > private Log log = LogFactory.getLog(Main.class); > > // ADJUST: number of threads to make requests on > > public static int NUM_PARALLEL_SECTION_REQUESTS = 20; > > public static HttpParams httpParams = new BasicHttpParams(); > > { > > httpParams.setBooleanParameter("http.protocol.expect- > > continue", false); > > // ADJUST: if this is included, will use 8888 as a > > proxy port. Charles Proxy defaults to this port. > > //httpParams.setParameter("http.route.default-proxy", > > new HttpHost("localhost", 8888)); > > } > > > protected class GetSection implements Callable<String> > > { > > protected int index; > > protected HttpClient client; > > protected String URL; > > public GetSection(int index, HttpClient client, String > > URL) > > { > > this.index = index; > > this.client = client; > > this.URL = URL; } > > public String call() throws Exception > > { > > HttpGet getSection = new > > HttpGet(URL); HttpResponse respSection = > > client.execute(getSection); > > String foo = > > IOUtils.toString(respSection.getEntity().getContent(), "UTF-8"); > > return foo; > > } } > > > public static void main(String[] args) throws Exception > > { new Main().maint(args); > > } > > > public void maint(String[] args) throws Exception { > > SchemeRegistry schemeRegistry = new SchemeRegistry(); > > schemeRegistry.register(new Scheme("http", > > PlainSocketFactory.getSocketFactory(), 80)); > > schemeRegistry.register(new Scheme("https", > > SSLSocketFactory.getSocketFactory(), 443)); > > HttpParams params = new BasicHttpParams(); > > ConnManagerParams.setMaxTotalConnections(params, > > NUM_PARALLEL_SECTION_REQUESTS); > > ConnManagerParams.setMaxConnectionsPerRoute(params, > > new ConnPerRouteBean(NUM_PARALLEL_SECTION_REQUESTS)); > > HttpProtocolParams.setVersion(params, > > HttpVersion.HTTP_1_1); > > ClientConnectionManager cm = new > > ThreadSafeClientConnManager(params, schemeRegistry); > > HttpClient client = new DefaultHttpClient(cm, > > httpParams); > > > ExecutorService es = > > Executors.newFixedThreadPool(NUM_PARALLEL_SECTION_REQUESTS); > > // ADJUST: total number of requests to make. > > int numSections = 100; > > ArrayList<Future<String>> futures = new > > ArrayList<Future<String>>(numSections); > > log.info("queuing requests"); > > for (int i = 0; i < numSections; i++) > > { > > // ADJUST: set a real hostname here > > futures.add(es.submit(new GetSection(i, > > client, "http://yourappid.appspot.com/sit/" + Integer.toString(i)))); > > // ADJUST: stagger initial requests with this > > sleep > > //Thread.sleep(200); > > } > > > es.shutdown(); > > > log.info("waiting for thread pool to finish"); > > while (!es.isTerminated()) > > Thread.sleep(500); > > > log.info("all requests queued"); > > > try > > { > > for (Future<String> future: futures) > > future.get(); > > log.info("got all futures"); > > } > > catch (ExecutionException e) > > { > > // TODO: not really sure what to do if cause > > is Throwable but not Exception > > if (e.getCause() instanceof Exception) > > throw (Exception)e.getCause(); > > } > > } > > > On Mar 2, 9:50 am, Eli Jones <eli.jo...@gmail.com> wrote: > > > What I'm suggesting is.. You need to create a simple test setup that > > > recreates this dynamic request limit error.. (It definitely should not > > > take 8mb of code). > > > > I will see if I can create a handler like the one you posted, deploy > > > it, and then run 30 seperate processes that keep getting from that > > > handler.. (I can write this up in less than 10kb or python code)... > > > > My guess is this will work. Without seeing sample code.. I can't tell > > > where you may be going wrong (or where GAE may be breaking) > > > > On 3/2/10, Gary Orser <garyor...@gmail.com> wrote: > > > > > Actually, 4 threads was before we optimized server side, and set up > > > > the test environment. > > > > > I have a tarball, about 8mb, with the test environment. (django and > > > > libraries, grrr) > > > > What is the best way to post this? I don't see any file attachments > > > > on groups. > > > > > Cheers, Gary > > > > > On Mar 2, 8:23 am, Eli Jones <eli.jo...@gmail.com> wrote: > > > >> Are these threads you're using (at this point, it really seems like > > you > > > >> should post some simplified code to illustrate the issue at hand) > > waiting > > > >> for their response before trying to get again? > > > > >> Posting some code to help recreate this issue will lead to a much > > faster > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.