Heh, yes.. I printed out your Java and it seemed perfectly fine after
reading through it several times... But Java is just wayy to busy
looking for my tastes.
Good luck getting some clarification on what's happening with this.
On 3/3/10, Gary Orser <garyor...@gmail.com> wrote:
> Eli,
>
> Python as usual is sooo much more elegant than java.
>
> I confirmed your test results on a billing enabled appspot.
>
> Cheers, Gary
>
> On Mar 2, 9:57 pm, Eli Jones <eli.jo...@gmail.com> wrote:
>> I did my own testing in python. And, I definitely couldn't get 30
>> simultaneous requests to work.
>>
>> This was done on a test app that does not have Billing Enabled.. so not
>> sure
>> if that affects the Dynamic Request Limit.
>>
>> Either way, it seems that for this test, the effective Limit was 16
>> simultaneous accesses. As soon as I modified my code to try 17, it
>> started
>> throwing a few Dynamic Request Limit errors.. though, the Logs page
>> AppEngine just refers to it as another Request (with code 500 and a time
>> of
>> 10 seconds).
>>
>> Here is the code for the handler.. after receiving a GET, it sleeps for 3
>> seconds and the responds with a 'hello':
>>
>> from google.appengine.ext import webapp
>> from google.appengine.ext.webapp.util import run_wsgi_app
>> import time
>>
>> class meTest(webapp.RequestHandler):
>> def get(self):
>> time.sleep(3)
>> meid = self.request.get('id')
>> self.response.out.write('hi! foo%s'%meid)
>>
>> application = webapp.WSGIApplication([('/test/meTest',meTest)],
>> debug = True)
>>
>> def main():
>> run_wsgi_app(application)
>>
>> if __name__ == "__main__":
>> main()
>>
>> And here is they python code I am running on my local machine to test. It
>> fires off 17 threads. Each thread requests a GET from the handler and
>> prints the response, three times in a row:
>>
>> import httplib
>> import threading
>>
>> class meThread(threading.Thread):
>> def run(self):
>> for i in range(3):
>> conn = httplib.HTTPConnection('datastoretester.appspot.com')
>> conn.request("GET","/test/meTest?id=%s" % self.getName())
>> response = conn.getresponse()
>> print response.read()
>> conn.close()
>>
>> for i in range(17):
>> meThread(name=str(i+1)).start()
>>
>> As soon as I scale the range back to (16), no more Dynamic Request Error.
>>
>> Special notes: I am not sure if there is some sort of built in
>> rate-limiting for requests from the same IP address.. but when sending 17
>> threads at the handler.. it is not responding in 3 seconds to their GETs..
>> It takes about 11 seconds.
>>
>> If you look in the Logs, for a run with 17 threads, you see this:
>>
>> 03-02 08:31PM 00.384 /test/meTest?id=16 200 12855ms 19cpu_ms 0kb gzip(gfe)
>> 03-02 08:31PM 01.266 /test/meTest?id=7 200 11973ms 19cpu_ms 0kb gzip(gfe)
>> 03-02 08:31PM 01.283 /test/meTest?id=15 500 10017ms 0cpu_ms 0kb gzip(gfe)
>> 03-02 08:30PM 58.280 /test/meTest?id=4 200 11952ms 0cpu_ms 0kb gzip(gfe)
>>
>> (Notice the 500 error in there as well.. that's the one that mentions the
>> Dynamic Request Limit).
>>
>> Now, if I run the test with only 4 threads, it looks nice and quick like
>> this:
>>
>> 03-02 08:35PM 25.602 /test/meTest?id=1 200 3009ms 0cpu_ms 0kb gzip(gfe)
>> 03-02 08:35PM 22.637 /test/meTest?id=2 200 3008ms 0cpu_ms 0kb gzip(gfe)
>> 03-02 08:35PM 22.623 /test/meTest?id=3 200 3009ms 0cpu_ms 0kb gzip(gfe)
>> 03-02 08:35PM 22.603 /test/meTest?id=4 200 3008ms 0cpu_ms 0kb gzip(gfe)
>>
>> Once I start testing with more than 4 threads, it starts to slow down in
>> its
>> response time...
>>
>> So, I'd guess something would need to be clarified.. is there some
>> internal
>> limiting going on per ip address? Does a "long" running process have a
>> lower simultaneous request limit?
>>
>> On Tue, Mar 2, 2010 at 12:21 PM, Gary Orser <garyor...@gmail.com> wrote:
>> > Eli,
>>
>> > You have the python request server.
>> > Here is the java client:
>> > You'll have to get the libraries yourself.
>>
>> > Cheers, Gary
>>
>> > import java.util.ArrayList;
>> > import java.util.concurrent.Callable;
>> > import java.util.concurrent.ExecutionException;
>> > import java.util.concurrent.ExecutorService;
>> > import java.util.concurrent.Executors;
>> > import java.util.concurrent.Future;
>>
>> > import org.apache.commons.io.IOUtils;
>> > import org.apache.http.HttpHost;
>> > import org.apache.http.HttpResponse;
>> > import org.apache.http.HttpVersion;
>> > import org.apache.http.client.HttpClient;
>> > import org.apache.http.client.methods.HttpGet;
>> > import org.apache.http.conn.ClientConnectionManager;
>> > import org.apache.http.conn.params.ConnManagerParams;
>> > import org.apache.http.conn.params.ConnPerRouteBean;
>> > import org.apache.http.conn.scheme.PlainSocketFactory;
>> > import org.apache.http.conn.scheme.Scheme;
>> > import org.apache.http.conn.scheme.SchemeRegistry;
>> > import org.apache.http.conn.ssl.SSLSocketFactory;
>> > import org.apache.http.impl.client.DefaultHttpClient;
>> > import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager;
>> > import org.apache.http.params.BasicHttpParams;
>> > import org.apache.http.params.HttpParams;
>> > import org.apache.http.params.HttpProtocolParams;
>>
>> > import org.apache.commons.logging.Log;
>> > import org.apache.commons.logging.LogFactory;
>>
>> > public class Main
>> > {
>> > private Log log = LogFactory.getLog(Main.class);
>> > // ADJUST: number of threads to make requests on
>> > public static int NUM_PARALLEL_SECTION_REQUESTS = 20;
>> > public static HttpParams httpParams = new BasicHttpParams();
>> > {
>> > httpParams.setBooleanParameter("http.protocol.expect-
>> > continue", false);
>> > // ADJUST: if this is included, will use 8888 as a
>> > proxy port. Charles Proxy defaults to this port.
>> > //httpParams.setParameter("http.route.default-proxy",
>> > new HttpHost("localhost", 8888));
>> > }
>>
>> > protected class GetSection implements Callable<String>
>> > {
>> > protected int index;
>> > protected HttpClient client;
>> > protected String URL;
>> > public GetSection(int index, HttpClient client, String
>> > URL)
>> > {
>> > this.index = index;
>> > this.client = client;
>> > this.URL = URL; }
>> > public String call() throws Exception
>> > {
>> > HttpGet getSection = new
>> > HttpGet(URL); HttpResponse respSection =
>> > client.execute(getSection);
>> > String foo =
>> > IOUtils.toString(respSection.getEntity().getContent(), "UTF-8");
>> > return foo;
>> > } }
>>
>> > public static void main(String[] args) throws Exception
>> > { new Main().maint(args);
>> > }
>>
>> > public void maint(String[] args) throws Exception {
>> > SchemeRegistry schemeRegistry = new SchemeRegistry();
>> > schemeRegistry.register(new Scheme("http",
>> > PlainSocketFactory.getSocketFactory(), 80));
>> > schemeRegistry.register(new Scheme("https",
>> > SSLSocketFactory.getSocketFactory(), 443));
>> > HttpParams params = new BasicHttpParams();
>> > ConnManagerParams.setMaxTotalConnections(params,
>> > NUM_PARALLEL_SECTION_REQUESTS);
>> > ConnManagerParams.setMaxConnectionsPerRoute(params,
>> > new ConnPerRouteBean(NUM_PARALLEL_SECTION_REQUESTS));
>> > HttpProtocolParams.setVersion(params,
>> > HttpVersion.HTTP_1_1);
>> > ClientConnectionManager cm = new
>> > ThreadSafeClientConnManager(params, schemeRegistry);
>> > HttpClient client = new DefaultHttpClient(cm,
>> > httpParams);
>>
>> > ExecutorService es =
>> > Executors.newFixedThreadPool(NUM_PARALLEL_SECTION_REQUESTS);
>> > // ADJUST: total number of requests to make.
>> > int numSections = 100;
>> > ArrayList<Future<String>> futures = new
>> > ArrayList<Future<String>>(numSections);
>> > log.info("queuing requests");
>> > for (int i = 0; i < numSections; i++)
>> > {
>> > // ADJUST: set a real hostname here
>> > futures.add(es.submit(new GetSection(i,
>> > client, "http://yourappid.appspot.com/sit/"; + Integer.toString(i))));
>> > // ADJUST: stagger initial requests with this
>> > sleep
>> > //Thread.sleep(200);
>> > }
>>
>> > es.shutdown();
>>
>> > log.info("waiting for thread pool to finish");
>> > while (!es.isTerminated())
>> > Thread.sleep(500);
>>
>> > log.info("all requests queued");
>>
>> > try
>> > {
>> > for (Future<String> future: futures)
>> > future.get();
>> > log.info("got all futures");
>> > }
>> > catch (ExecutionException e)
>> > {
>> > // TODO: not really sure what to do if cause
>> > is Throwable but not Exception
>> > if (e.getCause() instanceof Exception)
>> > throw (Exception)e.getCause();
>> > }
>> > }
>>
>> > On Mar 2, 9:50 am, Eli Jones <eli.jo...@gmail.com> wrote:
>> > > What I'm suggesting is.. You need to create a simple test setup that
>> > > recreates this dynamic request limit error.. (It definitely should not
>> > > take 8mb of code).
>>
>> > > I will see if I can create a handler like the one you posted, deploy
>> > > it, and then run 30 seperate processes that keep getting from that
>> > > handler.. (I can write this up in less than 10kb or python code)...
>>
>> > > My guess is this will work. Without seeing sample code.. I can't tell
>> > > where you may be going wrong (or where GAE may be breaking)
>>
>> > > On 3/2/10, Gary Orser <garyor...@gmail.com> wrote:
>>
>> > > > Actually, 4 threads was before we optimized server side, and set up
>> > > > the test environment.
>>
>> > > > I have a tarball, about 8mb, with the test environment. (django and
>> > > > libraries, grrr)
>> > > > What is the best way to post this? I don't see any file attachments
>> > > > on groups.
>>
>> > > > Cheers, Gary
>>
>> > > > On Mar 2, 8:23 am, Eli Jones <eli.jo...@gmail.com> wrote:
>> > > >> Are these threads you're using (at this point, it really seems like
>> > you
>> > > >> should post some simplified code to illustrate the issue at hand)
>> > waiting
>> > > >> for their response before trying to get again?
>>
>> > > >> Posting some code to help recreate this issue will lead to a much
>> > faster
>>
>> ...
>>
>> read more »
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Sent from my mobile device
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
