It's obviously very hard to diagnose this remotely. But if it's spending 8 out of 15 seconds waiting, then what about the other 7 seconds, where exactly are they lost? There's no point arguing that this should work (of course it should), IMO you need to start analysing. Maybe you did already, but it's not obvious from your posts what you did already, and what you didn't. In another thread you mentioned you didn't use appstats for a long time - I'd do that first thing. Also, I'd add very detailed manual logging, so you know what bits run fast, and which ones don't.
I've encountered plenty of GAE glitches in the past, but also quite a few things were our own fault. I'm not sure how performance analysis works on Python, but in Java it would be time to bring on the good old Profiler. I'd make sure that on localhost a request really just takes a few miliseconds, even under load, and then take it live, to a loadtesting environment and see what happens there under growing load. Given the numbers you posted, I'd not be surprised if some locking&synchronizing is part of the problem, but that's just a guess. Find out what operation really takes the most time, or at what load things start going wrong. On Monday, July 9, 2012 4:26:01 AM UTC+2, Richard wrote: > > Yep, I know about that. The internal limit is 1000msec, but you should > aim for 750msec just to 'be safe'. > > Here are my timings: > > ms=15376 cpu_ms=37 api_cpu_ms=17 cpm_usd=0.003428 pending_ms=8265 > > 37 msec of CPU .... after waiting 8.2 seconds to execute. > > Some math: > - 500 incoming requests > - 40 servers > = ~ 15 requests/server > ... but each request takes ~55 msec of CPU + API .... > > That's 825 msec for THE WHOLE LOT ! .... So why on earth is there 8 second > of waiting before that request was serviced ? > > > -R > > On Sunday, July 8, 2012 10:13:28 PM UTC-4, Per wrote: >> >> >> Not sure where I saw it, but I believe an application must respond within >> a second at most *on average*, or GAE reserves the right to throttle it. >> So, if you have some kind of slowdown inside your app, then requests start >> piling up. I'm not a Python developer, but if you had the equivalent of a >> synchronised map in there, with lots of concurrent access, then you might >> end up in a situation like this. Not sure about what's really to blame of >> course, but I'd strongly recommend setting up a controlled loadtesting >> environment (just copy the app, and start firing requests at it). Add >> plenty of logging to your app, enable appstats, and then slowly increase >> the load. Maybe even strip the application down, starting only with read >> access. I'm sure you will find something, and I would love to hear what it >> was! I wish you luck! >> >> >> >> >> >> On Monday, July 9, 2012 3:19:04 AM UTC+2, Richard wrote: >>> >>> I could if memcache actually worked. But it does not. I originally >>> tried to use it and found that I could not push the game state to memcache >>> and then have the other instances pull it. They would get versions of it >>> that were up to 5 minutes old. My timings are 5 second windows. >>> 5 secs to submit all scores >>> 5 secs to reap scores and calc leaderboards >>> 5 secs to fan out results to clients >>> >>> Experience shows that memcache is just broken for that sort of timing. >>> >>> As for using Go instead of Python. I am not sure I follow why Go should >>> be better. The lag is not coming from CPU or queries. >>> >>> Right now I am running 50 instances to serve 500 game clients. $48 for >>> the last 18 hours. 11% of my requests result in "Request was aborted". >>> Yeah, that is 12 THOUSAND fails. >>> >>> Back in the year 1995, ftp.cdrom.com could serve 2000 clients >>> simultaneously on a Pentium Pro 200MHz .... and I cannot serve 20 clients >>> on a 500Mhz virtual box ? >>> >>> I still contend there is some internal throttling going on somewhere. >>> >>> -R >>> >>> >>> >>> On Sunday, July 8, 2012 6:23:02 PM UTC-4, Kyle Finley wrote: >>>> >>>> Richard, >>>> >>>> Another option would be to move the Game State request to a >>>> Go<https://developers.google.com/appengine/docs/go/overview> instance, >>>> either as a >>>> backend<https://developers.google.com/appengine/docs/go/config/backends>or >>>> as a separate version. I believe a single Go instance should be able the >>>> handle 500 request / second. You could then share the Game State between >>>> the Python version and the Go version through Memcache, cacheing to >>>> instance memory every 5 sec. >>>> >>>> - Kyle >>>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_V2FQCZb_sQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
