[google-appengine] Simultaneous requests
Hello! I recently wrote an app running on GAE. It is a facebook statistics gatherer, which relies upon the Facebook API to function. Recently the app got a lot more users, and the requests per second "shot" to 4, which is the highest it has ever been. However AppEngine did not scale well at all, with over 25% of my requests being shot down because of the "Max Simultaneous request limit", which is said to be always caused by "Bad coding and long page request times", which is fair enough. The problem is my app relies on a Facebook API call called "feed.get ()", which returns a (long) JSON response detailing all the users actions in the last X days, from which my app generates statistics from. The getting of this data, along with the users friends and the users groups is very expensive (Takes up 99.9% of the request time). The facebook API fails as well sometimes as well, as shown by these logs: Request recieved Stream.get() failed us Stream.get() ran fine get_add_url() ran fine Friends.get() ran fine Groups.get() ran fine Network done in 9.25191187859 Loop and stats created in 0.000972032546997 seconds Created render info in 0.00014591217041 Rendered in 0.0445511341095 seconds Request finished in 9.29821896553 seconds As you can see 9.25 seconds is spent on getting the data only only 0.04 on processing it. The time taken varies greatly, sometimes it takes only 2 seconds in total and other times it hits up to 15 (With multiple API failures). My question is how can I try and avoid hitting the max request limit when it is a third party that is causing my long response times.? I could use some Ajax calls to get the stats, but that would double the requests needed (1 request to get the page, another to get the data). Is there any way to increase my request limit? Regards ~Tom -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Simultaneous Requests
I am currently working on a way to mass email [in a short period of time] using Google App Engine. I figured that as of now the best way to do this would be to run many requests in parallel. I was wondering if my app would accept many requests at once, and if so what would the limit be [in the paid version]? Would this work? Is this the best way to do this? As for implementing the parallel requests I figured the simplest way would be to use the python threading. I was wondering if using AJAX would be any better, or if it would even work for parallel requests?? Thanks, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
Re: [google-appengine] Simultaneous requests
Collect offline the Facebook data with cron and taskqueue requests. Setup the taskqueue to request only 1 every x seconds, to be sure the previous one terminated. Cache the results of these collects in datastore objects. When the user requests, show the datastore object. 2010/1/13 Tom : > Hello! > I recently wrote an app running on GAE. It is a facebook statistics > gatherer, which relies upon the Facebook API to function. > > Recently the app got a lot more users, and the requests per second > "shot" to 4, which is the highest it has ever been. However AppEngine > did not scale well at all, with over 25% of my requests being shot > down because of the "Max Simultaneous request limit", which is said to > be always caused by "Bad coding and long page request times", which is > fair enough. > > The problem is my app relies on a Facebook API call called "feed.get > ()", which returns a (long) JSON response detailing all the users > actions in the last X days, from which my app generates statistics > from. The getting of this data, along with the users friends and the > users groups is very expensive (Takes up 99.9% of the request time). > The facebook API fails as well sometimes as well, as shown by these > logs: > > Request recieved > Stream.get() failed us > Stream.get() ran fine > get_add_url() ran fine > Friends.get() ran fine > Groups.get() ran fine > Network done in 9.25191187859 > Loop and stats created in 0.000972032546997 seconds > Created render info in 0.00014591217041 > Rendered in 0.0445511341095 seconds > Request finished in 9.29821896553 seconds > > > As you can see 9.25 seconds is spent on getting the data only only > 0.04 on processing it. The time taken varies greatly, sometimes it > takes only 2 seconds in total and other times it hits up to 15 (With > multiple API failures). > > My question is how can I try and avoid hitting the max request limit > when it is a third party that is causing my long response times.? > I could use some Ajax calls to get the stats, but that would double > the requests needed (1 request to get the page, another to get the > data). > > Is there any way to increase my request limit? > > Regards > ~Tom > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
