Just out of curiosity - do book prices really change more than once every hour? Surely, you could have some kind of a hybrid approach where you pre-fetch the prices of 5-10% of the most common books, so you don't have to fetch them when a user wants to see the prices. This will alleviate some of the front-end instance time by offloading it to the backend that runs every hour. Also, perhaps not all suppliers change their prices every hour - maybe only some fraction of them do. So, run the backend only once a day for slow changing prices, and once an hour for faster moving prices, and for the 0.001% of the books that need more dynamic pricing, run a cron job every 2 minutes on a front-end, assuming it will get done in just a handful of seconds, so that the front-end can process user-facing requests at the same time.
I apologize if I don't fully understand your problem set, but I'm trying to throw out as many ideas as possible. On Tue, Sep 13, 2011 at 10:40 PM, ESPR!T <w...@espr.it> wrote: > With the backends I can't display data to use when he ask for them - > he can't wait next hour for getting the prices of books as he wants > them now. So I need to run the backend all the item and process price > requests as soon as possible (and as there is 5+ million books I can't > do a pre-fetch of some books on backend and then shut it down). > > I will try to implement that async url fetch for batch of requests, > that can by actually fast - just some suppliers are really slow and > some of them fast so maybe I could also do two queues (one for slow > response and one for fast) so the user get info from fast responsive > suppliers quickly and doesnt need to wait) - the only issue is that > technically I should process the pull queue from backend so that means > I would need it to run all the time ;/ (I really need to display the > prices as soon as possible and can't wait with processing). > > I will try to lower the latency so GAE can spin up more instance and > will see how it affect the instance time. > > Thank you guys! > > On Sep 14, 1:57 pm, Rishi Arora <rishi.ar...@ship-rack.com> wrote: > > My app is nearly identical to yours - several concurrent URL fetches are > > performed to "gather" content. And when users access my site, that > content > > is nicely formatted for them for display. My solution - part of if has > > already been suggested by Jeff - use async URL fetch. Spinning an > instance > > for 5 to 10 seconds waiting for your supplier's website to return book > > pricing data is a waste of resources. So, whenever you need to do a URL > > fetch, consider deferring it by queueing it up in a pull queue. Then > once > > you have enough deferred (10 is a good number), then call URL fetch > > simultaneously for all the 10 requests. Another optimization is to use > by > > the 9 free hours of backend time. I agree you can't have the backend > > running all the time. So, wake up the backend by a cron job that runs > once > > every hour. This will incur a minimum cost of 15 minutes per hour = 6 > > instance hours - which is under the free quota. Each time your backend > > wakes up, it looks up all the URL fetch requests deferred in the last > hour, > > and processes them. My app does exactly this, and it takes me about 45 > > seconds to fetch and process all data for ~300 URL fetches every hour. > > > > If you are attempting to stay within the free quota, absolutely use the > > backend hours in any way you can. it'll be a pity to not use those free > 9 > > instance hours. > > > > > > > > > > > > > > > > On Tue, Sep 13, 2011 at 8:43 PM, Tim Hoffman <zutes...@gmail.com> wrote: > > > Hi > > > > > You could submit the request via ajax back to your appengine app > > > and it can then do an async requuest on all the urls, . > > > > > In your case you have some of the info already and have to fetch some > of > > > it. > > > So it might be two ajax calls, one to get the list of books, the result > is > > > book prices for stuff you know, plus an indicator of the books that a > > > further request > > > will be required, your front end can then display the details you have, > > > submit another > > > ajax request to appengine to fetch results for the books you currently > have > > > no info on. > > > Which can then async urlfetch the rest of the details. > > > > > This way user gets some info straight away and you get to keep you > requests > > > to a minimum > > > and fill in the results later. > > > > > Just a thought ;-) > > > > > T > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To view this discussion on the web visit > > >https://groups.google.com/d/msg/google-appengine/-/3dA05F9-QDsJ. > > > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
