Hi Completely missed the tag (can't say I ever look at them)
Looking at the graph you are doing an awful lot of memcache gets and sets one after another and each rpc costs time (round trip etc) Just looking at the graph it tells me you code is behaving inefficiently. You want as few steps in the graph as possible as each one has a fairly fixed overhead in terms of roundtrip. For instance if there is 30ms base overhead for each round trip excluding any work, and you do 100 rpcs then thats an immediate 3000ms added to you processing time. These means start batching operations, for instance you can do get_multi and set_multi when using memcache. I am having trouble working out what you code is trying to do (hard to read here and incomplete), but basically you need to try and batch all operations. Its not clear why you have so many memcache.set's interleaved with all the gets. Again try and do it all at the end in a few operations. The evidence as to where all you time is going is in the graph. T On Monday, May 5, 2014 6:37:36 PM UTC+8, DiTieM wrote: > > You images are too small to see detail >> > > > <https://lh6.googleusercontent.com/-e3uAabHttKI/U2dpnjlQe_I/AAAAAAAAEmU/aYWYkuuQs3A/s1600/appstats.png> > Apologies, "Google Groups" resized. I did not notice till you mentioned. > The image is 1.7 mg and it was 1400 pixels wide. I cropped it, hope this > time it will be more visible. > > >> You probably need to look at the code that retrieves entities. Look at >> using larger batch sizes. You haven't mentioned what language or >> datastore access mechanism (ie on python, db or ndb) etc. >> > > the tag of the message: python27, I thought people used that for "saying > the language", I guess not easy to read. I am using ndb, as db is > deprecated. > > Removing some if-then-else, the code is like this: > > query = TATripDB.query( TATripDB.draft == 0 ) > > if request.offset: > cursor = Cursor( urlsafe = request.offset ) > else: > cursor = None > > # limit = request.limit > rows, next_cursor, more = query.fetch_page( 200, start_cursor = > cursor ) > > ret = [ ] > com_cache = { } > cur_cache = { } > > if rows: > unique_temporal_user = self.get_ta_profile( rows[ 0 > ].from_agency ) > > for r in rows: > ret.append( get_tatrip_answer( unique_temporal_user > # user_set[ r.key.parent( > ).id( ) ] > , r > , vlc_of_trip_key( > userID_key, r.key ) > , cur_cache > , com_cache ) ) > > return ExploreTATripA( trips = ret > , next_token = next_token > , cur_rates = cur_rates ) > > > The version of the memcache introduce some changes: > > tatrips_keys = memcache.get( 'TATRIPS_KEYS' ) > > if tatrips_keys: > rows = map( lambda x: self.get_tatrip_from_key( x > ), tatrips_keys ) > > I thought of using this method to avoid putting big object in memcache. > This is maybe what you refer as getting in larger batch size? Later I am > going to do some more experiments. Thank you for the suggestion! > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
