Re: [appengine-java] Multiple Async get vs one Sync batch get?
The first query, is a key-only merge-join query. Then I fetch the parents of the results (this is the technique suggested by Brett Slatkin at Building Scalable, Complex Apps on App Engine) When I get those parents, I get a key field from them and fetch it from the datastore. (I am making some changes to avoid this step) The last two queries have nothing to do with the above, so I can easily fetch them asynchronically. One of those is a count() of the first query. @Max: I really tried to optimize using multiple async gets, by the batch is way much faster. On Fri, Dec 17, 2010 at 9:30 PM, Max Ross (Google) < maxr+appeng...@google.com > wrote: > Interesting results! I've run similar tests with writes (put and delete) > and my results indicate that in most situations, multiple async writes will > perform better than one batch sync call. I can't think of any reason this > wouldn't be the case for reads as well. Are your entities all in different > entity groups? > > Thanks, > Max > > > On Fri, Dec 17, 2010 at 4:11 PM, Ikai Lan (Google) < > ikai.l+gro...@google.com > wrote: > >> Very cool. Thanks for posting! >> >> What kind of processing are you doing after you retrieve the entities? I'm >> surprised it made a difference. >> >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> Blogger: http://googleappengine.blogspot.com >> Reddit: http://www.reddit.com/r/appengine >> Twitter: http://twitter.com/app_engine >> >> >> >> On Fri, Dec 17, 2010 at 2:48 PM, Gal Dolber wrote: >> >>> Even better :) >>> >>> [image: Screen shot 2010-12-17 at 7.48.44 PM.png] >>> >>> >>> On Fri, Dec 17, 2010 at 7:31 PM, Gal Dolber wrote: >>> Here are my results if anyone interested. The idea was splitting a batch into multiple async gets so I could start processing each results while the others were being fetched. With the batch get I need to wait until I get all the results to do the next operations. At the end (on my situation) batching won. * * *Multiple async gets* [image: MultipleAsyncs.png] *Batching* [image: Batch.png] On Thu, Dec 16, 2010 at 4:48 PM, Gal Dolber wrote: > Yes, I am talking about datastore api. > > I made some isolated tests and the batch always wins (no by so much). > > But in my app, when I break the batch into multiple async gets, I can > start processing each async get result (and doing another api calls) > before... and it seems to be perform better. > > I'll keep doing tests and post if find anything interesting. > Thanks! > > > On Thu, Dec 16, 2010 at 2:42 PM, Ikai Lan (Google) < > ikai.l+gro...@google.com > wrote: > >> You're talking about the datastore API? Underneath the hood, >> synchronous gets just make asynchronous calls and block on Future.get() >> anyway. Making a single synchronous batch get would translate to a single >> asynchronous RPC with multiple keys vs. multiple asynchronous get RPCs. I >> suspect the single batch call will perform better from a CPU cost/quota >> perspective, but I'd defer to AppStats benchmarks as the authoritative >> answer. >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> Blogger: http://googleappengine.blogspot.com >> Reddit: http://www.reddit.com/r/appengine >> Twitter: http://twitter.com/app_engine >> >> >> >> On Wed, Dec 15, 2010 at 3:38 AM, Gal Dolber wrote: >> >>> Should I expect the similar performance? or the batch will always be >>> better? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine for Java" group. >>> To post to this group, send email to >>> google-appengine-j...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > > > -- > Guit: Elegant, beautiful, modular and *production ready* gwt > applications. > > http://code.google.com/p/guit/ > > > > > -- Guit: Elegant, beautiful, modular and *production ready* gwt applications. http://code.google.com/p/guit/ >>> >>> >>> -- >>> Guit: Elegant, beautiful, mo
Re: [appengine-java] Multiple Async get vs one Sync batch get?
Interesting results! I've run similar tests with writes (put and delete) and my results indicate that in most situations, multiple async writes will perform better than one batch sync call. I can't think of any reason this wouldn't be the case for reads as well. Are your entities all in different entity groups? Thanks, Max On Fri, Dec 17, 2010 at 4:11 PM, Ikai Lan (Google) > wrote: > Very cool. Thanks for posting! > > What kind of processing are you doing after you retrieve the entities? I'm > surprised it made a difference. > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger: http://googleappengine.blogspot.com > Reddit: http://www.reddit.com/r/appengine > Twitter: http://twitter.com/app_engine > > > > On Fri, Dec 17, 2010 at 2:48 PM, Gal Dolber wrote: > >> Even better :) >> >> [image: Screen shot 2010-12-17 at 7.48.44 PM.png] >> >> >> On Fri, Dec 17, 2010 at 7:31 PM, Gal Dolber wrote: >> >>> Here are my results if anyone interested. >>> >>> The idea was splitting a batch into multiple async gets so I could start >>> processing each results while the others were being fetched. >>> With the batch get I need to wait until I get all the results to do the >>> next operations. >>> >>> At the end (on my situation) batching won. >>> * >>> * >>> *Multiple async gets* >>> [image: MultipleAsyncs.png] >>> >>> *Batching* >>> >>> [image: Batch.png] >>> >>> >>> On Thu, Dec 16, 2010 at 4:48 PM, Gal Dolber wrote: >>> Yes, I am talking about datastore api. I made some isolated tests and the batch always wins (no by so much). But in my app, when I break the batch into multiple async gets, I can start processing each async get result (and doing another api calls) before... and it seems to be perform better. I'll keep doing tests and post if find anything interesting. Thanks! On Thu, Dec 16, 2010 at 2:42 PM, Ikai Lan (Google) < ikai.l+gro...@google.com > wrote: > You're talking about the datastore API? Underneath the hood, > synchronous gets just make asynchronous calls and block on Future.get() > anyway. Making a single synchronous batch get would translate to a single > asynchronous RPC with multiple keys vs. multiple asynchronous get RPCs. I > suspect the single batch call will perform better from a CPU cost/quota > perspective, but I'd defer to AppStats benchmarks as the authoritative > answer. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger: http://googleappengine.blogspot.com > Reddit: http://www.reddit.com/r/appengine > Twitter: http://twitter.com/app_engine > > > > On Wed, Dec 15, 2010 at 3:38 AM, Gal Dolber wrote: > >> Should I expect the similar performance? or the batch will always be >> better? >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- Guit: Elegant, beautiful, modular and *production ready* gwt applications. http://code.google.com/p/guit/ >>> >>> >>> -- >>> Guit: Elegant, beautiful, modular and *production ready* gwt >>> applications. >>> >>> http://code.google.com/p/guit/ >>> >>> >>> >>> >>> >> >> >> -- >> Guit: Elegant, beautiful, modular and *production ready* gwt applications. >> >> http://code.google.com/p/guit/ >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com > . > For mor
Re: [appengine-java] Multiple Async get vs one Sync batch get?
Yes, I am talking about datastore api. I made some isolated tests and the batch always wins (no by so much). But in my app, when I break the batch into multiple async gets, I can start processing each async get result (and doing another api calls) before... and it seems to be perform better. I'll keep doing tests and post if find anything interesting. Thanks! On Thu, Dec 16, 2010 at 2:42 PM, Ikai Lan (Google) > wrote: > You're talking about the datastore API? Underneath the hood, synchronous > gets just make asynchronous calls and block on Future.get() anyway. Making a > single synchronous batch get would translate to a single asynchronous RPC > with multiple keys vs. multiple asynchronous get RPCs. I suspect the single > batch call will perform better from a CPU cost/quota perspective, but I'd > defer to AppStats benchmarks as the authoritative answer. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger: http://googleappengine.blogspot.com > Reddit: http://www.reddit.com/r/appengine > Twitter: http://twitter.com/app_engine > > > > On Wed, Dec 15, 2010 at 3:38 AM, Gal Dolber wrote: > >> Should I expect the similar performance? or the batch will always be >> better? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- Guit: Elegant, beautiful, modular and *production ready* gwt applications. http://code.google.com/p/guit/ -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Multiple Async get vs one Sync batch get?
You're talking about the datastore API? Underneath the hood, synchronous gets just make asynchronous calls and block on Future.get() anyway. Making a single synchronous batch get would translate to a single asynchronous RPC with multiple keys vs. multiple asynchronous get RPCs. I suspect the single batch call will perform better from a CPU cost/quota perspective, but I'd defer to AppStats benchmarks as the authoritative answer. -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Wed, Dec 15, 2010 at 3:38 AM, Gal Dolber wrote: > Should I expect the similar performance? or the batch will always be > better? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Multiple Async get vs one Sync batch get?
Should I expect the similar performance? or the batch will always be better? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
