[google-appengine] Re: Large datastore queries much slower with Python than Java?
Darshan, Your API times being the same for both implementations suggests that you are indeed doing the same operations in the data store. I interpret the timings "datastore_v3.RunQuery real=122ms api=9179ms" and "datastore_v3.RunQuery real=377ms api=9179ms" as showing real= and api=, which also explains that the latter can be much more than the former: when using data access, I assume that the CPU time may be spent in several server nodes (the data store is distributed). In the logs accessible from the dashboard I usually see also the total CPU time (GAE API + user), which allows to calculate the CPU time spent in my own code. In addition to that I have my own timer that starts on entering my request handling code and ends on exit (allows to separate my own elapse time from the total elapse time per request), plus some counters for the datastore api calls. Cheers, R. On 12 Okt., 00:26, Darshan Shaligram wrote: > On Mon, Oct 11, 2010 at 5:18 PM, Eli Jones wrote: > > Well.. for one.. you are doing a datastore.query() instead of a db.query() > > Most all documentation on working with the datastore indicates to use db > > from google.appengine.ext instead of datastore from google.appengine.api. > > Maybe there is a difference in how they perform in this context? > > The performance of the high-level db api is pretty similar to the > performance of the low-level datastore API. I used the low-level api > so that I could keep the code reasonably similar for both Python and > Java. I'm much less familiar with the Java datastore API and I didn't > want to use Java layers that might muddy the waters performance-wise. > Having used the low-level api for Java, I used the closest > corresponding apis for Python. > > The reason I wrote these simple Python and Java projects was to > investigate datastore query performance issues I had in real Python > code (using google.appengine.ext.db), and comparing notes with a > colleague who was familiar with Java datastore performance. > > > Also, are you doing these tests on Appengine or in the Dev_appserver? > > These tests are on the appengine, not the dev server. I use the same > application name for both Java and Python versions, and different > versions for both (Java = v1, Python = v2) so that they share the same > datastore. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Large datastore queries much slower with Python than Java?
Hi I know this is a nit, but with the python version, is there any particular reason why you are doing objects = list(q.Get(fetch_size)) nfetched += len(objects) at line 108 in test_load_query.py the list() is redundant, the result from Get is for all purposes is a list or at least exceedingly list like that you wouldn't bother creating another list from it. objects = g.Get(fetch_size) nfetched += len(objects) It really won't make any difference to the performance. Rgds T On Oct 12, 6:26 am, Darshan Shaligram wrote: > On Mon, Oct 11, 2010 at 5:18 PM, Eli Jones wrote: > > Well.. for one.. you are doing a datastore.query() instead of a db.query() > > Most all documentation on working with the datastore indicates to use db > > from google.appengine.ext instead of datastore from google.appengine.api. > > Maybe there is a difference in how they perform in this context? > > The performance of the high-level db api is pretty similar to the > performance of the low-level datastore API. I used the low-level api > so that I could keep the code reasonably similar for both Python and > Java. I'm much less familiar with the Java datastore API and I didn't > want to use Java layers that might muddy the waters performance-wise. > Having used the low-level api for Java, I used the closest > corresponding apis for Python. > > The reason I wrote these simple Python and Java projects was to > investigate datastore query performance issues I had in real Python > code (using google.appengine.ext.db), and comparing notes with a > colleague who was familiar with Java datastore performance. > > > Also, are you doing these tests on Appengine or in the Dev_appserver? > > These tests are on the appengine, not the dev server. I use the same > application name for both Java and Python versions, and different > versions for both (Java = v1, Python = v2) so that they share the same > datastore. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Large datastore queries much slower with Python than Java?
On Tue, Oct 12, 2010 at 7:43 AM, Tim Hoffman wrote: > I know this is a nit, but with the python version, is there any > particular reason why you are doing > objects = list(q.Get(fetch_size)) > nfetched += len(objects) > at line 108 in test_load_query.py > the list() is redundant, the result from Get is for all purposes is a > list or at least exceedingly list like that you wouldn't bother > creating another list from it. You're right, that was unnecessary, removed. [...] > It really won't make any difference to the performance. Yes, confirmed. :-) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Large datastore queries much slower with Python than Java?
people say doing Batch Get and Puts does improve the performance batch get = db.get(list of model isntances) batch put = db.put(list of model instances) hope this helps -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
