By the way, I don't know that you can't improve the performance of your fetches - for example, serialization costs on tend to represent a much higher percentage of the fetch time in pythonland than javaland. If you're using python, you might be able to improve this by optimizing your data model. I don't know; you would need to profile carefully using appstats.
Jeff On Fri, Feb 17, 2012 at 11:55 AM, Andreas <a.schmi...@gmail.com> wrote: > i understand the scale of the datastore and of course it wont perform like > my little db on my machine with a few entities in there and im actually not > comparing my local environment with the GAE production environment. > > i was really expecting a lot more speed for a key query but i guess im > wrong. > still 3 seconds for 500 entities seems a little too much but this is > probably only how i was expecting it to perform. > > On Feb 16, 2012, at 7:21 PM, Jeff Schnitzer wrote: > > The datastore is not fast. And why do you think it would be? Because you > can fetch thousands of items out of RAM on your laptop in a fraction of a > second? > > GAE is a key/value store distributed on a gigantic cluster of probably > thousands of machines, each of which is busily working on not just your > load/store problem but thousands of other people's. Your laptop has the > dataset cached in RAM, and even if it didn't, your data is probably stored > sequentially on a single spindle. GAE has to fetch your entities from up > to 500 separate machines in the cluster, very likely off of disk. > > The problems you find at scale will not show up when you query mysql on > your laptop. GAE is already operating "at scale" so it performs in the > slow, but predictable-on-average way that gigantic computing architectures > do. > > Treat the datastore like a key/value store that likes really big entities. > There are things you can do to make fetching 500 keys at a time a little > faster (example: fetch in eventual consistency mode), but performance is > going to suck compared to what you are used to on other platforms... unless > you've actually run those other platforms at scale. The best thing to do > is adjust your architecture so you're fetching 1 fat entity instead of 500 > little ones. It's not always possible. > > Jeff > > On Thu, Feb 16, 2012 at 9:03 AM, Andreas <a.schmi...@gmail.com> wrote: > >> really? we are not speaking about fetching a million entities. >> i would expect to fetch 500 entities within a second if not a lot less. >> but obviously this is not the case. >> >> On Feb 16, 2012, at 1:52 AM, Jeff Schnitzer wrote: >> >> 3s for fetching 500 doesn't seem wildly out of sorts. >> >> You can speed it up a lot (at the cost of consistency) by doing an >> eventually consistent get instead of a strongly consistent get. I don't >> know the way to do that in Python. >> >> Jeff >> >> On Tue, Feb 14, 2012 at 4:05 PM, Andreas <a.schmi...@gmail.com> wrote: >> >>> im just profiling my app and i have to say i get surprising results. >>> >>> lets say i have a list with 500 keys as strings not key objects. >>> db.get(keylist) takes 3seconds??!!! >>> >>> how does this take that long? i mean i have the keys already. >>> would it be (significantly) faster with key objects? >>> >>> andreas >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
