i understand the scale of the datastore and of course it wont perform like my little db on my machine with a few entities in there and im actually not comparing my local environment with the GAE production environment.
i was really expecting a lot more speed for a key query but i guess im wrong. still 3 seconds for 500 entities seems a little too much but this is probably only how i was expecting it to perform. On Feb 16, 2012, at 7:21 PM, Jeff Schnitzer wrote: > The datastore is not fast. And why do you think it would be? Because you > can fetch thousands of items out of RAM on your laptop in a fraction of a > second? > > GAE is a key/value store distributed on a gigantic cluster of probably > thousands of machines, each of which is busily working on not just your > load/store problem but thousands of other people's. Your laptop has the > dataset cached in RAM, and even if it didn't, your data is probably stored > sequentially on a single spindle. GAE has to fetch your entities from up to > 500 separate machines in the cluster, very likely off of disk. > > The problems you find at scale will not show up when you query mysql on your > laptop. GAE is already operating "at scale" so it performs in the slow, but > predictable-on-average way that gigantic computing architectures do. > > Treat the datastore like a key/value store that likes really big entities. > There are things you can do to make fetching 500 keys at a time a little > faster (example: fetch in eventual consistency mode), but performance is > going to suck compared to what you are used to on other platforms... unless > you've actually run those other platforms at scale. The best thing to do is > adjust your architecture so you're fetching 1 fat entity instead of 500 > little ones. It's not always possible. > > Jeff > > On Thu, Feb 16, 2012 at 9:03 AM, Andreas <a.schmi...@gmail.com> wrote: > really? we are not speaking about fetching a million entities. > i would expect to fetch 500 entities within a second if not a lot less. > but obviously this is not the case. > > On Feb 16, 2012, at 1:52 AM, Jeff Schnitzer wrote: > >> 3s for fetching 500 doesn't seem wildly out of sorts. >> >> You can speed it up a lot (at the cost of consistency) by doing an >> eventually consistent get instead of a strongly consistent get. I don't >> know the way to do that in Python. >> >> Jeff >> >> On Tue, Feb 14, 2012 at 4:05 PM, Andreas <a.schmi...@gmail.com> wrote: >> im just profiling my app and i have to say i get surprising results. >> >> lets say i have a list with 500 keys as strings not key objects. >> db.get(keylist) takes 3seconds??!!! >> >> how does this take that long? i mean i have the keys already. >> would it be (significantly) faster with key objects? >> >> andreas >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
