The datastore is not fast. And why do you think it would be? Because you can fetch thousands of items out of RAM on your laptop in a fraction of a second?
GAE is a key/value store distributed on a gigantic cluster of probably thousands of machines, each of which is busily working on not just your load/store problem but thousands of other people's. Your laptop has the dataset cached in RAM, and even if it didn't, your data is probably stored sequentially on a single spindle. GAE has to fetch your entities from up to 500 separate machines in the cluster, very likely off of disk. The problems you find at scale will not show up when you query mysql on your laptop. GAE is already operating "at scale" so it performs in the slow, but predictable-on-average way that gigantic computing architectures do. Treat the datastore like a key/value store that likes really big entities. There are things you can do to make fetching 500 keys at a time a little faster (example: fetch in eventual consistency mode), but performance is going to suck compared to what you are used to on other platforms... unless you've actually run those other platforms at scale. The best thing to do is adjust your architecture so you're fetching 1 fat entity instead of 500 little ones. It's not always possible. Jeff On Thu, Feb 16, 2012 at 9:03 AM, Andreas <a.schmi...@gmail.com> wrote: > really? we are not speaking about fetching a million entities. > i would expect to fetch 500 entities within a second if not a lot less. > but obviously this is not the case. > > On Feb 16, 2012, at 1:52 AM, Jeff Schnitzer wrote: > > 3s for fetching 500 doesn't seem wildly out of sorts. > > You can speed it up a lot (at the cost of consistency) by doing an > eventually consistent get instead of a strongly consistent get. I don't > know the way to do that in Python. > > Jeff > > On Tue, Feb 14, 2012 at 4:05 PM, Andreas <a.schmi...@gmail.com> wrote: > >> im just profiling my app and i have to say i get surprising results. >> >> lets say i have a list with 500 keys as strings not key objects. >> db.get(keylist) takes 3seconds??!!! >> >> how does this take that long? i mean i have the keys already. >> would it be (significantly) faster with key objects? >> >> andreas >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
