I think this is actually an interesting question, and brings up a discussion worth having:
Is datastore performance reasonable? I don't want to make this a discussion of reliability, which is a separate issue. It just seems to me that the datastore is actually kinda pokey, taking seconds to write a few hundred entities. When people benchmark Tokyo Tyrant, I hear numbers thrown around like 22,000 writes/second sustained across 1M records: http://blog.hunch.se/2009/02/28-tokyo-cabinet You might argue that the theoretical scalability of BigTable's distributed store is higher... but we're talking about two full orders of magnitude difference. Will I ever near the 100-google-server equivalent load? Could I pay for it if I did? 100 CPUs (measured) running for 1 month is about $7,200. Actual CPU speed is at least twice the measured rate, so a single Tokyo Tyrant is theoretically equivalent to almost $15,000/month of appengine hosting. Ouch. Maybe this isn't an apples to apples comparison. Sure, there aren't extra indexes on those Tyrant entities... but to be honest, few of my entities have extra indexes. What other factors could change this analysis? Thoughts? BTW Tim, you may very well have quite a few indexes on your entities. In JDO, nearly all single fields are indexed by default. You must explicitly add an annotation to your fields to make them unindexed. With Objectify, you can declare your entity as @Indexed or @Unindexed and then use the same annotation on individual fields to override the default. Jeff On Wed, Feb 24, 2010 at 12:43 AM, Tim Cooper <tco...@gmail.com> wrote: > I have been trying to write 12,000 objects in a single page request. > These objects are all very small and the total amount of memory is not > large. There is no index on these objects - the only GQL queries I > make on them are based on the primary key. > > Ikai has said: "That is - if you have to delete or create 150 > persistent, indexed objects, you may want to rethink what problems you > are trying to solve." > > So I have been thinking about the problems I'm trying to solve, > including looking at the BuddyPoke blog and reading the GAE > documentation. I'm trying to populate the database with entries > relating to high school timetables. > > * I could do the writes asynchronously, but that looks like a lot of > additional effort. On my C++ app, writing the same information to my > laptop drive, this happens in under a second, because the amount of > data is actually quite small, but it times out on GAE. > * I am using pm.makePersistentAll(), but this doesn't help. > * There is no index on the objects - I access them only through the > primary key. (I'm pretty sure there's no index - but how can I > confirm this via the development server dashboard?) > * The objects constitute 12,000 entity groups. I could merge them > into fewer entity groups, but there's no natural groupings I could > use, so it could get quite complex to introduce a contrived grouping, > and also this would complicate the multi-user updating of the objects. > The AppEngine team seem to generally recommend using more entity > groups, but it's difficult to integrate that advice with the contrary > advice to use fewer entity groups for acceptable performance. > * I'd be happy if the GAE database was < 10 times slower than a > non-cloud RDBMS, but the way I'm using it, it's currently not. > > Does anyone have any advice? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
