Repeating a couple of things I mentioned to Greg in email, for the list discussion: A single-server SQL database with low load and tables with rows numbering in the thousands can do lots of cool things with that data in a reasonable amount of time. But many of the features of a SQL query engine have difficulty scaling to hundreds of machines, millions of records, and 50+ qps. The App Engine datastore is designed to scale, and inevitably lacks features of single-server SQL databases that don't scale. The datastore is designed to maintain its performance characteristics over very large data sets and heavy traffic, so performance with small data sets and low traffic may not always compare to a single-server SQL database under similar conditions. in the same sense that it won't compare to a RAM cache of the same data. I'll be the first to admit that this means some applications are not well-suited to App Engine. However, I wouldn't say that a table of 300 items of product data is necessarily impractical, even if it isn't as simple as rendering the results of a SQL query. One simple option in this case is to store an entity per item as you're doing now, but use range queries and multiple requests driven by JavaScript to build the page. (Watch how Google Reader renders lists of hundreds of items; you can barely tell the data wasn't all there on the first request.)
Going to the primary database to retrieve 300 items of 40 fields each is asking for 12,000 instantly dynamic fields for a single page. That's a *lot* to ask if the app expects to scale. Part of scaling data is determining how quickly updates need to be available, so you can pre-calculate and cache results to optimize queries, i.e. make data that doesn't need to be instantly dynamic less dynamic. In the case of a product catalog, most of the catalog data can be largely static. You can pre-build and store the product lists, then store dynamic data (like product availability) on separate entities queried at render time. The upcoming cron service will make this kind of pre-building and cache warm-up easier to do, and in some cases you can just update lists and caches when you update individual fields in primary storage. You have the right idea with wanting to cache this information, but you'll probably need to do something other than performing the 12,000-field query during a user request with the intent to cache. Incidentally, returning just keys for a query has been on our radar for a while, we just haven't gotten to it. I can't promise anything, but it's on our list. Feel free to file a feature request in the issue tracker to promote it. -- Dan On Thu, Jan 8, 2009 at 6:15 PM, Tzakie <gwood...@comexton.com> wrote: > > > > Quite frankly, I can't think of a Google web app that displays 100 of > > anything all at once... > > I'm getting the impression that people think what I'm asking for is > ridiculous > and off the radar. I sent you an e-mail with the urls of the current > app and > what I am working on. When you see it in context I think it looks > pretty > reasonable. > > E commerce apps particularly need long lists for a lot of things. On > the > e-commerce apps I do paging kills customers. They don't "next" they > just leave. > > Can't you guys make something that just returns the keys from a query? > That seems consistent with how I think big table works. > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
