I'd say the average number of line items is around 100. So, the slowest request (but also the most common) is to show the 5 most recent items from all recent user orders. In JDO, I was doing that via a query for the orders, then a order.getItems() for each, then iterating in reverse for a few orders. This turned out to be super slow past a certain average number of items per order. What I'm doing currently is memcaching the order with items collection manually, then using a keys only query for orders to load from cache where possible. This has gotten the request time down to about 1/6th of where it was previously, which means I still have users whose request takes around 5 seconds on average.
I think if I were to do it all over again I'd either embed the items right in the order, or not use JDO at all and try to do some more manual caching and batch loading. A lot of time was spent learning how JDO was loading the items collection and then trying to get JDO to cache that collection. It might be possible, but I couldn't figure it out. Thanks for the advice and links, they're very helpful. I'll definitely read up on Objectify and Twig for future projects. On Tue, Apr 6, 2010 at 6:28 PM, Jeff Schnitzer <j...@infohazard.org> wrote: > The most important number in my mind is the number of line items in an > order. If you're seeing 20s+ queries, it must be a pretty large > number. > > Objectify (and Twig) support queryable collections of embedded > objects, so you can put all the line items in a single order object. > There are some practical limits to this approach, though - the > collection can have at most 5k items and the total entity size cannot > exceed 1MB. Also, writing out indexed collections with 5,000 entries > is *very* expensive in terms of $, so if you have a high churn rate > you'll burn through your budget pretty fast. > > Obviously serialization is an option, but then you won't be able to > query for orders by line item data. > > If it makes the most sense to keep the line items as separate objects > (because of churn or simple volume of line items) you can still use > the memcache to your advantage. You'll want a bidirectional > relationship, with Keys stored in the Order pointing to the LineItems > as well as vice-versa. You can load an Order and use the LineItem > keys to perform a batch get, hopefully most of which will come out of > the memcache. > > It's hard to suggest a strategy without knowing the shape of the data, > the query expected profile, and the churn rate. > > Jeff > > On Tue, Apr 6, 2010 at 2:23 PM, Steve Pritchard <steve...@gmail.com> > wrote: > > Before proposing a solution, what is your expected active user base. > > 1000's or 1,000,000s or something in between. > > Steve > > > > On Apr 6, 3:50 pm, Matt Hall <matt.h...@gmail.com> wrote: > >> Thanks for the response, could you point me to somewhere where I can > >> read about batched queries by key in JDO? Or do I need to go to the > >> low level API for that? > >> > >> I think the conclusion I'm coming to here is 1) I shouldn't have made > >> the detail for each order as a separate object, (although this has > >> huge problems for querying later as in how many people have bought > >> item x), and 2) JDO managed relationships get slow with small numbers > >> of child objects, are hard to optimize, and probably not worth the > >> trouble overall. Is that pretty much what I should be concluding? > >> > >> Thanks again, I'm really trying to understand how to translate what's > >> a basic structure in RDMS land into something that will work in app > >> engine beyond toy implementations. > >> > >> On Apr 6, 12:54 pm, "Ikai L (Google)" <ika...@google.com> wrote: > >> > >> > For speed, you'll get the most gains if you denormalize when possible > - yes, > >> > the relational purists are going to riot, but you don't have many of > the > >> > benefits of normalization anyway such as foreign key constraints or > native > >> > joins. > >> > >> > If that isn't an option - and I really do recommend looking into it > where it > >> > makes sense - if you're able to retrieve keys, we may also be able to > make > >> > use of batch queries by key. > >> > >> > On Mon, Apr 5, 2010 at 2:59 PM, Matt Hall <matt.h...@gmail.com> > wrote: > >> > > Hi Everyone, > >> > >> > > I've been trying for the past while to get our relatively simple JDO > >> > > based site to run at a reasonable speed, but I'm failing and I'd > love > >> > > any suggestions. I'll describe it as succinctly as possible to > >> > > hopefully keep your interest :) > >> > >> > > * The basic structure is a User -> Order -> Line items type > >> > > structure. > >> > > * The main query performed is "Orders for a user." > >> > > * A user usually has around 10 orders, each order has under 100 > line > >> > > items. > >> > > * The Order -> Items relationship is done as a One-to-many owned > >> > > relationship. > >> > >> > > My problem is that the basic request the site handles: "Get all > orders > >> > > for a user and do something based on the line items" is very slow. > In > >> > > real response time, my average is now 1.5 seconds, and some of the > >> > > users are seeing 20 second response times. Clearly not ok. > >> > >> > > What appears to be slow, after profiling with the new app stats tool > >> > > is that each line item is getting loaded as it's used (lazy > loading), > >> > > which works great in some cases but terrible in this case since each > >> > > item takes 20-30ms to load. > >> > >> > > I have tried: > >> > >> > > * Moving items into the default fetch group for orders, seems to > have > >> > > no effect (may not even be possible due to limitations of joins) > >> > > * Level 2 caching with memcache - not much effect either since only > >> > > the individual line items are cached, not the whole List, so we > still > >> > > pay a price in round trip to the cache on each line item. > >> > > * A sort of wacky cache that I wrote on top of the JDO managed > >> > > relationship to cache the whole collection. This went poorly and > >> > > resulted in odd data inconsistencies, probably because I did > something > >> > > wrong but the whole thing felt wrong. > >> > >> > > So my question is - what am I doing wrong? What's the best/standard > >> > > way to represent a Master -> Detail type relationship, where the > total > >> > > items for a user will be relatively small but requests need to work > >> > > with a fair number of them. > >> > >> > > Thanks for any suggestions, I'd be happy to answer any questions if > >> > > what I've written isn't clear. > >> > >> > > Matt > >> > >> > > -- > >> > > You received this message because you are subscribed to the Google > Groups > >> > > "Google App Engine for Java" group. > >> > > To post to this group, send email to > >> > > google-appengine-j...@googlegroups.com. > >> > > To unsubscribe from this group, send email to > >> > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com><google-appengine-java%2B > unsubscr...@googlegroups.com> > >> > > . > >> > > For more options, visit this group at > >> > >http://groups.google.com/group/google-appengine-java?hl=en. > >> > >> > -- > >> > Ikai Lan > >> > Developer Programs Engineer, Google App Enginehttp:// > googleappengine.blogspot.com|http://twitter.com/app_engine > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > > To post to this group, send email to > google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
