Yes, you're encountering 'exploding indexes'. The number of index entries needed for a 3-term index is, for example, n*(n-1)*(n-2).
My suggestion would be this: - Keep indexes for one and two uses of the term list. - Keep a separate entity type that tracks how many occurrences there are of each term in the database. - To search, look up the terms, find the two least common ones, and query on those. Then filter the remaining results for any other terms. On Dec 7, 9:37 pm, Koen Bok <[EMAIL PROTECTED]> wrote: > First off, I _really_ like the idea of AppEngine. We're hosting most > of our sites (versionsapp.com, checkoutapp.com) on it and it works > great for that purpose. Impressed as we were, I started trying to > build our first serious webapp on AppEngine; a small webstore. What I > basically wanted to do is building a page where you could browse 50 > products at a time, optionally limited by a search or tag, based on > Django. All went well until I wanted to implement search... > > I grabbed a few example datasets from users of our Mac POS app, they > mostly have around a 1000 products with a name, brand and description. > I knew substring search was hard to implement, so I skipped that and > used the undocumented ext.search models with my own set of search > words, mostly around 40 per product. Because some searches took quite > long I decided to slice the results by 50 at a time, not impressive > but good enough. And because product descriptions don't change often I > could use memcache to cache each result 'slice'. > > To do so, I figured I needed to at least order by name, so the slicing > would make sense. Therefore I had to create an index on name, > searchIndex (set of searchable words). Worked great, until I tried to > type two words. I told me I had to create another index: name, > searchIndex, searchIndex. Seemed logical, but then people may want to > type more then 2 words at the same time so I added 5 indexes to get up > to five words. > > From that moment inserting products started to take more then a > second, which I guess was due to the indexing, but even worse, I > started to get errors like BadRequestError: Too many indexed > properties for entity: app. I found out the max indexes for an entity > is capped at 5000, and calculated that if I were to use max 3 words > for a search I could only add 17 searchable words in an items' index > (17*17*17=4913). I'm still not sure that math is 100% correct, but at > least I was able to insert most of my products again, but searching is > just horribly slow. > > So now I'm stuck with a _lot_ of limitations for a simple app: only 3 > search terms, no substring searching, 50 results at a time and in my > opinion pretty bad performance, especially when inserting. Maybe I'm > doing something horribly wrong, and I don't consider myself the best > db programmer out there but this just seems silly. > > Again, this is not to bash AppEngine because I really see the > potential and like the idea. But as it stands now it's just unusable > for us (a small indie dev company that wants their products to scale), > and it's hard for me to imagine that anyone else can live with these > limitations. I guess what I'm saying is: I love this thing to work for > us! We want to use it and pay for it! Seriously, please fix the > searching... you're Google for god sake :-) > > Kindest regards, > > Koen Bok - madebysofa.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
