On Fri, Sep 24, 2010 at 5:53 PM, Ice13ill <andrei.fifi...@gmail.com> wrote: > Well, that is what i'm using :) > FTS with self merged join (if that is what you're refering to) > > The problem here is with the build-in indexes (it should be able to > serve those queries that matches a text on a singe list property, for > a large number of entities right ?). As i understand, the FTS in GAE > is based on the ability of matching a list of keywords against the > list field of the entities you want to search (equality filter) right?
Hi Andrei, I am the author of that FTS blogpost mentioned before [1]. What you do can be done on GAE / FTS. The main obstacle you are currently facing is the exploding indexes problem. The good thing is: Using self merge joins you do not need explizit indexes. Everything you need is calculated internally via an algo called "zig zag". First of all I would recommend to watch Brett Slatkin's talk again. I watched Brett's talk also more than only one time to get the concepts. Stop thinking in explizit indexes and orderings - start thinking in Venn Diagramms where you specify the piece of the cake you want precisely without ordering. If you fully understood Brett's talk make also sure to design your search "bottom up". That means: Make sure you do not get back more than approx. 200 elements in one FTS search. That also means that you might include a timestamp or something to search only certain elements from today or any special day and such. No. No explicit index here. You simply save another String in your listproperties where you specify a timestamp such as "2010-09-29" what means today. At the end you can query all elements from today using that timestamp. No ordering and no index needed. Hope that helps :) Cheers, Raphael [1] http://googleappengine.blogspot.com/2010/04/making-your-app-searchable-using-self.html > > On Sep 24, 6:29 pm, Cyrille Vincey <crll...@gmail.com> wrote: >> I am afraid you will have to change your data model. >> By the way, I remember that someone shared so times ago his code for a >> full text search implementation in GAE. Maybe you should check this out. >> >> On 24/09/10 17:01, "Ice13ill" <andrei.fifi...@gmail.com> wrote: >> >> >Hmm... i understand the problem. But, if i want to query using a text >> >with multiple words and NO sorting order, i use the build-in index >> >supplied default right ? >> >So, what do i do for 90.000 entities ?? because it seams that the >> >build-in index is not powerful enough ! >> >> >"The built-in indices are not efficient enough for this query and your >> >data. Please add a composite index for this query.. An index is >> >missing but we are unable to tell you which one due to a bug in the >> >App Engine SDK. If your query only contains equality filters you most >> >likely need a composite index on all the properties referenced in >> >those filters." >> >> >So the solution is creating a index like this: >> >> >keyWordsList ▲ , keyWordsList ▲ , keyWordsList ▲ >> >> >Isn't this really the "build-in" index ? >> >> >On Sep 24, 5:16 pm, Cyrille Vincey <crll...@gmail.com> wrote: >> >> You're facing the exploding indexes problem. >> >> You should never put more than 1 list properties in one composite index. >> >> >> This problem is described in the GAE >> >>doc.http://code.google.com/intl/fr-FR/appengine/docs/python/datastore/que >> >>... >> >> dindexes.html#Big_Entities_and_Exploding_Indexes >> >> >> On 24/09/10 15:54, "Ice13ill" <andrei.fifi...@gmail.com> wrote: >> >> >> >I'm trying to implement a search based on a list of keywords (string). >> >> >The list is created based on the content of the objects' fields and i >> >> >use it when matching against a query text. >> >> >The problem is that i can't create indexes on App engine (i get Error >> >> >after about 30 mins, and i have to delete it). There are about 90,000 >> >> >entities of that kind and the index should looke like this: >> >> >> >keyWordsList ▲ , keyWordsList ▲ , keyWordsList ▲ , key ▼ >> >> >> >and i have about four of them. >> >> >> >My question: do i want to much ? :) I tried uploading about 4 indexes >> >> >and i got Error. I tried uploading 1 and worked (but needed about 2-3 >> >> >h). Is there a problem if i try to create 4 indexes of this magnitude >> >> >(and why)? Can i be sure that uploading one by one will work? Or maybe >> >> >this problem is related to AppEngine datastore problems ? >> >> >Please advice ! >> >> >> >-- >> >> >You received this message because you are subscribed to the Google >> >>Groups >> >> >"Google App Engine for Java" group. >> >> >To post to this group, send email to >> >> >google-appengine-j...@googlegroups.com. >> >> >To unsubscribe from this group, send email to >> >> >google-appengine-java+unsubscr...@googlegroups.com. >> >> >For more options, visit this group at >> >> >http://groups.google.com/group/google-appengine-java?hl=en. >> >> >-- >> >You received this message because you are subscribed to the Google Groups >> >"Google App Engine for Java" group. >> >To post to this group, send email to >> >google-appengine-j...@googlegroups.com. >> >To unsubscribe from this group, send email to >> >google-appengine-java+unsubscr...@googlegroups.com. >> >For more options, visit this group at >> >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
