So a list property with 5000 entries in it that is not part of any compound index then would not in itself create an exploding index situation, but would if it were part of a compound index, correct? Nor would querying on a list property with 5000 entries not part of any compound index using an equality filer, correct?
The reason I am asking all these questions is because I've implemented a fanout solution based on Brett Slatkin's youtube video - http://www.youtube.com/watch?v=AgaL6NGpkB8 - and was concerned that the principle's he promoted in the video using entitie with list properties as indexes to their parents aren't now problematic or considered bad practice or that changes have been made to the datastore where employing his technique would not create exploding indexes. Thanks in advance and for bearing with my many questions. Jeff On Tue, Oct 12, 2010 at 11:41 AM, Nick Johnson (Google) < nick.john...@google.com> wrote: > Hi Jeff, > > On Tue, Oct 12, 2010 at 4:32 PM, Jeff Schwartz <jefftschwa...@gmail.com>wrote: > >> Hi Nick, >> >> From the docs at >> http://code.google.com/appengine/docs/java/datastore/queriesandindexes.html#Big_Entities_and_Exploding_Indexes >> : >> >> "You could call an index that refers to multiple properties with multiple >> values an "exploding index," because it can get very large with just a few >> values." >> >> I believe the key words in the above are "multiple properties with >> multiple values" however I cannot see anywhere where there is a compound >> index using multiple properties that include the list property which >> according to the above is what would/could create an exploding index. I >> should preface this by saying that I am not a python developer so if there >> is something in the yaml file that does create this situation I apologize >> but would you mind pointing out what in the yaml file is? >> > > The same (list) property multiple times has the same effect as using > multiple different properties. When you include a list in a custom index, > all the values of the list must be indexed; when you include multiple lists, > or multiple instances of the same list, all unique combinations of values > must be indexed. > > -Nick Johnson > >> >> Thanks in advance. >> >> >> >> On Tue, Oct 12, 2010 at 11:22 AM, Nick Johnson (Google) < >> nick.john...@google.com> wrote: >> >>> Hi Jeff, >>> >>> X is a StringListProperty, and it's indexed on multiple times. The number >>> of index entries generated for multiple instances of the same list property >>> is n!/(n-m)!, where n is the number of elements in the list, and m is the >>> number of repeats in the index. >>> >>> -Nick Johnson >>> >>> >>> On Tue, Oct 12, 2010 at 4:18 PM, Jeff Schwartz >>> <jefftschwa...@gmail.com>wrote: >>> >>>> Hi Nick. It doesn't appear from the orignal poster's example that even >>>> if he had 5000 entries (the max) in his list property that that would cause >>>> an exploding index. Can you elaborate. >>>> >>>> >>>> >>>> On Tue, Oct 12, 2010 at 10:59 AM, Nick Johnson (Google) < >>>> nick.john...@google.com> wrote: >>>> >>>>> Hi Eric, >>>>> >>>>> This is a classic 'exploding' index. With 10 elements in the list, this >>>>> index will try to generate 10*9*8*7*6 = 30,240 index entries - far above >>>>> the >>>>> limit. It's not possible to build this index. Consider restructuring your >>>>> data, or using merge join queries. >>>>> >>>>> -Nick Johnson >>>>> >>>>> >>>>> On Tue, Oct 12, 2010 at 3:37 PM, Eric Ka Ka Ng <ngk...@gmail.com>wrote: >>>>> >>>>>> I would like to build an index for the following entity >>>>>> >>>>>> Class entityA (db.Model): >>>>>> X db.StringListProperty() >>>>>> Y db.IntegerProperty() >>>>>> Z db.IntegerProperty() >>>>>> >>>>>> >>>>>> >>>>>> in index.yaml >>>>>> >>>>>> - kind: entityA >>>>>> >>>>>> properties: >>>>>> >>>>>> - name: Y >>>>>> >>>>>> - name: X >>>>>> >>>>>> - name: X >>>>>> >>>>>> - name: X >>>>>> >>>>>> - name: X >>>>>> >>>>>> - name: X >>>>>> >>>>>> - name: Z >>>>>> >>>>>> direction: desc >>>>>> >>>>>> >>>>>> >>>>>> but the index cannot be built and is always resulted in Error State >>>>>> >>>>>> >>>>>> Thx now we can vaccum the 'error' index now without being stuck in the >>>>>> "building" state and have to send emails to G to ask to mark the index as >>>>>> 'Error' >>>>>> >>>>>> >>>>>> But I have tried several times to vaccum the index, then re-build, but >>>>>> always fail >>>>>> >>>>>> >>>>>> >>>>>> May i know why it keeps in failing? The StringList property X usually >>>>>> contains 5-10 strings. >>>>>> >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Eric >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> google-appeng...@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>>>> . >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland >>>>> Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 >>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>>> Number: 368047 >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To post to this group, send email to google-appengine@googlegroups.com >>>>> . >>>>> To unsubscribe from this group, send email to >>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>>> . >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>> >>>> >>>> >>>> -- >>>> Jeff >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to google-appeng...@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> . >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>> >>> >>> >>> -- >>> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. >>> :: Registered in Dublin, Ireland, Registration Number: 368047 >>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>> Number: 368047 >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> >> >> -- >> Jeff >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. > :: Registered in Dublin, Ireland, Registration Number: 368047 > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
