Nick is saying that the StringListProperty has values like: A, B, C. If you also have another property, say an IntegerProperty with value 7, and you create a compound index that contains the IntegerProperty and the StringListProperty you will be OK because the index will look like: 7-A 7-B 7-C
However, if the index contains the IntegerProperty and the StringListProperty twice the index looks something like: 7-A-A 7-A-B 7-A-C 7-B-B 7-B-C 7-C-C The issue is NOT from one multiple valued (ie ListProperty) being in a compound index once, but rather from one or more multiple valued properties appearing more than one time in the same compound index. Robert On Tue, Oct 12, 2010 at 12:00, Jeff Schwartz <jefftschwa...@gmail.com> wrote: > So a list property with 5000 entries in it that is not part of any compound > index then would not in itself create an exploding index situation, but > would if it were part of a compound index, correct? Nor would querying on a > list property with 5000 entries not part of any compound index using an > equality filer, correct? > > The reason I am asking all these questions is because I've implemented a > fanout solution based on Brett Slatkin's youtube video - > http://www.youtube.com/watch?v=AgaL6NGpkB8 - and was concerned that the > principle's he promoted in the video using entitie with list properties as > indexes to their parents aren't now problematic or considered bad practice > or that changes have been made to the datastore where employing his > technique would not create exploding indexes. > > Thanks in advance and for bearing with my many questions. > > Jeff > > > On Tue, Oct 12, 2010 at 11:41 AM, Nick Johnson (Google) > <nick.john...@google.com> wrote: >> >> Hi Jeff, >> >> On Tue, Oct 12, 2010 at 4:32 PM, Jeff Schwartz <jefftschwa...@gmail.com> >> wrote: >>> >>> Hi Nick, >>> >>> From the docs at >>> http://code.google.com/appengine/docs/java/datastore/queriesandindexes.html#Big_Entities_and_Exploding_Indexes: >>> >>> "You could call an index that refers to multiple properties with multiple >>> values an "exploding index," because it can get very large with just a few >>> values." >>> >>> I believe the key words in the above are "multiple properties with >>> multiple values" however I cannot see anywhere where there is a compound >>> index using multiple properties that include the list property which >>> according to the above is what would/could create an exploding index. I >>> should preface this by saying that I am not a python developer so if there >>> is something in the yaml file that does create this situation I apologize >>> but would you mind pointing out what in the yaml file is? >> >> The same (list) property multiple times has the same effect as using >> multiple different properties. When you include a list in a custom index, >> all the values of the list must be indexed; when you include multiple lists, >> or multiple instances of the same list, all unique combinations of values >> must be indexed. >> -Nick Johnson >>> >>> Thanks in advance. >>> >>> >>> On Tue, Oct 12, 2010 at 11:22 AM, Nick Johnson (Google) >>> <nick.john...@google.com> wrote: >>>> >>>> Hi Jeff, >>>> X is a StringListProperty, and it's indexed on multiple times. The >>>> number of index entries generated for multiple instances of the same list >>>> property is n!/(n-m)!, where n is the number of elements in the list, and m >>>> is the number of repeats in the index. >>>> -Nick Johnson >>>> >>>> On Tue, Oct 12, 2010 at 4:18 PM, Jeff Schwartz <jefftschwa...@gmail.com> >>>> wrote: >>>>> >>>>> Hi Nick. It doesn't appear from the orignal poster's example that even >>>>> if he had 5000 entries (the max) in his list property that that would >>>>> cause >>>>> an exploding index. Can you elaborate. >>>>> >>>>> >>>>> On Tue, Oct 12, 2010 at 10:59 AM, Nick Johnson (Google) >>>>> <nick.john...@google.com> wrote: >>>>>> >>>>>> Hi Eric, >>>>>> This is a classic 'exploding' index. With 10 elements in the list, >>>>>> this index will try to generate 10*9*8*7*6 = 30,240 index entries - far >>>>>> above the limit. It's not possible to build this index. Consider >>>>>> restructuring your data, or using merge join queries. >>>>>> -Nick Johnson >>>>>> >>>>>> On Tue, Oct 12, 2010 at 3:37 PM, Eric Ka Ka Ng <ngk...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> I would like to build an index for the following entity >>>>>>> Class entityA (db.Model): >>>>>>> X db.StringListProperty() >>>>>>> Y db.IntegerProperty() >>>>>>> Z db.IntegerProperty() >>>>>>> >>>>>>> >>>>>>> in index.yaml >>>>>>> >>>>>>> - kind: entityA >>>>>>> >>>>>>> properties: >>>>>>> >>>>>>> - name: Y >>>>>>> >>>>>>> - name: X >>>>>>> >>>>>>> - name: X >>>>>>> >>>>>>> - name: X >>>>>>> >>>>>>> - name: X >>>>>>> >>>>>>> - name: X >>>>>>> >>>>>>> - name: Z >>>>>>> >>>>>>> direction: desc >>>>>>> >>>>>>> but the index cannot be built and is always resulted in Error State >>>>>>> >>>>>>> Thx now we can vaccum the 'error' index now without being stuck in >>>>>>> the "building" state and have to send emails to G to ask to mark the >>>>>>> index >>>>>>> as 'Error' >>>>>>> >>>>>>> But I have tried several times to vaccum the index, then re-build, >>>>>>> but always fail >>>>>>> >>>>>>> May i know why it keeps in failing? The StringList property X usually >>>>>>> contains 5-10 strings. >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google App Engine" group. >>>>>>> To post to this group, send email to >>>>>>> google-appeng...@googlegroups.com. >>>>>>> To unsubscribe from this group, send email to >>>>>>> google-appengine+unsubscr...@googlegroups.com. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland >>>>>> Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 >>>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>>>> Number: 368047 >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> google-appeng...@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> google-appengine+unsubscr...@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> >>>>> >>>>> -- >>>>> Jeff >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To post to this group, send email to google-appeng...@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> google-appengine+unsubscr...@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>>> >>>> -- >>>> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland >>>> Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 >>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>> Number: 368047 >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to google-appeng...@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine+unsubscr...@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >>> >>> -- >>> Jeff >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. >> :: Registered in Dublin, Ireland, Registration Number: 368047 >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: >> 368047 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > > > -- > Jeff > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
