> > Generally speaking, If I'm not too mistaken, your approach to my scenario > wouldn't reduce write operations, it would just reduce index number, but in > return require manual indexing via additional models, kind of like moving > around the limitation, not that I'm against it
We were talking about reducing indexes, but that usually comes with more read or write operations. The reads operations can be reduced with memcache or other caching mechanism. What you can do is delete all the indexes you think you don't need and test your application in a dev environmet. If there are any index needed that you don't define, the app is gonna crash with a handy log message saying which index is needed. As a rule, i always expect that error development logs prior to add the index to my appengine-index.xml Doing that i'm totally sure i don't have unused indexes defined. Cheers and good luck with your index crusade! 2013/10/19 Kaan Soral <kaanso...@gmail.com> > In my scenario, there is a query with a sort order and there are optional > filters, each optional filter doubles the index number > > Generally speaking, If I'm not too mistaken, your approach to my scenario > wouldn't reduce write operations, it would just reduce index number, but in > return require manual indexing via additional models, kind of like moving > around the limitation, not that I'm against it > > An approach that keeps the simplicity but reduces the index number would > be great > > I wonder what would happen if I only indexed each filter/property in a > single index with the sort order and remove other combinations > I'm not sure what would happen with a query that misses only one property > from that index for example, does the system use that index, can error > occur with huge datasets etc. > > I'm probably not going to exceed the 200, but I was thinking about > refactoring a property from that index, which would cause a temporary > increase in index numbers before vacuuming, which might cause issues > > Thinking about my case now, it might be a good idea to think about > encoding the combinations and putting them inside a repeated property to be > queried > This wouldn't change the write operations, however would probably reduce > indexes drastically > > Thanks for the advice and thanks for making me think :) > > > On Saturday, October 19, 2013 1:24:32 PM UTC+3, Alejandro González Rodrigo > wrote: > >> I dont know what kind of model do you have and what you can do with it, >> but for example with this entity: >> * >> >> Record* >> @Id Long id >> String user >> Boolean newsletter; //<- indexed >> Boolean notifications; //<- indexed >> String choice; //A,B or C <- indexed >> >> How about splitting the entity this way? >> >> *RecordNewsletter* >> @Id Long id >> String user >> Boolean choiceA; >> Boolean choiceB; >> Boolean choiceC; >> >> *RecordNotifications* >> @Id Long id >> String user >> Boolean choiceA; >> Boolean choiceB; >> Boolean choiceC; >> >> >> As the 200 limit index count is only for composite indexes, with this >> model you have 0 indexes count. Yo have to do more queries to get your >> info, but with a good cache mechanism maybe is worth the datastore calls. >> This may not fit your model or your needs, but is a good example of how to >> reduce indexes drastically. >> >> Cheers! >> >> >> >> 2013/10/19 Kaan Soral <kaan...@gmail.com> >> >> The optimized/documented version of index.yaml is 138 indexes :) >>> >>> Generally the majority are from a single model with 2 boolean, one (3 >>> choiced) string property, generating 2^3x of the initial index number >>> >>> Since the filters are not always present, the devserver generates an >>> index for each situation, I'm not sure if it can be reduced without >>> NeedIndexError's >>> >>> >>> On Saturday, October 19, 2013 12:51:51 PM UTC+3, Alejandro González >>> Rodrigo wrote: >>> >>>> That 200 index limit quota can't be changed (at least to my knowledge). >>>> That limit is there for important reasons. If you reach the limit, maybe >>>> you are doing something wrong, or not in the best way. >>>> >>>> Did you consider using search API for removing some index? Bigquery or >>>> CloudSQL for making reports outside the blobstore and maybe remove some >>>> indexes? Are your data-model enough optimized for using the low number of >>>> indexes possible, using denormalization and ancestors queries to save >>>> indexes? Maybe in your model fits the Namespace aproach for multitency >>>> apps, saving again a lot of indexes? >>>> >>>> There are many tricks and considerations to take care of when modeling >>>> in a non relational database like the blobstore. >>>> >>>> Cheers! >>>> >>>> >>>> 2013/10/19 Kaan Soral <kaan...@gmail.com> >>>> >>>> with 138 indexes, I'm wondering this too :) >>>>> >>>>> >>>>> On Tuesday, September 3, 2013 6:50:42 AM UTC+3, aswath wrote: >>>>> >>>>>> Same here. I need to increase the number of indexes. Is this >>>>>> limitation removed for billing enabled apps? >>>>>> >>>>>> -Aswath >>>>>> >>>>>> >>>>>> On Tue, Sep 3, 2013 at 1:08 AM, kcunha.araujo <kcunha...@gmail.com>wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Anybody know how to increase quota index number? My app is on limite >>>>>>> 200. We need to increase it. >>>>>>> >>>>>>> Em quinta-feira, 12 de novembro de 2009 11h48min11s UTC-4, Orlando >>>>>>> escreveu: >>>>>>>> >>>>>>>> I have same problem: Error 400: --- begin server output --- >>>>>>>> Over index creation quota: The API call datastore_v3.CreateIndex() >>>>>>>> required more quota than is available. >>>>>>>> >>>>>>>> The application is 'testingryv'. I have tried to delete my useless >>>>>>>> indexes, but i got same problem. Any solution? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Orlando >>>>>>>> >>>>>>>> On 14 oct, 21:35, Dinesh Varadharajan >>>>>>>> <dinesh.varadhara...@**orangesca******pe.com> wrote: >>>>>>>> > I can delete and recreate apps.Thanks for your time.. >>>>>>>> > >>>>>>>> > Dinesh >>>>>>>> > On Oct 14, 6:02 pm, "Nick Johnson (Google)" < >>>>>>>> nick.john...@google.com> >>>>>>>> > wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > > On Wed, Oct 14, 2009 at 1:53 PM, Dinesh <dinesh.varadhara...@** >>>>>>>> orangesca******pe.com >>>>>>>> > >>>>>>>> > > > wrote: >>>>>>>> > >>>>>>>> > > > Hi Nick, >>>>>>>> > > > > Not functioning how? >>>>>>>> > > > I want to reuse existing apps, so I cleared data and indexes. >>>>>>>> I am not >>>>>>>> > > > seeing any data in data viewer or any index in indexes. But >>>>>>>> when >>>>>>>> > > > trying to import new app I am getting index quota error.. I >>>>>>>> just got >>>>>>>> > > > the alert saying deleting app is possible now. Thank god.. >>>>>>>> > >>>>>>>> > > Do you still need these apps reset, now that you can delete >>>>>>>> them and create >>>>>>>> > > new ones instead? >>>>>>>> > >>>>>>>> > > > > Are all of these apps out of index quota? >>>>>>>> > > > Yes >>>>>>>> > >>>>>>>> > > > > Requiring more than 100 indexes is usually a sign of a >>>>>>>> schema design that >>>>>>>> > > > > needs optimizing. We can increase the index quota, but only >>>>>>>> in >>>>>>>> > > > exceptional >>>>>>>> > > > > circumstances. >>>>>>>> > > > We store the data in expando model which is very huge.. The >>>>>>>> no of >>>>>>>> > > > columns at the minimum will be around 300. The auto generated >>>>>>>> indexes >>>>>>>> > > > are around 50% of total indexes. In addition, gae throws >>>>>>>> "need index >>>>>>>> > > > error" in lot of cases and we end up adding those indexes in >>>>>>>> to >>>>>>>> > > > index.yaml. So, it easily crosses the limit.. >>>>>>>> > >>>>>>>> > > Without knowing more about your data model, it's impossible to >>>>>>>> comment on >>>>>>>> > > how you can reduce your total number of indexes. >>>>>>>> > >>>>>>>> > > -Nick Johnson >>>>>>>> > >>>>>>>> > > > We don't use any of the inequality filters in our queries >>>>>>>> except in 2 >>>>>>>> > > > cases. >>>>>>>> > > > Thanks, >>>>>>>> > > > Dinesh >>>>>>>> > >>>>>>>> > > > On Oct 14, 4:33 pm, "Nick Johnson (Google)" < >>>>>>>> nick.john...@google.com> >>>>>>>> > > > wrote: >>>>>>>> > > > > Hi dinesh, >>>>>>>> > >>>>>>>> > > > > On Wed, Oct 14, 2009 at 12:00 PM, Dinesh < >>>>>>>> > >>>>>>>> > > > > dinesh.varadhara...@**orangescap******e.com> wrote: >>>>>>>> > >>>>>>>> > > > > > Hi Nick, >>>>>>>> > > > > > thanks for the immediate reply.. I cleared data and >>>>>>>> vacuumed indexes >>>>>>>> > > > > > in 4 of my apps. (Created new apps as old apps are not >>>>>>>> functioning). >>>>>>>> > >>>>>>>> > > > > Not functioning how? >>>>>>>> > >>>>>>>> > > > > > But still it is throwing the quota exception. The app ids >>>>>>>> are >>>>>>>> > >>>>>>>> > > > > > os-demo >>>>>>>> > > > > > osruntime >>>>>>>> > > > > > osclouddemo >>>>>>>> > > > > > osruntime >>>>>>>> > >>>>>>>> > > > > Are all of these apps out of index quota? >>>>>>>> > >>>>>>>> > > > > > And we might be terribly needing support for more than >>>>>>>> 100 indexes for >>>>>>>> > > > > > our apps and are ready to pay for the overuse. I don't >>>>>>>> think it is >>>>>>>> > > > > > part of billing. any thoughts of including it.. >>>>>>>> > >>>>>>>> > > > > Requiring more than 100 indexes is usually a sign of a >>>>>>>> schema design that >>>>>>>> > > > > needs optimizing. We can increase the index quota, but only >>>>>>>> in >>>>>>>> > > > exceptional >>>>>>>> > > > > circumstances. >>>>>>>> > >>>>>>>> > > > > > Thanks, >>>>>>>> > > > > > Dinesh >>>>>>>> > >>>>>>>> > > > > > On Oct 14, 3:52 pm, "Nick Johnson (Google)" < >>>>>>>> nick.john...@google.com> >>>>>>>> > > > > > wrote: >>>>>>>> > > > > > > Hi Dinesh, >>>>>>>> > > > > > > All apps are limited to 100 indexes. If you are getting >>>>>>>> this error on >>>>>>>> > > > an >>>>>>>> > > > > > app >>>>>>>> > > > > > > that has fewer than 100 indexes, please let me know >>>>>>>> which one it is, >>>>>>>> > > > and >>>>>>>> > > > > > I >>>>>>>> > > > > > > will reset your index count quota. >>>>>>>> > >>>>>>>> > > > > > > -Nick Johnson >>>>>>>> > >>>>>>>> > > > > > > On Wed, Oct 14, 2009 at 11:50 AM, Dinesh < >>>>>>>> > >>>>>>>> > > > > > > dinesh.varadhara...@**orangescap******e.com> wrote: >>>>>>>> > >>>>>>>> > > > > > > > Hi, >>>>>>>> > > > > > > > Suddenly I am getting >>>>>>>> > >>>>>>>> > > > > > > > "Over index creation quota: The API call >>>>>>>> datastore_v3.CreateIndex() >>>>>>>> > > > > > > > required more quota than is available." >>>>>>>> > >>>>>>>> > > > > > > > when trying to update indexes. however few of my >>>>>>>> older applications >>>>>>>> > > > > > > > has more than 100 indexes it it. Is this limitation >>>>>>>> newly >>>>>>>> > > > introduced >>>>>>>> > > > > > > > in gae. If so, how do I increase the quota. >>>>>>>> > >>>>>>>> > > > > > > > Dinesh >>>>>>>> > >>>>>>>> > > > > > > -- >>>>>>>> > > > > > > Nick Johnson, Developer Programs Engineer, App Engine >>>>>>>> > > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, >>>>>>>> Registration >>>>>>>> > > > > > Number: >>>>>>>> > > > > > > 368047 >>>>>>>> > >>>>>>>> > > > > -- >>>>>>>> > > > > Nick Johnson, Developer Programs Engineer, App Engine >>>>>>>> > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, >>>>>>>> Registration >>>>>>>> > > > Number: >>>>>>>> > > > > 368047 >>>>>>>> > >>>>>>>> > > -- >>>>>>>> > > Nick Johnson, Developer Programs Engineer, App Engine >>>>>>>> > > Google Ireland Ltd. :: Registered in Dublin, Ireland, >>>>>>>> Registration Number: >>>>>>>> > > 368047 >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google App Engine" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to google-appengi...@**googlegroups****.com. >>>>>>> To post to this group, send email to google-a...@googlegroups.**com. >>>>>>> >>>>>>> Visit this group at http://groups.google.com/**group**** >>>>>>> /google-appengine <http://groups.google.com/group/google-appengine>. >>>>>>> For more options, visit https://groups.google.com/**grou**** >>>>>>> ps/opt_out <https://groups.google.com/groups/opt_out>. >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to google-appengi...@**googlegroups**.com. >>>>> To post to this group, send email to google-a...@googlegroups.**com. >>>>> Visit this group at http://groups.google.com/**group** >>>>> /google-appengine <http://groups.google.com/group/google-appengine>. >>>>> For more options, visit >>>>> https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>> >>>> >>>> >>>> -- >>>> Alejandro González >>>> +34 666 57 79 13 >>>> <http://es.linkedin.com/pub/alejandro-gonzález-rodrigo/19/684/466/> >>>> >>>> <http://www.intelygenz.com/> >>>> If you have a dream <http://www.intelygenz.com/en/cases>* we can write >>>> the *code <http://www.thegameofcode.com/> >>>> >>>> >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengi...@**googlegroups.com. >>> To post to this group, send email to google-a...@googlegroups.**com. >>> Visit this group at >>> http://groups.google.com/**group/google-appengine<http://groups.google.com/group/google-appengine> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >> >> >> >> -- >> Alejandro González >> +34 666 57 79 13 >> <http://es.linkedin.com/pub/alejandro-gonzález-rodrigo/19/684/466/> >> >> <http://www.intelygenz.com/> >> If you have a dream <http://www.intelygenz.com/en/cases>* we can write >> the *code <http://www.thegameofcode.com/> >> >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > -- Alejandro González <alejandro.gonza...@intelygenz.com> +34 666 57 79 13 <http://es.linkedin.com/pub/alejandro-gonzález-rodrigo/19/684/466/> <http://www.intelygenz.com/> If you have a dream <http://www.intelygenz.com/en/cases>* we can write the * code <http://www.thegameofcode.com/> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
