I must "convert" 10TeraBytes of MySQL data into Search API indexed documents and I think I will definitely reach the 10GigaBytes index limit :). I actually don't know how much "Search API space" (documents+indexes) is needed for 10TB of MySQL data but I think it will be much more than 10GB. So I think I must sacrifice sorting and paginating if I will use more indexes... What do you thing?
On Friday, March 27, 2015 at 1:49:28 PM UTC+1, Kaan Soral wrote: > > No solution, have you reached the limit? > > As it seems, reaching the limit means doom, at that point you would have > to search for an external solution to re-create the entire index with, or > hope the limit can be increased, yet that only postpones the issue > > On Friday, March 27, 2015 at 1:29:21 PM UTC+2, Simeon Ivaylov Petrov wrote: >> >> If I search into multiple indexes I will not be able to sort and paginate >> all the merged results correctly, is that right? Is there a solution for >> this problem? >> >> On Friday, February 27, 2015 at 10:28:34 AM UTC+1, Carlos Lallana wrote: >>> >>> Hi Kaan, >>> >>> It is indeed not easy to reach the 10 GB size for an index, but yet not >>> impossible. As you can read here >>> <https://cloud.google.com/appengine/docs/quotas#search>, "when an app >>> tries to exceed this amount, an insufficient quota error is returned". >>> >>> Note that if you have purchased one of our Silver, Gold, or Platinum >>> support <https://cloud.google.com/support/packages> packages, you'll be >>> able to request a quota increase for this limit. >>> >>> But before that, I would suggest to shard (divide or split) your index >>> into multiple indexes. In order to do so, you should: >>> >>> 1. Create an additional index. >>> 2. Store a new document in the second index if needed. >>> 3. When you need to search for a document, search both indexes. For >>> better performance, you can make asynchronous search calls. >>> >>> According to what my colleague David (Google wise man) explained to me, >>> this approach will not scale indefinitely. Beyond two or maybe three >>> sharded indexes, this technique will become relatively inefficient. The >>> Search API is not designed for infinitely large document collections, so if >>> you plan to scale a great deal then you would need to consider alternative >>> architectures. However this technique will at least allow you to double >>> your search capacity. >>> >>> Hope that info helps! >>> >>> On Thursday, February 26, 2015 at 9:17:31 PM UTC+1, Kaan Soral wrote: >>>> >>>> Thanks for the reply >>>> >>>> I don't think 10gb is easy to reach, but growing is probably the aim of >>>> most of the appengine apps, and reaching that limit would probably be a >>>> nightmare >>>> >>>> What exactly happens at that point? >>>> >>>> I hope the system automatically purges the low-rank documents on it's >>>> own >>>> >>>> Is it also a hard-limit, or does an internal bell goes off, and someone >>>> warns you that you're first to reach the limit, and starts discussing what >>>> to do >>>> >>>> I really wish it was unlimited, just for the sake of the mind >>>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2c671ca9-2d34-4e6c-a9f6-6a1709e331f6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
