[google-appengine] Re: Search limit

['George (Cloud Platform Support)' via Google App Engine]

Hello Emlyn, 

The 15000 documents per minute quota is meant for the whole app. You can 
see in the document you link to that the action is named "Adding documents 
to Indexes", so generally, not one index or per index. 

In the Java "Quotas" document 
 the corresponding entry 
is "Maximum documents added or deleted 15,000 per minute". No mentioning of 
index(es) is made, value being defined per app. Similar statements can be 
read in the corresponding Python and Go documents. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/bec25533-ce04-4ae6-8f3c-d622a546d9bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Search limit

[Emlyn]

Thanks George. Not quite the news I wanted, but better to know.

On 21 June 2017 at 01:13, 'George (Cloud Platform Support)' via Google App
Engine  wrote:

> Hello Emlyn,
>
> The 15000 documents per minute quota is meant for the whole app. You can
> see in the document you link to that the action is named "Adding documents
> to Indexes", so generally, not one index or per index.
>
> In the Java "Quotas" document
>  the corresponding
> entry is "Maximum documents added or deleted 15,000 per minute". No
> mentioning of index(es) is made, value being defined per app. Similar
> statements can be read in the corresponding Python and Go documents.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/bec25533-ce04-4ae6-8f3c-
> d622a546d9bc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Emlyn

https://medium.com/the-infinite-machine - A publication about Google App
Engine
sutllang.com - My language sUTL
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPCWTQJYrqdkp34J3wGz%2BpXv6ryR3AtykWd0%2B%2Bbj5vbdmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Search limit

['George (Cloud Platform Support)' via Google App Engine]

Hi Emlyn, 

News is not entirely bad: "Note: Although these limits are enforced by the 
minute, the Cloud Platform Console displays the daily totals for each. 
Customers with Silver, Gold, or Platinum support can request higher 
throughput limits by contacting their support representative." This is to 
be read just below the quotas paragraph we speak about. In short, you may 
consider asking for a quota increase. 

Information on how to reduce your app's needs for paid resources, and so 
keep within quotas, you may consult the "Managing App Resources" 
documentation page 

. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Search limit

[Emlyn]

Thanks George, I read that, that's great. I may have to go for a support
level in the near future.

For anyone who's interested, I was able to fix my problems with the Search
API as follows:

My code enqueues a huge amount of tasks very rapidly, which modify
datastore objects. In the _post_put handlers, these also update search
indices.

I was having problems that the datastore updates were working, but the rate
was too high for the search indices; I was getting timeouts, quota
limitations, bad things :-)

I've changed my _post_put handlers to do the updates to search indices in
separate tasks, on a queue I've configured just for this. It's rate limited
to below the maximum rate for the search api (ie: maximum search throughput
is 15000/minute, or 250/s, so I've set it lower than this).

That's completely fixed the problem. And because I'm not hitting quota
limits, the real throughput is much higher than it was; I don't see this
queue hitting the limit and lagging behind the rest of the job.

Now I get to sleep comfortably at night :-) Thanks for the help.


On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google App
Engine  wrote:

> Hi Emlyn,
>
> News is not entirely bad: "Note: Although these limits are enforced by the
> minute, the Cloud Platform Console displays the daily totals for each.
> Customers with Silver, Gold, or Platinum support can request higher
> throughput limits by contacting their support representative." This is to
> be read just below the quotas paragraph we speak about. In short, you may
> consider asking for a quota increase.
>
> Information on how to reduce your app's needs for paid resources, and so
> keep within quotas, you may consult the "Managing App Resources"
> documentation page
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/623ea57b-1c0c-4454-a854-
> 5cb5fa84bd8b%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Emlyn

https://medium.com/the-infinite-machine - A publication about Google App
Engine
sutllang.com - My language sUTL
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Search limit

[Nickolas Daskalou]

That's a nice way of working around the rate limits Emlyn.

In your _post_put function you can also compare a hash of the last document
you put in the Search API index for the Datastore entity, and only send to
the Search API if the current document's hash is different.

To make it fast and cost-effective, you can use Memcache to store the hash
of the most recent document sent to the Search API for a particular
Datastore entity.

This method is most useful if your app does a lot of updates to Datastore
entities which would not actually change the Search API document at all
(e.g. when updating Datastore entity properties which are not indexed in
the Search API).

Nick


On 22 June 2017 at 11:42, Emlyn  wrote:

> Thanks George, I read that, that's great. I may have to go for a support
> level in the near future.
>
> For anyone who's interested, I was able to fix my problems with the Search
> API as follows:
>
> My code enqueues a huge amount of tasks very rapidly, which modify
> datastore objects. In the _post_put handlers, these also update search
> indices.
>
> I was having problems that the datastore updates were working, but the
> rate was too high for the search indices; I was getting timeouts, quota
> limitations, bad things :-)
>
> I've changed my _post_put handlers to do the updates to search indices in
> separate tasks, on a queue I've configured just for this. It's rate limited
> to below the maximum rate for the search api (ie: maximum search throughput
> is 15000/minute, or 250/s, so I've set it lower than this).
>
> That's completely fixed the problem. And because I'm not hitting quota
> limits, the real throughput is much higher than it was; I don't see this
> queue hitting the limit and lagging behind the rest of the job.
>
> Now I get to sleep comfortably at night :-) Thanks for the help.
>
>
> On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google App
> Engine  wrote:
>
>> Hi Emlyn,
>>
>> News is not entirely bad: "Note: Although these limits are enforced by
>> the minute, the Cloud Platform Console displays the daily totals for each.
>> Customers with Silver, Gold, or Platinum support can request higher
>> throughput limits by contacting their support representative." This is to
>> be read just below the quotas paragraph we speak about. In short, you may
>> consider asking for a quota increase.
>>
>> Information on how to reduce your app's needs for paid resources, and so
>> keep within quotas, you may consult the "Managing App Resources"
>> documentation page
>> 
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%
>> 40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Emlyn
>
> https://medium.com/the-infinite-machine - A publication about Google App
> Engine
> sutllang.com - My language sUTL
> https://plus.google.com/u/0/100281903174934656260 - Google+
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%
> 2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAOj3zuCVz3iR9HDYZdAHHF_LygKGa7HwtZZiKe-Swb6mVpL9sA%40mail.gmail.com.
For more options, visit https://groups.goog

Re: [google-appengine] Re: Search limit

[Emlyn]

I do this is a slightly more costly way Nick; I generate my new document,
read any existing document from the index, compare, only put on a change. I
like your memcache idea, I could possibly cache the last written version.


On 22 June 2017 at 11:38, Nickolas Daskalou  wrote:

> That's a nice way of working around the rate limits Emlyn.
>
> In your _post_put function you can also compare a hash of the last
> document you put in the Search API index for the Datastore entity, and only
> send to the Search API if the current document's hash is different.
>
> To make it fast and cost-effective, you can use Memcache to store the hash
> of the most recent document sent to the Search API for a particular
> Datastore entity.
>
> This method is most useful if your app does a lot of updates to Datastore
> entities which would not actually change the Search API document at all
> (e.g. when updating Datastore entity properties which are not indexed in
> the Search API).
>
> Nick
>
>
> On 22 June 2017 at 11:42, Emlyn  wrote:
>
>> Thanks George, I read that, that's great. I may have to go for a support
>> level in the near future.
>>
>> For anyone who's interested, I was able to fix my problems with the
>> Search API as follows:
>>
>> My code enqueues a huge amount of tasks very rapidly, which modify
>> datastore objects. In the _post_put handlers, these also update search
>> indices.
>>
>> I was having problems that the datastore updates were working, but the
>> rate was too high for the search indices; I was getting timeouts, quota
>> limitations, bad things :-)
>>
>> I've changed my _post_put handlers to do the updates to search indices in
>> separate tasks, on a queue I've configured just for this. It's rate limited
>> to below the maximum rate for the search api (ie: maximum search throughput
>> is 15000/minute, or 250/s, so I've set it lower than this).
>>
>> That's completely fixed the problem. And because I'm not hitting quota
>> limits, the real throughput is much higher than it was; I don't see this
>> queue hitting the limit and lagging behind the rest of the job.
>>
>> Now I get to sleep comfortably at night :-) Thanks for the help.
>>
>>
>> On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google
>> App Engine  wrote:
>>
>>> Hi Emlyn,
>>>
>>> News is not entirely bad: "Note: Although these limits are enforced by
>>> the minute, the Cloud Platform Console displays the daily totals for each.
>>> Customers with Silver, Gold, or Platinum support can request higher
>>> throughput limits by contacting their support representative." This is to
>>> be read just below the quotas paragraph we speak about. In short, you may
>>> consider asking for a quota increase.
>>>
>>> Information on how to reduce your app's needs for paid resources, and so
>>> keep within quotas, you may consult the "Managing App Resources"
>>> documentation page
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40
>>> googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Emlyn
>>
>> https://medium.com/the-infinite-machine - A publication about Google App
>> Engine
>> sutllang.com - My language sUTL
>> https://plus.google.com/u/0/100281903174934656260 - Google+
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%2BqEf%2Byz%
>> 2BxwKCn2GwT1EfGsuA%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
>