Re: [google-appengine] Best Practices for Search API?

[Emlyn]

"The search API is pretty cool. I hope Google makes an official service out
of it someday."

I second this! I live in fear of deprecation. How could we possibly replace
it?

On 10 June 2017 at 16:04, Jeff Schnitzer  wrote:

> No not at all; quite the opposite, really. I was storing word fragments
> for typeahead, eg “foobar” became [“f”, “fo”, “foo”, “foob”, “fooba”,
> “foobar”]. It was very expensive to index in the datastore and very cheap
> to index in the search api.
>
> The search API is pretty cool. I hope Google makes an official service out
> of it someday.
>
> Jeff
>
> On Fri, Jun 9, 2017 at 3:45 PM, Nickolas Daskalou 
> wrote:
>
>> Jeff, were you storing docs in the Search API somewhat intelligently,
>> e.g. using short field names, "0" instead of "false" for boolean values
>> etc., or you just sent documents to the Search API without much thought and
>> it was still efficiently storing them?
>>
>> Nick
>> On 10/06/2017 8:40 AM, "Jeff Schnitzer"  wrote:
>>
>>> The search index is incredibly efficient. I had some data I was indexing
>>> in the datastore, and the index was consuming 100GB+. When I moved it to
>>> the Search API, the index consumed a few GB. Afterwards I felt silly for
>>> asking for the quota raise in advance.
>>>
>>> YMMV, of course.
>>>
>>> Not a direct answer to your question but something to keep in mind. I
>>> was surprised by how small the index is.
>>>
>>> Jeff
>>>
>>> On Thu, Jun 8, 2017 at 11:34 PM, Emlyn  wrote:
>>>
 I've just noticed that the old 10GB limit on search indices can be
 raised on request to 200GB max, on an index by index basis.

 I've got a use case where I'd ideally like search indices to be
 unbounded, but 200GB will go a long way. Is there any penalty associated
 with this, though? Do the indices perform worse as they get larger? Does
 anyone know why there is a limit at all?

 --
 Emlyn

 http://point7.wordpress.com - My blog
 https://plus.google.com/u/0/100281903174934656260 - Google+

 --
 You received this message because you are subscribed to the Google
 Groups "Google App Engine" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to google-appengine+unsubscr...@googlegroups.com.
 To post to this group, send email to google-appengine@googlegroups.com.
 Visit this group at https://groups.google.com/group/google-appengine.
 To view this discussion on the web visit https://groups.google.com/d/ms
 gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP
 g_kvX4-aUm3w%40mail.gmail.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVX
>>> TSjQt1qYGNMCJQ%40mail.gmail.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/google-appengine/CAOj3zuDkQLAE%2BV5H1WWXxu4hj-MWNpddNiSh
>> C7R8YkWE5fug9Q%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit 

Re: [google-appengine] Best Practices for Search API?

[Jeff Schnitzer]

No not at all; quite the opposite, really. I was storing word fragments for
typeahead, eg “foobar” became [“f”, “fo”, “foo”, “foob”, “fooba”,
“foobar”]. It was very expensive to index in the datastore and very cheap
to index in the search api.

The search API is pretty cool. I hope Google makes an official service out
of it someday.

Jeff

On Fri, Jun 9, 2017 at 3:45 PM, Nickolas Daskalou  wrote:

> Jeff, were you storing docs in the Search API somewhat intelligently, e.g.
> using short field names, "0" instead of "false" for boolean values etc., or
> you just sent documents to the Search API without much thought and it was
> still efficiently storing them?
>
> Nick
> On 10/06/2017 8:40 AM, "Jeff Schnitzer"  wrote:
>
>> The search index is incredibly efficient. I had some data I was indexing
>> in the datastore, and the index was consuming 100GB+. When I moved it to
>> the Search API, the index consumed a few GB. Afterwards I felt silly for
>> asking for the quota raise in advance.
>>
>> YMMV, of course.
>>
>> Not a direct answer to your question but something to keep in mind. I was
>> surprised by how small the index is.
>>
>> Jeff
>>
>> On Thu, Jun 8, 2017 at 11:34 PM, Emlyn  wrote:
>>
>>> I've just noticed that the old 10GB limit on search indices can be
>>> raised on request to 200GB max, on an index by index basis.
>>>
>>> I've got a use case where I'd ideally like search indices to be
>>> unbounded, but 200GB will go a long way. Is there any penalty associated
>>> with this, though? Do the indices perform worse as they get larger? Does
>>> anyone know why there is a limit at all?
>>>
>>> --
>>> Emlyn
>>>
>>> http://point7.wordpress.com - My blog
>>> https://plus.google.com/u/0/100281903174934656260 - Google+
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP
>>> g_kvX4-aUm3w%40mail.gmail.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVX
>> TSjQt1qYGNMCJQ%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/CAOj3zuDkQLAE%2BV5H1WWXxu4hj-
> MWNpddNiShC7R8YkWE5fug9Q%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CADK-0uhhw4f%2BOwpJ%2Bws2S0ngLeJFWi-Li%3D%3DuOLd0hRODsUA7PQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Best Practices for Search API?

[Emlyn]

I've noticed the same thing Jeff, it's really brilliant. But what I'd like
to put into it is a pretty dense time series of real time events; it's
really difficult to put a sensible upper bound on how big that might get.

On 10 Jun. 2017 8:10 am, "Jeff Schnitzer"  wrote:

The search index is incredibly efficient. I had some data I was indexing in
the datastore, and the index was consuming 100GB+. When I moved it to the
Search API, the index consumed a few GB. Afterwards I felt silly for asking
for the quota raise in advance.

YMMV, of course.

Not a direct answer to your question but something to keep in mind. I was
surprised by how small the index is.

Jeff

On Thu, Jun 8, 2017 at 11:34 PM, Emlyn  wrote:

> I've just noticed that the old 10GB limit on search indices can be raised
> on request to 200GB max, on an index by index basis.
>
> I've got a use case where I'd ideally like search indices to be unbounded,
> but 200GB will go a long way. Is there any penalty associated with this,
> though? Do the indices perform worse as they get larger? Does anyone know
> why there is a limit at all?
>
> --
> Emlyn
>
> http://point7.wordpress.com - My blog
> https://plus.google.com/u/0/100281903174934656260 - Google+
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP
> g_kvX4-aUm3w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/
msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_
fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPAyf9BkjzR_f80ReRW6NAWw6oadSDsCOw1i5%2BMPNaGzFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Best Practices for Search API?

[Nickolas Daskalou]

Jeff, were you storing docs in the Search API somewhat intelligently, e.g.
using short field names, "0" instead of "false" for boolean values etc., or
you just sent documents to the Search API without much thought and it was
still efficiently storing them?

Nick
On 10/06/2017 8:40 AM, "Jeff Schnitzer"  wrote:

> The search index is incredibly efficient. I had some data I was indexing
> in the datastore, and the index was consuming 100GB+. When I moved it to
> the Search API, the index consumed a few GB. Afterwards I felt silly for
> asking for the quota raise in advance.
>
> YMMV, of course.
>
> Not a direct answer to your question but something to keep in mind. I was
> surprised by how small the index is.
>
> Jeff
>
> On Thu, Jun 8, 2017 at 11:34 PM, Emlyn  wrote:
>
>> I've just noticed that the old 10GB limit on search indices can be raised
>> on request to 200GB max, on an index by index basis.
>>
>> I've got a use case where I'd ideally like search indices to be
>> unbounded, but 200GB will go a long way. Is there any penalty associated
>> with this, though? Do the indices perform worse as they get larger? Does
>> anyone know why there is a limit at all?
>>
>> --
>> Emlyn
>>
>> http://point7.wordpress.com - My blog
>> https://plus.google.com/u/0/100281903174934656260 - Google+
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxP
>> g_kvX4-aUm3w%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_
> fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAOj3zuDkQLAE%2BV5H1WWXxu4hj-MWNpddNiShC7R8YkWE5fug9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Best Practices for Search API?

[Jeff Schnitzer]

The search index is incredibly efficient. I had some data I was indexing in
the datastore, and the index was consuming 100GB+. When I moved it to the
Search API, the index consumed a few GB. Afterwards I felt silly for asking
for the quota raise in advance.

YMMV, of course.

Not a direct answer to your question but something to keep in mind. I was
surprised by how small the index is.

Jeff

On Thu, Jun 8, 2017 at 11:34 PM, Emlyn  wrote:

> I've just noticed that the old 10GB limit on search indices can be raised
> on request to 200GB max, on an index by index basis.
>
> I've got a use case where I'd ideally like search indices to be unbounded,
> but 200GB will go a long way. Is there any penalty associated with this,
> though? Do the indices perform worse as they get larger? Does anyone know
> why there is a limit at all?
>
> --
> Emlyn
>
> http://point7.wordpress.com - My blog
> https://plus.google.com/u/0/100281903174934656260 - Google+
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yr
> yv3T2OaxPg_kvX4-aUm3w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.