What if all those fields are collapsed to one, like you suggest, but that one field is projected out of _source (think non-indexed json in a string field)? do you see a noticable performance gain then?
What if that field is set to be stored (and loaded using fields, not via _source)? what is the performance gain then? Fielddata and the doc_values optimization on top of them will not help you here, those data structures aren't being used for sending data out, only for aggregations and sorting. Also, using fielddata will require indexing those fields; it is apparent that you are not looking to be doing that. -- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Lucene.NET committer and PMC member On Tue, Apr 21, 2015 at 12:14 AM, Itai Frenkel <itaifren...@live.com> wrote: > Itamar, > > 1. The _source field includes many fields that are only being indexed, and > many fields that are only needed as a query search result. _source includes > them both.The projection from _source from the query result is too CPU > intensive to do during search time for each result, especially if the size > is big. > 2. I agree that adding another NoSQL could solve this problem, however it > is currently out of scope, as it would require syncing data with another > data store. > 3. Wouldn't a big stored field will bloat the lucene index size? Even if > not, isn't non_analyzed fields are destined to be (or already are) > doc_fields? > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: >> >> This is how _source works. doc_values don't make sense in this regard - >> what you are looking for is using stored fields and have the transform >> script write to that. Loading stored fields (even one field per hit) may be >> slower than loading and parsing _source, though. >> >> I'd just put this logic in the indexer, though. It will definitely help >> with other things as well, such as nasty huge mappings. >> >> Alternatively, find a way to avoid IO completely. How about using ES for >> search and something like riak for loading the actual data, if IO costs are >> so noticable? >> >> -- >> >> Itamar Syn-Hershko >> http://code972.com | @synhershko <https://twitter.com/synhershko> >> Freelance Developer & Consultant >> Lucene.NET committer and PMC member >> >> On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel <itaif...@live.com> wrote: >> >>> Hi, >>> >>> We are having a performance problem in which for each hit, elasticsearch >>> parses the entire _source then generates a new Json with only the requested >>> query _source fields. In order to overcome this issue we would like to use >>> mapping transform script that serializes the requested query fields (which >>> is known in advance) into a doc_value. Does that makes sense? >>> >>> The actual problem with the transform script is SecurityException that >>> does not allow using any json serialization mechanism. A binary >>> serialization would also be ok. >>> >>> >>> Itai >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuxvUoZ4L%2BUq0G82GLZKYfN-hj_e_gez6RsUc3hZeHbyw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
