[
https://issues.apache.org/jira/browse/SOLR-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823725#comment-13823725
]
Per Steffensen commented on SOLR-5444:
--------------------------------------
bq. You complained of OOM and slowness with docvalues in 4.4 (which loads
everything in RAM).
Guess your are referring to mailing-thread "DocValue on Strings slow and OOM"
on [email protected]
I complained about OOM and slowness for faceting on STRING docValues in 4.4,
yes. In the same mail I said that faceting on LONG docValues in 4.4 works
nicely - and FYI everything is NOT loaded into RAM when it comes to faceting on
docValue-longs in 4.4. There seem to be no (important) changes from 4.4 to 4.5
in the code related to faceting on docValue-LONGS. I might be wrong about that,
but the particular issue I raise here in SOLR-5444 has not been fixed in 4.5.
The OOM/slowness problem for faceting on STRING docValues in 4.4 might very
well be fixed in 4.5, and you are right that no one should work on trying to
fix it (in 4.4) before it has been confirmed that issue I raised in the
mailing-thread actually still exists in 4.5. But as I said in the
mailing-thread I will not go find out, because we are (probably) not going to
use it anyway, as we will change the c field (c_dstr_doc_sto) to become a long
field instead (c_dlng_doc_sto). So we are only going to facet on docValue-LONGS.
So to spell it out: *This issue SOLR-5444 has nothing to do with what I
complained about in the mailing-thread*. Actually in the mailing-thread I say
the faceting on LONG docValues in 4.4 works nicely. SOLR-5444 is about the fact
that it actually does not - under certain circumstances.
bq. I told you to use the latest version of lucene/solr: you ignored me and
refused to do that: your bad!
I did not ignore it! Actually I told you that I was not going to spend time on
going to 4.5 because the problem I complained about (faceting on STRING
docValues in 4.4) was not relevant for us any longer (at least for now). I
thought I would help out though, checking if the issues was actually fixed in
4.5 - if I could do that check fast. I would only be able to do it fast, if I
could just install a 4.5 on top of the 12 billion documents I had already
indexed with 4.4. Therefore I asked in the mailing-thread "... if we need to
reindex the 12billion documents again after upgrading to 4.5, or if we ought to
be able to deploy 4.5 on top of the already indexed documents". I got no
answer, so I guess you where not interested in me verifying that the problem
was actually fixed in 4.5.
bq. Nobody is going to waste time on things that have already been fixed.
It has not been fixed. You would have had a chance to realize that if you
actually read this issue.
I am not asking you to fix this one - I will be capable to doing that myself.
But there are several potential fixes, and because I actually work as you are
supposed to do in an open-source project, I ask for input about which fix is
the best (while laying out a few of the potential fixes). There might be people
out there that prefer one fix over another, for reasons that I might not know
of.
> Slow response on facet search, lots of facets, asking for few facets in
> response
> --------------------------------------------------------------------------------
>
> Key: SOLR-5444
> URL: https://issues.apache.org/jira/browse/SOLR-5444
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.4
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: docvalue, faceted-search, performance
> Fix For: 4.7
>
> Attachments: Profiiling_SimpleFacets_getListedTermCounts_path.png,
> Profiling_SimpleFacets_getTermCounts_path.png,
> Responsetime_func_of_facets_asked_for.png
>
>
> h5. Setup
> We have a 6-Solr-node (release 4.4.0) setup with 12 billion "small" documents
> loaded across 3 collections. The documents have the following fields
> * a_dlng_doc_sto (docvalue long)
> * b_dlng_doc_sto (docvalue long)
> * c_dstr_doc_sto (docvalue string)
> * timestamp_lng_ind_sto (indexed long)
> * d_lng_ind_sto (indexed long)
> From schema.xml
> {code}
> <dynamicField name="*_dstr_doc_sto" type="dstring" indexed="false"
> stored="true" required="true" docValues="true"/>
> <dynamicField name="*_lng_ind_sto" type="long" indexed="true"
> stored="true"/>
> <dynamicField name="*_dlng_doc_sto" type="dlng" indexed="false"
> stored="true" required="true" docValues="true"/>
> ...
> <fieldType name="dstring" class="solr.StrField" sortMissingLast="true"
> docValuesFormat="Disk"/>
> <fieldType name="dlng" class="solr.TrieLongField" precisionStep="0"
> positionIncrementGap="0" docValuesFormat="Disk"/>
> {code}
> timestamp_lng_ind_sto decides which collection documents go into
> We execute queries on the following format:
> * q=timestamp_lng_ind_sto:\[x TO y\] AND d_lng_ind_sto:(a OR b OR ... OR n)
> *
> facet=true&facet.field=a_dlng_doc_sto&facet.zeros=false&facet.mincount=1&facet.limit=<asked-for-facets>&rows=0&start=0
> h5. Problem
> We see very slow response-time when hitting large number of rows, spanning
> lots of facets, but only ask for "a few" of those facets
> h5. Concrete example of query to get some concrete numbers to look at
> With x and y plus a, b ... n set to values so that
> * The timestamp_lng_ind_sto:\[x TO y\] part of the search-criteria alone hit
> about 1.7 billion documents (actually all in one (containing 4.5 billion
> docs) of the three collections - but that is not important)
> * The d_lng_ind_sto:(a OR b OR ... OR n) part of the search-criteria alone
> hit about 500000 documents
> * The combined search-criteria (timestamp_lng_ind_sto AND'ed with
> d_lng_ind_sto) hit about 200000 documents
> The following graph shows responsetime as a function of <asked-for-facets>
> (in query)
> !Responsetime_func_of_facets_asked_for.png!
> Note that responsetime is high for "low" <asked-for-facets>, and that it
> increases fast (but linearly) in <asked-for-facets> up until
> <asked-for-facets> is somewhere inbetween 5000 (where responsetime is close
> to 1000 secs) and 10000 (where responsetime is about 5 secs). For values of
> <asked-for-facets> above 10000 responsetime stays "low" at between 1-10 secs
> Looking at the code and profiling it is clear that the change to better
> responsetime occurs when SimpleFacets.getFacetFieldCounts changes from using
> getListedTermCounts to using getTermCounts.
> The following image shows profiling information during a request with
> <asked-for-facets> at about 2000.
> !Profiiling_SimpleFacets_getListedTermCounts_path.png!
> Note that
> * SimpleFacets.getListedTermCounts is used (green box)
> * 91% of the time spent performing the query is spent in
> DocSetCollector-constructor (red box). During this concrete query 125000
> DocSetCollection-objects are created spending 710 secs all in all. Additional
> investigations show that the time is spent allocating huge int-arrays for the
> "scratch"-int-array. Several thousands of those DocSetCollection-constructors
> create int-arrays at size above 1 million - that takes time, and also leaves
> a nice little job of the GC'er afterwards.
> * The actual search-part of the query takes only 0.5% (4 secs) of the
> combined time executing the query (blue box)
> The following image shows profiling information during a request with
> <asked-for-facets> at about 10000
> !Profiling_SimpleFacets_getTermCounts_path.png!
> Note that
> * SimpleFacets.getTermCounts is used (green box)
> * The actual search-part of the query now takes 70% (11 secs) of the combined
> time executing the query (blue box)
> h5. What to do about this?
> * I am not sure why there are two paths that SimpleFacets.getFacetFieldCounts
> can take (getListedTermCounts or getTermCounts) - but I am pretty sure there
> is a good reason. It seems like getListedTermCounts is used when
> <asked-for-facets> is noticeable lower than the total number of facets hit
> (believe it is when <asked-for-facets> * 1.5 + 10 is below actual number of
> facets hit)
> * *One solution* could be to just drop the getListedTermCounts-path and
> always go getTermCounts, but that is probably not at good idea, because
> getListedTermCounts is probably there for a performance reason (in other
> scenarios)
> * The comment above DocSetCollection.scratch says
> {code}
> // in case there aren't that many hits, we may not want a very sparse
> // bit array. Optimistically collect the first few docs in an array
> // in case there are only a few.
> final int[] scratch;
> {code}
> The comment seems reasonable. But when we look at what values are used as
> "smallSetSize" for the DocSetCollection-constructor, it is always "maxDoc >>
> 6" (basically dividing by 64) - this value depends on maxDoc and will be high
> if maxDoc is high. In my case maxDoc is 50+ million a lot of the times
> resulting in "smallSetSize"s of 1+ million (that is not "a few"). I am very
> much in doubt why you want "smallSetSize" to increase as maxDoc increase -
> why not just always a low (fixed or something) value for "smallSetSize"? Is
> it ever a good idea with huge int-arrays for the "scratch"-array?
> * *Another solution* would be to never create "scratch"-arrays with size
> above e.g. 50
> * *There are probably several other potential solutions*
> I would really want your opinion on what solution to make, so that I do not
> unintentionally break good performance-optimizations, just because I missed
> some points explaining why the code is as it is today!?
> *Note* I have filed this as a 4.4 issues, because that is the platform I use
> for my tests etc. But I am sure the problem also exists on 4.5.1 (or whatever
> the latest 4.x release is)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
