Hi Jack,
I also run the test with queries that have query terms(with filter too).
Solr5 is faster compare to solr4 in the test. I got the queries set from
our production log, almost all of our queries have filter. So that suggest
to me that it is not the filter query that is slow.
I copy the fq query to the q field (i did not remove fq though), the solr5
is slightly faster than solr 4 for the query
solr4:
<?xml version="1.0" encoding="UTF-8"?><response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">64</int>
<lst name="params">
<str name="fl">id</str>
<str name="start">0</str>
<str name="q">+categoryIdsPath:1001</str>
<str name="debug">true</str>
<str name="fq">+categoryIdsPath:1001</str>
<str name="rows">2</str>
</lst>
</lst>
<result name="response" numFound="573467" start="0">
<doc>
<str name="id">36652255</str>
</doc>
<doc>
<str name="id">36651884</str>
</doc>
</result>
<lst name="debug">
<str name="rawquerystring">+categoryIdsPath:1001</str>
<str name="querystring">+categoryIdsPath:1001</str>
<str name="parsedquery">+categoryIdsPath:1001</str>
<str name="parsedquery_toString">+categoryIdsPath:1001</str>
<lst name="explain">
<str name="36652255">20.451632 = (MATCH)
weight(categoryIdsPath:1001 in 19) [], result of:
20.451632 = score(doc=19,freq=1.0 = termFreq=1.0
), product of:
4.522348 = queryWeight, product of:
4.522348 = idf(docFreq=610392, maxDocs=20670250)
1.0 = queryNorm
4.522348 = fieldWeight in 19, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
4.522348 = idf(docFreq=610392, maxDocs=20670250)
1.0 = fieldNorm(doc=19)</str>
<str name="36651884">20.451632 = (MATCH)
weight(categoryIdsPath:1001 in 44) [], result of:
20.451632 = score(doc=44,freq=1.0 = termFreq=1.0
), product of:
4.522348 = queryWeight, product of:
4.522348 = idf(docFreq=610392, maxDocs=20670250)
1.0 = queryNorm
4.522348 = fieldWeight in 44, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
4.522348 = idf(docFreq=610392, maxDocs=20670250)
1.0 = fieldNorm(doc=44)</str>
</lst>
<str name="QParser">LuceneQParser</str>
<arr name="filter_queries">
<str>+categoryIdsPath:1001</str>
</arr>
<arr name="parsed_filter_queries">
<str>+categoryIdsPath:1001</str>
</arr>
<lst name="timing">
<double name="time">63.0</double>
<lst name="prepare">
<double name="time">3.0</double>
<lst name="query">
<double name="time">3.0</double>
</lst>
<lst name="facet">
<double name="time">0.0</double>
</lst>
<lst name="mlt">
<double name="time">0.0</double>
</lst>
<lst name="highlight">
<double name="time">0.0</double>
</lst>
<lst name="stats">
<double name="time">0.0</double>
</lst>
<lst name="debug">
<double name="time">0.0</double>
</lst>
</lst>
<lst name="process">
<double name="time">60.0</double>
<lst name="query">
<double name="time">57.0</double>
</lst>
<lst name="facet">
<double name="time">0.0</double>
</lst>
<lst name="mlt">
<double name="time">0.0</double>
</lst>
<lst name="highlight">
<double name="time">0.0</double>
</lst>
<lst name="stats">
<double name="time">0.0</double>
</lst>
<lst name="debug">
<double name="time">3.0</double>
</lst>
</lst>
</lst>
</lst></response>
solr5:
<?xml version="1.0" encoding="UTF-8"?><response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">51</int>
<lst name="params">
<str name="fl">id</str>
<str name="start">0</str>
<str name="q">+categoryIdsPath:1001</str>
<str name="debug">true</str>
<str name="fq">+categoryIdsPath:1001</str>
<str name="rows">2</str>
</lst>
</lst>
<result name="response" numFound="566462" start="0">
<doc>
<str name="id">36652255</str>
</doc>
<doc>
<str name="id">36651884</str>
</doc>
</result>
<lst name="debug">
<str name="rawquerystring">+categoryIdsPath:1001</str>
<str name="querystring">+categoryIdsPath:1001</str>
<str name="parsedquery">+categoryIdsPath:1001</str>
<str name="parsedquery_toString">+categoryIdsPath:1001</str>
<lst name="explain">
<str name="36652255">20.420362 = weight(categoryIdsPath:1001
in 20) [], result of:
20.420362 = score(doc=20,freq=1.0), product of:
4.5188894 = queryWeight, product of:
4.5188894 = idf(docFreq=602005, maxDocs=20315855)
1.0 = queryNorm
4.5188894 = fieldWeight in 20, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
4.5188894 = idf(docFreq=602005, maxDocs=20315855)
1.0 = fieldNorm(doc=20)</str>
<str name="36651884">20.420362 = weight(categoryIdsPath:1001
in 49) [], result of:
20.420362 = score(doc=49,freq=1.0), product of:
4.5188894 = queryWeight, product of:
4.5188894 = idf(docFreq=602005, maxDocs=20315855)
1.0 = queryNorm
4.5188894 = fieldWeight in 49, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
4.5188894 = idf(docFreq=602005, maxDocs=20315855)
1.0 = fieldNorm(doc=49)</str>
</lst>
<str name="QParser">LuceneQParser</str>
<arr name="filter_queries">
<str>+categoryIdsPath:1001</str>
</arr>
<arr name="parsed_filter_queries">
<str>+categoryIdsPath:1001</str>
</arr>
<lst name="timing">
<double name="time">51.0</double>
<lst name="prepare">
<double name="time">1.0</double>
<lst name="query">
<double name="time">1.0</double>
</lst>
<lst name="facet">
<double name="time">0.0</double>
</lst>
<lst name="facet_module">
<double name="time">0.0</double>
</lst>
<lst name="mlt">
<double name="time">0.0</double>
</lst>
<lst name="highlight">
<double name="time">0.0</double>
</lst>
<lst name="stats">
<double name="time">0.0</double>
</lst>
<lst name="expand">
<double name="time">0.0</double>
</lst>
<lst name="debug">
<double name="time">0.0</double>
</lst>
</lst>
<lst name="process">
<double name="time">50.0</double>
<lst name="query">
<double name="time">48.0</double>
</lst>
<lst name="facet">
<double name="time">0.0</double>
</lst>
<lst name="facet_module">
<double name="time">0.0</double>
</lst>
<lst name="mlt">
<double name="time">0.0</double>
</lst>
<lst name="highlight">
<double name="time">0.0</double>
</lst>
<lst name="stats">
<double name="time">0.0</double>
</lst>
<lst name="expand">
<double name="time">0.0</double>
</lst>
<lst name="debug">
<double name="time">2.0</double>
</lst>
</lst>
</lst>
</lst></response>
On Fri, Nov 6, 2015 at 12:12 PM, Jack Krupansky <jack.krupan...@gmail.com>
wrote:
> Just to be clear, I was suggesting that the filter query (fq) was slow, not
> the MatchAllDocsQuery, which should be just as speedy as before. You can
> test for yourself whether the MADQ by itself is any slower.
>
> You could also test using the fq as the main query (q) - with no fq
> parameter, and see if that is a lot faster, both with old and new Solr.
>
> -- Jack Krupansky
>
> On Fri, Nov 6, 2015 at 3:01 PM, wei <sw90...@gmail.com> wrote:
>
> > Thanks Jack and Shawn. I checked these Jira tickets, but I am not sure if
> > the slowness of MatchAllDocsQuery is also caused by the removal of
> > fieldcache. Can someone please explain a little bit?
> >
> > Thanks,
> > Wei
> >
> > On Fri, Nov 6, 2015 at 7:15 AM, Shawn Heisey <apa...@elyograg.org>
> wrote:
> >
> > > On 11/5/2015 10:25 PM, Jack Krupansky wrote:
> > > > I vaguely recall some discussion concerning removal of the field
> cache
> > in
> > > > Lucene.
> > >
> > > The FieldCache wasn't exactly *removed* ... it's more like it was
> > > renamed, improved, and sort of hidden in a miscellaneous package. Some
> > > things still require this functionality, so they use the hidden class
> > > instead, which was changed to use the DocValues API.
> > >
> > > https://issues.apache.org/jira/browse/LUCENE-5666
> > >
> > > I am not qualified to discuss LUCENE-5666 beyond what I wrote in the
> > > paragraph above, and it's possible that some of what I said is wrong
> > > because I do not really understand the APIs involved.
> > >
> > > The change has caused problems for Solr. End result from Solr's
> > > perspective: Certain things which used to work perfectly fine (mostly
> > > facets and grouping) in Solr 4.x have one of two problems in 5.x:
> > > Either they don't work at all, or performance has gone way down. Some
> > > of these problems are documented in Jira. These are the issues I know
> > > about:
> > >
> > > https://issues.apache.org/jira/browse/SOLR-8088
> > > https://issues.apache.org/jira/browse/SOLR-7495
> > > https://issues.apache.org/jira/browse/SOLR-8096
> > >
> > > For fields where adding docValues is a viable option (most field types
> > > other than solr.TextField), adding docValues and reindexing is very
> > > likely to solve those problems.
> > >
> > > Sometimes adding docValues won't work, either because the field type
> > > doesn't allow it, or because it's the indexed terms that are needed,
> not
> > > the original field value. For those situations, there is currently no
> > > solution.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
