Thanks Erick. The 1500 fields is a design that I inherited. I'm trying to figure out why it was done as such and what it will take to fix it.
What about my other question: how does one go about debugging performance issues in Solr to find out where time is mostly spent? How do I know my Solr parameters, such as cache and what have you are set right? From what I see, we are using the defaults off solrconfig.xml. I'm on Solr 5.2 Steve On Thu, Nov 19, 2015 at 11:36 PM, Erick Erickson <erickerick...@gmail.com> wrote: > An fq is still a single entry in your filterCache so from that > perspective it's the same. > > And to create that entry, you're still using all the underlying fields > to search, so they have to be loaded just like they would be in a q > clause. > > But really, the fundamental question here is why your design even has > 1,500 fields and, more specifically, why you would want to search them > all at once. From a 10,000 ft. view, that's a very suspect design. > > Best, > Erick > > On Thu, Nov 19, 2015 at 4:06 PM, Walter Underwood <wun...@wunderwood.org> > wrote: > > The implementation for fq has changed from 4.x to 5.x, so I’ll let > someone else answer that in detail. > > > > In 4.x, the result of each filter query can be cached. After that, they > are quite fast. > > > > wunder > > Walter Underwood > > wun...@wunderwood.org > > http://observer.wunderwood.org/ (my blog) > > > > > >> On Nov 19, 2015, at 3:59 PM, Steven White <swhite4...@gmail.com> wrote: > >> > >> Thanks Walter. I see your point. Does this apply to fq as will? > >> > >> Also, how does one go about debugging performance issues in Solr to find > >> out where time is mostly spent? > >> > >> Steve > >> > >> On Thu, Nov 19, 2015 at 6:54 PM, Walter Underwood < > wun...@wunderwood.org> > >> wrote: > >> > >>> With one field in qf for a single-term query, Solr is fetching one > posting > >>> list. With 1500 fields, it is fetching 1500 posting lists. It could > easily > >>> be 1500 times slower. > >>> > >>> It might be even slower than that, because we can’t guarantee that: a) > >>> every algorithm in Solr is linear, b) that all those lists will fit in > >>> memory. > >>> > >>> wunder > >>> Walter Underwood > >>> wun...@wunderwood.org > >>> http://observer.wunderwood.org/ (my blog) > >>> > >>> > >>>> On Nov 19, 2015, at 3:46 PM, Steven White <swhite4...@gmail.com> > wrote: > >>>> > >>>> Hi everyone > >>>> > >>>> What is considered too many fields for qf and fq? On average I will > have > >>>> 1500 fields in qf and 100 in fq (all of which are OR'ed). Assuming I > can > >>>> (I have to check with the design) for qf, if I cut it down to 1 field, > >>> will > >>>> I see noticeable performance improvement? It will take a lot of > effort > >>> to > >>>> test this which is why I'm asking first. > >>>> > >>>> As is, I'm seeing 2-5 sec response time for searches on an index of 1 > >>>> million records with total index size (on disk) of 4 GB. I gave Solr > 2 > >>> GB > >>>> of RAM (also tested at 4 GB) in both cases Solr didn't use more then 1 > >>> GB. > >>>> > >>>> Thanks in advanced > >>>> > >>>> Steve > >>> > >>> > > >
