Despite docvalues provide NRT faceting with a great performance (since 5.4), enum method is still really important for edge cases (many docs, small num of terms). Also, Solr's UnIvertedField had a really smart BigTerms strategy, when fattest terms were counted by enum and remaining ones with fc. DocValues needs to evolve much until it's capable to provide such level of flexibility. I might confuse something again, but I a little bit worry about existence BigTerms/UnIvertedField in recent versions.
On Wed, Sep 23, 2015 at 12:44 PM, Alessandro Benedetti < benedetti.ale...@gmail.com> wrote: > Well done Mikhail, > curious to see the performance! > > Apart the disk usage ( of course building docValues will cost more space), > taking in consideration the Field cardinality, in the past when the field > cardinality was low ( few unique values in the field), the enum approach > was suggested ( so DocValues were non necessary strictly talking about > faceting). > > After this improvement, in your opinion, what is the current status ? after > the 5.4 release, will enum faceting approach make sense anymore ? > Has anyone benchmarked this ? > > Cheers > > 2015-09-23 9:02 GMT+01:00 Mikhail Khludnev <mkhlud...@griddynamics.com>: > > > Uwe, > > > > I'm sorry for confusion https://issues.apache.org/jira/browse/SOLR-7730 > > goes in 5.4 only. Hence, to get fast DV facets you need to apply patch > > (it's pretty small). > > Accelerating non-DV facets is not so clear so far. Please show profiler > > snapshot for non-DV facets if you wish to go this way. > > > > > > On Tue, Sep 22, 2015 at 5:36 PM, Uwe Reh <r...@hebis.uni-frankfurt.de> > > wrote: > > > > > The exact version as shown by the UI is: > > > - solr-impl 5.3.0 1696229 - noble - 2015-08-17 17:10:43 > > > - lucene-impl 5.3.0 1696229 - noble - 2015-08-17 16:59:03 > > > > > > Unfortunately my skills in debugging are limited. So I'm not sure > about a > > > 'deeper caller stack'. > > > Did you mean the attached snapshot from VirtualVM, a stack trace like > > > below or something else? Please give me a hint. > > > > > > uwe > > > > > > "qtp1734853116-68" #68 prio=5 os_prio=64 tid=0x00000000117fd800 > nid=0x77 > > >> runnable [0xfffffd7f991fc000] > > >> java.lang.Thread.State: RUNNABLE > > >> at java.util.HashMap.resize(HashMap.java:734) > > >> at java.util.HashMap.putVal(HashMap.java:662) > > >> at java.util.HashMap.put(HashMap.java:611) > > >> at > > >> > > > org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:344) > > >> at > > >> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:366) > > >> at > > >> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:304) > > >> at > > >> > > > org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:245) > > >> at > > >> > > > org.apache.lucene.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:237) > > >> at > > >> > > > org.apache.lucene.index.SlowCompositeReaderWrapper.getSortedSetDocValues(SlowCompositeReaderWrapper.java:174) > > >> at > > >> > > > org.apache.solr.request.DocValuesFacets.getCounts(DocValuesFacets.java:72) > > >> at > > >> > > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:492) > > >> at > > >> > > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:385) > > >> at > > >> org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:628) > > >> at > > >> org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:619) > > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > >> at > > >> org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:573) > > >> at > > >> > > > org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:644) > > >> at > > >> > > > org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:294) > > >> at > > >> > > > org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:256) > > >> at > > >> > > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:285) > > >> at > > >> > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) > > >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068) > > >> at > > >> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669) > > >> at > > >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462) > > >> at > > >> > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210) > > >> at > > >> > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179) > > >> at > > >> > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) > > >> at > > >> > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > > >> at > > >> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > >> at > > >> > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) > > >> at > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) > > >> at > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > > >> at > > >> > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > > >> at > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > > >> at > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > > >> at > > >> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > >> at > > >> > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) > > >> at > > >> > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > > >> at > > >> > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > > >> at org.eclipse.jetty.server.Server.handle(Server.java:499) > > >> at > > >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > > >> at > > >> > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > > >> at org.eclipse.jetty.io > > >> .AbstractConnection$2.run(AbstractConnection.java:540) > > >> at > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > > >> at > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > > >> at java.lang.Thread.run(Thread.java:745) > > >> > > > > > > > > > > > > Am 22.09.2015 um 12:56 schrieb Mikhail Khludnev: > > > > > >> It's quite strange > > >> https://issues.apache.org/jira/browse/SOLR-7730 significantly > optimized > > >> DV > > >> facets at 5.3.0 exactly by avoiding FileInfo merge. > > >> Would you mind to provide deeper caller stack for > > >> > org.apache.lucene.index.FileInfos.MultibleFields.getMergedFieldInfos()? > > >> Or a time spend in SlowCompositeReaderWrapper, DocValuesFacets, > > >> MultiDocValues and their hot methods. > > >> Which version you exactly on? and how do you know that? > > >> Thanks > > >> > > >> > > > > > > > > > -- > > Sincerely yours > > Mikhail Khludnev > > Principal Engineer, > > Grid Dynamics > > > > <http://www.griddynamics.com> > > <mkhlud...@griddynamics.com> > > > > > > -- > -------------------------- > > Benedetti Alessandro > Visiting card - http://about.me/alessandro_benedetti > Blog - http://alexbenedetti.blogspot.co.uk > > "Tyger, tyger burning bright > In the forests of the night, > What immortal hand or eye > Could frame thy fearful symmetry?" > > William Blake - Songs of Experience -1794 England > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
