[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831022#comment-16831022
]
Gus Heck commented on SOLR-8291:
--------------------------------
Just hit this recently. [~ronnnnnnn]'s explanation seems to jibe with what I
was finding as I prowled the code. He found the spot where the bitsets were
left null which I hadn't found yet, so kudos and thanks to him there :). It
looks like there have been no major changes to the logic of the code in
sortDocSet since LUCENE-2831 in 2011, and the export handler code seems to only
date back to 2014, so the assumptions in ExportQParserPlugin don't seem to
match the previously established behavior of not calling getLeafCollector after
the last matching doc. Given the history, I'd say that the assumption by
ExportQParserPlugin that it can create the array of bit sets in advance based
on the leaf count is the underlying flaw.
{code:java}
public TopDocsCollector getTopDocsCollector(int len,
QueryCommand cmd,
IndexSearcher searcher) throws
IOException {
int leafCount = searcher.getTopReaderContext().leaves().size();
FixedBitSet[] sets = new FixedBitSet[leafCount];
return new ExportCollector(sets);
} {code}
This makes the problem specific to the Export handler code path, and general
search path stuff like sortDocSet() probably should be left alone. A quick
browse into half a dozen other TopDocsCollectors don't show any use of the leaf
count similar to this, but I did not do an exhaustive search.
Two fixes come to mind within the export handler code, the patch supplied by
Ron, tweaked to add the null check to the loop condition, or an else block
break to avoid useless iterations. The second is for ExportQParserPlugin to
populate with empty bitsets, which is unattractive for memory reasons, but
solves the problem at the source.
The first solution (a version of Ron's patch) amounts to ExportWriter being
more tolerant in what it accepts and short circuiting earlier, so I think
that's the preferable path vs creating more objects for GC. I'll be testing
such a patch on the system that exposed this for me soon.
> NPE calling export handler when useFilterForSortedQuery=true
> ------------------------------------------------------------
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 5.2.1
> Reporter: Ray
> Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> <useFilterForSortedQuery>true</useFilterForSortedQuery>
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.<init>(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
