Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
On Tue, Nov 30, 2010 at 7:51 PM, Martin Grotzke martin.grot...@googlemail.com wrote: On Tue, Nov 30, 2010 at 3:09 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke martin.grot...@googlemail.com wrote: Still I'm wondering, why this issue does not occur with the plain example solr setup with 2 indexed docs. Any explanation? It's an old option you have in your solrconfig.xml that causes a different code path to be followed in Solr: !-- An optimization that attempts to use a filter to satisfy a search. If the requested sort does not include score, then the filterCache will be checked for a filter matching the query. If found, the filter will be used as the source of document ids, and then the sort will be applied to that. -- useFilterForSortedQuerytrue/useFilterForSortedQuery Most apps would be better off commenting that out or setting it to false. It only makes sense when a high number of queries will be duplicated, but with different sorts. Great, this sounds really promising, would be a very easy fix. I need to check this tomorrow on our test/integration server if changing this does the trick for us. I just verified this fix on our test/integration system and it works - cool! Thanx a lot for this hint, cheers, Martin
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Hmm this is in fact a regression. TopFieldCollector expects (but does not verify) that numHits is 0. I guess to fix this we could fix TopFieldCollector.create to return a NullCollector when numHits is 0. But: why is your app doing this? Ie, if numHits (rows) is 0, the only useful thing you can get is totalHits? Still I think we should fix it in Lucene -- it's a nuisance to push such corner case checks up into the apps. I'll open an issue... Mike On Mon, Nov 29, 2010 at 7:14 AM, Martin Grotzke martin.grot...@googlemail.com wrote: Hi, after an upgrade from solr-1.3 to 1.4.1 we're getting an ArrayIndexOutOfBoundsException for a query with rows=0 and a sort param specified: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84) at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) The query is e.g.: /select/?sort=popularity+descrows=0start=0q=foo When this is changed to rows=1 or when the sort param is removed the exception is gone and everything's fine. With a clean 1.4.1 installation (unzipped, started example and posted two documents as described in the tutorial) this issue is not reproducable. Does anyone have a clue what might be the reason for this and how we could fix this on the solr side? Of course - for a quick fix - I'll change our app so that there's no sort param specified when rows=0. Thanx cheers, Martin -- Martin Grotzke http://twitter.com/martin_grotzke
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
On Tue, Nov 30, 2010 at 10:29 AM, Michael McCandless luc...@mikemccandless.com wrote: Hmm this is in fact a regression. TopFieldCollector expects (but does not verify) that numHits is 0. I guess to fix this we could fix TopFieldCollector.create to return a NullCollector when numHits is 0. Fixing this in lucene/solr sounds good :-) Still I'm wondering, why this issue does not occur with the plain example solr setup with 2 indexed docs. Any explanation? But: why is your app doing this? Ie, if numHits (rows) is 0, the only useful thing you can get is totalHits? Actually I don't know this (yet). Normally our search logic should optimize this and ignore a requested sorting with rows=0, but there seems to be a case that circumvents this - still figuring out. Still I think we should fix it in Lucene -- it's a nuisance to push such corner case checks up into the apps. I'll open an issue... Just for the record, this is https://issues.apache.org/jira/browse/LUCENE-2785 One question: as leaving out sorting leads to better performance, this should also be true for rows=0. Or is lucene/solr already that clever that it makes this optimization (ignoring sort) automatically? Do I understand it correctly, that the solution with the null collector would make this optimiztion? We're just asking ourselves if we should go ahead and analyze and fix this in our app or wait for a patch for solr/lucene. What do you think? Is there s.th. like a timeframe when there's an agreement on the correct solution and a patch available? Thanx cheers, Martin Mike On Mon, Nov 29, 2010 at 7:14 AM, Martin Grotzke martin.grot...@googlemail.com wrote: Hi, after an upgrade from solr-1.3 to 1.4.1 we're getting an ArrayIndexOutOfBoundsException for a query with rows=0 and a sort param specified: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84) at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) The query is e.g.: /select/?sort=popularity+descrows=0start=0q=foo When this is changed to rows=1 or when the sort param is removed the exception is gone and everything's fine. With a clean 1.4.1 installation (unzipped, started example and posted two documents as described in the tutorial) this issue is not reproducable. Does anyone have a clue what might be the reason for this and how we could fix this on the solr side? Of course - for a quick fix - I'll change our app so that there's no sort param specified when rows=0. Thanx cheers, Martin -- Martin Grotzke http://twitter.com/martin_grotzke -- Martin Grotzke http://www.javakaffee.de/blog/
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke martin.grot...@googlemail.com wrote: Still I'm wondering, why this issue does not occur with the plain example solr setup with 2 indexed docs. Any explanation? It's an old option you have in your solrconfig.xml that causes a different code path to be followed in Solr: !-- An optimization that attempts to use a filter to satisfy a search. If the requested sort does not include score, then the filterCache will be checked for a filter matching the query. If found, the filter will be used as the source of document ids, and then the sort will be applied to that. -- useFilterForSortedQuerytrue/useFilterForSortedQuery Most apps would be better off commenting that out or setting it to false. It only makes sense when a high number of queries will be duplicated, but with different sorts. But: why is your app doing this? Ie, if numHits (rows) is 0, the only useful thing you can get is totalHits? Actually I don't know this (yet). Normally our search logic should optimize this and ignore a requested sorting with rows=0, but there seems to be a case that circumvents this - still figuring out. Still I think we should fix it in Lucene -- it's a nuisance to push such corner case checks up into the apps. I'll open an issue... Just for the record, this is https://issues.apache.org/jira/browse/LUCENE-2785 One question: as leaving out sorting leads to better performance, this should also be true for rows=0. Or is lucene/solr already that clever that it makes this optimization (ignoring sort) automatically? Solr has always special-cased this case and avoided sorting altogether (for the normal code path... but overlooked it when useFilterForSortedQuery=true. -Yonik http://www.lucidimagination.com
Re: ArrayIndexOutOfBoundsException for query with rows=0 and sort param
On Tue, Nov 30, 2010 at 3:09 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Tue, Nov 30, 2010 at 8:24 AM, Martin Grotzke martin.grot...@googlemail.com wrote: Still I'm wondering, why this issue does not occur with the plain example solr setup with 2 indexed docs. Any explanation? It's an old option you have in your solrconfig.xml that causes a different code path to be followed in Solr: !-- An optimization that attempts to use a filter to satisfy a search. If the requested sort does not include score, then the filterCache will be checked for a filter matching the query. If found, the filter will be used as the source of document ids, and then the sort will be applied to that. -- useFilterForSortedQuerytrue/useFilterForSortedQuery Most apps would be better off commenting that out or setting it to false. It only makes sense when a high number of queries will be duplicated, but with different sorts. Great, this sounds really promising, would be a very easy fix. I need to check this tomorrow on our test/integration server if changing this does the trick for us. Though, I just enabled useFilterForSortedQuery in the solr 1.4.1 example and tested rows=0 with a sort param, which doesn't fail - a correct/valid result is returned. Is there any condition that has to be met additionally to produce the error? One question: as leaving out sorting leads to better performance, this should also be true for rows=0. Or is lucene/solr already that clever that it makes this optimization (ignoring sort) automatically? Solr has always special-cased this case and avoided sorting altogether Great, good to know! Cheers, Martin
ArrayIndexOutOfBoundsException for query with rows=0 and sort param
Hi, after an upgrade from solr-1.3 to 1.4.1 we're getting an ArrayIndexOutOfBoundsException for a query with rows=0 and a sort param specified: java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.lucene.search.FieldComparator$StringOrdValComparator.copy(FieldComparator.java:660) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:84) at org.apache.solr.search.SolrIndexSearcher.sortDocSet(SolrIndexSearcher.java:1391) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:872) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) The query is e.g.: /select/?sort=popularity+descrows=0start=0q=foo When this is changed to rows=1 or when the sort param is removed the exception is gone and everything's fine. With a clean 1.4.1 installation (unzipped, started example and posted two documents as described in the tutorial) this issue is not reproducable. Does anyone have a clue what might be the reason for this and how we could fix this on the solr side? Of course - for a quick fix - I'll change our app so that there's no sort param specified when rows=0. Thanx cheers, Martin -- Martin Grotzke http://twitter.com/martin_grotzke
