[ https://issues.apache.org/jira/browse/NUTCH-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1251: --------------------------------- Fix Version/s: (was: 1.5) 1.6 20120304-push-1.6 > Deletion of duplicates fails with > org.apache.solr.client.solrj.SolrServerException > ---------------------------------------------------------------------------------- > > Key: NUTCH-1251 > URL: https://issues.apache.org/jira/browse/NUTCH-1251 > Project: Nutch > Issue Type: Bug > Components: indexer > Affects Versions: 1.4 > Environment: Any crawl where the number of URLs in Solr exceeds 1024 > (the default max number of clusters in Lucene boolean query). > Reporter: Arkadi Kosmynin > Priority: Critical > Fix For: 1.6 > > > Deletion of duplicates fails. This happens because the "get all" query used > to get Solr index size is "id:[* TO *]", which is a range query. Lucene is > trying to expand it to a Boolean query and gets as many clauses as there are > ids in the index. This is too many in a real situation and it throws an > exception. > To correct this problem, change the "get all" query (SOLR_GET_ALL_QUERY) to > "\*:\*", which is the standard Solr "get all" query. > Indexing log extract: > java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error > executing query > at > org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:236) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) > Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing > query > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118) > at > org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:234) > ... 3 more > Caused by: org.apache.solr.common.SolrException: Internal Server Error > Internal Server Error > request: http://localhost:8081/arch/select?q=id:[* TO > *]&fl=id,boost,tstamp,digest&start=0&rows=82938&wt=javabin&version=2 > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430) > at > org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244) > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) > ... 5 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira