You can only put 2 billion documents in one core. This error message is the clue:
Too many documents, composite IndexReaders cannot exceed 2147483647 You will need to shard the collection. You might have multiple shards per node, but you will probably need 50-100 shards and lots of servers. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ On Feb 4, 2015, at 2:07 PM, Jack Krupansky <jack.krupan...@gmail.com> wrote: > What's your cluster size? The 2 billion limit is per-node. > > My personal recommendation is that you don't load more than 100 million > documents per node. You need to do a proof of concept test to verify > whether your particular data would support a higher number or not. > Ultimately, it will not be a matter of how many documents you can load, but > whether the query response latency for those documents is sufficient. > > -- Jack Krupansky > > On Wed, Feb 4, 2015 at 4:54 PM, Arumugam, Suresh <suresh.arumu...@emc.com> > wrote: > >> Hi All, >> >> >> >> We are trying to load 14+ Billion documents into Solr. But we are failing >> to load them into Solr. >> >> >> >> Solr version: *4.8.0* >> >> Analyzer used: *ClassicTokenizer for index as well as query.* >> >> >> >> Can someone help me in getting into the core of this issue? >> >> >> >> For 14+ Billion document load, we are loading 2Billion batches using the >> dataimport with single thread. >> >> >> >> First batch completed successfully & added 2 Billion >> documents >> >> Second batch, dataimport is showing as successful >> completion. But the no of documents is still 2 Billion with the following >> exception in the logs. >> >> >> >> *Screenshot of Dataimport handler:* >> >> >> >> >> >> *Exception:* >> >> >> >> 2/4/2015, 1:09:06 PM ERROR SolrWriter Exception while solr >> commit. >> >> org.apache.solr.common.SolrException: Error opening new searcher >> >> at >> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550) >> >> at >> org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662) >> >> at >> org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1406) >> >> at >> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:611) >> >> at >> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) >> >> at >> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) >> >> at >> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458) >> >> at >> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435) >> >> at >> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157) >> >> at >> org.apache.solr.handler.dataimport.SolrWriter.commit(SolrWriter.java:108) >> >> at >> org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:309) >> >> at >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:257) >> >> at >> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444) >> >> at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:485) >> >> at >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464) >> >> Caused by: java.lang.IllegalArgumentException: Too many documents, >> composite IndexReaders cannot exceed 2147483647 >> >> at >> org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77) >> >> at >> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:369) >> >> at >> org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:43) >> >> at >> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:117) >> >> at >> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385) >> >> at >> org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289) >> >> at >> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274) >> >> at >> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251) >> >> at >> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1461) >> >> >> >> >> Please let me know if we need more details on this. >> >> >> >> Thanks in advance. >> >> >> >> Regards, >> >> Suresh.A, >> >> EMC Corporation. >>
