Exception while loading 2 Billion + Documents in Solr 4.8.0

[Arumugam, Suresh]

Hi All,

We are trying to load 14+ Billion documents into Solr. But we are failing to 
load them into Solr.

Solr version: 4.8.0
Analyzer used: ClassicTokenizer for index as well as query.

Can someone help me in getting into the core of this issue?

For 14+ Billion document load, we are loading 2Billion batches using the 
dataimport with single thread.

                First batch completed successfully & added 2 Billion documents
                Second batch, dataimport is showing as successful completion. 
But the no of documents is still 2 Billion with the following exception in the 
logs.

Screenshot of Dataimport handler:

[cid:image001.jpg@01D04082.14887E00]

Exception:

2/4/2015, 1:09:06 PM     ERROR SolrWriter           Exception while solr commit.
org.apache.solr.common.SolrException: Error opening new searcher
                at 
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550)
                at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662)
                at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1406)
                at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:611)
                at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
                at 
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
                at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458)
                at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435)
                at 
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
                at 
org.apache.solr.handler.dataimport.SolrWriter.commit(SolrWriter.java:108)
                at 
org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:309)
                at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:257)
                at 
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444)
                at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:485)
                at 
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)
Caused by: java.lang.IllegalArgumentException: Too many documents, composite 
IndexReaders cannot exceed 2147483647
                at 
org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
                at 
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:369)
                at 
org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:43)
                at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:117)
                at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385)
                at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
                at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274)
                at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
                at 
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1461)

Please let me know if we need more details on this.

Thanks in advance.

Regards,
Suresh.A,
EMC Corporation.