[
https://issues.apache.org/jira/browse/SOLR-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6065:
---------------------------
Description:
LUCENE-5843 recently fixed IndexWriter to throw a specific exception in the
event that an index grows too big, preventing people from creating indexes they
can't later search. That issue also added a new test-only setter that can be
used to lower the limit for hte purposes of testing.
For the Solr side of things, we should primarily focus on making sure we have
some cloud based tests that set the lower limit and then verify that users get
clean error messages when they exceed it for a single shard.
{panel:title=original bug report}
yamazaki reported an error on solr-user where, on opening a new searcher, he
got an IAE from BaseCompositeReader because the numDocs was greater then
Integer.MAX_VALUE.
I'm surprised that in a straight forward setup (ie: no "AddIndex" merging)
IndexWriter will even let you add more docs then max int. We should
investigate if this makes sense and either add logic in IndexWriter to prevent
this from happening, or add logic to Solr's UpdateHandler to prevent things
from getting that far.
ie: we should be failing to "add" too many documents, and leaving the index
usable -- not accepting the add and leaving hte index in an unusable state.
stack trace reported by user...
{noformat}
ERROR org.apache.solr.core.CoreContainer – Unable to create core: collection1
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
at
org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:796)
... 13 more
Caused by: org.apache.solr.common.SolrException: Error opening Reader
at
org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
at
org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
at
org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1414)
... 15 more
Caused by: java.lang.IllegalArgumentException: Too many documents,
composite IndexReaders cannot exceed 2147483647
at
org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:368)
at
org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:42)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:71)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
at
org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
... 18 more
{noformat}
{panel}
was:
yamazaki reported an error on solr-user where, on opening a new searcher, he
got an IAE from BaseCompositeReader because the numDocs was greater then
Integer.MAX_VALUE.
I'm surprised that in a straight forward setup (ie: no "AddIndex" merging)
IndexWriter will even let you add more docs then max int. We should
investigate if this makes sense and either add logic in IndexWriter to prevent
this from happening, or add logic to Solr's UpdateHandler to prevent things
from getting that far.
ie: we should be failing to "add" too many documents, and leaving the index
usable -- not accepting the add and leaving hte index in an unusable state.
stack trace reported by user...
{noformat}
ERROR org.apache.solr.core.CoreContainer – Unable to create core: collection1
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
at
org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:796)
... 13 more
Caused by: org.apache.solr.common.SolrException: Error opening Reader
at
org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
at
org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
at
org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1414)
... 15 more
Caused by: java.lang.IllegalArgumentException: Too many documents,
composite IndexReaders cannot exceed 2147483647
at
org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:368)
at
org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:42)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:71)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
at
org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
... 18 more
{noformat}
Summary: Solr should give you clear error if you try to add too many
docs (was: Solr / IndexWriter should prevent you from adding docs if it
creates an index to big to open)
> Solr should give you clear error if you try to add too many docs
> ----------------------------------------------------------------
>
> Key: SOLR-6065
> URL: https://issues.apache.org/jira/browse/SOLR-6065
> Project: Solr
> Issue Type: Bug
> Reporter: Hoss Man
>
> LUCENE-5843 recently fixed IndexWriter to throw a specific exception in the
> event that an index grows too big, preventing people from creating indexes
> they can't later search. That issue also added a new test-only setter that
> can be used to lower the limit for hte purposes of testing.
> For the Solr side of things, we should primarily focus on making sure we have
> some cloud based tests that set the lower limit and then verify that users
> get clean error messages when they exceed it for a single shard.
> {panel:title=original bug report}
> yamazaki reported an error on solr-user where, on opening a new searcher, he
> got an IAE from BaseCompositeReader because the numDocs was greater then
> Integer.MAX_VALUE.
> I'm surprised that in a straight forward setup (ie: no "AddIndex" merging)
> IndexWriter will even let you add more docs then max int. We should
> investigate if this makes sense and either add logic in IndexWriter to
> prevent this from happening, or add logic to Solr's UpdateHandler to prevent
> things from getting that far.
> ie: we should be failing to "add" too many documents, and leaving the index
> usable -- not accepting the add and leaving hte index in an unusable state.
> stack trace reported by user...
> {noformat}
> ERROR org.apache.solr.core.CoreContainer – Unable to create core: collection1
> org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
> at
> org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
> at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
> at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:796)
> ... 13 more
> Caused by: org.apache.solr.common.SolrException: Error opening Reader
> at
> org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
> at
> org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
> at
> org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1414)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Too many documents,
> composite IndexReaders cannot exceed 2147483647
> at
> org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
> at
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:368)
> at
> org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:42)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:71)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
> at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
> at
> org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
> ... 18 more
> {noformat}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
