[
https://issues.apache.org/jira/browse/SOLR-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509155#comment-17509155
]
Shawn Heisey commented on SOLR-16107:
-------------------------------------
This looks like a support request. For the Solr project, Jira is not a support
portal. It is a bug tracker and a place to track implementation of new
features. Nobody has determined that there is a bug that needs fixing or a new
feature that needs implementing, so this is out of place in Jira. I will be
closing this issue. For support, we have mailing lists, a slack workspace, and
an IRC channel.
If you run out of disk space, anything can happen, including index corruption.
Lucene (the search library that Solr uses) always recommends that you should
have enough disk space available to hold three times your index size, because
3x the index on a temporary basis is sometimes possible depending on the exact
update pattern being employed. Most people only ever see 2x the index size,
but 3x CAN happen.
There's no generic guide that will tell you how much disk space you need. It
is highly dependent on the schema being employed, the nature of the data being
indexed, and exactly how you do your indexing. If anyone can tell you how big
a Solr index for sitecore can get, it will be the sitecore project.
> No Space left on the device exception
> -------------------------------------
>
> Key: SOLR-16107
> URL: https://issues.apache.org/jira/browse/SOLR-16107
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
> Affects Versions: 6.6.5
> Environment: RHEL 3.10.0-957.12.2.el7.x86_64
> java.version:1.8.0_282
> OpenJDK 64-Bit Server VM (25.282-b08) for linux-amd64 JRE (1.8.0_282-b08)
> sitecore version - 8.2 update 7/solr 6.6.5
> AWS EC2 Solr Master/Slave1/Slave2
>
> Reporter: Pravinkumar Bhagat
> Priority: Major
> Labels: RHE, aws, file-system, indexing, storage
> Attachments: solr.log
>
>
> Getting "No Space left on the device" in logs and it reaches full disk size
> for solr Master instance after couple of weeks.
> Common Exceptions seen in logs below:-
> 1> Exception writing document id
> sitecore://master/\{f7283eb0-515e-4cfc-b851-b398dfb9206c}?lang=fr-ca&ver=1&ndx=sitecore_master_index
> to the index; possible analysis error.
> 2> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> 3> java.io.IOException: No space left on device
> 4> org.apache.lucene.index.CorruptIndexException
> Already increased disk size couple of times ,initially we were having 150 GB
> then we got the error so we increased it to 250 GB then again we got the same
> error and now its 500GB for Solr AWS EC2 Master Server having linux os. When
> solr goes out of space , we have seen excessive logging with log files data
> around 20 to 30 GB.
> Every time when this happens we are also facing index data corrruption and
> needed to delete solr logs and indexes data folder and restart solr server
> followed by rebuilding whole indexes again fixes the problem.
> Configuration:-
> Sitecore 8.2 Update 7
> Solr 6.6.5
> stand alone replication setup having One Solr Master and 2 Slave instances
> Solr hosted on AWS EC2 boxes having RHE linux OS with 4 processors and 16 GB
> RAM , heap size 3 GB [JVM Memory].
> Master solr instance: disk space : 500 GB
> root directory size[/tmp] : 81 GB
> Slave1 solr instance: disk space: 250 GB
> Slave2 solr instance: disk Space: 250 GB
> Max size of one of index is around 20 GB to 25 GB
> Total 25,00000 documents
> Daily Import Utility scheduller create/update approx. 50 to 200 items
> 2022-03-10 14:06:11.537 ERROR (qtp232824863-2108) [
> x:sitecore_master_index] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: Exception writing document id
> sitecore://master/\{f7283eb0-515e-4cfc-b851-b398dfb9206c}?lang=fr-ca&ver=1&ndx=sitecore_master_index
> to the index; possible analysis error.
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
> at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)
> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter
> is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1567)
> at
> org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924)
> at
> org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913)
> at
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
> ... 42 more
> Caused by: java.io.IOException: No space left on device
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
> at java.nio.channels.Channels.writeFully(Channels.java:101)
> at java.nio.channels.Channels.access$000(Channels.java:61)
> at java.nio.channels.Channels$1.write(Channels.java:174)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419)
> at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> at
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
> at
> org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
> at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
> at org.apache.lucene.codecs.lucene50.ForUtil.writeBlock(ForUtil.java:175)
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:237)
> at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:141)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:866)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
> at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
