[ https://issues.apache.org/jira/browse/SOLR-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shawn Heisey resolved SOLR-16107. --------------------------------- Resolution: Invalid Jira is not a support portal. > No Space left on the device exception > ------------------------------------- > > Key: SOLR-16107 > URL: https://issues.apache.org/jira/browse/SOLR-16107 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ > Affects Versions: 6.6.5 > Environment: RHEL 3.10.0-957.12.2.el7.x86_64 > java.version:1.8.0_282 > OpenJDK 64-Bit Server VM (25.282-b08) for linux-amd64 JRE (1.8.0_282-b08) > sitecore version - 8.2 update 7/solr 6.6.5 > AWS EC2 Solr Master/Slave1/Slave2 > > Reporter: Pravinkumar Bhagat > Priority: Major > Labels: RHE, aws, file-system, indexing, storage > Attachments: solr.log > > > Getting "No Space left on the device" in logs and it reaches full disk size > for solr Master instance after couple of weeks. > Common Exceptions seen in logs below:- > 1> Exception writing document id > sitecore://master/\{f7283eb0-515e-4cfc-b851-b398dfb9206c}?lang=fr-ca&ver=1&ndx=sitecore_master_index > to the index; possible analysis error. > 2> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > 3> java.io.IOException: No space left on device > 4> org.apache.lucene.index.CorruptIndexException > Already increased disk size couple of times ,initially we were having 150 GB > then we got the error so we increased it to 250 GB then again we got the same > error and now its 500GB for Solr AWS EC2 Master Server having linux os. When > solr goes out of space , we have seen excessive logging with log files data > around 20 to 30 GB. > Every time when this happens we are also facing index data corrruption and > needed to delete solr logs and indexes data folder and restart solr server > followed by rebuilding whole indexes again fixes the problem. > Configuration:- > Sitecore 8.2 Update 7 > Solr 6.6.5 > stand alone replication setup having One Solr Master and 2 Slave instances > Solr hosted on AWS EC2 boxes having RHE linux OS with 4 processors and 16 GB > RAM , heap size 3 GB [JVM Memory]. > Master solr instance: disk space : 500 GB > root directory size[/tmp] : 81 GB > Slave1 solr instance: disk space: 250 GB > Slave2 solr instance: disk Space: 250 GB > Max size of one of index is around 20 GB to 25 GB > Total 25,00000 documents > Daily Import Utility scheduller create/update approx. 50 to 200 items > 2022-03-10 14:06:11.537 ERROR (qtp232824863-2108) [ > x:sitecore_master_index] o.a.s.h.RequestHandlerBase > org.apache.solr.common.SolrException: Exception writing document id > sitecore://master/\{f7283eb0-515e-4cfc-b851-b398dfb9206c}?lang=fr-ca&ver=1&ndx=sitecore_master_index > to the index; possible analysis error. > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206) > at > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748) > at > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > at > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > at org.eclipse.jetty.server.Server.handle(Server.java:534) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter > is closed > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749) > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1567) > at > org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924) > at > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913) > at > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302) > at > org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239) > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194) > ... 42 more > Caused by: java.io.IOException: No space left on device > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > at java.nio.channels.Channels.writeFully(Channels.java:101) > at java.nio.channels.Channels.access$000(Channels.java:61) > at java.nio.channels.Channels$1.write(Channels.java:174) > at > org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419) > at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126) > at > org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53) > at > org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73) > at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52) > at org.apache.lucene.codecs.lucene50.ForUtil.writeBlock(ForUtil.java:175) > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:237) > at > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:141) > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:866) > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101) > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) > at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org