FWIW, we now have a hypothetical suspect. We are getting these errors on three CentOS7 hosts, each of which recently had antivirus software installed.
-----Original Message----- From: Oakley, Craig (NIH/NLM/NCBI) [C] [mailto:craig.oak...@nih.gov] Sent: Thursday, May 11, 2017 11:03 AM To: solr-user@lucene.apache.org Subject: RE: Underlying file changed by an external force None of them have dataDir properties: they just use the "data" subdirectory in the same directory as the core.properties -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, May 10, 2017 6:59 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Underlying file changed by an external force bq: All the core.properties files are each in their own directory with no overlap Not quite what I was asking. By definition, all core.properties are in their own directory. In fact Solr stops looking down the tree when it finds the first directory with core.properties in it and immediately moves on to the next sibling directory. _Inside_ the core.properties files, are there any dataDir properties pointing to the same place as in any other core.properties? Note, dataDir properties usually aren't even present unless you did something special so don't be surprised if there's nothing there. Best, Erick On Wed, May 10, 2017 at 10:56 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <craig.oak...@nih.gov> wrote: >> You need to look at all of your core.properties files and see if any of them >> point to the same data directory. > > All the core.properties files are each in their own directory with no overlap. > >> Second: if you issue a "kill -9" you can leave write locks lingering. > > We manage our Solr instances with supervisor, which can send a "kill -9" if > "kill -6" does not suffice; but the problem tends to manifest itself at some > time other than startup > > The Solr version is 5.4.1, in case that is relevant. > > > -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Thursday, May 04, 2017 3:20 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Underlying file changed by an external force > > You need to look at all of your core.properties files and see if any > of them point to the same data directory. > > Second: if you issue a "kill -9" you can leave write locks lingering. > > Best, > Erick > > On Thu, May 4, 2017 at 11:00 AM, Oakley, Craig (NIH/NLM/NCBI) [C] > <craig.oak...@nih.gov> wrote: >> We have been having problems with different collections on different >> SolrCloud clusters, all seeming to be related to the write.lock file with >> stack traces similar to the following. Are there any suggestions what might >> be the cause and what might be the solution? Thanks >> >> >> org.apache.lucene.store.AlreadyClosedException: Underlying file changed by >> an external force at 2017-04-13T20:43:08.630152Z, >> (lock=NativeFSLock(path=/data/solr/biosample/dba_test_shard1_replica1/data/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 >> exclusive valid],ctime=2017-04-13T20:43:08.630152Z)) >> >> at >> org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) >> >> at >> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37) >> >> at >> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:732) >> >> at >> org.apache.lucene.index.IndexFileDeleter.deletePendingFiles(IndexFileDeleter.java:503) >> >> at >> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:448) >> >> at >> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2099) >> >> at >> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2041) >> >> at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1083) >> >> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1125) >> >> at >> org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:131) >> >> at >> org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:183) >> >> at >> org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:207) >> >> at org.apache.solr.core.SolrCore.reload(SolrCore.java:472) >> >> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:849) >> >> at >> org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:768) >> >> at >> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230) >> >> at >> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184) >> >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156) >> >> at >> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664) >> >> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438) >> >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223) >> >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) >> >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) >> >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> >> at >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) >> >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) >> >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) >> >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) >> >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >> >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> >> at >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) >> >> at >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) >> >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >> >> at org.eclipse.jetty.server.Server.handle(Server.java:499) >> >> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) >> >> at >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) >> >> at >> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >> >> at java.lang.Thread.run(Thread.java:745) >>
