On Tuesday, March 19, 2019 at 11:35:47 AM UTC-4, Alan Orth wrote: > > Dear list, > > In the last few months we've been having issues with Solr throwing "Error > creating core" errors one out of every two times we start Tomcat. This > results in our statistics from previous years being inaccessible. For > example, from the Solr log yesterday: > > 2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error > creating core [statistics-2018]: Error opening new searcher > ... > Caused by: org.apache.solr.common.SolrException: Error opening new > searcher > at > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677) > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845) > ... 31 more > Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain > timed out: NativeFSLock@/dspace/solr/statistics-2018/data/index/write.lock > > Some old Stack Overflow posts recommend setting Solr's address space to > unlimited (ulimit -v unlimited), but the issue still occurs seemingly > randomly for us. >
I think that raising ulimit would only help if Tomcat's JVM memory settings add up to more than the OS was configured to give the process. Those recommendations may have been for stand-alone Solr running in its own Jetty instance. > Now I usually just try to restart Tomcat again, or sometimes I shut it > down cleanly, delete all the Solr write locks, and then start Tomcat back > up. It's starting to feel a bit superstitious... maybe I should try to kill > a chicken. > > We are running DSpace 5.8 with Tomcat 7.0.93 on Ubuntu 16.04. We upgraded > from DSpace 5.5 in late 2018 and 2019 was the first year that the yearly > stats-util sharding completed successfully (fixed in DSpace 5.7). I believe > our problems are related to the existence of these shards. It feels like > there is some kind of *race condition* because it only every so often and > it's not always the same core that Solr is refusing to create. Sometimes > it's statistics-2018, statistics-2015, etc. > So, sometimes Solr doesn't remove all of its locks when shut down. Does it log ([DSpace]log/solr*.log) anything interesting at shutdown? does Tomcat? > On this note, is there any reason we are still using Solr 4.10.2 with > DSpace 5.x and 6.x? The Solr project issued two bug fix releases in that > series—4.10.3 and 4.10.4—and there are about fifty bug fixes in those > releases, some of which address memory leaks and shard handling. See the > change logs for 4.10.3¹ and 4.10.4². As an experiment I just bumped the > version to 4.10.3 in my test environment and DSpace starts up... so that's > a good sign. I will do more testing of ingests, indexing, etc and report > back. > > I think the reason is that nobody went looking for newer releases. Your experiment is definitely worth trying, and your results will be interesting. -- All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
