Assuming this relates to the original thread
https://lists.apache.org/thread/58wjltm7h304g76xfnm640n37qo5bjhq
java.lang.IllegalArgumentException: Unknown directory:
> > > NRTCachingDirectory(MMapDirectory@
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92
> <mailto:
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
>
>
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > > at
> > org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
This looks like close might be called multiple times? At least this looks
like getCoreStatus is looking at CachingDirectoryFactory.release which then
fails.... I don't have an idea of why this might be, but could be an area
to look into.
Kevin Risden
On Mon, Feb 6, 2023 at 11:30 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<[email protected]> wrote:
> This error has happened yet again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Thursday, October 13, 2022 6:09 PM
> To: [email protected]
> Subject: RE: IllegalArgumentException: Unknown directory
>
> This error has happened again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <[email protected]>
> Sent: Friday, August 05, 2022 10:47 PM
> To: [email protected]
> Subject: RE: IllegalArgumentException: Unknown directory
>
> This error has happened again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Monday, May 02, 2022 2:29 PM
> To: [email protected]
> Subject: Re: IllegalArgumentException: Unknown directory
>
> This has happened several more times, and I have notice something else
> which might be a clue:
>
> The problem happened over the weekend; and the first erroneous complaint
> about
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> (as though it did not exist) occurred 17 minutes after
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties
> was updated. Last month, the problem occurred just 40 seconds after
> replication.properties was updated.
>
> What known connections might there be between replication.properties and
> snapshot_metadata?
>
> -----Original Message-----
> From: matthew sporleder <[email protected]>
> Sent: Monday, March 28, 2022 9:41 AM
> To: [email protected]
> Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> My only other guess (and I mean wild guess) is that because the error is
> for a *lock* that it is actually a timeout or similar locking error with a
> bad error message.
>
> On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <[email protected]> wrote:
>
> > Thanks for the quick reply
> >
> > grep -i -c OutOfMemory solr.log.20220* shows zero
> >
> > Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
> >
> > Let me know if you have any other suggestions
> >
> > Thanks again
> >
> > -----Original Message-----
> > From: matthew sporleder <[email protected]>
> > Sent: Friday, March 25, 2022 11:00 AM
> > To: [email protected]
> > Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> >
> >
> > Is there an OOM anywhere in that log? I've definitely seen java lose
> track
> > of things during a slow-moving oom.
> >
> > Also definitely check dmesg for anything in that same timeframe if you
> > still have the logs.
> >
> > On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> > <[email protected]> wrote:
> >
> > > I have a core which ceased responding either to select or to
> admin/core:
> > > restarting the Solr instance resolved the problem, but I am wondering
> > > whether there is some configuration which may need to be tweaked. Below
> > is
> > > a portion of solr.log from the time when the problem began. Please note
> > > that the directory
> > >
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > does indeed exist
> > >
> > > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.c.S.Request
> > > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> > >
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > > hits=94 status=0 QTime=2610
> > > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.r.SolrJmxReporter Closing reporter
> > > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName
> =
> > > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > > null, agent id = null] for registry
> > > solr.core.run_sel_cache.shard1.replica_n3 /
> > > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > > com.codahale.metrics.MetricRegistry@5211d667>
> > > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.c.S.Request
> > > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> > >
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > > hits=94 status=0 QTime=593
> > > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> > SKIPPED
> > > (unnecessary).
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > > listeners, but a live one attempted to unregister!
> > > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> > o.a.s.c.SolrCore
> > > java.lang.IllegalArgumentException: Unknown directory:
> > > NRTCachingDirectory(MMapDirectory@
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92
> <mailto:
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > > at
> > >
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > > at
> > org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > > at
> > > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > > at
> > >
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > > at
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at
> > >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > > at
> > >
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > > at
> > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > > at
> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > > at
> > >
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > > at org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > > at org.eclipse.jetty.io
> > > .FillInterest.fillable(FillInterest.java:103)
> > > at org.eclipse.jetty.io
> > > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> > with
> > > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > > params={action=clusterstatus&wt=json} status=0 QTime=13
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > > listeners, but a live one attempted to unregister!
> > >
> >
>
