jonasplaum opened a new issue, #5521: URL: https://github.com/apache/couchdb/issues/5521
## Description A running nouveau node stopped working. In the logs there is the following error for every request: ``` Apr 28 14:50:27 db02 sh[655]: ERROR [2025-04-28 12:50:27,945] io.dropwizard.jersey.errors.IllegalStateExceptionMapper: Error handling a request: 76ba79d0331ef673 Apr 28 14:50:27 db02 sh[655]: ! org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90DocValuesMetadata vs expected codec=Lucene90DocValuesData (resource=ByteBufferIndexInput(path="/HDD02/couchdb-nouveau/[email protected]/shards/e0000000-efffffff/evidence-2025-04.1738413941/63d1ee259368069b85f2525040a2174f3618ab304d554f624208e93f1bd886b0/index/9/_1xdx_Lucene90_0.dvd")) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.<init>(Lucene90DocValuesProducer.java:114) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat.fieldsProducer(Lucene90DocValuesFormat.java:154) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:305) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:374) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:52) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:69) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:197) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:113) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:179) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:286) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.openSegmentStates(IndexWriter.java:6479) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.forceApply(IndexWriter.java:6253) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.tryApply(IndexWriter.java:6187) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$12(IndexWriter.java:2818) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter$EventQueue.processEventsInternal(IndexWriter.java:338) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter$EventQueue.processEvents(IndexWriter.java:327) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:6030) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:586) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:144) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:52) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.lucene9.Lucene9Index.doSearch(Lucene9Index.java:206) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.core.Index.search(Index.java:81) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.resources.IndexResource.lambda$searchIndex$3(IndexResource.java:111) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.core.IndexManager.with(IndexManager.java:150) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.resources.IndexResource.searchIndex(IndexResource.java:110) Apr 28 14:50:27 db02 sh[655]: ! at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) Apr 28 14:50:27 db02 sh[655]: ! ... 70 common frames omitted Apr 28 14:50:27 db02 sh[655]: ! Causing: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:917) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:930) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.lucene.index.IndexWriter.getDocStats(IndexWriter.java:6517) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.lucene9.Lucene9Index.doNumDocs(Lucene9Index.java:126) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.core.Index.info(Index.java:48) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.resources.IndexResource.lambda$getIndexInfo$1(IndexResource.java:83) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.core.IndexManager.with(IndexManager.java:150) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.resources.IndexResource.getIndexInfo(IndexResource.java:82) Apr 28 14:50:27 db02 sh[655]: ! at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) Apr 28 14:50:27 db02 sh[655]: ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Apr 28 14:50:27 db02 sh[655]: ! at java.base/java.lang.reflect.Method.invoke(Method.java:569) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:379) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312) Apr 28 14:50:27 db02 sh[655]: ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:36) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) Apr 28 14:50:27 db02 sh[655]: ! at org.apache.couchdb.nouveau.core.UserAgentFilter.doFilter(UserAgentFilter.java:45) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.metrics.jetty11.InstrumentedHandler.handle(InstrumentedHandler.java:313) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) Apr 28 14:50:27 db02 sh[655]: ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.Server.handle(Server.java:563) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) Apr 28 14:50:27 db02 sh[655]: ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) Apr 28 14:50:27 db02 sh[655]: ! at java.base/java.lang.Thread.run(Thread.java:840) ``` Its always the same shard, so only this one shard of one database is impacted. ## Your Environment CouchDB 3.4.2 with nouveau running on debian12 using java 17 ## Additional Context A second nouveau server with the same data works fine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
