Hi Mihai, Yes I think I see. You probably had those event dead letters present in 3.7.4 when it was still Elasticsearch. Then you migrated to 3.8, where James migrated from Elasticsearch to Opensearch, and the group name of those events changed to something else.
The recurrent problem with the event dead letter api is that when a group name changes, if you still have events with the older name, the system does not understand where it comes from anymore and you have the ClassNotFound errors you experienced. Maybe at this stage the best is to delete those events. Can you try to delete all events of that group, like this: https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#deleting-all-events-of-a-group ? I checked the code quickly, there does not seem to be trying a deserialization for the delete, might solve the problem (of course the dead letter events might be lost as well then). We could clearly do something better here with this API, like rely on something else than the class path of the group class as a group (which breaks everytime there is a path change), or not being so strict in some of those methods... Hope this helps you a bit though. Best regards, Rene. On Jun 5, 2023 5:24 PM, from Mihai Zamfira Hi, I recently upgraded from 3.7.4 to james 3.8 . The server reports warnings like below "[WARN ] o.a.j.PeriodicalHealthChecks - DEGRADED: EventDeadLettersHealthCheck : EventDeadLetters contain events. This might indicate transient failure on event processing” When I try to see the details of the events using the webadmin API "curl -XGET http: //ip:port/events/deadLetter/groups" I get the following error as a response { "statusCode": 500, "type": "ServerError", "message": "WebAdmin encountered an unexpected internal error", "details": null } >From the java trace I cannot understand why it's trying to use elasticsearch >when this is a new opensearch installation. 13:13:57.700 [INFO ] o.a.j.w.m.LoggingRequestFilter - WebAdmin request received 13:13:57.712 [ERROR] s.h.m.GeneralError - java.lang.ClassNotFoundException: org.apache.james.mailbox.elasticsearch.v7.events.ElasticSearchListeningMessageSearchIndex$ElasticSearchListeningMessageSearchIndexGroup at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) at org.apache.james.events.Group.deserialize(Group.java:48) ... 39 common frames omitted Wrapped by: org.apache.james.events.Group$GroupDeserializationException: java.lang.ClassNotFoundException: org.apache.james.mailbox.elasticsearch.v7.events.ElasticSearchListeningMessageSearchIndex$ElasticSearchListeningMessageSearchIndexGroup at org.apache.james.events.Group.deserialize(Group.java:51) at org.apache.james.events.CassandraEventDeadLettersGroupDAO.lambda$retrieveAllGroups$0(CassandraEventDeadLettersGroupDAO.java:79) at com.github.fge.lambdas.functions.FunctionChainer.doApply(FunctionChainer.java:20) at com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:17) ... 36 common frames omitted Wrapped by: com.github.fge.lambdas.ThrownByLambdaException: org.apache.james.events.Group$GroupDeserializationException: java.lang.ClassNotFoundException: org.apache.james.mailbox.elasticsearch.v7.events.ElasticSearchListeningMessageSearchIndex$ElasticSearchListeningMessageSearchIndexGroup at com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:21) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106) at com.datastax.dse.driver.internal.core.cql.reactive.ReactiveResultSetSubscription.doOnNext(ReactiveResultSetSubscription.java:364) at com.datastax.dse.driver.internal.core.cql.reactive.ReactiveResultSetSubscription.drain(ReactiveResultSetSubscription.java:236) at com.datastax.dse.driver.internal.core.cql.reactive.ReactiveResultSetSubscription.lambda$fetchNextPageAndEnqueue$2(ReactiveResultSetSubscription.java:358) at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.setFinalResult(CqlRequestHandler.java:324) at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.access$1500(CqlRequestHandler.java:95) at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler$NodeResponseCallback.onResponse(CqlRequestHandler.java:655) at com.datastax.oss.driver.internal.core.channel.InFlightHandler.channelRead(InFlightHandler.java:257) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) Suppressed: java.lang.Exception: #block terminated with an error at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) at reactor.core.publisher.Mono.block(Mono.java:1710) at org.apache.james.webadmin.service.EventDeadLettersService.listGroupsAsStrings(EventDeadLettersService.java:52) at org.apache.james.webadmin.routes.EventDeadLettersRoutes.listGroups(EventDeadLettersRoutes.java:90) at spark.ResponseTransformerRouteImpl$1.handle(ResponseTransformerRouteImpl.java:47) at spark.http.matching.Routes.execute(Routes.java:61) at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:134) at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1598) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) Any help is appreciated, thank you!
