The cluster wants to transport an exception to your web app container, and the web app does not have access to elasticsearch jar.
You should have a look at the ES server logs, if there are any exceptions, to find the real problem. Then, after fixing the real problem, you should try to configure your web app so the elasticsearch jar is in the classpath. To your question about WebAppClassLoader: ES uses the thread context class loader https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/Classes.java#L56 Also, do you use any ES plugins? If plugins throw exceptions, they must also be available in the web app. Jörg On Tue, Aug 5, 2014 at 4:57 PM, Earle Nietzel <earle.niet...@gmail.com> wrote: > Using elasticsearch embedded in tomcat 7 where we have custom classloader > that shares spring application beans with many webapps. The API's to these > implementations are in shared but the implementations are in a separate > classloader "ComponentLoader". Our search implementation is loaded from > ComponentLoader where elasticsearch has been promoted as our default search > engine. > > Everything works fine on a single node but when in a cluster seeing the > following ClassNotFoundException issue when shards are trying to update on > other nodes. > > Caused by: java.lang.ClassNotFoundException: > org.elasticsearch.transport.RemoteTransportException > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702) > > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547) > > at > org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93) > > at > org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67) > > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) > > I thought this might have been related to > https://github.com/elasticsearch/elasticsearch/issues/4634 but that > doesn't seem to be the case. > > My next idea is to look at why > org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is > invoking WebappClassLoader instead of our custom ComponentLoader and coerce > it to use ours. > > But I wanted to get some opinions on this strategy as I am new to > elasticsearch ;) > > my thanks, > Earle > > > Full stack trace: > > 2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1] > org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to > start shard > org.elasticsearch.indices.recovery.RecoveryFailedException: > [sakai_index][0]: Recovery failed from > [app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false} > into > [app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false} > > at > org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307) > > at > org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65) > > at > org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to > deserialize exception response from stream > Caused by: org.elasticsearch.transport.TransportSerializationException: > Failed to deserialize exception response from stream > at > org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169) > > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) > > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) > > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) > > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.InvalidClassException: failed to read class descriptor > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167) > > ... 23 more > Caused by: java.lang.ClassNotFoundException: > org.elasticsearch.transport.RemoteTransportException > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702) > > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547) > > at > org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93) > > at > org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67) > > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) > ... 28 more > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/17542a40-7b6c-4c31-9e3e-1fdd2fe7f90b%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/17542a40-7b6c-4c31-9e3e-1fdd2fe7f90b%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFrqmCV70PUab5oicBDfygO6DnFasmgJyZ3Q6PtoMCCnQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
