Re: java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException
Hi Jörg,
I was able to resolve this by moving elasticsearch and its dependencies
into tomcats shared/lib. Since Tomcats shared classloader is always in the
chain of classloaders to search the ClassNotFoundException's are no more.
Though there shouldn't have been any reason for elasticsearch to not work
in the custom ComponentLoader, this did work in the 0.90.x series. We only
experienced these classloader issues when ES was sending shard data to
another node in the cluster which lead me to beleive that the Netty
Transport for some reason was not using the classloader from the original
ImmutableSettings.
Its my guess that the ImmutableSettings object was not present which is why
it chose to use use the Classes.getDefaultClassLoader().
Thanks for your direction was very helpful in figuring this out,
Earle
On Tuesday, August 5, 2014 11:39:28 AM UTC-4, Jörg Prante wrote:
>
> The cluster wants to transport an exception to your web app container, and
> the web app does not have access to elasticsearch jar.
>
> You should have a look at the ES server logs, if there are any exceptions,
> to find the real problem.
>
> Then, after fixing the real problem, you should try to configure your web
> app so the elasticsearch jar is in the classpath.
>
> To your question about WebAppClassLoader: ES uses the thread context class
> loader
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/Classes.java#L56
>
> Also, do you use any ES plugins? If plugins throw exceptions, they must
> also be available in the web app.
>
> Jörg
>
>
> On Tue, Aug 5, 2014 at 4:57 PM, Earle Nietzel > wrote:
>
>> Using elasticsearch embedded in tomcat 7 where we have custom classloader
>> that shares spring application beans with many webapps. The API's to these
>> implementations are in shared but the implementations are in a separate
>> classloader "ComponentLoader". Our search implementation is loaded from
>> ComponentLoader where elasticsearch has been promoted as our default search
>> engine.
>>
>> Everything works fine on a single node but when in a cluster seeing the
>> following ClassNotFoundException issue when shards are trying to update on
>> other nodes.
>>
>> Caused by: java.lang.ClassNotFoundException:
>> org.elasticsearch.transport.RemoteTransportException
>> at
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
>>
>>
>> at
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
>>
>>
>> at
>> org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
>>
>>
>> at
>> org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
>>
>>
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
>>
>> I thought this might have been related to
>> https://github.com/elasticsearch/elasticsearch/issues/4634 but that
>> doesn't seem to be the case.
>>
>> My next idea is to look at why
>> org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is
>> invoking WebappClassLoader instead of our custom ComponentLoader and coerce
>> it to use ours.
>>
>> But I wanted to get some opinions on this strategy as I am new to
>> elasticsearch ;)
>>
>> my thanks,
>> Earle
>>
>>
>> Full stack trace:
>>
>> 2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1]
>> org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to
>> start shard
>> org.elasticsearch.indices.recovery.RecoveryFailedException:
>> [sakai_index][0]: Recovery failed from
>> [app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false}
>>
>> into
>> [app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false}
>>
>>
>> at
>> org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
>>
>>
>> at
>> org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
>>
>>
>> at
>> org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException
Using elasticsearch embedded in tomcat 7 where we have custom classloader
that shares spring application beans with many webapps. The API's to these
implementations are in shared but the implementations are in a separate
classloader "ComponentLoader". Our search implementation is loaded from
ComponentLoader where elasticsearch has been promoted as our default search
engine.
Everything works fine on a single node but when in a cluster seeing the
following ClassNotFoundException issue when shards are trying to update on
other nodes.
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.transport.RemoteTransportException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
I thought this might have been related
to https://github.com/elasticsearch/elasticsearch/issues/4634 but that
doesn't seem to be the case.
My next idea is to look at why
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is
invoking WebappClassLoader instead of our custom ComponentLoader and coerce
it to use ours.
But I wanted to get some opinions on this strategy as I am new to
elasticsearch ;)
my thanks,
Earle
Full stack trace:
2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1]
org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to
start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[sakai_index][0]: Recovery failed from
[app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false}
into
[app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize exception response from stream
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.
