I am trying to use Spark 3.5.3 in an old CDH cluster. The previous version
on this CDH cluster of Spark was 3.3.2.
When I submit a task to Yarn, netty prompts a conversion error:
Caused by: java.lang.ClassCastException:
> io.netty.util.Recycler$DefaultHandle cannot be cast to
> io.netty.util.Recycler$EnhancedHandle
>
> at
> io.netty.channel.ChannelOutboundBuffer$Entry.(ChannelOutboundBuffer.java:827)
>
>
> at
> io.netty.channel.ChannelOutboundBuffer$Entry.(ChannelOutboundBuffer.java:806)
>
>
> at
> io.netty.channel.ChannelOutboundBuffer$Entry$1.newObject(ChannelOutboundBuffer.java:810)
>
>
> at
> io.netty.channel.ChannelOutboundBuffer$Entry$1.newObject(ChannelOutboundBuffer.java:807)
>
>
> at
> io.netty.util.internal.ObjectPool$RecyclerObjectPool$1.newObject(ObjectPool.java:77)
I checked the Netty version for Spark3.5.3 and it is 4.1.96, while my old
CDH cluster is using version 4.1.17.
But I checked the error reporting location code for both versions, and I
found that the code in 4.1.17 converts *DefaultHandle* to *EnhancedHandle* as
well.
The code in netty *io.netty.channel.ChannelOutboundBuffer.Entry* is same:
private Entry(Handle<Entry> handle) {
> this.handle = (EnhancedHandle<Entry>) handle;
> }
>
Any ideas on how to solve this problem please.
