I am running a production job for at least 1 year, and I got to day this
error:
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
Adjusted frame length exceeds 2147483647: 2969686273 - discarded
(connection to
'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
10.208.65.38:40737')
Nothing was changed in the code for a long time. What's causing this error
and how to fix it? I am running Flink 1.10.3 on YARN.
This is the full stack trace:
2022-03-15 03:22:13
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
Adjusted frame length exceeds 2147483647: 2969686273 - discarded
(connection to
'flink-session-playback-prod-1641716499-sw-6q8p.c.data-prod-292614.internal/
10.208.65.38:40737')
at org.apache.flink.runtime.io.network.netty.
CreditBasedPartitionRequestClientHandler.exceptionCaught(
CreditBasedPartitionRequestClientHandler.java:165)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:297)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:276)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.fireExceptionCaught(
AbstractChannelHandlerContext.java:268)
at org.apache.flink.shaded.netty4.io.netty.channel.
ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter
.java:143)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeExceptionCaught(
AbstractChannelHandlerContext.java:297)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.notifyHandlerException(
AbstractChannelHandlerContext.java:831)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:376)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:360)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
.java:352)
at org.apache.flink.shaded.netty4.io.netty.channel.
DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:
1421)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:374)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:360)
at org.apache.flink.shaded.netty4.io.netty.channel.
DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.
AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKey(NioEventLoop.java:697)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKeysOptimized(NioEventLoop.java:632)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
.processSelectedKeys(NioEventLoop.java:549)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(
NioEventLoop.java:511)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.
ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.shaded.netty4.io.netty.handler.codec.
TooLongFrameException: Adjusted frame length exceeds 2147483647: 2969686273
- discarded
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:513)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder
.java:491)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.exceededFrameLength(
LengthFieldBasedFrameDecoder.java:378)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:421)
at org.apache.flink.runtime.io.network.netty.
NettyMessage$NettyMessageDecoder.decode(NettyMessage.java:214)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:334)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder
.java:505)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at org.apache.flink.shaded.netty4.io.netty.handler.codec.
ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at org.apache.flink.shaded.netty4.io.netty.channel.
AbstractChannelHandlerContext.invokeChannelRead(
AbstractChannelHandlerContext.java:374)
... 14 more
