Hi, from time to time my job is stopping to process messages with warn
message listed below. Tried to increase jobmanager.memory.process.size and
taskmanager.memory.process.size but it didn't help.
What else can I try? "Framework Off-heap" is 128mb now as seen is task
manager dashboard and Task Off-heap is 0b. Documentation says that "You
should only change this value if you are sure that the Flink framework needs
more memory." And I'm not sure about it.
Flink version is 1.13.2.
2021-11-29 14:06:53,659 WARN
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline [] - An
exceptionCaught() event was fired, and it reached at the tail of the
pipeline. It usually means the last handler in the pipeline did not handle
the exception.
org.apache.hbase.thirdparty.io.netty.channel.ChannelPipelineException:
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded
() has thrown an exception; removed.
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHand
lerAdded0(DefaultChannelPipeline.java:624)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst
(DefaultChannelPipeline.java:181)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst
(DefaultChannelPipeline.java:358)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst
(DefaultChannelPipeline.java:339)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnect
ion.java:215)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$600(NettyRpcConnection
.java:76)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcC
onnection.java:289)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcC
onnection.java:277)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyLi
stener0(DefaultPromise.java:578)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyLi
steners0(DefaultPromise.java:571)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyLi
stenersNow(DefaultPromise.java:550)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyLi
steners(DefaultPromise.java:491)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue
0(DefaultPromise.java:616)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSucce
ss0(DefaultPromise.java:605)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySucce
ss(DefaultPromise.java:104)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySucces
s(DefaultChannelPromise.java:84)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$Abstract
NioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$Abstract
NioUnsafe.finishConnect(AbstractNioChannel.java:335)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelecte
dKey(NioEventLoop.java:707)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelecte
dKeysOptimized(NioEventLoop.java:655)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelecte
dKeys(NioEventLoop.java:581)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLo
op.java:493)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecut
or$4.run(SingleThreadEventExecutor.java:989)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(T
hreadExecutorMap.java:74)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable
.run(FastThreadLocalRunnable.java:30)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
~[?:?]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena$DirectArena.allocateDi
rect(PoolArena.java:632)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena$DirectArena.newChunk(P
oolArena.java:607)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena.allocateNormal(PoolAre
na.java:202)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena.tcacheAllocateSmall(Po
olArena.java:172)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.jav
a:134)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.jav
a:126)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.PooledByteBufAllocator.newDirect
Buffer(PooledByteBufAllocator.java:395)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directB
uffer(AbstractByteBufAllocator.java:187)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directB
uffer(AbstractByteBufAllocator.java:178)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.buffer(
AbstractByteBufAllocator.java:115)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.writeRespons
e(NettyHBaseSaslRpcClientHandler.java:79)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded
(NettyHBaseSaslRpcClientHandler.java:115)
~[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261c
aa9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.c
allHandlerAdded(AbstractChannelHandlerContext.java:938)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHand
lerAdded0(DefaultChannelPipeline.java:609)
[blob_p-6eb282e9e614ab47d8c0b446632a1a9cba8a3955-6e6e09bc9b5fae2679cbbb261ca
a9da2:?]
... 25 more
