看看当时的负载呢？有没有过高的情况，是什么原因。然后监控下网络和磁盘
在 2021-03-09 14:57:43，"yidan zhao" <hinobl...@gmail.com> 写道：
>而且大家推荐怎么设置呢，我可能默认就G1了。不清楚G1是否也需要精调。
>我目前设置的内存还是比较大的。（50G的，100G的TaskManager都有），这么大heap，是否需要特别设置啥呢？
>
>或者是否有必要拆小，比如设置10Gheap，然后把taskmanager数量提上去。
>
>yidan zhao <hinobl...@gmail.com> 于2021年3月9日周二 下午2:56写道：
>
>> 好的，我会看下。
>> 然后我今天发现我好多个集群GC collector不一样。
>> 目前发现3种，默认的是G1。flink conf中配置了env.java.opts:
>> "-XX:-OmitStackTraceInFastThrow"的情况出现了2种，一种是Parallel GC with 83
>> threads，还有一种是Mark Sweep Compact GC。
>> 大佬们，Flink是根据内存大小有什么动态调整吗。
>>
>>
>> 不使用G1我大概理解了，可能设置了java.opts这个是覆盖，不是追加。本身我只是希望设置下-XX:-OmitStackTraceInFastThrow而已。
>>
>>
>> 杨杰 <471419...@qq.com> 于2021年3月8日周一 下午3:09写道：
>>
>>> Hi,
>>>
>>> 可以排查下 GC 情况，频繁 FGC 也会导致这些情况。
>>>
>>> Best,
>>> jjiey
>>>
>>> > 2021年3月8日 14:37，yidan zhao <hinobl...@gmail.com> 写道：
>>> >
>>> >
>>> 如题，我有个任务频繁发生该异常然后重启。今天任务启动1h后，看了下WEB-UI的检查点也没，restored达到了8已经。然后Exception页面显示该错误，估计大多数都是因为该错误导致的restore。
>>> > 除此外，就是 ‘Job leader for job id eb5d2893c4c6f4034995b9c8e180f01e lost
>>> > leadership’ 错导致任务重启。
>>> >
>>> > 下面给出刚刚的一个错误日志（环境flink1.12，standalone集群，5JM+5TM，JM和TM混部在相同机器）：
>>> > 2021-03-08 14:31:40
>>> > org.apache.flink.runtime.io
>>> .network.netty.exception.RemoteTransportException:
>>> > Error at remote task manager '10.35.185.38/10.35.185.38:2016'.
>>> > at org.apache.flink.runtime.io.network.netty.
>>> > CreditBasedPartitionRequestClientHandler.decodeMsg(
>>> > CreditBasedPartitionRequestClientHandler.java:294)
>>> > at org.apache.flink.runtime.io.network.netty.
>>> > CreditBasedPartitionRequestClientHandler.channelRead(
>>> > CreditBasedPartitionRequestClientHandler.java:183)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:379)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:365)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
>>> > .java:357)
>>> > at org.apache.flink.runtime.io.network.netty.
>>> > NettyMessageClientDecoderDelegate.channelRead(
>>> > NettyMessageClientDecoderDelegate.java:115)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:379)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:365)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
>>> > .java:357)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:
>>> > 1410)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:379)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeChannelRead(
>>> > AbstractChannelHandlerContext.java:365)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.epoll.
>>> > AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(
>>> > AbstractEpollStreamChannel.java:792)
>>> > at
>>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop
>>> > .processReady(EpollEventLoop.java:475)
>>> > at
>>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop
>>> > .run(EpollEventLoop.java:378)
>>> > at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>>> > SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>>> > at org.apache.flink.shaded.netty4.io.netty.util.internal.
>>> > ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>>> > at java.lang.Thread.run(Thread.java:748)
>>> > Caused by: org.apache.flink.runtime.io.network.partition.
>>> > ProducerFailedException: org.apache.flink.util.FlinkException:
>>> JobManager
>>> > responsible for eb5d2893c4c6f4034995b9c8e180f01e lost the leadership.
>>> > at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue
>>> > .writeAndFlushNextMessageIfPossible(PartitionRequestQueue.java:221)
>>> > at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue
>>> > .enqueueAvailableReader(PartitionRequestQueue.java:108)
>>> > at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue
>>> > .userEventTriggered(PartitionRequestQueue.java:170)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:346)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:332)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.fireUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:324)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter
>>> > .java:117)
>>> > at org.apache.flink.shaded.netty4.io.netty.handler.codec.
>>> > ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:346)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:332)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.fireUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:324)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline
>>> > .java:1428)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:346)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> > AbstractChannelHandlerContext.invokeUserEventTriggered(
>>> > AbstractChannelHandlerContext.java:332)
>>> > at org.apache.flink.shaded.netty4.io.netty.channel.
>>> >
>>> DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:
>>> > 913)
>>> > at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue
>>> > .lambda$notifyReaderNonEmpty$0(PartitionRequestQueue.java:87)
>>> > at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>>> > AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>>> > at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>>> >
>>> SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
>>> > at
>>> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop
>>> > .run(EpollEventLoop.java:387)
>>> > ... 3 more
>>> > Caused by: org.apache.flink.util.FlinkException: JobManager responsible
>>> for
>>> > eb5d2893c4c6f4034995b9c8e180f01e lost the leadership.
>>> > at org.apache.flink.runtime.taskexecutor.TaskExecutor
>>> > .disconnectJobManagerConnection(TaskExecutor.java:1422)
>>> > at org.apache.flink.runtime.taskexecutor.TaskExecutor.access$1300(
>>> > TaskExecutor.java:174)
>>> > at org.apache.flink.runtime.taskexecutor.
>>> > TaskExecutor$JobLeaderListenerImpl.lambda$null$2(TaskExecutor.java:1856)
>>> > at java.util.Optional.ifPresent(Optional.java:159)
>>> > at org.apache.flink.runtime.taskexecutor.
>>> > TaskExecutor$JobLeaderListenerImpl.lambda$jobManagerLostLeadership$3(
>>> > TaskExecutor.java:1855)
>>> > at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(
>>> > AkkaRpcActor.java:404)
>>> > at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(
>>> > AkkaRpcActor.java:197)
>>> > at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(
>>> > AkkaRpcActor.java:154)
>>> > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
>>> > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
>>> > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>> > at akka.japi.pf
>>> .UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
>>> > at
>>> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
>>> > at
>>> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>>> > at
>>> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
>>> > at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
>>> > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
>>> > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
>>> > at akka.actor.ActorCell.invoke(ActorCell.scala:561)
>>> > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
>>> > at akka.dispatch.Mailbox.run(Mailbox.scala:225)
>>> > at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
>>> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> > at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool
>>> > .java:1339)
>>> > at
>>> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> > at
>>> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread
>>> > .java:107)
>>> > Caused by: java.lang.Exception: Job leader for job id
>>> > eb5d2893c4c6f4034995b9c8e180f01e lost leadership.
>>> > ... 24 more
>>> >
>>> >
>>> > （1）zookeeper的超时设置的是60s，感觉网络异常zk超时不至于60s都不够。
>>> > （2）akka.ask.timeout: 60s
>>> > taskmanager.network.request-backoff.max： 60000
>>> > akka此参数之前也调整为60s了。
>>> >
>>> > 如上信息，希望社区同学们给点思路。
>>> >
>>>
>>>
