不是，是用per-job方式提交的
在 2020-05-15 14:14:20，"shao.hongxiao" <17611022...@163.com> 写道：
>你的是batch 模式吗
>
>
>
>
>| |
>邵红晓
>|
>|
>邮箱：17611022...@163.com
>|
>
>签名由 网易邮箱大师 定制
>
>在2020年05月15日 15:05，Jeff 写道：
>hi all,
>最近上线一批新任务后taskmanager频繁挂掉,很可能是OOM问题，操作系统日志里没找到相关记录，flink日志只找到如下部分，但还是不确定是什么原因，请问要怎么确定原因呢?
>
>
>
>
>id, channel, rowtime) -> select: (appid, channel, rowtime, 1 AS $f3)
>b91d36766995398a9b0c9416ac1fb6bc.
>2020-05-14 08:55:30,504 ERROR org.apache.flink.runtime.taskmanager.Task - Task
>did not exit gracefully within 180 + seconds.
>2020-05-14 08:55:30,505 ERROR
>org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit
>gracefully within 180 + seconds.
>2020-05-14 08:55:30,505 ERROR
>org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error occurred
>while executing the TaskManager. Shutting it down...
>2020-05-14 08:55:30,505 INFO
>org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopping TaskExecutor
>akka.tcp://flink@172.24.150.24:48412/user/taskmanager_0.
>2020-05-14 08:55:30,508 INFO
>org.apache.flink.runtime.taskexecutor.JobLeaderService - Stop job leader
>service.
>2020-05-14 08:55:30,510 INFO
>org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager - Shutting
>down TaskExecutorLocalStateStoresManager.
>2020-05-14 08:55:30,512 INFO
>org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager removed
>spill file directory /tmp/flink-io-c882cdf4-840c-4c62-800e-0ca3226be20b
>2020-05-14 08:55:30,512 INFO
>org.apache.flink.runtime.io.network.NetworkEnvironment - Shutting down the
>network environment and its components.
>2020-05-14 08:55:30,514 INFO
>org.apache.flink.runtime.io.network.netty.NettyClient - Successful shutdown
>(took 2 ms).
>2020-05-14 08:55:30,517 INFO
>org.apache.flink.runtime.io.network.netty.NettyServer - Successful shutdown
>(took 2 ms).
>2020-05-14 08:55:30,545 INFO
>org.apache.flink.runtime.taskexecutor.JobLeaderService - Stop job leader
>service.
>2020-05-14 08:55:30,546 INFO org.apache.flink.runtime.filecache.FileCache -
>removed file cache directory
>/tmp/flink-dist-cache-3e0d4351-d5aa-4358-a028-9a59f398d92f
>2020-05-14 08:55:30,550 INFO
>org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopped TaskExecutor
>akka.tcp://flink@172.24.150.24:48412/user/taskmanager_0.
>2020-05-14 08:55:30,552 INFO org.apache.flink.runtime.blob.PermanentBlobCache
>- Shutting down BLOB cache
>2020-05-14 08:55:30,554 INFO org.apache.flink.runtime.blob.TransientBlobCache
>- Shutting down BLOB cache
>2020-05-14 08:55:30,563 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
>- Stopping Akka RPC service.
>2020-05-14 08:55:30,566 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote
>daemon.
>2020-05-14 08:55:30,567 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut
>down; proceeding with flushing remote transports.
>2020-05-14 08:55:30,570 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote
>daemon.
>2020-05-14 08:55:30,571 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut
>down; proceeding with flushing remote transports.
>2020-05-14 08:55:30,571 INFO org.apache.flink.runtime.taskmanager.Task -
>Source: KafkaTableSource switched from RUNNING to FAILED.
>java.lang.RuntimeException: segment has been freed
>at
>org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
>at
>org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
>at
>org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
>at
>org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>at
>org.apache.flink.table.runtime.CRowWrappingCollector.collect(CRowWrappingCollector.scala:37)
>at
>org.apache.flink.table.runtime.CRowWrappingCollector.collect(CRowWrappingCollector.scala:28)
>at DataStreamCalcRule$2658.processElement(Unknown Source)
>at
>org.apache.flink.table.runtime.CRowProcessRunner.processElement(CRowProcessRunner.scala:66)
>at
>org.apache.flink.table.runtime.CRowProcessRunner.processElement(CRowProcessRunner.scala:35)
>at
>org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
>at
>org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
>at
>org.apache.flink.table.runtime.CRowWrappingCollector.collect(CRowWrappingCollector.scala:37)
>at
>org.apache.flink.table.runtime.CRowWrappingCollector.collect(CRowWrappingCollector.scala:28)
>at DataStreamSourceConversion$2595.processElement(Unknown Source)
>at
>org.apache.flink.table.runtime.CRowOutputProcessRunner.processElement(CRowOutputProcessRunner.scala:70)
>at
>org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
>at
>org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
>at
>org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
>at
>org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
>at
>org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
>at
>org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
>at
>org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.emitRecord(KafkaFetcher.java:185)
>at
>org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:150)
>at
>org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:711)
>at
>org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)
>at
>org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)
>at
>org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)
>at
>org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:302)
>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>at java.lang.Thread.run(Thread.java:745)
>Caused by: java.lang.IllegalStateException: segment has been freed
>at
>org.apache.flink.core.memory.HybridMemorySegment.put(HybridMemorySegment.java:228)
>at
>org.apache.flink.core.memory.HybridMemorySegment.put(HybridMemorySegment.java:381)
>at
>org.apache.flink.runtime.io.network.buffer.BufferBuilder.append(BufferBuilder.java:85)
>at
>org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.copyToBufferBuilder(SpanningRecordSerializer.java:95)
>at
>org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:178)
>at
>org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:162)
>at
>org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:128)
>at
>org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
>... 39 more
>2020-05-14 08:55:30,572 INFO org.apache.flink.runtime.taskmanager.Task -
>Freeing task resources for Source: KafkaTableSource
>2020-05-14 08:55:30,572 ERROR org.apache.flink.runtime.taskmanager.Task -
>FATAL - exception in resource cleanup of task Source: KafkaTableSource.
>java.lang.IllegalStateException: Memory manager has been shut down.
>at
>org.apache.flink.runtime.memory.MemoryManager.releaseAll(MemoryManager.java:480)
>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:828)
>at java.lang.Thread.run(Thread.java:745)
>2020-05-14 08:55:30,572 ERROR
>org.apache.flink.runtime.taskexecutor.TaskExecutor - FATAL - exception in
>resource cleanup of task Source: KafkaTableSource(.
>java.lang.IllegalStateException: Memory manager has been shut down.
>at
>org.apache.flink.runtime.memory.MemoryManager.releaseAll(MemoryManager.java:480)
>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:828)
>at java.lang.Thread.run(Thread.java:745)
>2020-05-14 08:55:30,572 ERROR
>org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error occurred
>while executing the TaskManager. Shutting it down...
>java.lang.IllegalStateException: Memory manager has been shut down.
>at
>org.apache.flink.runtime.memory.MemoryManager.releaseAll(MemoryManager.java:480)
>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:828)
>at java.lang.Thread.run(Thread.java:745)
>2020-05-14 08:55:30,587 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
>2020-05-14 08:55:30,587 INFO
>akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
>2020-05-14 08:55:30,600 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
>- Stopped Akka RPC service.
