toolongframeexception

[Jiacheng Jiang]

大家好：

   我搭建了一个单机standealone 的flink用来测试mysql 
CDC，我发现每天晚上4点和4.30，我的taskmanager和jobmanager日志都有akka的TooLongFrameException，是BlobServerConnection报出来的，那个时候可能有定时任务在操作大量操作mysql数据库，我想问：

  1.  TooLongFrameException可能是mysql操作比较大造成的吗？如果是，有解决方法吗？
  2.  BlobServer是干嘛的？我一直认为blobserver是用来放job jar之类的，应该和cdc job的数据无关吧
  3.  哪些情况可能造成TooLongFrameException？

感谢大家

安装operator

[Jiacheng Jiang]

请问没有helm的情况下能否安装flink-k8s-operator？安装operator是否可以不要clusterrole

flink on k8s的taskmanager为啥不是Replica Sets

[Jiacheng Jiang]

请问大伙：

Flink on k8s里面，为什么taskmanager是单独的pod，而不是做成Replica Sets 利用副本数来控制taskmanager的数量





从 Windows 版邮件发送

??????Re:Re: flink-1.10 checkpoint ?????? NullPointerException

[??????(Jiacheng Jiang)]

hi



----
??: "chenkaibit"https://github.com/apache/flink/blob/release-1.10.0/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L1421
kafka??wordcount??kafka??checkpoint??
|CheckpointingMode|ExactlyOnce|
|Interval|5s|
|Timeout|10m0s|
|MinimumPauseBetweenCheckpoints|0ms|
|MaximumConcurrentCheckpoints|1|


5377??checkpointNPE


https://github.com/yuchuanchen/flink/commit/e5122d9787be1fee9bce141887e0d70c9b0a4f19NPE;


??2020-04-2110:21:56??chenkaibithttps://github.com/apache/flink/blob/aa4eb8f0c9ce74e6b92c3d9be5dc8e8cb536239d/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L1349





From:chenkaibit

?????? flink on yarn ????

[??????(Jiacheng Jiang)]

hi Xintong:
 
directgc??maxDirectMemorysystem.gc()??maxDirectMemorykill??




----
??: 
   "user-zh"

flink on yarn ????

[??????(Jiacheng Jiang)]

??flink on 
yarnflink??1.7.2??on 
yarntaskmanager.heap.sizecontainer??container??kill??taskmanager.heap.size
 = xmx+maxDirectMemorySizexmx = taskmanager.heap.size - cutoff 
- network??maxDirectMemorySize = cutoff + 
network??networkmaxDirectMemorySize??cutoffcutoff??maxDirectMemorySize??cutoffmaxDirectMemorySize??metaspacecontainerkill??

????????????????????????

[??????(Jiacheng Jiang)]

??ok??FlinkKafkaProducerpreCommit??KafkaProducer??flush??kafka??checkpointjob??preCommit??commit??




--  --
??: ""

????????????????????????

[??????(Jiacheng Jiang)]

??ok??FlinkKafkaProducerpreCommit??KafkaProducer??flush??kafka??checkpointjob??preCommit??commit??




----
??: ""

liststate

[??????(Jiacheng Jiang)]

  
??RocksDBListState#get??list??next??

??????flink??????????????????

[??????(Jiacheng Jiang)]

checkpoint




----
??: "??"

??????RichMapFunction??????

[??????(Jiacheng Jiang)]

flink 1.10??slot??tmcluster.evenly-spread-out-slots: true






----
??: "xue...@outlook.com"https://go.microsoft.com/fwlink/?LinkId=550986gt;

??????flink on yarn????????kerberos??kafka

[??????(Jiacheng Jiang)]

krb5.conf??FLINK_ENV_JAVA_OPTS=-Djava.security.krb5.conf=x/krb5.conf




----
??: "zjfpla...@hotmail.com"

??????webui??????job

[??????(Jiacheng Jiang)]

hi all
  flink web ui??jobweb 
ui??job??savepointweb??job??

?????? flink 1.10????????

[??????(Jiacheng Jiang)]

direct??nativeunsafe.allocateMemoryDirectByteBuffer??newdirectMaxDirectMemorySize,jvm??DirectByteBuffer




----
??: "Xintong Song"http://apache-flink.147419.n8.nabble.com/Flink-tt1869.html
gt;nbsp;<
http://apache-flink.147419.n8.nabble.com/Flink-tt1869.html#a1872amp;gt;
gt;
gt;

gt;nbsp;Onamp;nbsp;Tue,amp;nbsp;Apramp;nbsp;28,amp;nbsp;2020amp;nbsp;atamp;nbsp;9:32amp;nbsp;AMamp;nbsp;??(Jiachengamp;nbsp;Jiang)amp;nbsp;<
gt;nbsp;920334...@qq.comamp;gt;
gt;nbsp;wrote:
gt;
gt;nbsp;amp;gt;amp;nbsp;hiamp;amp;nbsp;Xintong
gt;nbsp;amp;gt;amp;nbsp;amp;amp;nbsp;amp;nbsp;amp;amp;nbsp;
gt;

gt;nbsp;amp;gt;amp;nbsp;flink1.10??-Xmx,-Xms,-XX:MaxDirectMomerySize,-XX:MaxMetaspaceSizemanaged

gt;nbsp;amp;gt;amp;nbsp;memorymanagedamp;nbsp;memory??jvmmetaspace??managed
gt;nbsp;amp;gt;amp;nbsp;memoryjvm
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;

gt;nbsp;amp;gt;amp;nbsp;--amp;amp;nbsp;amp;amp;nbsp;--

gt;nbsp;amp;gt;amp;nbsp;??:amp;nbsp;amp;quot;Xintongamp;nbsp;Songamp;quot;<
tonysong...@gmail.com
gt;nbsp;amp;amp;gt;;
gt;nbsp;amp;gt;amp;nbsp;:amp;nbsp;2020??4??27??(??)amp;nbsp;6:56
gt;nbsp;amp;gt;amp;nbsp;??:amp;nbsp;amp;quot;user-zhamp;quot;<
user-zh@flink.apache.orgamp;amp;gt;;

gt;nbsp;amp;gt;amp;nbsp;:amp;nbsp;Re:amp;nbsp;flinkamp;nbsp;1.10
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;nbsp;Frameworkamp;amp;nbsp;Off-heapamp;amp;nbsp;??amp;amp;nbsp;Taskamp;amp;nbsp;Off-Heapamp;amp;nbsp;??Network
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;nbsp;bufferamp;amp;nbsp;pool??
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;Thankamp;amp;nbsp;you~
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;Xintongamp;amp;nbsp;Song
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;Onamp;amp;nbsp;Mon,amp;amp;nbsp;Apramp;amp;nbsp;27,amp;amp;nbsp;2020amp;amp;nbsp;atamp;amp;nbsp;6:14amp;amp;nbsp;PMamp;amp;nbsp;??(Jiachengamp;amp;nbsp;Jiang)amp;amp;nbsp;<
gt;nbsp;amp;gt;amp;nbsp;920334...@qq.comamp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;wrote:
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;Thankamp;amp;amp;nbsp;youamp;amp;amp;nbsp;Xintong.??
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;-XX:MaxDirectMemorySizeFrameworkamp;amp;nbsp;+amp;amp;nbsp;Taskamp;amp;nbsp;Off-Heapamp;amp;nbsp;+amp;amp;nbsp;Networkamp;amp;nbsp;Memory
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;MaxDirectMemorySize=Frameworkamp;amp;amp;nbsp;Off-Heapamp;amp;amp;nbsp;+amp;amp;nbsp;Taskamp;amp;nbsp;Off-Heapamp;amp;nbsp;+amp;amp;nbsp;Network
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;Memory??MaxDirectMemorySize=10??10=1+1+8??10=1+8+1,??2??3??task

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;off-heapjob
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;--amp;amp;amp;nbsp;amp;amp;amp;nbsp;--
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;??:amp;amp;nbsp;amp;amp;quot;Xintongamp;amp;nbsp;Songamp;amp;quot;<
gt;nbsp;tonysong...@gmail.com
gt;nbsp;amp;gt;amp;nbsp;amp;amp;amp;gt;;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;:amp;amp;nbsp;2020??4??27??(??)amp;amp;nbsp;12:06

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;??:amp;amp;nbsp;amp;amp;quot;user-zhamp;amp;quot;<
gt;nbsp;user-zh@flink.apache.orgamp;amp;amp;gt;;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;:amp;amp;nbsp;Re:amp;amp;nbsp;flinkamp;amp;nbsp;1.10
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;??
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;
gt;

gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;amp;amp;nbsp;Flinkamp;amp;amp;nbsp;1.10amp;amp;amp;nbsp;amp;amp;amp;nbsp;TMamp;amp;amp;nbsp;amp;amp;amp;nbsp;heap,
gt;nbsp;amp;gt;amp;nbsp;amp;amp;gt;
gt;nbsp;amp;gt;
gt;

??????flink on kubernetes ????????????????

[??????(Jiacheng Jiang)]

Map??es




----
??: "a511955993"

?????? flink 1.10????????

[??????(Jiacheng Jiang)]

hi Xintong??flink??managed 
memoryflink??native memory




----
??: "Xintong Song"http://apache-flink.147419.n8.nabble.com/Flink-tt1869.html

?????? flink 1.10????????

[??????(Jiacheng Jiang)]

hi Xintong??java??native memoryapi




----
??: "Xintong Song"http://apache-flink.147419.n8.nabble.com/Flink-tt1869.html

execution.checkpointing.tolerable-failed-checkpoints ????

[??????(Jiacheng Jiang)]

hi
 
flink-conf.yaml??execution.checkpointing.tolerable-failed-checkpoints??
 
1000checkpoint??job??
best
jungglge

??????flink????????hdfs parquest,hdfs????ha??????????

[??????(Jiacheng Jiang)]

??fs.defaultFS??



----
??: "196371551"<196371...@qq.com; 
: 2020??4??30??(??) 8:50
??: "user"

?????? ????????

[??????(Jiacheng Jiang)]

java.lang.OutOfMemoryError: unable to create newnative 
thread??XmxMaxDirectMemorySize??




----
??:"LakeShen"http://apache-flink.147419.n8.nabble.com/flink-kafka-td2386.html
  [2] 
http://apache-flink.147419.n8.nabble.com/Flink-Kafka-td2390.html
  On Sun, Apr 26, 2020 at 10:58 yanggang_it_job 

?????? flink 1.10????????

[??????(Jiacheng Jiang)]

ThankyouXintong.??
-XX:MaxDirectMemorySizeFramework + Task Off-Heap + Network Memory
MaxDirectMemorySize=FrameworkOff-Heap+ Task Off-Heap + Network 
Memory??MaxDirectMemorySize=10??10=1+1+8??10=1+8+1,??2??3??task
 off-heapjob



----
??: "Xintong Song"https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_detail.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_trouble.html#outofmemoryerror-direct-buffer-memory
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_setup.html#%E6%89%98%E7%AE%A1%E5%86%85%E5%AD%98

OnMon,Apr27,2020at10:31AM??(JiachengJiang)<920334...@qq.com
wrote:

hi


nbsp;nbsp;??standalone
flinkjob??tm??8gflink1.10??taskmanager.memory.flink.size??10g??job??jobjava.lang.OutOfMemoryError:
Directbuffermemory??1??job??Direct
Memory??flink1.10standaloneclusterslot
??slotATaskManagerwiththreeslots,forexample,will
dedicate1/3ofitsmanagedmemorytoeachslot.nbsp;??itsmanaged
memory??taskmanager.memory.managed.size??


bestnbsp;
Jungle

flink 1.10????????

[??????(Jiacheng Jiang)]

hi


 ??standalone 
flinkjob??tm??8gflink1.10??taskmanager.memory.flink.size??10g??job??jobjava.lang.OutOfMemoryError:
 Direct buffer memory??1??job??Direct 
Memory??flink1.10standalone 
clusterslot
??slotA TaskManager with three slots, for example, 
will dedicate 1/3 of its managed memory to each slot.??its managed 
memory??taskmanager.memory.managed.size??


best
Jungle

?????? flink1.10????win10????Standlone????

[??????(Jiacheng Jiang)]

Causedby:org.apache.flink.configuration.IllegalConfigurationException:
Thenetworkmemorymin(64mb)andmax(1gb)mismatch,thenetworkmemory
hastoberesolvedandsettoafixedvaluebeforetaskexecutorstarts




??1.10??




----
??:""

??????flink1.10????win10????Standlone????

[??????(Jiacheng Jiang)]

??




----
??: ""

??????1.10????????????--??????????????

[??????(Jiacheng Jiang)]

??MailboxProcessorstreamtask??processInputMailboxDefaultAction??MailboxProcessorInputStatus
 status = 
inputProcessor.processInput();inputProcessor??StreamOneInputProcessor??InputStatus
 status = 
input.emitNext(output);input??StreamTaskNetworkInputprocessElement??StreamTaskAbstractInvokable??StreamTask??invoke()??runMailboxLoop()??StreamTask??invoke()



----
??: ""<17626017...@163.com; 
: 2020??4??19??(??) 5:37
??: "user-zh"

??????Flink?????? ?????? Keyed Watermarks

[??????(Jiacheng Jiang)]

??




----
??:"Utopia"https://bigdata.cs.ut.ee/keyed-watermarks-partition-aware-watermark-generation-apache-flink



Best regards
Utopia