Re: Zigzag shape in TM JVM used memory

2021-04-08 Thread Piotr Nowojski
Hi, I don't think there is a Flink specific answer to this question. Just do what you would normally do with a normal Java application running inside a JVM. If there is an OOM on heap space, you can either try to bump the heap space, or reduce usage of it. The only Flink specific part is probably

Re: Zigzag shape in TM JVM used memory

2021-04-07 Thread Lu Niu
Hi, Piotr Thanks for replying. I asked this because such a pattern might imply memory oversubscription. For example, I tuned down the memory of one app from heap 2.63GB to 367MB and the job still runs fine: before: https://drive.google.com/file/d/1o8k9Vv3yb5gXITi4GvmlXMteQcRfmOhr/view?usp=sharing

Re: Zigzag shape in TM JVM used memory

2021-04-06 Thread Piotr Nowojski
Hi, this should be posted on the user mailing list not the dev. Apart from that, this looks like normal/standard behaviour of JVM, and has very little to do with Flink. Garbage Collector (GC) is kicking in when memory usage is approaching some threshold: https://www.google.com/search?q=jvm+heap+m

Re: Zigzag shape in TM JVM used memory

2021-04-05 Thread Lu Niu
Hi, we need to update our email system then :) . Here are the links: https://drive.google.com/file/d/1lZ5_P8_NqsN1JeLzutGj4DxkyWJN75mR/view?usp=sharing https://drive.google.com/file/d/1J6c6rQJwtDp1moAGlvQyLQXTqcuG4HjL/view?usp=sharing https://drive.google.com/file/d/1-R2KzsABC471AEjkF5qTm5O3V47

Re: Zigzag shape in TM JVM used memory

2021-04-04 Thread Yun Gao
Hi Lu, The image seems not be able to shown due to the mail server limitation, could you upload it somewhere and paste the link here ? Logically, I think zigzag usually due to there are some small object get created and eliminated soon in the heap. Are you running a SQL job or a DataStream jo

Zigzag shape in TM JVM used memory

2021-04-04 Thread Lu Niu
Hi, Flink dev We observed that the TM JVM used memory metric shows zigzag shape among lots of our applications, although these applications are quite different in business logic. The upper bound is close to the max heap size. Is this expected in flink application? Or does flink internally aggressi