Re: Out of memory in heap memory when working with state

2022-09-06 Thread Hangxiang Yu
Hi, lan. I guess you are using the old version of flink. You could use RocksDBStateBackend[1] in the new version. It will put the state into disk when the state is large which could avoid using too much memory. BTW, In the current internal mechanism, the state on the external storage like s3 is

Out of memory in heap memory when working with state

2022-09-05 Thread lan tran
Hi team,Currently, I was facing the OutOfMemoryError: Java heap space. This was some how due to the fact that I was storing the state on FileSystem. With the FsStateBackend, the working state for each task manager is in memory (on the JVM heap), and state backups (checkpoints) go to a distributed