Hi rui, The 'state.backend.fs.memory-threshold' configures the threshold below which state is stored as part of the metadata, rather than in separate files. So as a result the JM will use its memory to merge small checkpoint files and write them into one file. Currently the FLIP-306[1][2] is proposed to merge small checkpoint files without consuming JM memory. This feature is currently being worked on and is targeted for the next minor release (1.19).
Best, Zakelly [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints [2] https://issues.apache.org/jira/browse/FLINK-32070 On Fri, Oct 13, 2023 at 6:28 PM rui chen <ruichen...@gmail.com> wrote: > > We found that for some tasks, the JM memory continued to increase. I set > the parameter of state.backend.fs.memory-threshold to 0, and the JM memory > would no longer increase, but many small files might be written in this > way. Does the community have any optimization plan for this area?
