With the FsStateBackend you could also try increasing the value
of state.backend.fs.memory-threshold [1]. Only those state chunks that are
larger than this value are stored in separate files; smaller chunks go into
the checkpoint metadata file. The default is 1KB, increasing this
should reduce
Hi
Yes, for your use case, if you do not have large state size, you can try to
use FsStateBackend.
Best,
Congxian
ysnakie 于2020年4月27日周一 下午3:42写道：
> Hi
> If I use FsStateBackend instead of RocksdbFsStateBackend, will the open
> files decrease significantly? I dont have large state size.
>
>
Hi
If there are indeed so many files need to upload to hdfs, then currently we
do not have any solutions to limit the open files, there exist an issue[1]
wants to fix this problem, and a pr for it, maybe you can try the attached
pr to try it can solve your problem.
[1]
Hi everyone
We have a Flink Job to write files to HDFS's different directories. It will open many files due to its high parallelism. I also found that if using rocksdb state backend, it will have even more files open during the checkpointing. We use yarn to
