Hi Siew,
As far as I know, if the state size is larger than the tmpfs memory size,
which means tmpfs will be occupied fully leading to "IOException: No space left
on device". However, you just need to call
RocksDBStateBackend#setDbStoragePaths method to re-configure the local DB
storage dir to a larger disk volume and don't worry about the data completeness
as the doc of RocksDBStateBackend#getDbStoragePaths method said:
"Under these directories on the TaskManager, RocksDB stores its SST files and
metadata files. These directories do not need to be persistent, they can be
ephermeral, meaning that they are lost on a machine failure, because state in
RocksDB is persisted in checkpoints."
Regards,
Yun Tang
________________________________
发件人: Siew Wai Yow <wai_...@hotmail.com>
发送时间: 2018年7月6日 13:58
收件人: user@flink.apache.org
主题: flink rocksdb where to configure mount point
Hi,
We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When
the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.
RocksDBStateBackend rocksdb = new RocksDBStateBackend(new
FsStateBackend(hdfs://), true);
env.setStateBackend(rocksdb);
Questions,
1. What happen if large state is larger than the tmpfs memory size? Will
flink do something?
2. Is it possible to configure rocksdb dir via flink?
Thank you.
Regards,
Yow
