[
https://issues.apache.org/jira/browse/FLINK-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhijiang updated FLINK-5138:
----------------------------
Description:
Users may specify the state size in *setResources* API, then the state backend
would roughly estimate the different kinds of memory usages based on the state
size. Currently the specific state backend ({{FsStateBackend}},
{{MemoryStateBackend}}, etc) can implement the default unknown resource for it
to work around.
There are two advantages to do this:
- This estimate memory will be considered together when requesting resource
for container, so the total memory usage will not exceed the container limit.
- Especially for *RocksDB* backend, the proper memory setting for *RocksDB*
may get better performance in read and write operation.
was:
Users may specify the state size in *setResources* API, then the state backend
would roughly estimate the different kinds of memory usages based on the state
size. This part of estimate memory would be aggregated with other memories in
*ResourceSpec*.
There are two advantages to do this:
- For *RocksDB* backend, the proper memory setting for *RocksDB* may get
better performance in read and write operation.
- This estimate memory will be considered together when requesting resource
for container, so the total memory usage will not exceed the container limit.
> Add the method in state backend to estimate memory by hint of state size in
> ResourceSpec
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-5138
> URL: https://issues.apache.org/jira/browse/FLINK-5138
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Reporter: zhijiang
> Assignee: zhijiang
>
> Users may specify the state size in *setResources* API, then the state
> backend would roughly estimate the different kinds of memory usages based on
> the state size. Currently the specific state backend ({{FsStateBackend}},
> {{MemoryStateBackend}}, etc) can implement the default unknown resource for
> it to work around.
> There are two advantages to do this:
> - This estimate memory will be considered together when requesting resource
> for container, so the total memory usage will not exceed the container limit.
> - Especially for *RocksDB* backend, the proper memory setting for *RocksDB*
> may get better performance in read and write operation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
