[
https://issues.apache.org/jira/browse/FLINK-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216416#comment-17216416
]
Yu Li commented on FLINK-15532:
-------------------------------
According to the [current
fix|https://github.com/facebook/rocksdb/commit/06c2935d83e5b9c9ede477fdcb922b8cd9e966a5]
in RocksDB community:
{quote}
// We keep the handle even if insertion fails and a null handle is
// returned, so that when memory shrinks, we don't release extra
// entries from cache.
// Ideally we should prevent this allocation from happening if
// this insertion fails. However, the callers to this code path
// are not able to handle failures properly. We'll need to improve
// it in the future.
{quote}
it's still not a complete one that could strictly restrict the memory usage, so
there are still more works required to get this JIRA done.
[~lipeidian] could you share more details about the problems you encountered?
AFAICS, the existing solution to control RocksDB memory usage by using managed
memory (which does some pre-calculation to reserve buffers to work around the
issue here, more details refer to FLINK-15512) works in most cases (we debugged
many reported K8S OOMKill issues but RocksDB memory management works fine in
most of them and is not the root cause).
> Enable strict capacity limit for memory usage for RocksDB
> ---------------------------------------------------------
>
> Key: FLINK-15532
> URL: https://issues.apache.org/jira/browse/FLINK-15532
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Major
>
> Currently, due to the limitation of RocksDB (see
> [issue-6247|https://github.com/facebook/rocksdb/issues/6247]), we cannot
> create a strict-capacity-limit LRUCache which shared among rocksDB
> instance(s).
> This issue tracks this problem and offer the ability of strict mode once we
> could enable this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
