[
https://issues.apache.org/jira/browse/FLINK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853850#comment-17853850
]
Xintong Song edited comment on FLINK-35557 at 6/11/24 1:26 AM:
---------------------------------------------------------------
I think this is by design. As suggested by the name, the memory is expected to
be shared within the slot. Take RocksDBStateBackend as an example, when
there're multiple states in one slot, there will only be one cache area and one
write buffer area shared by all the states. Given that states are initialized
concurrently, there's no guarantee which state will be initialized first, thus
we introduced shared resources to only allocate the memory when the first state
is initialized.
If the memory consumer does not want this sharing behavior, it can simply call
allocatePages / reserveMemory instead of
getSharedMemoryResourceForManagedMemory.
was (Author: xintongsong):
I think this is by design. As suggested by the name, the memory is expected to
be shared within the slot. Take RocksDBStateBackend as an example, when
there're multiple states in one slot, there will only be one cache area and one
write buffer area share by all the states. Given that states are initialized
concurrently, there's no guarantee which state will be initialized first, thus
we introduced shared resources to only allocate the memory when the first state
is initialized.
If the memory consumer does not want this sharing behaviors, it can simply call
allocatePages / reserveMemory instead of
getSharedMemoryResourceForManagedMemory.
> MemoryManager only reserves memory per consumer type once
> ---------------------------------------------------------
>
> Key: FLINK-35557
> URL: https://issues.apache.org/jira/browse/FLINK-35557
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Runtime / Task
> Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
> Reporter: Roman Khachatryan
> Assignee: Roman Khachatryan
> Priority: Major
> Fix For: 1.20.0, 1.19.2
>
>
> # In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
> [create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
> a reserve function
> # The function
> [decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
> the available Slot memory and fails if there's not enough memory
> # We pass it to {{SharedResources.getOrAllocateSharedResource}}
> # In {{SharedResources.getOrAllocateSharedResource}} , we check if the
> resource (memory) was already reserved by some key (e.g.
> {{{}state-rocks-managed-memory{}}})
> # If not, we create a new one and call the reserve function
> # If the resource was already reserved (not null), we do NOT reserve the
> memory again:
> [https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
> So there will be only one (first) memory reservation for rocksdb for example,
> no matter how many state backends in a slot are created. Meaning that managed
> memory limits are not applied
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
