[jira] [Commented] (FLINK-35557) MemoryManager only reserves memory per consumer type once
[
https://issues.apache.org/jira/browse/FLINK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853850#comment-17853850
]
Xintong Song commented on FLINK-35557:
--
I think this is by design. As suggested by the name, the memory is expected to
be shared within the slot. Take RocksDBStateBackend as an example, when
there're multiple states in one slot, there will only be one cache area and one
write buffer area share by all the states. Given that states are initialized
concurrently, there's no guarantee which state will be initialized first, thus
we introduced shared resources to only allocate the memory when the first state
is initialized.
If the memory consumer does not want this sharing behaviors, it can simply call
allocatePages / reserveMemory instead of
getSharedMemoryResourceForManagedMemory.
> MemoryManager only reserves memory per consumer type once
> -
>
> Key: FLINK-35557
> URL: https://issues.apache.org/jira/browse/FLINK-35557
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Runtime / Task
>Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Fix For: 1.20.0, 1.19.2
>
>
> # In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
> [create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
> a reserve function
> # The function
> [decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
> the available Slot memory and fails if there's not enough memory
> # We pass it to {{SharedResources.getOrAllocateSharedResource}}
> # In {{SharedResources.getOrAllocateSharedResource}} , we check if the
> resource (memory) was already reserved by some key (e.g.
> {{{}state-rocks-managed-memory{}}})
> # If not, we create a new one and call the reserve function
> # If the resource was already reserved (not null), we do NOT reserve the
> memory again:
> [https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
> So there will be only one (first) memory reservation for rocksdb for example,
> no matter how many state backends in a slot are created. Meaning that managed
> memory limits are not applied
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35557) MemoryManager only reserves memory per consumer type once
[
https://issues.apache.org/jira/browse/FLINK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853588#comment-17853588
]
Hong Liang Teoh commented on FLINK-35557:
-
Changing this to 1.19.2 release as this is a long-standing bug without a fix
atm.
> MemoryManager only reserves memory per consumer type once
> -
>
> Key: FLINK-35557
> URL: https://issues.apache.org/jira/browse/FLINK-35557
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Runtime / Task
>Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Fix For: 1.20.0, 1.19.2
>
>
> # In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
> [create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
> a reserve function
> # The function
> [decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
> the available Slot memory and fails if there's not enough memory
> # We pass it to {{SharedResources.getOrAllocateSharedResource}}
> # In {{SharedResources.getOrAllocateSharedResource}} , we check if the
> resource (memory) was already reserved by some key (e.g.
> {{{}state-rocks-managed-memory{}}})
> # If not, we create a new one and call the reserve function
> # If the resource was already reserved (not null), we do NOT reserve the
> memory again:
> [https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
> So there will be only one (first) memory reservation for rocksdb for example,
> no matter how many state backends in a slot are created. Meaning that managed
> memory limits are not applied
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35557) MemoryManager only reserves memory per consumer type once
[
https://issues.apache.org/jira/browse/FLINK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853161#comment-17853161
]
Roman Khachatryan commented on FLINK-35557:
---
I guess it doesn't manifest itself thanks to JM telling TM to reserve the right
amount via ResourceProfile.
> MemoryManager only reserves memory per consumer type once
> -
>
> Key: FLINK-35557
> URL: https://issues.apache.org/jira/browse/FLINK-35557
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Runtime / Task
>Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Fix For: 1.20.0, 1.19.1
>
>
> # In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
> [create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
> a reserve function
> # The function
> [decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
> the available Slot memory and fails if there's not enough memory
> # We pass it to {{SharedResources.getOrAllocateSharedResource}}
> # In {{SharedResources.getOrAllocateSharedResource}} , we check if the
> resource (memory) was already reserved by some key (e.g.
> {{{}state-rocks-managed-memory{}}})
> # If not, we create a new one and call the reserve function
> # If the resource was already reserved (not null), we do NOT reserve the
> memory again:
> [https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
> So there will be only one (first) memory reservation for rocksdb for example,
> no matter how many state backends in a slot are created. Meaning that managed
> memory limits are not applied
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
