[
https://issues.apache.org/jira/browse/FLINK-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982195#comment-16982195
]
Zhu Zhu edited comment on FLINK-14936 at 11/26/19 7:08 AM:
-----------------------------------------------------------
Thanks [~azagrebin] for the explanation.
I think you are right that we can have {{computeMemorySizeForOperator}} in the
{{AbstractStreamOperator}}(or a util) so that an operator can use it to reserve
managed memory. In this way operator developers even do not need to be aware of
the operator managed memory fraction in {{StreamConfig}}. This should work at
the moment.
The limitation is that it cannot be used for other purposes, like computing
managed memory size for the StateBackend.
The proposed {{MemoryManager#computeMemorySize}} is actually not related for a
particular usage of managed memory. It can be used by an operator's processing
logic, or other components like state backend. Having it paired with
{{#reserveMemory}} is just the the other pair {{(#computeNumberOfPages,
#allocatePages)}}.
By supporting it directly, {{MemoryManager}} methods {{#getMemorySizeByType}}
and {{#getMemorySize}} can be private since they are exposed for this purpose
only ATM. (Correct me if I'm wrong here)
In general, I think we should have a {{#computeMemorySize()}} in
{{AbstractStreamOperator}} so that different operators do not need to the same
things to fetch the fractions from {{StreamConfig}} and compute the memory size.
For {{MemoryManager#computeMemorySize(fraction, memoryType)}}, I think we can
also have it to further avoid code duplication and to not expose
{{#getMemorySizeByType}} and {{#getMemorySize}}.
was (Author: zhuzh):
Thanks [~azagrebin] for the explanation.
I think you are right that we can have {{computeMemorySizeForOperator}} in the
{{AbstractStreamOperator}}(or a util) so that an operator can use it to reserve
managed memory. In this way operator developers even do not need to be aware of
the operator managed memory fraction in {{StreamConfig}}. This should work at
the moment.
The limitation is that it cannot be used for other purposes, like computing
managed memory size for the StateBackend.
The proposed {{MemoryManager#computeMemorySize}} is actually not related for a
particular usage of managed memory. It can be used by an operator's processing
logic, or other components like state backend. Having it paired with
{{#reserveMemory}} is just the the other pair {{(#computeNumberOfPages,
#allocatePages)}}.
By supporting it directly, {{MemoryManager}} methods {{#getMemorySizeByType}}
and {{#getMemorySize}} can be private since they are exposed for this purpose
only ATM. (Correct me if I'm wrong here)
In general, I think we should have a {{#computeMemorySize()}} in
{{AbstractStreamOperator}} so that different operators do not need to the same
things to fetch the fractions from {{StreamConfig}} and compute the memory size.
For {{MemoryManager#computeMemorySize(fraction, memoryType)}}, I think we can
have it to further avoid code duplication and to not expose
{{#getMemorySizeByType}} and {{#getMemorySize}}.
> Introduce MemoryManager#computeMemorySize to calculate managed memory from a
> fraction
> -------------------------------------------------------------------------------------
>
> Key: FLINK-14936
> URL: https://issues.apache.org/jira/browse/FLINK-14936
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
>
> A MemoryManager#computeMemorySize(double fraction) is needed to calculate
> managed memory bytes from a fraction.
> It can be helpful for operators to get the memory size it can reserve and for
> further #reserveMemory. (Similar to #computeNumberOfPages).
> Here are two cases that may need this method in near future:
> 1. [Python operator memory
> management|https://lists.apache.org/thread.html/dd4dedeb9354c2ee559cd2f15629c719853915b5efb31a0eafee9361@%3Cdev.flink.apache.org%3E]
> 2. [Statebackend memory
> management|https://issues.apache.org/jira/browse/FLINK-14883]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
