[ https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738986#comment-14738986 ]
Maximilian Michels commented on FLINK-2641: ------------------------------------------- The new configuration paramters are - taskmanager.memory.off-heap: Boolean - taskmanager.memory.off-heap-ratio: Float These have to interplay with these existing config entries: - taskmanager.memory.size: Int - taskmanager.memory.fraction: Float - taskmanager.heap.mb: Int This works as follows: * If {{taskmanager.memory.size}} is set, the task manager allocates the number of memory either on the JVM heap or as direct memory depending on whether {{taskmanager.memory.off-heap}} is set to true. * If {{taskmanager.memory.size}} is not set, the memory allocation is slightly more involved: ** In the standard heap mode ({{taskmanager.memory.off-heap}} set to false), the task manager gets the free heap memory during startup, multiplies it by {{taskmanager.memory.fraction}}, and allocates the resulting memory size as heap memory. ** In off-heap mode ({{taskmanager.memory.off-heap}} set to true), the task manager gets the maximum available allocatable memory (set by the -Xmx parameter), multiplies it by the {{taskmanager.memory.off-heap-ratio}}, and allocates the resulting memory size as direct (off-heap) memory size. In order to be able to switch from heap to off-heap memory usage and not having to adjust any memory settings, the {{taskmanager.heap.mb}} setting has to be dynamically adjusted: * If {{taskmanager.memory.off-heap}} is set and ** {{taskmanager.memory.size}} is set, then the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb - taskmanager.memory.size}}. ** {{taskmanager.memory.size}} is not set, the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb * (1 - (1/taskmanager.memory.off-heap-ratio))}}. Note that {{taskmanager.memory.off-heap-ratio > 1.0}}. ----- Up for discussion: It might confuse users that {{taskmanager.heap.mb}} specifies the JVM heap size but we use it to calculate the total memory (on and off-heap combined). Therefore, I'd propose to add another config entry which let's you specify the JVM process memory: {{taskmanager.jvm.mb}} would set the maximum memory which is then split up between JVM heap and off-heap memory. The old parameter can still be used to explicitly set the heap memory. Since we introduce new config parameters anyways with the off-heap memory, I think this is a fair compromise. {{taskmanager.heap.mb}} will be deprecated in favor of {{taskmanager.jvm.mb}}. Then we should also rename {{jobmanager.heap.mb}} to {{jobmanager.jvm.mb}}. Since I'm not entirely sure about the name and whether we should introduce additional config parameters, I will leave that up for discussion now and implement the config parameter handling with the default {{taskmanager.heap.mb}} parameter and its dynamic adjustment. > Integrate the off-heap memory configuration with the TaskManager start script > ----------------------------------------------------------------------------- > > Key: FLINK-2641 > URL: https://issues.apache.org/jira/browse/FLINK-2641 > Project: Flink > Issue Type: New Feature > Components: Start-Stop Scripts > Affects Versions: 0.10 > Reporter: Stephan Ewen > Assignee: Maximilian Michels > Fix For: 0.10 > > > The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and > {{-XX:MaxDirectMemorySize}} parameters according to the off-heap memory > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)