[ https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745098#comment-14745098 ]
ASF GitHub Bot commented on FLINK-2641: --------------------------------------- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488999 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi - if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]]; then - echo "[ERROR] Configured TaskManager JVM heap size is not a number. Please set '$KEY_TASKM_HEAP_MB' in $FLINK_CONF_FILE." + if [[ ! ${FLINK_TM_HEAP} =~ ${IS_NUMBER} ]] || [[ "${FLINK_TM_HEAP}" -lt "0" ]]; then + echo "[ERROR] Configured TaskManager JVM heap size is not a number. Please set '${KEY_TASKM_MEM_SIZE}' in ${FLINK_CONF_FILE}." exit 1 fi - if [ "$FLINK_TM_HEAP" -gt 0 ]; then - export JVM_ARGS="$JVM_ARGS -Xms"$FLINK_TM_HEAP"m -Xmx"$FLINK_TM_HEAP"m" + if [ "${FLINK_TM_HEAP}" -gt "0" ]; then + + TM_HEAP_SIZE=${FLINK_TM_HEAP} + TM_OFFHEAP_SIZE=0 + # some space for Netty initilization + NETTY_BUFFERS=$((1024 * 1024)) + + if useOffHeapMemory; then + if [[ "${FLINK_TM_MEM_MANAGED_SIZE}" -gt "0" ]]; then + # We split up the total memory in heap and off-heap memory + if [[ "${FLINK_TM_HEAP}" -le "${FLINK_TM_MEM_MANAGED_SIZE}" ]]; then + echo "[ERROR] Configured TaskManager memory size ('${KEY_TASKM_MEM_SIZE}') must be larger than the managed memory size ('${KEY_TASKM_MEM_MANAGED_SIZE}')." + exit 1 + fi + TM_OFFHEAP_SIZE=${FLINK_TM_MEM_MANAGED_SIZE} + TM_HEAP_SIZE=$((FLINK_TM_HEAP - FLINK_TM_MEM_MANAGED_SIZE)) + else + # We calculate the memory using a fraction of the total memory + if [[ `bc -l <<< "${FLINK_TM_MEM_MANAGED_FRACTION} >= 1.0"` != "0" ]]; then + echo "[ERROR] Configured TaskManager managed memory fraction is not a valid value. Please set '${KEY_TASKM_MEM_MANAGED_FRACTION}' in ${FLINK_CONF_FILE}" + exit 1 + fi + # recalculate the JVM heap memory by taking the off-heap ratio into account + TM_OFFHEAP_SIZE=`printf '%.0f\n' $(bc -l <<< "${FLINK_TM_HEAP} * ${FLINK_TM_MEM_MANAGED_FRACTION}")` + TM_HEAP_SIZE=$((FLINK_TM_HEAP - TM_OFFHEAP_SIZE)) + fi + fi + + export JVM_ARGS="${JVM_ARGS} -Xms${TM_HEAP_SIZE}M -Xmx${TM_HEAP_SIZE}M -XX:MaxDirectMemorySize=$((TM_OFFHEAP_SIZE * 1024 * 1024 + FLINK_TM_MEM_NETWORK_SIZE + NETTY_BUFFERS))" --- End diff -- Append "M" to -XX:MaxDirectMemorySize parameter? > Integrate the off-heap memory configuration with the TaskManager start script > ----------------------------------------------------------------------------- > > Key: FLINK-2641 > URL: https://issues.apache.org/jira/browse/FLINK-2641 > Project: Flink > Issue Type: New Feature > Components: Start-Stop Scripts > Affects Versions: 0.10 > Reporter: Stephan Ewen > Assignee: Maximilian Michels > Fix For: 0.10 > > > The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and > {{-XX:MaxDirectMemorySize}} parameters according to the off-heap memory > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)