[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15101945#comment-15101945 ] Ufuk Celebi commented on FLINK-3073: Minor fix added in 21882af > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057463#comment-15057463 ] Henry Saputra commented on FLINK-3073: -- The only thing I am bit concern is how fast we add and remove feature, in this case the streaming mode, which could add con > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048739#comment-15048739 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1433 > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048738#comment-15048738 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163264394 Manually merged > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048632#comment-15048632 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163221930 `preallocate` can be seen as one word, no? > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048527#comment-15048527 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163194389 is `pre-allocate` also ok? btw, we have a wild mixture of option styles, there is camelCase, under_score, and with-hyphens. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048508#comment-15048508 ] ASF GitHub Bot commented on FLINK-3073: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163192234 +1 from my side, after the switch to "preallocate" > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048475#comment-15048475 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163187358 PR looks good. +1 for `taskmanager.memory.preallocate`. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048458#comment-15048458 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163184597 Can do, yes. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048456#comment-15048456 ] ASF GitHub Bot commented on FLINK-3073: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163184249 Looks good! One thing that just came to my mind is whether we should call the config flag "taskmanager.memory.preallocate" (default false). I think preallocate is somehow easier to get than lazy allocate... > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048426#comment-15048426 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-163177539 Any objections against me merging this now? > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046719#comment-15046719 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162846459 I also discovered some other stuff, for example, the off-heap memory usage is not printed by the `MemoryLogger` (max is fixing this) and some stats in the web fronted about TaskManagers are wrong. For example "Free Memory" is simply the size of the JVM Heap. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046718#comment-15046718 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162846304 Any more comments? The log messages now look like this: ``` 11:28:58,212 INFO org.apache.flink.runtime.taskmanager.TaskManager - Limiting managed memory to 0.7 of the currently free heap space (295 MB), memory will be allocated lazily. ``` or ``` 11:30:09,859 INFO org.apache.flink.runtime.taskmanager.TaskManager - Using 0.7 of the currently free heap space for managed heap memory (295 MB). ``` > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044690#comment-15044690 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162469441 Did the changes to boolean option and merging of log messages. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044649#comment-15044649 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162464044 @StephanEwen So you want to merge the two messages @rmetzger posted? I'm on it. I'll also change the option to a boolean `lazy_alloc`, no need for the String option. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044007#comment-15044007 ] ASF GitHub Bot commented on FLINK-3073: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162334879 As followup to this, I would suggest two changes: 1. Adjust the log message, to reflect that not 0.7 are either used immediately (eager allocation) or that the managed memory is capped at 0.7 * total memory. That makes it clear from reading the log that the memory is not pre-allocated. 2. The web frontend should display for managed memory: (used / max). I have talked to users who were confused, thinking despite starting the system in streaming mode, it pre-allocated all the heap way, > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044003#comment-15044003 ] ASF GitHub Bot commented on FLINK-3073: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1433#issuecomment-162334401 Tested the change on a cluster with a job reading some 100GBs of data ... looks good .. ``` 19:07:19,537 INFO org.apache.flink.runtime.taskmanager.TaskManager - Using 0.7 of the currently free heap space for Flink managed heap memory (9001 MB). 19:07:19,538 INFO org.apache.flink.runtime.taskmanager.TaskManager - MemoryManager will be initialized with lazy memory allocation. ``` > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037911#comment-15037911 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46563274 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then # TM_MAX_OFFHEAP_SIZE="8388607T" -if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then +if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; then --- End diff -- You might want to allocate lazily in batch jobs with off-heap memory. In the current state of the pull request, we could run into out of memory errors then. On the other hand, I see that we don't want to subtract heap memory in streaming with lazy memory allocation (because we don't allocate managed memory for streaming jobs for now). > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037843#comment-15037843 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46557510 --- Diff: docs/setup/cluster_setup.md --- @@ -236,34 +236,18 @@ bin/start-cluster.sh To stop Flink, there is also a `stop-cluster.sh` script. - -### Starting Flink in the streaming mode - -~~~bash -bin/start-cluster-streaming.sh -~~~ - -The streaming mode changes the startup behavior of Flink: The system is not -bringing up the managed memory services with preallocated memory at the beginning. -Flink streaming is not using the managed memory employed by the batch operators. -By not starting these services with preallocated memory, streaming jobs can benefit -from more heap space being available. - -Note that you can still start batch jobs in the streaming mode. The memory manager -will then allocate memory segments from the Java heap as needed. - ### Optional: Adding JobManager/TaskManager instances to a cluster You can add both TaskManager or JobManager instances to your running cluster with the `bin/taskmanager.sh` and `bin/jobmanager.sh` scripts. Adding a TaskManager -bin/taskmanager.sh (start [batch|streaming])|stop|stop-all) +bin/taskmanager.sh start|stop|stop-all Adding a JobManager -bin/jobmanager.sh (start cluster [batch|streaming])|stop|stop-all) +bin/jobmanager.sh start|stop|stop-all --- End diff -- Could you preserve the "cluster" or "local" parameter? > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037865#comment-15037865 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46559041 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1617,7 +1598,16 @@ object TaskManager { } } -val preAllocateMemory: Boolean = streamingMode == StreamingMode.BATCH_ONLY +val allocationMode = configuration.getString( + ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY, + ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION); +val preAllocateMemory: Boolean = allocationMode.equals("eager") --- End diff -- will do > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037873#comment-15037873 ] ASF GitHub Bot commented on FLINK-3073: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46560453 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1617,7 +1598,16 @@ object TaskManager { } } -val preAllocateMemory: Boolean = streamingMode == StreamingMode.BATCH_ONLY +val allocationMode = configuration.getString( + ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY, + ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION); +val preAllocateMemory: Boolean = allocationMode.equals("eager") --- End diff -- There is already a lazy allocation flag (boolean) Can you reuse that simply? > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037855#comment-15037855 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46558585 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then # TM_MAX_OFFHEAP_SIZE="8388607T" -if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then +if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; then --- End diff -- are you sure? I just preserved the previous behavior. (eager == batch_mode, lazy == streaming_mode) > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037856#comment-15037856 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46558633 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1617,7 +1598,16 @@ object TaskManager { } } -val preAllocateMemory: Boolean = streamingMode == StreamingMode.BATCH_ONLY +val allocationMode = configuration.getString( + ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY, + ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION); +val preAllocateMemory: Boolean = allocationMode.equals("eager") --- End diff -- Could we verify whether we only have "eager" or "lazy" and throw an Exception otherwise? > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037853#comment-15037853 ] ASF GitHub Bot commented on FLINK-3073: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46558419 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then # TM_MAX_OFFHEAP_SIZE="8388607T" -if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then +if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; then --- End diff -- For the off-heap memory case, the JVM heap memory should be reduced regardless of eager or lazy memory allocation. I think this can be changed to ```bash if useOffHeapMemory; then ``` > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037846#comment-15037846 ] ASF GitHub Bot commented on FLINK-3073: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1433#discussion_r46557639 --- Diff: docs/setup/cluster_setup.md --- @@ -236,34 +236,18 @@ bin/start-cluster.sh To stop Flink, there is also a `stop-cluster.sh` script. - -### Starting Flink in the streaming mode - -~~~bash -bin/start-cluster-streaming.sh -~~~ - -The streaming mode changes the startup behavior of Flink: The system is not -bringing up the managed memory services with preallocated memory at the beginning. -Flink streaming is not using the managed memory employed by the batch operators. -By not starting these services with preallocated memory, streaming jobs can benefit -from more heap space being available. - -Note that you can still start batch jobs in the streaming mode. The memory manager -will then allocate memory segments from the Java heap as needed. - ### Optional: Adding JobManager/TaskManager instances to a cluster You can add both TaskManager or JobManager instances to your running cluster with the `bin/taskmanager.sh` and `bin/jobmanager.sh` scripts. Adding a TaskManager -bin/taskmanager.sh (start [batch|streaming])|stop|stop-all) +bin/taskmanager.sh start|stop|stop-all Adding a JobManager -bin/jobmanager.sh (start cluster [batch|streaming])|stop|stop-all) +bin/jobmanager.sh start|stop|stop-all --- End diff -- fixed it > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037830#comment-15037830 ] ASF GitHub Bot commented on FLINK-3073: --- GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1433 [FLINK-3073] Replace Streaming Mode by Memory Allocation Mode Before, streaming mode (either batch or streaming) would specify how memory is allocated on task managers. This introduces a new configuration value taskmanager.memory.allocation that can take values "lazy" or "eager". This controls how memory is allocated. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink stream-mode-default Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1433 commit bea4c112b7dbf0b7465b7419251173a742514967 Author: Aljoscha Krettek Date: 2015-12-01T15:13:55Z [FLINK-3073] Replace Streaming Mode by Memory Allocation Mode Before, streaming mode (either batch or streaming) would specify how memory is allocated on task managers. This introduces a new configuration value taskmanager.memory.allocation that can take values "lazy" or "eager". This controls how memory is allocated. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035879#comment-15035879 ] Maximilian Michels commented on FLINK-3073: --- +1 There is no difference between streaming and batch apart from the prellocation of the memory. If there is ever one, we can introduce a dedicated batch/streaming option again. Please provide an option in the config to preallocate the managed memory. I agree it should be turned off by default. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034151#comment-15034151 ] Robert Metzger commented on FLINK-3073: --- I would remove all the code for specific startup modes and just use the lazy allocating memory manager by default. > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033528#comment-15033528 ] Aljoscha Krettek commented on FLINK-3073: - Should we keep the distinction between streaming-mode and batch-mode nn case we have some future systems that would only need to be started if using the cluster primarily for batch or for streaming. Or should I change the name of the config value and internal fields and such to something like "memory pre-allocation mode". Because right now, it only changes the way how managed memory is allocated, either lazily (streaming) or upon startup (batch). > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger >Assignee: Aljoscha Krettek > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3073) Activate streaming mode by default
[ https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029808#comment-15029808 ] Aljoscha Krettek commented on FLINK-3073: - +1 > Activate streaming mode by default > -- > > Key: FLINK-3073 > URL: https://issues.apache.org/jira/browse/FLINK-3073 > Project: Flink > Issue Type: Improvement > Components: TaskManager >Reporter: Robert Metzger > Fix For: 1.0.0 > > > Currently, TaskManagers are still started in the batch mode. > I have the impression that more users are actually using Flink for stream > processing, and, the streaming mode also allows batch workloads. > It would be nice to change that for the 1.0 release -- This message was sent by Atlassian JIRA (v6.3.4#6332)