question: jvm heap size per task?
Hi, Greetings, hope this is the proper place to ask questions, apologize if not. We have a shared flink cluster running with docker, want to set different heap size per task(some tasks require larger heap size, while most tasks only need a little), is it feasible? I've gone through [1], [2] and [3], my current understanding is all task managers have the same heap size which is set by `taskmanager.heap.size` and is fixed ever since the task manager is started, and all tasks running in that task manager will share that heap size. Am I understanding it correctly? And any approaches to our needs? Thanks in advance! [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html [2]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html [3]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors Thanks, Li
Re: question: jvm heap size per task?
Hi Li, It is true that currently all the task managers have the same heap size, and it's fixed ever since started. Unfortunately, your needs cannot be satisfied at the moment. Heap size of task managers cannot be changed once started, because flink task managers run in JVMs and JVM does not support resizing after started. However, there is an ongoing approach towards your needs. The community is working on fine-grained resource management, which in general allows specify per task/operator resource requirements and allocate task manager resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for more details. Another related effort is pluggable slot manager [3], which allows having pluggable resource scheduling strategies such as launch task managers with customized resources according to the tasks' requirements. Thank you~ Xintong Song [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation [3] https://issues.apache.org/jira/browse/FLINK-14106 On Wed, Dec 25, 2019 at 4:18 PM Li Zhao wrote: > Hi, > > Greetings, hope this is the proper place to ask questions, apologize if > not. > > We have a shared flink cluster running with docker, want to set different > heap size per task(some tasks require larger heap size, while most tasks > only need a little), is it feasible? > > I've gone through [1], [2] and [3], my current understanding is all task > managers have the same heap size which is set by `taskmanager.heap.size` > and is fixed ever since the task manager is started, and all tasks running > in that task manager will share that heap size. > > Am I understanding it correctly? And any approaches to our needs? > > Thanks in advance! > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html > [2]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html > [3]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > Thanks, > Li >
Re: question: jvm heap size per task?
Understood, thank you for the quick response! Thanks, Li Xintong Song 于2019年12月25日周三 下午5:05写道： > Hi Li, > > It is true that currently all the task managers have the same heap size, > and it's fixed ever since started. Unfortunately, your needs cannot be > satisfied at the moment. > > Heap size of task managers cannot be changed once started, because flink > task managers run in JVMs and JVM does not support resizing after started. > > However, there is an ongoing approach towards your needs. The community is > working on fine-grained resource management, which in general allows > specify per task/operator resource requirements and allocate task manager > resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for > more details. Another related effort is pluggable slot manager [3], which > allows having pluggable resource scheduling strategies such as launch task > managers with customized resources according to the tasks' requirements. > > Thank you~ > > Xintong Song > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > [3] https://issues.apache.org/jira/browse/FLINK-14106 > > On Wed, Dec 25, 2019 at 4:18 PM Li Zhao wrote: > >> Hi, >> >> Greetings, hope this is the proper place to ask questions, apologize if >> not. >> >> We have a shared flink cluster running with docker, want to set different >> heap size per task(some tasks require larger heap size, while most tasks >> only need a little), is it feasible? >> >> I've gone through [1], [2] and [3], my current understanding is all task >> managers have the same heap size which is set by `taskmanager.heap.size` >> and is fixed ever since the task manager is started, and all tasks running >> in that task manager will share that heap size. >> >> Am I understanding it correctly? And any approaches to our needs? >> >> Thanks in advance! >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html >> [2]: >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html >> [3]: >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >> >> Thanks, >> Li >> >
