Hi KristoffSC As Zhu said, Flink enables slot sharing[1] by default. This feature is nothing to do with the resource of your cluster. The benefit of this feature is written in [1] as well. I mean, it will not detect how many slots in your cluster and adjust its behavior toward this number. If you want to make the best use of your cluster, you can increase the parallelism of the vertex that has the largest parallelism or "disable" the slot sharing by [2]. IMO, the first way matches your purpose.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html#task-slots-and-resources [2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/#task-chaining-and-resource-groups Best, Yangze Guo On Fri, Jan 10, 2020 at 6:49 PM KristoffSC <krzysiek.chmielew...@gmail.com> wrote: > > Hi Zhu Zhu, > well In my last test I did not change the job config, so I did not change > the parallelism level of any operator and I did not change policy regarding > slot sharing (it stays as default one). Operator Chaining is set to true > without any extra actions like "start new chain, disable chain etc" > > What I assume however is that Flink will try find most efficient way to use > available resources during job submission. > > In the first case, where I had only 6 task managers (which matches max > parallelism of my JobVertex), Flink reused some TaskSlots. Adding extra task > slots did was not effective because reason described by David. This is > understandable. > > However, I was assuming that if I submit my job on a cluster that have more > task managers than 6, Flink will not share task slots by default. That did > not happen. Flink deployed the job in the same way regardless of extra > resources. > > > So the conclusion is that simple job resubmitting will not work in this case > and actually I cant have any certainty that it will. Since in my case Flink > still reuses slot task. > > If this would be the production case, I would have to do a test job > submission on testing env and potentially change the job. Not the config, > but adding slot sharing groups etc. > So if this would be the production case I will not be able to react fast, I > would have to deploy new version of my app/job which could be problematic. > > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
