Hello community, I am trying to upgrade a Flink Yarn session cluster running BEAM pipelines from version 1.2.0 to 1.6.3.
Here is my session start command: yarn-session.sh -d -n 4 -jm 1024 -tm 3072 -s 7 Because of the dynamic resource allocation, no taskmanager gets created initially. Now once I submit a job with parallelism 5, I see that 1 task-manager gets created and all 5 parallel instances are scheduled on the same taskmanager( because I have 7 slots). This can create hot spot as only one physical node ( out of 4 in my case) is utilized for processing. I noticed the legacy mode, which would provision all task managers at cluster creation, but since legacy mode is expected to go away soon, I didn't want to try that route. Is there a way I can configure the multiple jobs or parallel instances of same job spread across all the available Yarn nodes and continue using the 'new' mode ? Thanks, Jins George
