Hi, Currently, Flink does not support automatic scaling of the YARN containers. There are certainly plans to add this feature: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/add-some-new-api-to-the-scheduler-in-the-job-manager-tp7153p7448.html
Adding an API for manually starting and stopping containers is something we can do very easily, something dynamic is a bit harder. The feature is something I was always interested to work on, but nobody really asked for it. But it seems that more and more people are asking for it. If this is a requirement for you to use Flink, I'm certainly willing to add the feature into Flink. Best, Robert On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <p...@datameer.com> wrote: > Hi, > > I have started a Flink YARN session using yarn-session.sh and the > configuration of number of YARN container seems to be pretty static. > > Is it possible to have Flink adjust the number of containers depending on > the actual workload. E.g. stop containers that are idle for too long and > start them up again when more jobs get submitted? > Or are there any plans to have more dynamic resource management? > > Thanks, > —Peter
