Re: Task managers run on separate nodes in a cluster

2018-09-16 Thread Martin Eden
Thanks for the feedback Liu and Till. @Liu Yeah that would work but unfortunately we run other services on the cluster so it's not really an option. @Till Will have a look and see how much time I can dedicate to this. M On Mon, Sep 17, 2018 at 7:21 AM Till Rohrmann wrote: > Hi Martin, > > I'm no

Re: Task managers run on separate nodes in a cluster

2018-09-16 Thread Till Rohrmann
Hi Martin, I'm not aware that the community is actively working on enabling the balanced host attribute constraint. If you wanna give it a try, then I'm happy to review your contribution. Cheers, Till On Mon, Sep 17, 2018 at 5:28 AM Renjie Liu wrote: > Hi, Martin: > I think a better solution w

Re: Task managers run on separate nodes in a cluster

2018-09-16 Thread Renjie Liu
Hi, Martin: I think a better solution would be to set the number of cores of each container equals to that of a physical server if this mesos cluster is dedicated to your flink cluster. On Mon, Sep 17, 2018 at 5:28 AM Martin Eden wrote: > Hi Till, > > I was able to use mesos.constraints.hard.hos

Re: Task managers run on separate nodes in a cluster

2018-09-16 Thread Martin Eden
Hi Till, I was able to use mesos.constraints.hard.hostattribute to run all task managers on a particular host in my cluster. However, after looking a bit at the code, I'm not sure we can use mesos.constraints.hard.hostattribute for load balancing Flink task managers evenly across hosts in a Mesos

Re: Task managers run on separate nodes in a cluster

2018-09-14 Thread Till Rohrmann
Hi Martin, Flink supports the mesos.constraints.hard.hostattribute to specify task constraints based on agent attributes [1]. I think you could use them to control the task placement. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#mesos-constraints-hard-hostattribute

Re: Task managers run on separate nodes in a cluster

2018-09-14 Thread Martin Eden
Thanks Vino! On Fri, Sep 14, 2018 at 3:37 AM vino yang wrote: > Hi Martin, > > Till has done most of the work of Flink on Mesos. Ping Till for you. > > Thanks, vino. > > Martin Eden 于2018年9月12日周三 下午11:21写道: > >> Hi all, >> >> We're using Flink 1.3.2 with DCOS / Mesos. >> >> We have a 3 node clu

Re: Task managers run on separate nodes in a cluster

2018-09-13 Thread vino yang
Hi Martin, Till has done most of the work of Flink on Mesos. Ping Till for you. Thanks, vino. Martin Eden 于2018年9月12日周三 下午11:21写道: > Hi all, > > We're using Flink 1.3.2 with DCOS / Mesos. > > We have a 3 node cluster and are running the Flink DCOS package (Flink > Mesos framework) configured w

Task managers run on separate nodes in a cluster

2018-09-12 Thread Martin Eden
Hi all, We're using Flink 1.3.2 with DCOS / Mesos. We have a 3 node cluster and are running the Flink DCOS package (Flink Mesos framework) configured with 3 Task Managers. Our goal is to run each of them on separate hosts for better load balancing but it seems the task managers end up running on