Hi David, Thank you for your replay! Presumably I wasn’t clear in my previous post. Here an example to visualize what I'm trying to figure out:
Imagine we have a data flow propagating massages through a Kafka-Cluster which is happen to consist of 3 brokers (3 partitions, 3 replica). If one of those brokers goes down, Kafka does two things: - Broker rebalancing - Rebalancing the consumer within a group Now when marathon starts the failed broker again, some messages could get duplicated or missed… That is exactly what I would like to avoid (requirement). Make sense? Does someone have experience with Kafka on DC/OS + Marathon on a production environment and supports Exactly-Ones Semantic? Which case would you recommend? 1. Kafka on DC/OS + Marathon using Mesos private nodes (+ microservices on the public nodes) 2. Kafka on separate DC/OS-Cluster ? i.e. micro services have a different DC/OS Cluster 3. Kafka -Cluster on its own Cheers, Valentin > Am 02.10.2017 um 16:35 schrieb David Garcia <dav...@spiceworks.com>: > > I’m not sure how your requirements of Kafka are related to your requirements > for marathon. Kafka is a streaming-log system and marathon is a scheduler. > Mesos, as your resource manager, simply “manages” resources. Are you asking > about multitenancy? If so, I highly recommend that you separate your Kafka > cluster (and zookeeper) from your other services. Kafka leverages the OS > page cache to optimize read performance and it seems likely this would > interfere with Mesos resource management policy. > > -David > > On 10/2/17, 6:39 AM, "Valentin Forst" <valen...@aseno.de> wrote: > > Hi there, > > Working in a huge compony we are about to install Kafka on DC/OS (Mesos) > and intend to use Marathon as a Scheduler. Since I am new to DC/OS and > Marathon, I was wondering if this is a recommended way of using Kafka in the > production environment. > > My doubts are: > - Kafka manages Broker rebalancing (e.g. Failover, etc.) using its own > semantic. Can I trust Marathon that it will match the requirements here? > - Since our Container Platform - DC/OS is going to be used by other „micro > services“ - soon or later this is going to raise a performance issue. Should > we better use a dedicated DC/OS instance for our Kafka-Cluster? Or > Kafka-Cluster on its own? > - Is there something else we should consider important if using Kafka on > DC/OS + Marathon? > > > Thanks in advance for your time. > Valentin > > >
