Hello Olof, the new consumer on trunk has a feature for subscribing to a specific partition https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L200 for use by launching on Mesos. In this case the "rebalancing" doesn't happen since every instance is on a partition (so if you have 100 partitions you would have 100 instances running (or less with some basic business logic (e.g. 10 instances each would own 10 partitions, etc). This is great because if you lose 1 instance EVERY instance doesn't have to rebalance and they just keep running. Awesome!
Making mesos/kafka provide the ability for consumers to launch also is an interesting idea. Since each consumer is a custom application things like configuration management would be specific and different so there are a lot of nuances to consider. I am not saying it can't be done but we need to make sure the implementation is useful and generic enough for this. Perhaps we could provide the ability to supply the tgz for folks custom consumer so we can launch it (via the kafka scheduler) for you. I think though to-do this there would be a lot more than just scheduler changes we would likely have to right a new executor and some API so your consumer can get data from the executor or something. We can noodle on it. It would be great to hear more about what uses cases you (or anyone else) have in mind so we can see how it might work based on other implementations we work, see and know about. For what we do now on Mesos (for producers and consumers) is either run them as custom frameworks (because they do special things (e.g. Storm, Spark)) or via Marathon. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Thu, Jun 11, 2015 at 12:54 PM, Johansson, Olof < olof.johans...@thingworx.com> wrote: > Hi, > > I just started to look into the New Kafka Consumer where it's easier to > assign TopicPartitions to individual consumer instances. Ideally we would > like the assignment of TopicPartitions to be "smart". That is, be able to > apply similar scheduling, operations, and constraints as is implemented by > Mesos-Kafka today when it manages Kafka Brokers, and be able to assign > partitions based on the state of the consumers and consumer groups in the > cluster. > > Are there any plans to extend mesos-kafka to manage Consumers and not only > Brokers? If so, have the goals of such an effort been discussed? > > (I understand that due to what's currently available in Kafka it's not > easy to do. But it doesn't hurt to ask. :-) ) > > Thanks! > Olof >
