Hi Aditya, Thanks for the write-up. I like the idea of marking specific configurations as dynamic, I think this is just what we need.
Few comments: 1. I have deep concerns about managing configuration in ZooKeeper. First, Producers and Consumers shouldn't depend on ZK at all, this seems to add back a dependency we are trying to get away from. Second, no one manages configuration in ZK, which means that configuration management tools will no longer work smoothly with Kafka (Puppet, Chef, Ansible, Salt, etc). Also, it means that users will need to edit configuration with ZK commands which are not friendly (although we can work around them by providing tools). I'd prefer if we continue managing configuration in the existing file and somehow get Kafka to reread the file. Options can include: 1) New wire protocol for "refresh config" that will cause broker to re-read configuration 2) Thread that re-reads configs every 5 minutes 3) Single flag in ZK that says "re-read config file". 4) There are probably more... I realize that this doesn't solve the "centralized config management" problem - but I suspect this problem has already been solved outside of Kafka by now. 2. In addition to the configuration storage question, there's the extra step of "OK, we got new configs - now what?" which seems missing from the KIP - how do we propagate configuration changes to broker components? I'd like to see a generic way that everyone with dynamic config will implement, rather than having everyone make up something new. 3. I'd also like to see a list of existing configs that can / should become dynamic. Gwen On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Hey everyone, > > Wrote up a KIP to update topic, client and broker configs dynamically via > Zookeeper. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > Please read and provide feedback. > > Thanks, > Aditya > > PS: I've intentionally kept this discussion separate from KIP-5 since I'm > not sure if that is actively being worked on and I wanted to start with a > clean slate. >
