Andrii, Thanks for updating the KIP. Few comments -
Would you mind updating the Compatibility, Migration Plan section as well? We should discuss the new config store options - I'm not so sure that zookeeper is such a slam dunk choice for storing the configs. 1. Firstly, we have seen quite a few problems with zkClient which is currently the client we use to talk to ZooKeeper. In our observation, the extra layer of indirection it provides seemed to be useful when we were starting off but the more production experience we got, the more problems popped up. These include missed watches <https://issues.apache.org/jira/browse/KAFKA-1155> in some cases, no timeouts <https://issues.apache.org/jira/browse/KAFKA-1907>, limited APIs etc. This makes it much trickier to depend on ZooKeeper for important state information that needs to propagate in the cluster. We may have to investigate writing our own utility for interacting with ZooKeeper but even then it is not going to get much easier since we still have to solve the corner case issues we ran into with zkClient. 2. Secondly, this adds yet another piece of important state information we store in ZooKeeper assuming that zookeeper is the easiest alternative for key-value storage. However, we have gotten burnt by this in the past and still are. For example, admin commands like preferred replica election and partition reassignment spend quite some time updating the progress of the admin operation in ZooKeeper and hence slowing down the operation since ZooKeeper is not optimized for sudden spikes in write throughput. Also, I don't know if the argument that the information is necessarily safer in ZooKeeper is quite convincing, especially, with the successful move of consumer offsets from ZooKeeper to Kafka. It may not be a bad option to consider storing configs in a special durable Kafka topic. I'm not necessarily advocating this but would definitely like to discuss it. Before we move forward on adding more important state information in ZooKeeper, I think we should make an effort at reducing some technical debt we have accumulated and clean that up first. On Mon, Feb 16, 2015 at 9:18 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Hi all, > > I've added some details to Proposed Changes paragraph. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management > > Please let me know if something else require description / details. > > Thanks, > Andrii > > On Sat, Jan 24, 2015 at 1:39 AM, Jay Kreps <j...@confluent.io> wrote: > > > Cool. Yeah sorry to nag about these KIPs, and I hope it doesn't come > across > > the wrong way. But the hope I really have for these is that they are > > complete enough that even highly involved users can see and understand > the > > change, motivation, etc. I think that will do a ton to help extend the > > community. > > > > -Jay > > > > On Thu, Jan 22, 2015 at 10:22 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > There is still some to-dos to be done in > > > https://reviews.apache.org/r/29513/diff/ to use changing to ConfigDef > > > https://reviews.apache.org/r/30126/diff/ once that is in. > > > > > > We can get more written up on it, will do. > > > > > > On Fri, Jan 23, 2015 at 12:05 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > > > > > > > Hey Joe, > > > > > > > > Can you fill in this KIP? The purpose of these KIPs is to give a full > > > > overview of the feature, how it will work, be implemented, the > > > > considerations involved, etc. There is only like one sentence on this > > > which > > > > isn't enough for anyone to know what you are thinking. > > > > > > > > Moving off of configs to something that I'm guessing would be > > > > Zookeeper-based (?) is a massive change so we really need to describe > > > this > > > > in a way that can be widely circulated. > > > > > > > > I actually think this would be a good idea. But there are a ton of > > > > advantages to good old fashioned text files in terms of config > > management > > > > and change control. And trying to support both may or may not be > > better. > > > > > > > > -Jay > > > > > > > > > > > > On Wed, Jan 21, 2015 at 10:34 PM, Joe Stein <joe.st...@stealth.ly> > > > wrote: > > > > > > > > > Created a KIP > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-5+-+Broker+Configuration+Management > > > > > > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1786 > > > > > > > > > > /******************************************* > > > > > Joe Stein > > > > > Founder, Principal Consultant > > > > > Big Data Open Source Security LLC > > > > > http://www.stealth.ly > > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop > > > > > > > ********************************************/ > > > > > > > > > > > > > > > -- Thanks, Neha
