Theoretically, using just the broker id and zk connect string it should be possible for the broker to read all configs from Zookeeper. Like Ashish said, we should probably take a look and make sure.
Additionally, we've spoken about making config changes only through a broker API. However, we also need a way to change properties even if a specific broker, controller or entire cluster is down or unable to accept config change requests for any reason. This implies that we need a mechanism to make config changes by talking to zookeeper directly and that we cant rely solely on the broker/controller API. Aditya ________________________________________ From: Ashish Singh [asi...@cloudera.com] Sent: Thursday, May 07, 2015 8:19 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-21 Configuration Management Agreed :). However, the other concerns remain. Do you think just providing zk info to broker will be sufficient? I will myself spend some to look into the existing required confine. On Thursday, May 7, 2015, Jun Rao <j...@confluent.io> wrote: > Ashish, > > 3. This is true. However, using files has the same problem. You can't store > the location of the file in the file itself. The location of the file has > to be passed out of band into Kafka. > > Thanks, > > Jun > > On Wed, May 6, 2015 at 6:34 PM, Ashish Singh <asi...@cloudera.com > <javascript:;>> wrote: > > > Hey Jun, > > > > Where does the broker get the info, which zk it needs to talk to? > > > > On Wednesday, May 6, 2015, Jun Rao <j...@confluent.io <javascript:;>> > wrote: > > > > > Ashish, > > > > > > 3. Just want to clarify. Why can't you store ZK connection config in > ZK? > > > This is a property for ZK clients, not ZK server. > > > > > > Thanks, > > > > > > Jun > > > > > > On Wed, May 6, 2015 at 5:48 PM, Ashish Singh <asi...@cloudera.com > <javascript:;> > > > <javascript:;>> wrote: > > > > > > > I too would like to share some concerns that we came up with while > > > > discussing the effect of moving configs to zookeeper will have. > > > > > > > > 1. Kafka will start to become a configuration management tool to some > > > > degree, and be subject to all the things such tools are commonly > asked > > to > > > > do. Kafka'll likely need to re-implement the role / group / service > > > > hierarchy that CM uses. Kafka'll need some way to conveniently dump > its > > > > configs so they can be re-imported later, as a backup tool. People > will > > > > want this to be audited, which means you'd need distinct logins for > > > > different people, and user management. You can try to push some of > this > > > > stuff onto tools like CM, but this is Kafka going out of its way to > be > > > > difficult to manage, and most projects don't want to do that. Being > > > unique > > > > in how configuration is done is strictly a bad thing for both > > integration > > > > and usability. Probably lots of other stuff. Seems like a bad > > direction. > > > > > > > > 2. Where would the default config live? If we decide on keeping the > > > config > > > > files around just for getting the default config, then I think on > > > restart, > > > > the config file will be ignored. This creates an obnoxious asymmetry > > for > > > > how to configure Kafka the first time and how you update it. You have > > to > > > > learn 2 ways of making config changes. If there was a mistake in your > > > > original config file, you can't just edit the config file and > restart, > > > you > > > > have to go through the API. Reading configs is also more irritating. > > This > > > > all creates a learning curve for users of Kafka that will make it > > harder > > > to > > > > use than other projects. This is also a backwards-incompatible > change. > > > > > > > > 3. All Kafka configs living in ZK is strictly impossible, since at > the > > > very > > > > least ZK connection configs cannot be stored in ZK. So you will have > a > > > file > > > > where some values are in effect but others are not, which is again > > > > confusing. Also, since you are still reading the config file on first > > > > start, there are still multiple sources of truth, or at least the > > > > appearance of such to the user. > > > > > > > > On Wed, May 6, 2015 at 5:33 PM, Jun Rao <j...@confluent.io > <javascript:;> > > <javascript:;>> > > > wrote: > > > > > > > > > One of the Chef users confirmed that Chef integration could still > > work > > > if > > > > > all configs are moved to ZK. My rough understanding of how Chef > works > > > is > > > > > that a user first registers a service host with a Chef server. > After > > > > that, > > > > > a Chef client will be run on the service host. The user can then > push > > > > > config changes intended for a service/host to the Chef server. The > > > server > > > > > is then responsible for pushing the changes to Chef clients. Chef > > > clients > > > > > support pluggable logic. For example, it can generate a config file > > > that > > > > > Kafka broker will take. If we move all configs to ZK, we can > > customize > > > > the > > > > > Chef client to use our config CLI to make the config changes in > > Kafka. > > > In > > > > > this model, one probably doesn't need to register every broker in > > Chef > > > > for > > > > > the config push. Not sure if Puppet works in a similar way. > > > > > > > > > > Also for storing the configs, we probably can't store the > > broker/global > > > > > level configs in Kafka itself (e.g. in a special topic). The reason > > is > > > > that > > > > > in order to start a broker, we likely need to make some broker > level > > > > config > > > > > changes (e.g., the default log.dir may not be present, the default > > port > > > > may > > > > > not be available, etc). If we need a broker to be up to make those > > > > changes, > > > > > we get into this chicken and egg problem. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Tue, May 5, 2015 at 4:14 PM, Gwen Shapira < > gshap...@cloudera.com <javascript:;> > > > <javascript:;>> > > > > > wrote: > > > > > > > > > > > Sorry I missed the call today :) > > > > > > > > > > > > I think an additional requirement would be: > > > > > > Make sure that traditional deployment tools (Puppet, Chef, etc) > are > > > > still > > > > > > capable of managing Kafka configuration. > > > > > > > > > > > > For this reason, I'd like the configuration refresh to be pretty > > > close > > > > to > > > > > > what most Linux services are doing to force a reload of > > > configuration. > > > > > > AFAIK, this involves handling HUP signal in the main thread to > > reload > > > > > > configuration. Then packaging scripts can add something nice like > > > > > "service > > > > > > kafka reload". > > > > > > > > > > > > (See Apache web server: > > > > > > > > https://github.com/apache/httpd/blob/trunk/build/rpm/httpd.init#L101 > > > ) > > > > > > > > > > > > Gwen > > > > > > > > > > > > > > > > > > On Tue, May 5, 2015 at 8:54 AM, Joel Koshy <jjkosh...@gmail.com > <javascript:;> > > > <javascript:;>> > > > > wrote: > > > > > > > > > > > > > Good discussion. Since we will be talking about this at 11am, I > > > > wanted > > > > > > > to organize these comments into requirements to see if we are > all > > > on > > > > > > > the same page. > > > > > > > > > > > > > > REQUIREMENT 1: Needs to accept dynamic config changes. This > needs > > > to > > > > > > > be general enough to work for all configs that we envision may > > need > > > > to > > > > > > > accept changes at runtime. e.g., log (topic), broker, client > > > > (quotas), > > > > > > > etc.. possible options include: > > > > > > > - ZooKeeper watcher > > > > > > > - Kafka topic > > > > > > > - Direct RPC to controller (or config coordinator) > > > > > > > > > > > > > > The current KIP is really focused on REQUIREMENT 1 and I think > > that > > > > is > > > > > > > reasonable as long as we don't come up with something that > > requires > > > > > > > significant re-engineering to support the other requirements. > > > > > > > > > > > > > > REQUIREMENT 2: Provide consistency of configs across brokers > > > (modulo > > > > > > > per-broker overrides) or at least be able to verify > consistency. > > > > What > > > > > > > this effectively means is that config changes must be seen by > all > > > > > > > brokers eventually and we should be able to easily compare the > > full > > > > > > > config of each broker. > > > > > > > > > > > > > > REQUIREMENT 3: Central config store. Needs to work with plain > > > > > > > file-based configs and other systems (e.g., puppet). Ideally, > > > should > > > > > > > not bring in other dependencies (e.g., a DB). Possible options: > > > > > > > - ZooKeeper > > > > > > > - Kafka topic > > > > > > > - other? E.g. making it pluggable? > > > > > > > > > > > > > > Any other requirements? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > On Tue, May 05, 2015 at 01:38:09AM +0000, Aditya Auradkar > wrote: > > > > > > > > Hey Neha, > > > > > > > > > > > > > > > > Thanks for the feedback. > > > > > > > > 1. In my earlier exchange with Jay, I mentioned the broker > > > writing > > > > > all > > > > > > > it's configs to ZK (while respecting the overrides). Then ZK > can > > be > > > > > used > > > > > > to > > > > > > > view all configs. > > > > > > > > > > > > > > > > 2. Need to think about this a bit more. Perhaps we can > discuss > > > this > > > > > > > during the hangout tomorrow? > > > > > > > > > > > > > > > > 3 & 4) I viewed these config changes as mainly administrative > > > > > > > operations. In the case, it may be reasonable to assume that > the > > ZK > > > > > port > > > > > > is > > > > > > > available for communication from the machine these commands are > > > run. > > > > > > Having > > > > > > > a ConfigChangeRequest (or similar) is nice to have but having a > > new > > > > API > > > > > > and > > > > > > > sending requests to controller also change how we do topic > based > > > > > > > configuration currently. I was hoping to keep this KIP as > minimal > > > as > > > > > > > possible and provide a means to represent and modify client and > > > > broker > > > > > > > based configs in a central place. Are there any concerns if we > > > tackle > > > > > > these > > > > > > > things in a later KIP? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Aditya > > > > > > > > ________________________________________ > > > > > > > > From: Neha Narkhede [n...@confluent.io <javascript:;> > <javascript:;>] > > > > > > > > Sent: Sunday, May 03, 2015 9:48 AM > > > > > > > > To: dev@kafka.apache.org <javascript:;> <javascript:;> > > > > > > > > Subject: Re: [DISCUSS] KIP-21 Configuration Management > > > > > > > > > > > > > > > > Thanks for starting this discussion, Aditya. Few > > > questions/comments > > > > > > > > > > > > > > > > 1. If you change the default values like it's mentioned in > the > > > KIP, > > > > > do > > > > > > > you > > > > > > > > also overwrite the local config file as part of updating the > > > > default > > > > > > > value? > > > > > > > > If not, where does the admin look to find the default values, > > ZK > > > or > > > > > > local > > > > > > > > Kafka config file? What if a config value is different in > both > > > > > places? > > > > > > > > > > > > > > > > 2. I share Gwen's concern around making sure that popular > > config > > > > > > > management > > > > > > > > tools continue to work with this change. Would love to see > how > > > each > > > > > of > > > > > > > > those would work with the proposal in the KIP. I don't know > > > enough > > > > > > about > > > > > > > > each of the tools but seems like in some of the tools, you > have > > > to > > > > > > define > > > > > > > > some sort of class with parameter names as config names. How > > will > > > > > such > > > > > > > > tools find out about the config values? In Puppet, if this > > means > > > > that > > > > > > > each > > > > > > > > Puppet agent has to read it from ZK, this means the ZK port > has > > > to > > > > be > > > > > > > open > > > > > > > > to pretty much every machine in the DC. This is a bummer and > a > > > very > > > > > > > > confusing requirement. Not sure if this is really a problem > or > > > not > > > > > > (each > > > > > > > of > > > > > > > > those tools might behave differently), though pointing out > that > > > > this > > > > > is > > > > > > > > something worth paying attention to. > > > > > > > > > > > > > > > > 3. The wrapper tools that let users read/change config tools > > > should > > > > > not > > > > > > > > depend on ZK for the reason mentioned above. It's a pain to > > > assume > > > > > that > > > > > > > the > > > > > > > > ZK port is open from any machine that needs to run this tool. > > > > Ideally > > > > > > > what > > > > > > > > users want is a REST API to the brokers to change or read the > > > > config > > > > > > (ala > > > > > > > > Elasticsearch), but in the absence of the REST API, we should > > > think > > > > > if > > > > > > we > > > > > > > > can write the tool such that it just requires talking to the > > > Kafka > > > > > > broker > > > > > > > > port. This will require a config RPC. > > > > > > > > > > > > > > > > 4. Not sure if KIP is the right place to discuss the design > of > > > > > > > propagating > > > > > > > > the config changes to the brokers, but have you thought about > > > just > > > > > > > letting > > > > > > > > the controller oversee the config changes and propagate via > RPC > > > to > > > > > the > > > > > > > > brokers? That way, there is an easier way to express config > > > changes > > > > > > that > > > > > > > > require all brokers to change it for it to be called > complete. > > > > Maybe > > > > > > this > > > > > > > > is not required, but it is hard to say if we don't discuss > the > > > full > > > > > set > > > > > > > of > > > > > > > > configs that need to be dynamic. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Neha > > > > > > > > > > > > > > > > On Fri, May 1, 2015 at 12:53 PM, Jay Kreps < > > jay.kr...@gmail.com <javascript:;> > > > <javascript:;>> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hey Aditya, > > > > > > > > > > > > > > > > > > This is a great! A couple of comments: > > > > > > > > > > > > > > > > > > 1. Leaving the file config in place is definitely the least > > > > > > > disturbance. > > > > > > > > > But let's really think about getting rid of the files and > > just > > > > have > > > > > > one > > > > > > > > > config mechanism. There is always a tendency to make > > everything > > > > > > > pluggable > > > > > > > > > which so often just leads to two mediocre solutions. Can we > > do > > > > the > > > > > > > exercise > > > > > > > > > of trying to consider fully getting rid of file config and > > > seeing > > > > > > what > > > > > > > goes > > > > > > > > > wrong? > > > > > > > > > > > > > > > > > > 2. Do we need to model defaults? The current approach is > that > > > if > > > > > you > > > > > > > have a > > > > > > > > > global config x it is overridden for a topic xyz by > > > > /topics/xyz/x, > > > > > > and > > > > > > > I > > > > > > > > > think this could be extended to /brokers/0/x. I think this > is > > > > > > simpler. > > > > > > > We > > > > > > > > > need to specify the precedence for these overrides, e.g. if > > you > > > > > > > override at > > > > > > > > > the broker and topic level I think the topic level takes > > > > > precedence. > > > > > > > > > > > > > > > > > > 3. I recommend we have the producer and consumer config > just > > be > > > > an > > > > > > > override > > > > > > > > > under client.id. The override is by client id and we can > > have > > > > > > separate > > > > > > > > > properties for controlling quotas for producers and > > consumers. > > > > > > > > > > > > > > > > > > 4. Some configs can be changed just by updating the > > reference, > > > > > others > > > > > > > may > > > > > > > > > require some action. An example of this is if you want to > > > disable > > > > > log > > > > > > > > > compaction (assuming we wanted to make that dynamic) we > need > > to > > > > > call > > > > > > > > > shutdown() on the cleaner. I think it may be required to > > > > register a > > > > > > > > > listener callback that gets called when the config changes. > > > > > > > > > > > > > > > > > > 5. For handling the reference can you explain your plan a > > bit? > > > > > > > Currently we > > > > > > > > > have an immutable KafkaConfig object with a bunch of vals. > > That > > > > or > > > > > > > > > individual values in there get injected all over the code > > > base. I > > > > > was > > > > > > > > > thinking something like this: > > > > > > > > > a. We retain the KafkaConfig object as an immutable object > > just > > > > as > > > > > > > today. > > > > > > > > > b. It is no longer legit to grab values out fo that config > if > > > > they > > > > > > are > > > > > > > > > changeable. > > > > > > > > > c. Instead of making KafkaConfig itself mutable we make > > > > > > > KafkaConfiguration > > > > > > > > > which has a single volatile reference to the current > > > KafkaConfig. > > > > > > > > > KafkaConfiguration is what gets passed into various > > components. > > > > So > > > > > to > > > > > > > > > access a config you do something like > > config.instance.myValue. > > > > When > > > > > > the > > > > > > > > > config changes the config manager updates this reference. > > > > > > > > > d. The KafkaConfiguration is the thing that allows doing > the > > > > > > > > > configuration.onChange("my.config", callback) > > > > > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar < > > > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hey everyone, > > > > > > > > > > > > > > > > > > > > Wrote up a KIP to update topic, client and broker configs > > > > > > > dynamically via > > > > > > > > > > Zookeeper. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > > > > > > > Please read and provide feedback. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > > > PS: I've intentionally kept this discussion separate from > > > KIP-5 > > > > > > > since I'm > > > > > > > > > > not sure if that is actively being worked on and I wanted > > to > > > > > start > > > > > > > with a > > > > > > > > > > clean slate. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Thanks, > > > > > > > > Neha > > > > > > > > > > > > > > -- > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > Ashish > > > > > > > > > > > > > -- > > Ashish 🎤h > > > -- Ashish 🎤h
