Aditya,
In the following, we should encode the config properties as a json map to
be consistent with topic config.
"Internally, the znodes are comma-separated key-value pairs where key
represents the configuration property to change.
{"version": x, "config" : {X1=Y1, X2=Y2..}}"
Thanks,
Jun
On Fri, May 15, 2015 at 1:09 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:
> Yes we did. I just overlooked that line.. cleaning it up now.
>
> Aditya
>
> ________________________________________
> From: Gwen Shapira [gshap...@cloudera.com]
> Sent: Friday, May 15, 2015 12:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-21 Configuration Management
>
> The wiki says:
> "There will be 3 paths within config
> /config/clients/<client_id>
> /config/topics/<topic_name>
> /config/brokers/<broker_id>
> Didn't we decide that brokers will not be configured dynamically, rather we
> will keep the config in the file?
>
> On Fri, May 15, 2015 at 10:46 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Updated the wiki to capture our recent discussions. Please read.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> >
> > Thanks,
> > Aditya
> >
> > ________________________________________
> > From: Joel Koshy [jjkosh...@gmail.com]
> > Sent: Tuesday, May 12, 2015 1:09 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-21 Configuration Management
> >
> > The lack of audit could be addressed to some degree by an internal
> > __config_changes topic which can have very long retention. Also, per
> > the hangout summary that Gwen sent out it appears that we decided
> > against supporting SIGHUP/dynamic configs for the broker.
> >
> > Thanks,
> >
> > Joel
> >
> > On Tue, May 12, 2015 at 11:05:06AM -0700, Neha Narkhede wrote:
> > > Thanks for chiming in, Todd!
> > >
> > > Agree that the lack of audit and rollback is a major downside of moving
> > all
> > > configs to ZooKeeper. Being able to configure dynamically created
> > entities
> > > in Kafka is required though. So I think what Todd suggested is a good
> > > solution to managing all configs - catching SIGHUP for broker configs
> and
> > > storing dynamic configs in ZK like we do today.
> > >
> > > On Tue, May 12, 2015 at 10:30 AM, Jay Kreps <jay.kr...@gmail.com>
> wrote:
> > >
> > > > Hmm, here is how I think we can change the split brain proposal to
> > make it
> > > > a bit better:
> > > > 1. Get rid of broker overrides, this is just done in the config file.
> > This
> > > > makes the precedence chain a lot clearer (e.g. zk always overrides
> > file on
> > > > a per-entity basis).
> > > > 2. Get rid of the notion of dynamic configs in ConfigDef and in the
> > broker.
> > > > All overrides are dynamic and all server configs are static.
> > > > 3. Create an equivalent of LogConfig for ClientConfig and any future
> > config
> > > > type we make.
> > > > 4. Generalize the TopicConfigManager to handle multiple types of
> > overrides.
> > > >
> > > > What we haven't done is try to think through how the pure zk approach
> > would
> > > > work.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > > On Mon, May 11, 2015 at 10:53 PM, Ashish Singh <asi...@cloudera.com>
> > > > wrote:
> > > >
> > > > > I agree with the Joel's suggestion on keeping broker's configs in
> > > > > config file and clients/topics config in ZK. Few other projects,
> > Apache
> > > > > Solr for one, also does something similar for its configurations.
> > > > >
> > > > > On Monday, May 11, 2015, Gwen Shapira <gshap...@cloudera.com>
> wrote:
> > > > >
> > > > > > I like this approach (obviously).
> > > > > > I am also OK with supporting broker re-read of config file based
> > on ZK
> > > > > > watch instead of SIGHUP, if we see this as more consistent with
> the
> > > > rest
> > > > > of
> > > > > > our code base.
> > > > > >
> > > > > > Either is fine by me as long as brokers keep the file and just do
> > > > refresh
> > > > > > :)
> > > > > >
> > > > > > On Tue, May 12, 2015 at 2:54 AM, Joel Koshy <jjkosh...@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > >
> > > > > > > So the general concern here is the dichotomy of configs (which
> we
> > > > > > > already have - i.e., in the form of broker config files vs
> topic
> > > > > > > configs in zookeeper). We (at LinkedIn) had some discussions on
> > this
> > > > > > > last week and had this very question for the operations team
> > whose
> > > > > > > opinion is I think to a large degree a touchstone for this
> > decision:
> > > > > > > "Has the operations team at LinkedIn experienced any pain so
> far
> > with
> > > > > > > managing topic configs in ZooKeeper (while broker configs are
> > > > > > > file-based)?" It turns out that ops overwhelmingly favors the
> > current
> > > > > > > approach. i.e., service configs as file-based configs and
> > > > client/topic
> > > > > > > configs in ZooKeeper is intuitive and works great. This may be
> > > > > > > somewhat counter-intuitive to devs, but this is one of those
> > > > decisions
> > > > > > > for which ops input is very critical - because for all
> practical
> > > > > > > purposes, they are the users in this discussion.
> > > > > > >
> > > > > > > If we continue with this dichotomy and need to support dynamic
> > config
> > > > > > > for client/topic configs as well as select service configs then
> > there
> > > > > > > will need to be dichotomy in the config change mechanism as
> well.
> > > > > > > i.e., client/topic configs will change via (say) a ZooKeeper
> > watch
> > > > and
> > > > > > > the service config will change via a config file re-read (on
> > SIGHUP)
> > > > > > > after config changes have been pushed out to local files. Is
> > this a
> > > > > > > bad thing? Personally, I don't think it is - i.e. I'm in favor
> of
> > > > this
> > > > > > > approach. What do others think?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Joel
> > > > > > >
> > > > > > > On Mon, May 11, 2015 at 11:08:44PM +0300, Gwen Shapira wrote:
> > > > > > > > What Todd said :)
> > > > > > > >
> > > > > > > > (I think my ops background is showing...)
> > > > > > > >
> > > > > > > > On Mon, May 11, 2015 at 10:17 PM, Todd Palino <
> > tpal...@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > > > >
> > > > > > > > > I understand your point here, Jay, but I disagree that we
> > can't
> > > > > have
> > > > > > > two
> > > > > > > > > configuration systems. We have two different types of
> > > > configuration
> > > > > > > > > information. We have configuration that relates to the
> > service
> > > > > itself
> > > > > > > (the
> > > > > > > > > Kafka broker), and we have configuration that relates to
> the
> > > > > content
> > > > > > > within
> > > > > > > > > the service (topics). I would put the client configuration
> > > > (quotas)
> > > > > > in
> > > > > > > the
> > > > > > > > > with the second part, as it is dynamic information. I just
> > don't
> > > > > see
> > > > > > a
> > > > > > > good
> > > > > > > > > argument for effectively degrading the configuration for
> the
> > > > > service
> > > > > > > > > because of trying to keep it paired with the configuration
> of
> > > > > dynamic
> > > > > > > > > resources.
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Mon, May 11, 2015 at 11:33 AM, Jay Kreps <
> > jay.kr...@gmail.com
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I totally agree that ZK is not in-and-of-itself a
> > configuration
> > > > > > > > > management
> > > > > > > > > > solution and it would be better if we could just keep all
> > our
> > > > > > config
> > > > > > > in
> > > > > > > > > > files. Anyone who has followed the various config
> > discussions
> > > > > over
> > > > > > > the
> > > > > > > > > past
> > > > > > > > > > few years of discussion knows I'm the biggest proponent
> of
> > > > > > immutable
> > > > > > > > > > file-driven config.
> > > > > > > > > >
> > > > > > > > > > The analogy to "normal unix services" isn't actually
> quite
> > > > right
> > > > > > > though.
> > > > > > > > > > The problem Kafka has is that a number of the
> configurable
> > > > > entities
> > > > > > > it
> > > > > > > > > > manages are added dynamically--topics, clients, consumer
> > > > groups,
> > > > > > etc.
> > > > > > > > > What
> > > > > > > > > > this actually resembles is not a unix services like HTTPD
> > but a
> > > > > > > database,
> > > > > > > > > > and databases typically do manage config dynamically for
> > > > exactly
> > > > > > the
> > > > > > > same
> > > > > > > > > > reason.
> > > > > > > > > >
> > > > > > > > > > The last few emails are arguing that files > ZK as a
> config
> > > > > > > solution. I
> > > > > > > > > > agree with this, but that isn't really the question,
> > right?The
> > > > > > > reality is
> > > > > > > > > > that we need to be able to configure dynamically created
> > > > entities
> > > > > > > and we
> > > > > > > > > > won't get a satisfactory solution to that using files
> (e.g.
> > > > rsync
> > > > > > is
> > > > > > > not
> > > > > > > > > an
> > > > > > > > > > acceptable topic creation mechanism). What we are
> > discussing is
> > > > > > > having a
> > > > > > > > > > single config mechanism or multiple. If we have multiple
> > you
> > > > need
> > > > > > to
> > > > > > > > > solve
> > > > > > > > > > the whole config lifecycle problem for both--management,
> > audit,
> > > > > > > rollback,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Gwen, you were saying we couldn't get rid of the
> > configuration
> > > > > > file,
> > > > > > > not
> > > > > > > > > > sure if I understand. Is that because we need to give the
> > URL
> > > > for
> > > > > > ZK?
> > > > > > > > > > Wouldn't the same argument work to say that we can't use
> > > > > > > configuration
> > > > > > > > > > files because we have to specify the file path? I think
> we
> > can
> > > > > just
> > > > > > > give
> > > > > > > > > > the server the same --zookeeper argument we use
> everywhere
> > > > else,
> > > > > > > right?
> > > > > > > > > >
> > > > > > > > > > -Jay
> > > > > > > > > >
> > > > > > > > > > On Sun, May 10, 2015 at 11:28 AM, Todd Palino <
> > > > tpal...@gmail.com
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I've been watching this discussion for a while, and I
> > have to
> > > > > > jump
> > > > > > > in
> > > > > > > > > and
> > > > > > > > > > > side with Gwen here. I see no benefit to putting the
> > configs
> > > > > into
> > > > > > > > > > Zookeeper
> > > > > > > > > > > entirely, and a lot of downside. The two biggest
> > problems I
> > > > > have
> > > > > > > with
> > > > > > > > > > this
> > > > > > > > > > > are:
> > > > > > > > > > >
> > > > > > > > > > > 1) Configuration management. OK, so you can write glue
> > for
> > > > Chef
> > > > > > to
> > > > > > > put
> > > > > > > > > > > configs into Zookeeper. You also need to write glue for
> > > > Puppet.
> > > > > > And
> > > > > > > > > > > Cfengine. And everything else out there. Files are an
> > > > industry
> > > > > > > standard
> > > > > > > > > > > practice, they're how just about everyone handles it,
> and
> > > > > there's
> > > > > > > > > reasons
> > > > > > > > > > > for that, not just "it's the way it's always been
> done".
> > > > > > > > > > >
> > > > > > > > > > > 2) Auditing. Configuration files can easily be managed
> > in a
> > > > > > source
> > > > > > > > > > > repository system which tracks what changes were made
> > and who
> > > > > > made
> > > > > > > > > them.
> > > > > > > > > > It
> > > > > > > > > > > also easily allows for rolling back to a previous
> > version.
> > > > > > > Zookeeper
> > > > > > > > > does
> > > > > > > > > > > not.
> > > > > > > > > > >
> > > > > > > > > > > I see absolutely nothing wrong with putting the quota
> > > > (client)
> > > > > > > configs
> > > > > > > > > > and
> > > > > > > > > > > the topic config overrides in Zookeeper, and keeping
> > > > everything
> > > > > > > else
> > > > > > > > > > > exactly where it is, in the configuration file. To
> handle
> > > > > > > > > configurations
> > > > > > > > > > > for the broker that can be changed at runtime without a
> > > > > restart,
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > > > use the industry standard practice of catching SIGHUP
> and
> > > > > > > rereading the
> > > > > > > > > > > configuration file at that point.
> > > > > > > > > > >
> > > > > > > > > > > -Todd
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, May 10, 2015 at 4:00 AM, Gwen Shapira <
> > > > > > > gshap...@cloudera.com <javascript:;>>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I am still not clear about the benefits of managing
> > > > > > > configuration in
> > > > > > > > > > > > ZooKeeper vs. keeping the local file and adding a
> > "refresh"
> > > > > > > mechanism
> > > > > > > > > > > > (signal, protocol, zookeeper, or other).
> > > > > > > > > > > >
> > > > > > > > > > > > Benefits of staying with configuration file:
> > > > > > > > > > > > 1. In line with pretty much any Linux service that
> > exists,
> > > > so
> > > > > > > admins
> > > > > > > > > > > have a
> > > > > > > > > > > > lot of related experience.
> > > > > > > > > > > > 2. Much smaller change to our code-base, so easier to
> > > > patch,
> > > > > > > review
> > > > > > > > > and
> > > > > > > > > > > > test. Lower risk overall.
> > > > > > > > > > > >
> > > > > > > > > > > > Can you walk me over the benefits of using Zookeeper?
> > > > > > Especially
> > > > > > > > > since
> > > > > > > > > > it
> > > > > > > > > > > > looks like we can't get rid of the file entirely?
> > > > > > > > > > > >
> > > > > > > > > > > > Gwen
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 7, 2015 at 3:33 AM, Jun Rao <
> > j...@confluent.io
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > One of the Chef users confirmed that Chef
> integration
> > > > could
> > > > > > > still
> > > > > > > > > > work
> > > > > > > > > > > if
> > > > > > > > > > > > > all configs are moved to ZK. My rough understanding
> > of
> > > > how
> > > > > > Chef
> > > > > > > > > works
> > > > > > > > > > > is
> > > > > > > > > > > > > that a user first registers a service host with a
> > Chef
> > > > > > server.
> > > > > > > > > After
> > > > > > > > > > > > that,
> > > > > > > > > > > > > a Chef client will be run on the service host. The
> > user
> > > > can
> > > > > > > then
> > > > > > > > > push
> > > > > > > > > > > > > config changes intended for a service/host to the
> > Chef
> > > > > > server.
> > > > > > > The
> > > > > > > > > > > server
> > > > > > > > > > > > > is then responsible for pushing the changes to Chef
> > > > > clients.
> > > > > > > Chef
> > > > > > > > > > > clients
> > > > > > > > > > > > > support pluggable logic. For example, it can
> > generate a
> > > > > > config
> > > > > > > file
> > > > > > > > > > > that
> > > > > > > > > > > > > Kafka broker will take. If we move all configs to
> > ZK, we
> > > > > can
> > > > > > > > > > customize
> > > > > > > > > > > > the
> > > > > > > > > > > > > Chef client to use our config CLI to make the
> config
> > > > > changes
> > > > > > in
> > > > > > > > > > Kafka.
> > > > > > > > > > > In
> > > > > > > > > > > > > this model, one probably doesn't need to register
> > every
> > > > > > broker
> > > > > > > in
> > > > > > > > > > Chef
> > > > > > > > > > > > for
> > > > > > > > > > > > > the config push. Not sure if Puppet works in a
> > similar
> > > > way.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also for storing the configs, we probably can't
> > store the
> > > > > > > > > > broker/global
> > > > > > > > > > > > > level configs in Kafka itself (e.g. in a special
> > topic).
> > > > > The
> > > > > > > reason
> > > > > > > > > > is
> > > > > > > > > > > > that
> > > > > > > > > > > > > in order to start a broker, we likely need to make
> > some
> > > > > > broker
> > > > > > > > > level
> > > > > > > > > > > > config
> > > > > > > > > > > > > changes (e.g., the default log.dir may not be
> > present,
> > > > the
> > > > > > > default
> > > > > > > > > > port
> > > > > > > > > > > > may
> > > > > > > > > > > > > not be available, etc). If we need a broker to be
> up
> > to
> > > > > make
> > > > > > > those
> > > > > > > > > > > > changes,
> > > > > > > > > > > > > we get into this chicken and egg problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 5, 2015 at 4:14 PM, Gwen Shapira <
> > > > > > > > > gshap...@cloudera.com <javascript:;>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry I missed the call today :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think an additional requirement would be:
> > > > > > > > > > > > > > Make sure that traditional deployment tools
> > (Puppet,
> > > > > Chef,
> > > > > > > etc)
> > > > > > > > > are
> > > > > > > > > > > > still
> > > > > > > > > > > > > > capable of managing Kafka configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For this reason, I'd like the configuration
> > refresh to
> > > > be
> > > > > > > pretty
> > > > > > > > > > > close
> > > > > > > > > > > > to
> > > > > > > > > > > > > > what most Linux services are doing to force a
> > reload of
> > > > > > > > > > > configuration.
> > > > > > > > > > > > > > AFAIK, this involves handling HUP signal in the
> > main
> > > > > thread
> > > > > > > to
> > > > > > > > > > reload
> > > > > > > > > > > > > > configuration. Then packaging scripts can add
> > something
> > > > > > nice
> > > > > > > like
> > > > > > > > > > > > > "service
> > > > > > > > > > > > > > kafka reload".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (See Apache web server:
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > https://github.com/apache/httpd/blob/trunk/build/rpm/httpd.init#L101
> > > > > > > > > > > )
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Gwen
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 5, 2015 at 8:54 AM, Joel Koshy <
> > > > > > > jjkosh...@gmail.com <javascript:;>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Good discussion. Since we will be talking about
> > this
> > > > at
> > > > > > > 11am, I
> > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > to organize these comments into requirements to
> > see
> > > > if
> > > > > we
> > > > > > > are
> > > > > > > > > all
> > > > > > > > > > > on
> > > > > > > > > > > > > > > the same page.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > REQUIREMENT 1: Needs to accept dynamic config
> > > > changes.
> > > > > > This
> > > > > > > > > needs
> > > > > > > > > > > to
> > > > > > > > > > > > > > > be general enough to work for all configs that
> we
> > > > > > envision
> > > > > > > may
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > accept changes at runtime. e.g., log (topic),
> > broker,
> > > > > > > client
> > > > > > > > > > > > (quotas),
> > > > > > > > > > > > > > > etc.. possible options include:
> > > > > > > > > > > > > > > - ZooKeeper watcher
> > > > > > > > > > > > > > > - Kafka topic
> > > > > > > > > > > > > > > - Direct RPC to controller (or config
> > coordinator)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The current KIP is really focused on
> REQUIREMENT
> > 1
> > > > and
> > > > > I
> > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > reasonable as long as we don't come up with
> > something
> > > > > > that
> > > > > > > > > > requires
> > > > > > > > > > > > > > > significant re-engineering to support the other
> > > > > > > requirements.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > REQUIREMENT 2: Provide consistency of configs
> > across
> > > > > > > brokers
> > > > > > > > > > > (modulo
> > > > > > > > > > > > > > > per-broker overrides) or at least be able to
> > verify
> > > > > > > > > consistency.
> > > > > > > > > > > > What
> > > > > > > > > > > > > > > this effectively means is that config changes
> > must be
> > > > > > seen
> > > > > > > by
> > > > > > > > > all
> > > > > > > > > > > > > > > brokers eventually and we should be able to
> > easily
> > > > > > compare
> > > > > > > the
> > > > > > > > > > full
> > > > > > > > > > > > > > > config of each broker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > REQUIREMENT 3: Central config store. Needs to
> > work
> > > > with
> > > > > > > plain
> > > > > > > > > > > > > > > file-based configs and other systems (e.g.,
> > puppet).
> > > > > > > Ideally,
> > > > > > > > > > > should
> > > > > > > > > > > > > > > not bring in other dependencies (e.g., a DB).
> > > > Possible
> > > > > > > options:
> > > > > > > > > > > > > > > - ZooKeeper
> > > > > > > > > > > > > > > - Kafka topic
> > > > > > > > > > > > > > > - other? E.g. making it pluggable?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any other requirements?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Joel
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, May 05, 2015 at 01:38:09AM +0000,
> Aditya
> > > > > Auradkar
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > Hey Neha,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the feedback.
> > > > > > > > > > > > > > > > 1. In my earlier exchange with Jay, I
> > mentioned the
> > > > > > > broker
> > > > > > > > > > > writing
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > > it's configs to ZK (while respecting the
> > overrides).
> > > > > Then
> > > > > > > ZK
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > view all configs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. Need to think about this a bit more.
> > Perhaps we
> > > > > can
> > > > > > > > > discuss
> > > > > > > > > > > this
> > > > > > > > > > > > > > > during the hangout tomorrow?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3 & 4) I viewed these config changes as
> mainly
> > > > > > > administrative
> > > > > > > > > > > > > > > operations. In the case, it may be reasonable
> to
> > > > assume
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > ZK
> > > > > > > > > > > > > port
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > available for communication from the machine
> > these
> > > > > > > commands are
> > > > > > > > > > > run.
> > > > > > > > > > > > > > Having
> > > > > > > > > > > > > > > a ConfigChangeRequest (or similar) is nice to
> > have
> > > > but
> > > > > > > having a
> > > > > > > > > > new
> > > > > > > > > > > > API
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > sending requests to controller also change how
> > we do
> > > > > > topic
> > > > > > > > > based
> > > > > > > > > > > > > > > configuration currently. I was hoping to keep
> > this
> > > > KIP
> > > > > as
> > > > > > > > > minimal
> > > > > > > > > > > as
> > > > > > > > > > > > > > > possible and provide a means to represent and
> > modify
> > > > > > > client and
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > based configs in a central place. Are there any
> > > > > concerns
> > > > > > > if we
> > > > > > > > > > > tackle
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > things in a later KIP?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Aditya
> > > > > > > > > > > > > > > > ________________________________________
> > > > > > > > > > > > > > > > From: Neha Narkhede [n...@confluent.io
> > > > > <javascript:;>]
> > > > > > > > > > > > > > > > Sent: Sunday, May 03, 2015 9:48 AM
> > > > > > > > > > > > > > > > To: dev@kafka.apache.org <javascript:;>
> > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-21 Configuration
> > > > > Management
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for starting this discussion, Aditya.
> > Few
> > > > > > > > > > > questions/comments
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. If you change the default values like it's
> > > > > mentioned
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > KIP,
> > > > > > > > > > > > > do
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > also overwrite the local config file as part
> of
> > > > > > updating
> > > > > > > the
> > > > > > > > > > > > default
> > > > > > > > > > > > > > > value?
> > > > > > > > > > > > > > > > If not, where does the admin look to find the
> > > > default
> > > > > > > values,
> > > > > > > > > > ZK
> > > > > > > > > > > or
> > > > > > > > > > > > > > local
> > > > > > > > > > > > > > > > Kafka config file? What if a config value is
> > > > > different
> > > > > > in
> > > > > > > > > both
> > > > > > > > > > > > > places?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. I share Gwen's concern around making sure
> > that
> > > > > > popular
> > > > > > > > > > config
> > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > tools continue to work with this change.
> Would
> > love
> > > > > to
> > > > > > > see
> > > > > > > > > how
> > > > > > > > > > > each
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > those would work with the proposal in the
> KIP.
> > I
> > > > > don't
> > > > > > > know
> > > > > > > > > > > enough
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > each of the tools but seems like in some of
> the
> > > > > tools,
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > some sort of class with parameter names as
> > config
> > > > > > names.
> > > > > > > How
> > > > > > > > > > will
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > tools find out about the config values? In
> > Puppet,
> > > > if
> > > > > > > this
> > > > > > > > > > means
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > Puppet agent has to read it from ZK, this
> > means the
> > > > > ZK
> > > > > > > port
> > > > > > > > > has
> > > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > open
> > > > > > > > > > > > > > > > to pretty much every machine in the DC. This
> > is a
> > > > > > bummer
> > > > > > > and
> > > > > > > > > a
> > > > > > > > > > > very
> > > > > > > > > > > > > > > > confusing requirement. Not sure if this is
> > really a
> > > > > > > problem
> > > > > > > > > or
> > > > > > > > > > > not
> > > > > > > > > > > > > > (each
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > those tools might behave differently), though
> > > > > pointing
> > > > > > > out
> > > > > > > > > that
> > > > > > > > > > > > this
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > something worth paying attention to.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. The wrapper tools that let users
> read/change
> > > > > config
> > > > > > > tools
> > > > > > > > > > > should
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > depend on ZK for the reason mentioned above.
> > It's a
> > > > > > pain
> > > > > > > to
> > > > > > > > > > > assume
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > ZK port is open from any machine that needs
> to
> > run
> > > > > this
> > > > > > > tool.
> > > > > > > > > > > > Ideally
> > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > users want is a REST API to the brokers to
> > change
> > > > or
> > > > > > > read the
> > > > > > > > > > > > config
> > > > > > > > > > > > > > (ala
> > > > > > > > > > > > > > > > Elasticsearch), but in the absence of the
> REST
> > API,
> > > > > we
> > > > > > > should
> > > > > > > > > > > think
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > can write the tool such that it just requires
> > > > talking
> > > > > > to
> > > > > > > the
> > > > > > > > > > > Kafka
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > port. This will require a config RPC.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4. Not sure if KIP is the right place to
> > discuss
> > > > the
> > > > > > > design
> > > > > > > > > of
> > > > > > > > > > > > > > > propagating
> > > > > > > > > > > > > > > > the config changes to the brokers, but have
> you
> > > > > thought
> > > > > > > about
> > > > > > > > > > > just
> > > > > > > > > > > > > > > letting
> > > > > > > > > > > > > > > > the controller oversee the config changes and
> > > > > propagate
> > > > > > > via
> > > > > > > > > RPC
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > brokers? That way, there is an easier way to
> > > > express
> > > > > > > config
> > > > > > > > > > > changes
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > require all brokers to change it for it to be
> > > > called
> > > > > > > > > complete.
> > > > > > > > > > > > Maybe
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > is not required, but it is hard to say if we
> > don't
> > > > > > > discuss
> > > > > > > > > the
> > > > > > > > > > > full
> > > > > > > > > > > > > set
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > configs that need to be dynamic.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Neha
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, May 1, 2015 at 12:53 PM, Jay Kreps <
> > > > > > > > > > jay.kr...@gmail.com <javascript:;>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hey Aditya,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is a great! A couple of comments:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Leaving the file config in place is
> > definitely
> > > > > the
> > > > > > > least
> > > > > > > > > > > > > > > disturbance.
> > > > > > > > > > > > > > > > > But let's really think about getting rid of
> > the
> > > > > files
> > > > > > > and
> > > > > > > > > > just
> > > > > > > > > > > > have
> > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > config mechanism. There is always a
> tendency
> > to
> > > > > make
> > > > > > > > > > everything
> > > > > > > > > > > > > > > pluggable
> > > > > > > > > > > > > > > > > which so often just leads to two mediocre
> > > > > solutions.
> > > > > > > Can we
> > > > > > > > > > do
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > exercise
> > > > > > > > > > > > > > > > > of trying to consider fully getting rid of
> > file
> > > > > > config
> > > > > > > and
> > > > > > > > > > > seeing
> > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > goes
> > > > > > > > > > > > > > > > > wrong?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. Do we need to model defaults? The
> current
> > > > > approach
> > > > > > > is
> > > > > > > > > that
> > > > > > > > > > > if
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > have a
> > > > > > > > > > > > > > > > > global config x it is overridden for a
> topic
> > xyz
> > > > by
> > > > > > > > > > > > /topics/xyz/x,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think this could be extended to
> > /brokers/0/x. I
> > > > > think
> > > > > > > this
> > > > > > > > > is
> > > > > > > > > > > > > > simpler.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > need to specify the precedence for these
> > > > overrides,
> > > > > > > e.g. if
> > > > > > > > > > you
> > > > > > > > > > > > > > > override at
> > > > > > > > > > > > > > > > > the broker and topic level I think the
> topic
> > > > level
> > > > > > > takes
> > > > > > > > > > > > > precedence.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. I recommend we have the producer and
> > consumer
> > > > > > config
> > > > > > > > > just
> > > > > > > > > > be
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > override
> > > > > > > > > > > > > > > > > under client.id. The override is by client
> > id
> > > > and
> > > > > we
> > > > > > > can
> > > > > > > > > > have
> > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > properties for controlling quotas for
> > producers
> > > > and
> > > > > > > > > > consumers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4. Some configs can be changed just by
> > updating
> > > > the
> > > > > > > > > > reference,
> > > > > > > > > > > > > others
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > require some action. An example of this is
> > if you
> > > > > > want
> > > > > > > to
> > > > > > > > > > > disable
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > compaction (assuming we wanted to make that
> > > > > dynamic)
> > > > > > we
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > call
> > > > > > > > > > > > > > > > > shutdown() on the cleaner. I think it may
> be
> > > > > required
> > > > > > > to
> > > > > > > > > > > > register a
> > > > > > > > > > > > > > > > > listener callback that gets called when the
> > > > config
> > > > > > > changes.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 5. For handling the reference can you
> explain
> > > > your
> > > > > > > plan a
> > > > > > > > > > bit?
> > > > > > > > > > > > > > > Currently we
> > > > > > > > > > > > > > > > > have an immutable KafkaConfig object with a
> > bunch
> > > > > of
> > > > > > > vals.
> > > > > > > > > > That
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > individual values in there get injected all
> > over
> > > > > the
> > > > > > > code
> > > > > > > > > > > base. I
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > thinking something like this:
> > > > > > > > > > > > > > > > > a. We retain the KafkaConfig object as an
> > > > immutable
> > > > > > > object
> > > > > > > > > > just
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > today.
> > > > > > > > > > > > > > > > > b. It is no longer legit to grab values out
> > fo
> > > > that
> > > > > > > config
> > > > > > > > > if
> > > > > > > > > > > > they
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > changeable.
> > > > > > > > > > > > > > > > > c. Instead of making KafkaConfig itself
> > mutable
> > > > we
> > > > > > make
> > > > > > > > > > > > > > > KafkaConfiguration
> > > > > > > > > > > > > > > > > which has a single volatile reference to
> the
> > > > > current
> > > > > > > > > > > KafkaConfig.
> > > > > > > > > > > > > > > > > KafkaConfiguration is what gets passed into
> > > > various
> > > > > > > > > > components.
> > > > > > > > > > > > So
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > access a config you do something like
> > > > > > > > > > config.instance.myValue.
> > > > > > > > > > > > When
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > config changes the config manager updates
> > this
> > > > > > > reference.
> > > > > > > > > > > > > > > > > d. The KafkaConfiguration is the thing that
> > > > allows
> > > > > > > doing
> > > > > > > > > the
> > > > > > > > > > > > > > > > > configuration.onChange("my.config",
> callback)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > -Jay
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 28, 2015 at 3:57 PM, Aditya
> > Auradkar
> > > > <
> > > > > > > > > > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Wrote up a KIP to update topic, client
> and
> > > > broker
> > > > > > > configs
> > > > > > > > > > > > > > > dynamically via
> > > > > > > > > > > > > > > > > > Zookeeper.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Please read and provide feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Aditya
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > PS: I've intentionally kept this
> discussion
> > > > > > separate
> > > > > > > from
> > > > > > > > > > > KIP-5
> > > > > > > > > > > > > > > since I'm
> > > > > > > > > > > > > > > > > > not sure if that is actively being worked
> > on
> > > > and
> > > > > I
> > > > > > > wanted
> > > > > > > > > > to
> > > > > > > > > > > > > start
> > > > > > > > > > > > > > > with a
> > > > > > > > > > > > > > > > > > clean slate.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Neha
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Joel
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Ashish ?h
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Neha
> >
>
>
