For the sequential config/changes/config_change_XX znode, do we have any manners to do cleaning in order to avoid the change-log from growing indefinitely?
Guozhang On Thu, May 28, 2015 at 5:02 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > I still have a couple of questions: > 1. Are we introducing a new Java API for the config change protocol and if > so where will that appear? Is that going to be part of the java api in the > admin api kip? Let's document that. > 2. The proposed JSON format uses camel case for field names, is that what > we've used for other JSON in zookeeper? > 3. This changes the format of the notifications, right? How will we > grandfather in the old format? Clusters will have existing change > notifications in the old format so I think the new code will need to be > able to read those? > > -Jay > > On Thu, May 28, 2015 at 11:41 AM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > > > bump > > > > ________________________________________ > > From: Aditya Auradkar > > Sent: Tuesday, May 26, 2015 1:16 PM > > To: dev@kafka.apache.org > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > Hey everyone, > > > > Completed the changes to KIP-4. After today's hangout, there doesn't > > appear to be anything remaining to discuss on this KIP. > > Please vote so we can formally close this. > > > > Thanks, > > Aditya > > > > ________________________________________ > > From: Aditya Auradkar > > Sent: Thursday, May 21, 2015 11:26 AM > > To: dev@kafka.apache.org > > Subject: RE: [VOTE] KIP-21 Dynamic Configuration > > > > I think we should remove the config part in TopicMetadataResponse. It's > > probably cleaner if Alter and Describe are the only way to view and > modify > > configs but I don't feel very strongly about it. > > > > Re-summarizing the proposed changes to KIP-4: > > - Change AlterTopic to not allow setting configs. Config changes will > flow > > through AlterConfig. CreateTopic will still allow setting configs as it > is > > nice to be able to specify configs while creating the topic. > > - TopicMetadataResponse shoudn't return config for the topic. > > DescribeConfig is the way to go. > > - Change "InvalidTopicConfiguration" error code to "InvalidEntityConfig" > > as proposed in KIP-21. > > > > Aditya > > > > ________________________________________ > > From: Jun Rao [j...@confluent.io] > > Sent: Thursday, May 21, 2015 10:50 AM > > To: dev@kafka.apache.org > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > What about TopicMetadataResponse in KIP-4? Do we remove the config part > in > > it? > > > > Thanks, > > > > Jun > > > > On Thu, May 21, 2015 at 10:25 AM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Hey Jun, > > > > > > I've added a section on error codes on the KIP-21 wiki. > > > > > > Here are the proposed changes to KIP-4. I'll update the wiki shortly. > > > - Change AlterTopic to not allow setting configs. Config changes will > > flow > > > through AlterConfig. CreateTopic will still allow setting configs as it > > is > > > nice to be able to specify configs while creating the topic. > > > - Change "InvalidTopicConfiguration" error code to > "InvalidEntityConfig" > > > as proposed in KIP-21. > > > > > > > > > Thanks, > > > Aditya > > > > > > ________________________________________ > > > From: Jun Rao [j...@confluent.io] > > > Sent: Thursday, May 21, 2015 8:41 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > Aditya, > > > > > > For completeness, could you list the set of error codes in the wiki? > > Also, > > > could you summarize the changes that are needed for the requests listed > > in > > > KIP-4 and update the wiki accordingly? > > > > > > Thanks, > > > > > > Jun > > > > > > On Tue, May 19, 2015 at 10:33 PM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > Thanks Andrii. I'll make the changes. > > > > > > > > I've also updated KIP-21 to include the new config requests. Take a > > look > > > > and vote. > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > Aditya > > > > ________________________________________ > > > > From: Andrii Biletskyi [andrii.bilets...@stealth.ly] > > > > Sent: Tuesday, May 19, 2015 2:26 PM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > Hi, > > > > > > > > Sorry I wasn't able to participate. I don't have objections about > > > removing > > > > config changes from AlterTopic (as I understand both AddedConfig and > > > > DeletedConfig) - you are welcome to update the KIP page. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > On Tue, May 19, 2015 at 11:40 PM, Aditya Auradkar < > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > Updating the discussion with the latest comments. > > > > > > > > > > 1. We discussed adding 2 new API's (AlterConfig and > DescribeConfig). > > > I'll > > > > > update KIP-21 with details on these. > > > > > 2. Discussed during the KIP hangout. We are in agreement. > > > > > > > > > > (1) has a dependency on KIP-4 being completed. Rest of the work in > > the > > > > KIP > > > > > can be implemented independently. Any concerns if we tackle it as > two > > > > > separate work items implementation wise? > > > > > > > > > > We also discussed changing the AlterTopic command in KIP-4 to not > > > include > > > > > config changes. Instead, all config changes will pass through the > > newly > > > > > proposed AlterConfig. If no-one objects, I can make some changes to > > > KIP-4 > > > > > to reflect this. > > > > > > > > > > Aditya > > > > > > > > > > ________________________________________ > > > > > From: Jay Kreps [jay.kr...@gmail.com] > > > > > Sent: Tuesday, May 19, 2015 10:51 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > Hey Aditya, > > > > > > > > > > Two comments: > > > > > > > > > > 1. Yeah we need to reconcile this with the APIs in KIP-4. I think > it > > > does > > > > > make sense to allow setting config during topic creation. I agree > > with > > > > your > > > > > summary that having alter topic and alter config may be confusing, > > but > > > > > there are also some non-config changes such as replication factor > and > > > > > partition count that alter topic can carry out. What is the final > > state > > > > you > > > > > are proposing? > > > > > > > > > > 2. This is implementation related so probably can be removed from > the > > > KIP > > > > > entirely, but you seem to be proposing a separate config manager > for > > > each > > > > > config override type. Should we just generalize TopicConfigManager > to > > > be > > > > > ConfigOverrideManager and have it handle all the override types we > > will > > > > > have? I think I may just be unclear on what you are proposing... > > > > > > > > > > -Jay > > > > > > > > > > On Mon, May 18, 2015 at 1:34 PM, Aditya Auradkar < > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > Yeah, that was just a typo. I've fixed it. Thanks for calling it > > out. > > > > > > > > > > > > In KIP-4, I believe we have 3 types of requests: CreateTopic, > > > > AlterTopic > > > > > > and DeleteTopic. The topic configs are a sub-type of the Create > and > > > > Alter > > > > > > commands. I think it would be nice to simply have a AlterConfig > > > command > > > > > > that can alter any type of config rather than having a specific > > > > > > ClientConfig. > > > > > > > > > > > > AlterConfig => [ConfigType [AddedConfigEntry] [DeletedConfig]] > > > > > > ConfigType => string > > > > > > AddedConfigEntry => ConfigKey ConfigValue > > > > > > ConfigKey => string > > > > > > ConfigValue => string > > > > > > DeletedConfig => string > > > > > > > > > > > > The downside of this approach is that we will have 2 separate > ways > > of > > > > > > changing topic configs (AlterTopic and AlterConfig). While a > > general > > > > > > AlterConfig only makes sense if we plan to have more than two > types > > > of > > > > > > entity configs.. it's definitely more future proof. Thoughts? > > > > > > > > > > > > Aditya > > > > > > > > > > > > ________________________________________ > > > > > > From: Todd Palino [tpal...@gmail.com] > > > > > > Sent: Monday, May 18, 2015 12:39 PM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [VOTE] KIP-21 Dynamic Configuration > > > > > > > > > > > > Agree with Jun here on the JSON format. I think your intention > was > > > > likely > > > > > > to have actual JSON here and it was just a typo in the wiki? > > > > > > > > > > > > -Todd > > > > > > > > > > > > On Mon, May 18, 2015 at 12:07 PM, Jun Rao <j...@confluent.io> > > wrote: > > > > > > > > > > > > > Aditya, > > > > > > > > > > > > > > Another thing to consider. In KIP-4, we are adding a new RPC > > > request > > > > to > > > > > > > change and retrieve topic configs. Do we want to add a similar > > RPC > > > > > > request > > > > > > > to change configs per client id? If so, do we want to > introduce a > > > > > > separate > > > > > > > new request or have a combined new request for both topic and > > > client > > > > id > > > > > > > level config changes? > > > > > > > > > > > > > > A minor point in the wiki, for the json format in ZK, we should > > > > change > > > > > > > {X1=Y1, > > > > > > > X2=Y2..} to a json map, right? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > On Mon, May 18, 2015 at 9:48 AM, Aditya Auradkar < > > > > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration > > > > > > > > > > > > > > > > Aditya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- -- Guozhang
