Rajini, that was not my intent, the intent was to give a user of this api an insight in to what their topic will look like once created. as things stand now a user is unable to (easily) have any knowledge of what their topic configs will be before doing a `CREATE_TOPICS`. as i mentioned in the KIP, another option would be to have the `CreateTopicsOptions.validateOnly=true` version return data, but seems more invasive/questionable.
dan On Wed, Dec 6, 2017 at 5:10 AM, Rajini Sivaram <rajinisiva...@gmail.com> wrote: > Hi Dan, > > Thank you for the KIP. KIP-226 (https://cwiki.apache.org/ > confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration) proposes > to add an option to include all synonyms of a config option when describing > configs. This includes any defaults. For example (using Dong's example), if > you have topicA with min.cleanable.dirty.ratio=0.6 as an override and the > broker default is 0.5, AdminClient#describeConfigs with synonyms would > return the actual value in use as the config value (I,e. > min.cleanable.dirty.ratio=0.6). And the synonyms list would contain (in > the > order of precedence in which these configs are applied): > > 1. min.cleanable.dirty.ratio=0.6, config source=TOPIC_CONFIG > 2. log.min.cleanable.dirty.ratio=0.5, config > source=STATIC_BROKER_CONFIG > > > KIP-226 doesn't give you exactly what you are proposing in this KIP, but it > gives the mapping of configs. My concern with this KIP is that it assumes > that if the broker config is static, i.e. if the current value of > log.min.cleanable.dirty.ratio=0.6, you can safely create a topic with > default min.cleanable.dirty.ratio relying on that the value to be applied > all the time. This will not work with dynamic broker configs if the broker > defaults can be updated at any time. > > > On Mon, Dec 4, 2017 at 11:22 PM, dan <dan.norw...@gmail.com> wrote: > > > for point 1 i agree, its not too strong. only addition i could come up > with > > is that it allows any utility to have better forwards compatability. a > cli > > written that can inspect how a topic will be created would be able to > give > > insight/expectations about configs that didn't exist at compilation time. > > > > for point 2, i am thinking about a situation where the user creating > topics > > and the user managing brokers are different people with different > > skills/abilities. > > > > so in the given example lets assume a user creating the topic does not > > *really* understand what that config means, so they are likely to just > > leave it as default. and are happy for their admin to change these on the > > broker as needed. > > > > but say we have another user who is creating a topic who has much more > > experience and has done testing, they will be able to determine what the > > value is on the cluster and check to see if it matches expectations or > > needs to be set. possibly if this is set to something incorrect for their > > usecase they will have a reason to verify and speak with the admin about > > their usecase. > > > > > > moreover, i think without this added capability it makes it > > difficult/impossible to accurately use broker defaults for topics. right > > now a user is left to either decide configs a priori and lose this > > functionality, or guess/check what they need to set and end in a possibly > > bad situation until they can get their *live* topic configured. > > > > dan > > > > On Mon, Dec 4, 2017 at 2:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hey Dan, > > > > > > Thanks again for the update:) > > > > > > I am not sure I fully understand the points (1) and (2) in the "Always > > > Configure ALL Configs For a Topic". In my previous question, I don't > mean > > > that users should specify full list of topics configs. I mean that user > > can > > > specify the full list of topic configs he/she wants to override. > > > > > > Your KIP proposes to allow user to query the default topic configs. In > my > > > understanding, users want to know the default configs only if he/she > > > already has a specific list of (config, value) pairs for which he/she > > wants > > > to override. The workflow will be that user queries the default > configs, > > > compares the default value with that specific list, and selectively > > > override the configs whose value is different from the default value. > > > Therefore, the alternative solution does not suffer the problem (1) > > because > > > user needs to know that specific list of (config, value) pair anyway. > > Does > > > this make sense? > > > > > > Regarding problem (2), I think you are saying that if the user wants to > > set > > > the min.cleanable.dirty.ratio to be 0.5 and the default value is > > currently > > > set to 0.5, then user would not want to explicitly override the config > > > during topic creation. The purpose is for the min.cleanable.dirty.ratio > > of > > > this topic to be changed to 0.6 if admin change the default > > > min.cleanable.dirty.ratio to 0.6 in the future. > > > > > > But I am not sure this is a reasonable example. If user wants to > > > compare default value of min.cleanable.dirty.ratio with its expected > > value > > > 0.5, then it seems reasonable for user to override it to be 0.5 during > > > topic creation if the default value is currently 0.6. The question is, > > why > > > would user not want to override the default value to be 0.5 if the > > default > > > value is changed from 0.5 to 0.6 later? > > > > > > Thanks, > > > Dong > > > > > > > > > On Mon, Dec 4, 2017 at 2:36 PM, dan <dan.norw...@gmail.com> wrote: > > > > > > > updated again :) > > > > > > > > by having users always set all configs you lose the ability to use > the > > > > broker defaults as intended, since topic configs are overlaid. > example > > in > > > > the kip doc. > > > > > > > > dan > > > > > > > > On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin <lindon...@gmail.com> > wrote: > > > > > > > > > Hey Dan, > > > > > > > > > > Thanks for the update. I just want to push the discussion a bit > > > further. > > > > > Another alternative, which currently is not described in the KIP, > is > > > for > > > > > user to always create the topic with the full list of configs it > may > > > want > > > > > to override. Can you help me understand what is the drawback of > this > > > > > approach? > > > > > > > > > > Thanks, > > > > > Dong > > > > > > > > > > On Mon, Dec 4, 2017 at 11:30 AM, dan <dan.norw...@gmail.com> > wrote: > > > > > > > > > > > Dong, > > > > > > > > > > > > i added a section on current state and workarounds along with my > > > > > arguments > > > > > > for why they are less than optimal to the wiki. but the jist of > it > > is > > > > you > > > > > > can end up with messages in your topic in an incorrect/invalid > > state > > > if > > > > > you > > > > > > do this. > > > > > > > > > > > > thanks, > > > > > > dan > > > > > > > > > > > > On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin <lindon...@gmail.com> > > > wrote: > > > > > > > > > > > > > Hey Dan, > > > > > > > > > > > > > > Thanks for the KIP. Can you help me understand the motivation > by > > > > > > providing > > > > > > > a use-case that can not be easily completed without this KIP? > > > > > > > > > > > > > > It seems that most users will simply create the topic without > > > > worrying > > > > > > > about the default configs. If a user has specific requirement > for > > > the > > > > > > > default configs, he/she can use AdminClient.describeConfigs() > and > > > > > > > AdminClient.alterConfigs() after the topic has been created. > This > > > > seems > > > > > > to > > > > > > > work well. Did I miss something? > > > > > > > > > > > > > > Thanks, > > > > > > > Dong > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 9:25 AM, dan <dan.norw...@gmail.com> > > wrote: > > > > > > > > > > > > > > > I would like to start a discussion about KIP-234 > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > 234%3A+add+support+for+getting+topic+defaults+from+ > AdminClient > > > > > > > > > > > > > > > > thanks > > > > > > > > dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
