Hey Dan, I think you are saying that, if user can read the default config before creating the topic, then this user can make better decision in what configs need to be overwritten. The question here is, how user is going to use this config to make the decision.
In my understanding, user will compare the default value with expected value, and override the config to be expected value if they are different. If this is the only way that the default value can affect user's decision, then it seems OK for user to directly override the config to the expected value. I am wondering if this solution has some drawback. On the other hand, maybe there is a more advanced way that the default value can affect the user's decision. It may be useful to understand this use-case more specifically. Could you help provide a specific example here? Thanks, Dong On Wed, Dec 6, 2017 at 11:12 AM, dan <dan.norw...@gmail.com> wrote: > Rajini, > > that was not my intent, the intent was to give a user of this api an > insight in to what their topic will look like once created. as things stand > now a user is unable to (easily) have any knowledge of what their topic > configs will be before doing a `CREATE_TOPICS`. as i mentioned in the KIP, > another option would be to have the `CreateTopicsOptions. > validateOnly=true` > version return data, but seems more invasive/questionable. > > dan > > On Wed, Dec 6, 2017 at 5:10 AM, Rajini Sivaram <rajinisiva...@gmail.com> > wrote: > > > Hi Dan, > > > > Thank you for the KIP. KIP-226 (https://cwiki.apache.org/ > > confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration) > proposes > > to add an option to include all synonyms of a config option when > describing > > configs. This includes any defaults. For example (using Dong's example), > if > > you have topicA with min.cleanable.dirty.ratio=0.6 as an override and the > > broker default is 0.5, AdminClient#describeConfigs with synonyms would > > return the actual value in use as the config value (I,e. > > min.cleanable.dirty.ratio=0.6). And the synonyms list would contain (in > > the > > order of precedence in which these configs are applied): > > > > 1. min.cleanable.dirty.ratio=0.6, config source=TOPIC_CONFIG > > 2. log.min.cleanable.dirty.ratio=0.5, config > > source=STATIC_BROKER_CONFIG > > > > > > KIP-226 doesn't give you exactly what you are proposing in this KIP, but > it > > gives the mapping of configs. My concern with this KIP is that it assumes > > that if the broker config is static, i.e. if the current value of > > log.min.cleanable.dirty.ratio=0.6, you can safely create a topic with > > default min.cleanable.dirty.ratio relying on that the value to be applied > > all the time. This will not work with dynamic broker configs if the > broker > > defaults can be updated at any time. > > > > > > On Mon, Dec 4, 2017 at 11:22 PM, dan <dan.norw...@gmail.com> wrote: > > > > > for point 1 i agree, its not too strong. only addition i could come up > > with > > > is that it allows any utility to have better forwards compatability. a > > cli > > > written that can inspect how a topic will be created would be able to > > give > > > insight/expectations about configs that didn't exist at compilation > time. > > > > > > for point 2, i am thinking about a situation where the user creating > > topics > > > and the user managing brokers are different people with different > > > skills/abilities. > > > > > > so in the given example lets assume a user creating the topic does not > > > *really* understand what that config means, so they are likely to just > > > leave it as default. and are happy for their admin to change these on > the > > > broker as needed. > > > > > > but say we have another user who is creating a topic who has much more > > > experience and has done testing, they will be able to determine what > the > > > value is on the cluster and check to see if it matches expectations or > > > needs to be set. possibly if this is set to something incorrect for > their > > > usecase they will have a reason to verify and speak with the admin > about > > > their usecase. > > > > > > > > > moreover, i think without this added capability it makes it > > > difficult/impossible to accurately use broker defaults for topics. > right > > > now a user is left to either decide configs a priori and lose this > > > functionality, or guess/check what they need to set and end in a > possibly > > > bad situation until they can get their *live* topic configured. > > > > > > dan > > > > > > On Mon, Dec 4, 2017 at 2:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > > > Hey Dan, > > > > > > > > Thanks again for the update:) > > > > > > > > I am not sure I fully understand the points (1) and (2) in the > "Always > > > > Configure ALL Configs For a Topic". In my previous question, I don't > > mean > > > > that users should specify full list of topics configs. I mean that > user > > > can > > > > specify the full list of topic configs he/she wants to override. > > > > > > > > Your KIP proposes to allow user to query the default topic configs. > In > > my > > > > understanding, users want to know the default configs only if he/she > > > > already has a specific list of (config, value) pairs for which he/she > > > wants > > > > to override. The workflow will be that user queries the default > > configs, > > > > compares the default value with that specific list, and selectively > > > > override the configs whose value is different from the default value. > > > > Therefore, the alternative solution does not suffer the problem (1) > > > because > > > > user needs to know that specific list of (config, value) pair anyway. > > > Does > > > > this make sense? > > > > > > > > Regarding problem (2), I think you are saying that if the user wants > to > > > set > > > > the min.cleanable.dirty.ratio to be 0.5 and the default value is > > > currently > > > > set to 0.5, then user would not want to explicitly override the > config > > > > during topic creation. The purpose is for the > min.cleanable.dirty.ratio > > > of > > > > this topic to be changed to 0.6 if admin change the default > > > > min.cleanable.dirty.ratio to 0.6 in the future. > > > > > > > > But I am not sure this is a reasonable example. If user wants to > > > > compare default value of min.cleanable.dirty.ratio with its expected > > > value > > > > 0.5, then it seems reasonable for user to override it to be 0.5 > during > > > > topic creation if the default value is currently 0.6. The question > is, > > > why > > > > would user not want to override the default value to be 0.5 if the > > > default > > > > value is changed from 0.5 to 0.6 later? > > > > > > > > Thanks, > > > > Dong > > > > > > > > > > > > On Mon, Dec 4, 2017 at 2:36 PM, dan <dan.norw...@gmail.com> wrote: > > > > > > > > > updated again :) > > > > > > > > > > by having users always set all configs you lose the ability to use > > the > > > > > broker defaults as intended, since topic configs are overlaid. > > example > > > in > > > > > the kip doc. > > > > > > > > > > dan > > > > > > > > > > On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin <lindon...@gmail.com> > > wrote: > > > > > > > > > > > Hey Dan, > > > > > > > > > > > > Thanks for the update. I just want to push the discussion a bit > > > > further. > > > > > > Another alternative, which currently is not described in the KIP, > > is > > > > for > > > > > > user to always create the topic with the full list of configs it > > may > > > > want > > > > > > to override. Can you help me understand what is the drawback of > > this > > > > > > approach? > > > > > > > > > > > > Thanks, > > > > > > Dong > > > > > > > > > > > > On Mon, Dec 4, 2017 at 11:30 AM, dan <dan.norw...@gmail.com> > > wrote: > > > > > > > > > > > > > Dong, > > > > > > > > > > > > > > i added a section on current state and workarounds along with > my > > > > > > arguments > > > > > > > for why they are less than optimal to the wiki. but the jist of > > it > > > is > > > > > you > > > > > > > can end up with messages in your topic in an incorrect/invalid > > > state > > > > if > > > > > > you > > > > > > > do this. > > > > > > > > > > > > > > thanks, > > > > > > > dan > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin <lindon...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > Hey Dan, > > > > > > > > > > > > > > > > Thanks for the KIP. Can you help me understand the motivation > > by > > > > > > > providing > > > > > > > > a use-case that can not be easily completed without this KIP? > > > > > > > > > > > > > > > > It seems that most users will simply create the topic without > > > > > worrying > > > > > > > > about the default configs. If a user has specific requirement > > for > > > > the > > > > > > > > default configs, he/she can use AdminClient.describeConfigs() > > and > > > > > > > > AdminClient.alterConfigs() after the topic has been created. > > This > > > > > seems > > > > > > > to > > > > > > > > work well. Did I miss something? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dong > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 9:25 AM, dan <dan.norw...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > > > I would like to start a discussion about KIP-234 > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > > 234%3A+add+support+for+getting+topic+defaults+from+ > > AdminClient > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
