Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

On Fri, Dec 15, 2017, at 08:56, dan wrote: > i didn't think it would be appropriate to return `num.partitions` as a > Config for a Topic since that is not a valid config for that resource > type. > > yes, that is another option i have on the wiki in the Rejected > Alternatives section (though,

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

i didn't think it would be appropriate to return `num.partitions` as a Config for a Topic since that is not a valid config for that resource type. yes, that is another option i have on the wiki in the Rejected Alternatives section (though, its really just an alternative until something else

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

On Wed, Dec 13, 2017, at 10:00, dan wrote: > > Why not just return > > org.apache.kafka.clients.admin.Config like describeConfigs does? > > brokers have a `num.partitions` config that does not map to a valid > `Config` entry for a topic. Hi Dan, Sorry if I'm misunderstanding something, but why

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

> Why not just return > org.apache.kafka.clients.admin.Config like describeConfigs does? brokers have a `num.partitions` config that does not map to a valid `Config` entry for a topic. another added benefit to using `NewTopic` may be (future kip) having the cluster return the actual replica

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

On Tue, Dec 12, 2017, at 19:02, Ewen Cheslack-Postava wrote: > re: API versions, I actually wasn't sure if we needed it or not. I'm fine > if people would prefer just bumping it, but I was actually curious if we > could get away without bumping it. I don't know the behavior of the > broker code

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

re: API versions, I actually wasn't sure if we needed it or not. I'm fine if people would prefer just bumping it, but I was actually curious if we could get away without bumping it. I don't know the behavior of the broker code paths for this well enough to know what types of errors those non-null

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Colin/Ewen, i will add changes to bump the API version. any preferences on the return type for the new method? tbh it seems like returning a NewTopic could make sense because the ConfigResource for a TOPIC type does not let me encode `numPartitions` thanks dan On Mon, Dec 11, 2017 at 7:22 PM,

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hi Dan, The KIP looks good overall. On Mon, Dec 11, 2017, at 18:28, Ewen Cheslack-Postava wrote: > I think the key point is when the kafka admin and user creating topics > differ. I think a more realistic example of Dan's point (2) is for > retention. I know that realistically, admins aren't

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

I think the key point is when the kafka admin and user creating topics differ. I think a more realistic example of Dan's point (2) is for retention. I know that realistically, admins aren't just going to randomly drop the broker defaults from 1w to 1d without warning anyone (they'd likely be

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Dong, I agree that it *may* be better for a user to be explicit, however there are a couple reasons they may not. 1) a user doesn't even know what the options are. imagine writing a tool for users to create topics that steps them through things: $ kafka-topics.sh --create Give your topic a name:

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, I think you are saying that, if user can read the default config before creating the topic, then this user can make better decision in what configs need to be overwritten. The question here is, how user is going to use this config to make the decision. In my understanding, user will

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Rajini, that was not my intent, the intent was to give a user of this api an insight in to what their topic will look like once created. as things stand now a user is unable to (easily) have any knowledge of what their topic configs will be before doing a `CREATE_TOPICS`. as i mentioned in the

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hi Dan, Thank you for the KIP. KIP-226 (https://cwiki.apache.org/ confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration) proposes to add an option to include all synonyms of a config option when describing configs. This includes any defaults. For example (using Dong's example), if you

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

for point 1 i agree, its not too strong. only addition i could come up with is that it allows any utility to have better forwards compatability. a cli written that can inspect how a topic will be created would be able to give insight/expectations about configs that didn't exist at compilation

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks again for the update:) I am not sure I fully understand the points (1) and (2) in the "Always Configure ALL Configs For a Topic". In my previous question, I don't mean that users should specify full list of topics configs. I mean that user can specify the full list of topic

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

updated again :) by having users always set all configs you lose the ability to use the broker defaults as intended, since topic configs are overlaid. example in the kip doc. dan On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin wrote: > Hey Dan, > > Thanks for the update. I just

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks for the update. I just want to push the discussion a bit further. Another alternative, which currently is not described in the KIP, is for user to always create the topic with the full list of configs it may want to override. Can you help me understand what is the drawback of this

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Dong, i added a section on current state and workarounds along with my arguments for why they are less than optimal to the wiki. but the jist of it is you can end up with messages in your topic in an incorrect/invalid state if you do this. thanks, dan On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks for the KIP. Can you help me understand the motivation by providing a use-case that can not be easily completed without this KIP? It seems that most users will simply create the topic without worrying about the default configs. If a user has specific requirement for the default

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

updated to be more explicit On Mon, Dec 4, 2017 at 9:55 AM, Ted Yu wrote: > In the example, resources is derived from ConfigResource. > It would be clearer if you show the derivation. > > Cheers > > On Mon, Dec 4, 2017 at 9:51 AM, dan wrote: > > > I

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

In the example, resources is derived from ConfigResource. It would be clearer if you show the derivation. Cheers On Mon, Dec 4, 2017 at 9:51 AM, dan wrote: > I added sample usage as well as a WIP commit to the KIP. > > dan > > On Mon, Dec 4, 2017 at 9:36 AM, Ted Yu

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

I added sample usage as well as a WIP commit to the KIP. dan On Mon, Dec 4, 2017 at 9:36 AM, Ted Yu wrote: > Can you add some code snippet showing how ConfigResource delivers topic > defaults with the proposed change ? > > > Cheers > > On Mon, Dec 4, 2017 at 9:25 AM, dan

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Can you add some code snippet showing how ConfigResource delivers topic defaults with the proposed change ? Cheers On Mon, Dec 4, 2017 at 9:25 AM, dan wrote: > I would like to start a discussion about KIP-234 > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >