On Fri, Dec 15, 2017, at 08:56, dan wrote:
> i didn't think it would be appropriate to return `num.partitions` as a
> Config for a Topic since that is not a valid config for that resource
> type.
>
> yes, that is another option i have on the wiki in the Rejected
> Alternatives section (though, its really just an alternative until something
> else
> passes voting :). we still have the issue with return type here, NewTopic has
> the info i want to return, but not necessarily the name i'd want...
Hi Dan,
Hmm. I don't think NewTopic is really the right thing to return,
though. NewTopic is structured so that it can be vague... you can
specify just replication factor, or a full assignment. You can specify
some configuration keys, or no configuration keys at all.
NewTopic will probably grow more optional fields in the future. But the
key thing is that NewTopic represents a request to create something, not
a description of something that actually exists.
> it could
> make sense for a createTopics() to return a TopicDescription and we could
> add the config/replication info to that class, but we'd then need to add
> all this info for all describeTopics() calls. not sure of the
> ramifications/overhead here.
Replication info is already part of TopicDescription, right? There is a
list of TopicPartitionInfo there.
Config already encapsulates everything you would want to know about
topic configs, I think.
We don't really need to have a new type that encloses both descriptions
and configs. We can just have
CreateTopicsResult {
KafkaFuture<TopicDescription> description(String topic);
KafkaFuture<Config> config(String topic);
Map<String, KafkaFuture<Void>> values();
KafkaFuture<Void> all();
}
And then people who care about the config of the topic they just created
can call result.config("myTopicName").get(), and those who don't care
can ignore it.
This would be really nice... it lets you figure out what the partition
assignment and config actually are when you actually create a topic.
Plus it extends the existing createTopics(validateOnly=true)
functionality to be even more useful.
best,
Colin
>
> dan
>
> On Wed, Dec 13, 2017 at 10:10 AM, Colin McCabe <cmcc...@apache.org>
> wrote:
>
> > On Wed, Dec 13, 2017, at 10:00, dan wrote:
> > > > Why not just return
> > > > org.apache.kafka.clients.admin.Config like describeConfigs does?
> > >
> > > brokers have a `num.partitions` config that does not map to a valid
> > > `Config` entry for a topic.
> >
> > Hi Dan,
> >
> > Sorry if I'm misunderstanding something, but why not map it to
> > num.partitions?
> >
> > >
> > > another added benefit to using `NewTopic` may be (future kip) having the
> > > cluster return the actual replica mappings it would create (i have no
> > > idea if this is actually possible)
> >
> > A better way of doing that would probably be extending
> > CreateTopicsRequest so that it returns partition assignment information
> > to the caller. Then using validOnly = true to get this information.
> >
> > Actually, come to think of it, maybe we should be doing that for this
> > KIP too. Why not have CreateTopicsRequest return the config that was
> > used, plus the partition assignment that was made? We don't create
> > topics that often, so the extra space on the wire should not be a
> > concern.
> >
> > best,
> > Colin
> >
> > >
> > > dan
> > >
> > > On Wed, Dec 13, 2017 at 9:55 AM, Colin McCabe <cmcc...@apache.org>
> > wrote:
> > >
> > > > On Tue, Dec 12, 2017, at 19:02, Ewen Cheslack-Postava wrote:
> > > > > re: API versions, I actually wasn't sure if we needed it or not. I'm
> > fine
> > > > > if people would prefer just bumping it, but I was actually curious
> > if we
> > > > > could get away without bumping it. I don't know the behavior of the
> > > > > broker code paths for this well enough to know what types of errors
> > those
> > > > > non-null assertions get converted into.
> > > >
> > > > There's no advantage to trying to keep the API version number the same,
> > > > though. Since we have bidirectional client compatibility now, the
> > > > clients and the server will just negotiate whatever version they need.
> > > > New clients can still talk to older brokers that don't support this
> > > > feature.
> > > >
> > > > If you don't bump the API version, the best case scenario is that you
> > > > get a disconnect exception and the end-user is left confused about why.
> > > > The worse-case scenario is that you crash the broker (but probably not,
> > > > since you'd just get an NPE in serde, I think). If you bump the
> > version
> > > > number, you can provide a proper UnsupportedVersionException when the
> > > > feature is not supported.
> > > >
> > > > > For the return type, NewTopic seems reasonable and kind of intuitive
> > --
> > > > > basically a description of the NewTopic you would get. The only
> > reason I
> > > > > would be wary of reusing it is that what we don't want people doing
> > is
> > > > > taking that and passing it directly into AdminClient.createTopics
> > since
> > > > > we don't want them explicitly overriding all the defaults.
> > > >
> > > > Yeah. Another thing is that NewTopic has a lot of stuff related to
> > > > replication that doesn't seem relevant here. For example, when
> > creating
> > > > NewTopic, you have the option of either setting replicationFactor, or
> > > > setting up a specific replica assignment. Why not just return
> > > > org.apache.kafka.clients.admin.Config like describeConfigs does?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > -Ewen
> > > > >
> > > > > On Tue, Dec 12, 2017 at 2:32 PM, dan <dan.norw...@gmail.com> wrote:
> > > > >
> > > > > > Colin/Ewen,
> > > > > >
> > > > > > i will add changes to bump the API version.
> > > > > >
> > > > > > any preferences on the return type for the new method? tbh it seems
> > > > like
> > > > > > returning a NewTopic could make sense because the ConfigResource
> > for a
> > > > > > TOPIC type does not let me encode `numPartitions`
> > > > > >
> > > > > > thanks
> > > > > > dan
> > > > > >
> > > > > > On Mon, Dec 11, 2017 at 7:22 PM, Colin McCabe <cmcc...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Hi Dan,
> > > > > > >
> > > > > > > The KIP looks good overall.
> > > > > > >
> > > > > > > On Mon, Dec 11, 2017, at 18:28, Ewen Cheslack-Postava wrote:
> > > > > > > > I think the key point is when the kafka admin and user creating
> > > > topics
> > > > > > > > differ. I think a more realistic example of Dan's point (2) is
> > for
> > > > > > > > retention. I know that realistically, admins aren't just going
> > to
> > > > > > > > randomly
> > > > > > > > drop the broker defaults from 1w to 1d without warning anyone
> > > > (they'd
> > > > > > > > likely be fired...). But as a user, I may not know the broker
> > > > configs,
> > > > > > if
> > > > > > > > admins have overridden them, etc. I may want a *minimum* of,
> > e.g.,
> > > > 2d.
> > > > > > > > But if the broker defaults are higher such that the admins are
> > > > > > confident
> > > > > > > the
> > > > > > > > cluster can handle 1w, I'd rather just fall back on the default
> > > > value.
> > > > > > >
> > > > > > > Right. I think this API addresses a similar set of use-cases as
> > > > adding
> > > > > > > the "validateOnly" boolean for createTopics. You shouldn't have
> > to
> > > > > > > create a topic to know whether it was possible to create it, or
> > what
> > > > the
> > > > > > > retention will end up being, etc. etc.
> > > > > > >
> > > > > > > > Now, there's arguably a better solution for that case -- allow
> > > > topic
> > > > > > > > configs to express a *minimum* value (or maximum depending on
> > the
> > > > > > > > particular config), with the broker config taking precedence
> > if it
> > > > has
> > > > > > a
> > > > > > > > smaller value (or larger in the case of maximums). This lets
> > you
> > > > > > express
> > > > > > > > your minimum requirements but allows the cluster to do more if
> > > > that's
> > > > > > the
> > > > > > > > default. However, that would represent a much more significant
> > and
> > > > > > > > invasive change, and honestly I think it is more likely to
> > confuse
> > > > > > users.
> > > > > > >
> > > > > > > There always need to be topic defaults, though. If we add a
> > foobar
> > > > > > > configuration for topics, existing topics will need to get
> > > > grandfathered
> > > > > > > in with a default foobar. And they won't be able to set min and
> > max
> > > > > > > ranges, because foobars didn't exist back when the old topics
> > were
> > > > > > > created.
> > > > > > >
> > > > > > > >
> > > > > > > > @Dan, regarding compatibility, this changes behavior without
> > > > revving
> > > > > > the
> > > > > > > > request version number, which normally we only do for things
> > that
> > > > are
> > > > > > > > reasonably considered bugfixes or were it has no compatibility
> > > > > > > > implications. In this case, older brokers talking to newer
> > > > AdminClients
> > > > > > > > will presumably return some error. Do we know what the non-null
> > > > > > assertion
> > > > > > > > gets converted to and if we're happy with the behavior (i.e.
> > will
> > > > > > > > applications be able to do something reasonable, distinguish it
> > > > from
> > > > > > some
> > > > > > > > completely unrelated error, etc)? Similarly, it's obviously
> > only
> > > > one
> > > > > > > > implementation using the KIP-4 APIs, but do we know what
> > > > client-side
> > > > > > > > validation AdminClient is already doing and whether old
> > > > AdminClients
> > > > > > > > talking to new brokers will see a change in behavior (or do
> > they do
> > > > > > > > client-side validation such that old clients simply wouldn't
> > have
> > > > > > access
> > > > > > > > to this new functionality)?
> > > > > > >
> > > > > > > I think we should bump the API version for this or add a new API
> > key.
> > > > > > > Nothing good is going to happen by pretending like this is
> > compatible
> > > > > > > with existing brokers.
> > > > > > >
> > > > > > > Also, I think it would be better just to have a separate
> > function in
> > > > > > > AdminClient rather than overloading the behavior of NULL in
> > > > > > > describeConfigs. It's not really that much more effort to have
> > > > another
> > > > > > > AdminClient function, and it's a lot simpler for devs to
> > understand
> > > > than
> > > > > > > magical NULL behavior in describeConfigs.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > -Ewen
> > > > > > > >
> > > > > > > > On Mon, Dec 11, 2017 at 2:11 PM, dan <dan.norw...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Dong,
> > > > > > > > >
> > > > > > > > > I agree that it *may* be better for a user to be explicit,
> > > > however
> > > > > > > there
> > > > > > > > > are a couple reasons they may not.
> > > > > > > > > 1) a user doesn't even know what the options are. imagine
> > > > writing a
> > > > > > > tool
> > > > > > > > > for users to create topics that steps them through things:
> > > > > > > > >
> > > > > > > > > $ kafka-topics.sh --create
> > > > > > > > > Give your topic a name: my-fav-topic
> > > > > > > > > How many partitions do you want [12]:
> > > > > > > > > What is the minimum in set replica size [2]:
> > > > > > > > > What is the maximum message size [1MB]:
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > > 2) a user wants to use broker defaults within reason. say
> > they
> > > > thinke
> > > > > > > they
> > > > > > > > > want min.cleanable.dirty.ratio=.5 and the default is .6.
> > maybe
> > > > thats
> > > > > > > fine,
> > > > > > > > > or even better for them. since the person maintaining the
> > actual
> > > > > > > cluster
> > > > > > > > > has put thought in to this config. and as the maintainer
> > keeps
> > > > > > working
> > > > > > > on
> > > > > > > > > making the cluster run better they can change and tune
> > things on
> > > > the
> > > > > > > > > cluster level as needed.
> > > > > > > > >
> > > > > > > > > dan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Dec 6, 2017 at 11:51 AM, Dong Lin <
> > lindon...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Dan,
> > > > > > > > > >
> > > > > > > > > > I think you are saying that, if user can read the default
> > > > config
> > > > > > > before
> > > > > > > > > > creating the topic, then this user can make better
> > decision in
> > > > what
> > > > > > > > > configs
> > > > > > > > > > need to be overwritten. The question here is, how user is
> > > > going to
> > > > > > > use
> > > > > > > > > this
> > > > > > > > > > config to make the decision.
> > > > > > > > > >
> > > > > > > > > > In my understanding, user will compare the default value
> > with
> > > > > > > expected
> > > > > > > > > > value, and override the config to be expected value if
> > they are
> > > > > > > > > different.
> > > > > > > > > > If this is the only way that the default value can affect
> > > > user's
> > > > > > > > > decision,
> > > > > > > > > > then it seems OK for user to directly override the config
> > to
> > > > the
> > > > > > > expected
> > > > > > > > > > value. I am wondering if this solution has some drawback.
> > > > > > > > > >
> > > > > > > > > > On the other hand, maybe there is a more advanced way that
> > the
> > > > > > > default
> > > > > > > > > > value can affect the user's decision. It may be useful to
> > > > > > understand
> > > > > > > this
> > > > > > > > > > use-case more specifically. Could you help provide a
> > specific
> > > > > > example
> > > > > > > > > here?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dong
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 6, 2017 at 11:12 AM, dan <
> > dan.norw...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Rajini,
> > > > > > > > > > >
> > > > > > > > > > > that was not my intent, the intent was to give a user of
> > > > this api
> > > > > > > an
> > > > > > > > > > > insight in to what their topic will look like once
> > created.
> > > > as
> > > > > > > things
> > > > > > > > > > stand
> > > > > > > > > > > now a user is unable to (easily) have any knowledge of
> > what
> > > > their
> > > > > > > topic
> > > > > > > > > > > configs will be before doing a `CREATE_TOPICS`. as i
> > > > mentioned in
> > > > > > > the
> > > > > > > > > > KIP,
> > > > > > > > > > > another option would be to have the `CreateTopicsOptions.
> > > > > > > > > > > validateOnly=true`
> > > > > > > > > > > version return data, but seems more
> > invasive/questionable.
> > > > > > > > > > >
> > > > > > > > > > > dan
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 6, 2017 at 5:10 AM, Rajini Sivaram <
> > > > > > > > > rajinisiva...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Dan,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for the KIP. KIP-226 (
> > https://cwiki.apache.org/
> > > > > > > > > > > > confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+
> > > > > > Configuration)
> > > > > > > > > > > proposes
> > > > > > > > > > > > to add an option to include all synonyms of a config
> > option
> > > > > > when
> > > > > > > > > > > describing
> > > > > > > > > > > > configs. This includes any defaults. For example (using
> > > > Dong's
> > > > > > > > > > example),
> > > > > > > > > > > if
> > > > > > > > > > > > you have topicA with min.cleanable.dirty.ratio=0.6 as
> > an
> > > > > > > override and
> > > > > > > > > > the
> > > > > > > > > > > > broker default is 0.5, AdminClient#describeConfigs with
> > > > > > synonyms
> > > > > > > > > would
> > > > > > > > > > > > return the actual value in use as the config value
> > (I,e.
> > > > > > > > > > > > min.cleanable.dirty.ratio=0.6). And the synonyms list
> > > > would
> > > > > > > contain
> > > > > > > > > > (in
> > > > > > > > > > > > the
> > > > > > > > > > > > order of precedence in which these configs are
> > applied):
> > > > > > > > > > > >
> > > > > > > > > > > > 1. min.cleanable.dirty.ratio=0.6, config
> > > > source=TOPIC_CONFIG
> > > > > > > > > > > > 2. log.min.cleanable.dirty.ratio=0.5, config
> > > > > > > > > > > > source=STATIC_BROKER_CONFIG
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > KIP-226 doesn't give you exactly what you are
> > proposing in
> > > > this
> > > > > > > KIP,
> > > > > > > > > > but
> > > > > > > > > > > it
> > > > > > > > > > > > gives the mapping of configs. My concern with this KIP
> > is
> > > > that
> > > > > > it
> > > > > > > > > > assumes
> > > > > > > > > > > > that if the broker config is static, i.e. if the
> > current
> > > > value
> > > > > > of
> > > > > > > > > > > > log.min.cleanable.dirty.ratio=0.6, you can safely
> > create a
> > > > > > topic
> > > > > > > > > with
> > > > > > > > > > > > default min.cleanable.dirty.ratio relying on that the
> > > > value to
> > > > > > be
> > > > > > > > > > applied
> > > > > > > > > > > > all the time. This will not work with dynamic broker
> > > > configs if
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > defaults can be updated at any time.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 4, 2017 at 11:22 PM, dan <
> > > > dan.norw...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > for point 1 i agree, its not too strong. only
> > addition i
> > > > > > could
> > > > > > > come
> > > > > > > > > > up
> > > > > > > > > > > > with
> > > > > > > > > > > > > is that it allows any utility to have better forwards
> > > > > > > > > compatability.
> > > > > > > > > > a
> > > > > > > > > > > > cli
> > > > > > > > > > > > > written that can inspect how a topic will be created
> > > > would be
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > > give
> > > > > > > > > > > > > insight/expectations about configs that didn't exist
> > at
> > > > > > > compilation
> > > > > > > > > > > time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > for point 2, i am thinking about a situation where
> > the
> > > > user
> > > > > > > > > creating
> > > > > > > > > > > > topics
> > > > > > > > > > > > > and the user managing brokers are different people
> > with
> > > > > > > different
> > > > > > > > > > > > > skills/abilities.
> > > > > > > > > > > > >
> > > > > > > > > > > > > so in the given example lets assume a user creating
> > the
> > > > topic
> > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > *really* understand what that config means, so they
> > are
> > > > > > likely
> > > > > > > to
> > > > > > > > > > just
> > > > > > > > > > > > > leave it as default. and are happy for their admin to
> > > > change
> > > > > > > these
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > broker as needed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > but say we have another user who is creating a topic
> > who
> > > > has
> > > > > > > much
> > > > > > > > > > more
> > > > > > > > > > > > > experience and has done testing, they will be able to
> > > > > > determine
> > > > > > > > > what
> > > > > > > > > > > the
> > > > > > > > > > > > > value is on the cluster and check to see if it
> > matches
> > > > > > > expectations
> > > > > > > > > > or
> > > > > > > > > > > > > needs to be set. possibly if this is set to something
> > > > > > > incorrect for
> > > > > > > > > > > their
> > > > > > > > > > > > > usecase they will have a reason to verify and speak
> > with
> > > > the
> > > > > > > admin
> > > > > > > > > > > about
> > > > > > > > > > > > > their usecase.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > moreover, i think without this added capability it
> > makes
> > > > it
> > > > > > > > > > > > > difficult/impossible to accurately use broker
> > defaults
> > > > for
> > > > > > > topics.
> > > > > > > > > > > right
> > > > > > > > > > > > > now a user is left to either decide configs a priori
> > and
> > > > lose
> > > > > > > this
> > > > > > > > > > > > > functionality, or guess/check what they need to set
> > and
> > > > end
> > > > > > in
> > > > > > > a
> > > > > > > > > > > possibly
> > > > > > > > > > > > > bad situation until they can get their *live* topic
> > > > > > configured.
> > > > > > > > > > > > >
> > > > > > > > > > > > > dan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 4, 2017 at 2:50 PM, Dong Lin <
> > > > > > lindon...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Dan,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks again for the update:)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am not sure I fully understand the points (1) and
> > > > (2) in
> > > > > > > the
> > > > > > > > > > > "Always
> > > > > > > > > > > > > > Configure ALL Configs For a Topic". In my previous
> > > > > > question,
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > mean
> > > > > > > > > > > > > > that users should specify full list of topics
> > configs.
> > > > I
> > > > > > mean
> > > > > > > > > that
> > > > > > > > > > > user
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > specify the full list of topic configs he/she
> > wants to
> > > > > > > override.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Your KIP proposes to allow user to query the
> > default
> > > > topic
> > > > > > > > > configs.
> > > > > > > > > > > In
> > > > > > > > > > > > my
> > > > > > > > > > > > > > understanding, users want to know the default
> > configs
> > > > only
> > > > > > if
> > > > > > > > > > he/she
> > > > > > > > > > > > > > already has a specific list of (config, value)
> > pairs
> > > > for
> > > > > > > which
> > > > > > > > > > he/she
> > > > > > > > > > > > > wants
> > > > > > > > > > > > > > to override. The workflow will be that user
> > queries the
> > > > > > > default
> > > > > > > > > > > > configs,
> > > > > > > > > > > > > > compares the default value with that specific
> > list, and
> > > > > > > > > selectively
> > > > > > > > > > > > > > override the configs whose value is different from
> > the
> > > > > > > default
> > > > > > > > > > value.
> > > > > > > > > > > > > > Therefore, the alternative solution does not
> > suffer the
> > > > > > > problem
> > > > > > > > > (1)
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > user needs to know that specific list of (config,
> > > > value)
> > > > > > pair
> > > > > > > > > > anyway.
> > > > > > > > > > > > > Does
> > > > > > > > > > > > > > this make sense?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding problem (2), I think you are saying that
> > if
> > > > the
> > > > > > > user
> > > > > > > > > > wants
> > > > > > > > > > > to
> > > > > > > > > > > > > set
> > > > > > > > > > > > > > the min.cleanable.dirty.ratio to be 0.5 and the
> > default
> > > > > > > value is
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > > set to 0.5, then user would not want to explicitly
> > > > override
> > > > > > > the
> > > > > > > > > > > config
> > > > > > > > > > > > > > during topic creation. The purpose is for the
> > > > > > > > > > > min.cleanable.dirty.ratio
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > this topic to be changed to 0.6 if admin change the
> > > > default
> > > > > > > > > > > > > > min.cleanable.dirty.ratio to 0.6 in the future.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > But I am not sure this is a reasonable example. If
> > user
> > > > > > > wants to
> > > > > > > > > > > > > > compare default value of min.cleanable.dirty.ratio
> > > > with its
> > > > > > > > > > expected
> > > > > > > > > > > > > value
> > > > > > > > > > > > > > 0.5, then it seems reasonable for user to override
> > it
> > > > to be
> > > > > > > 0.5
> > > > > > > > > > > during
> > > > > > > > > > > > > > topic creation if the default value is currently
> > 0.6.
> > > > The
> > > > > > > > > question
> > > > > > > > > > > is,
> > > > > > > > > > > > > why
> > > > > > > > > > > > > > would user not want to override the default value
> > to
> > > > be 0.5
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > value is changed from 0.5 to 0.6 later?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Dong
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 2:36 PM, dan <
> > > > dan.norw...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > updated again :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > by having users always set all configs you lose
> > the
> > > > > > > ability to
> > > > > > > > > > use
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > broker defaults as intended, since topic configs
> > are
> > > > > > > overlaid.
> > > > > > > > > > > > example
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the kip doc.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > dan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin <
> > > > > > > lindon...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey Dan,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the update. I just want to push the
> > > > > > > discussion a
> > > > > > > > > bit
> > > > > > > > > > > > > > further.
> > > > > > > > > > > > > > > > Another alternative, which currently is not
> > > > described
> > > > > > in
> > > > > > > the
> > > > > > > > > > KIP,
> > > > > > > > > > > > is
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > user to always create the topic with the full
> > list
> > > > of
> > > > > > > configs
> > > > > > > > > > it
> > > > > > > > > > > > may
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to override. Can you help me understand what
> > is the
> > > > > > > drawback
> > > > > > > > > of
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > approach?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Dong
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 11:30 AM, dan <
> > > > > > > dan.norw...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Dong,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > i added a section on current state and
> > > > workarounds
> > > > > > > along
> > > > > > > > > with
> > > > > > > > > > > my
> > > > > > > > > > > > > > > > arguments
> > > > > > > > > > > > > > > > > for why they are less than optimal to the
> > wiki.
> > > > but
> > > > > > the
> > > > > > > > > jist
> > > > > > > > > > of
> > > > > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > can end up with messages in your topic in an
> > > > > > > > > > incorrect/invalid
> > > > > > > > > > > > > state
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > do this.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > thanks,
> > > > > > > > > > > > > > > > > dan
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin <
> > > > > > > > > > lindon...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey Dan,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the KIP. Can you help me
> > understand
> > > > the
> > > > > > > > > > motivation
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > providing
> > > > > > > > > > > > > > > > > > a use-case that can not be easily completed
> > > > without
> > > > > > > this
> > > > > > > > > > KIP?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It seems that most users will simply
> > create the
> > > > > > topic
> > > > > > > > > > without
> > > > > > > > > > > > > > > worrying
> > > > > > > > > > > > > > > > > > about the default configs. If a user has
> > > > specific
> > > > > > > > > > requirement
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > default configs, he/she can use
> > > > > > > > > > AdminClient.describeConfigs()
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > AdminClient.alterConfigs() after the topic
> > has
> > > > been
> > > > > > > > > > created.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > seems
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > work well. Did I miss something?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Dong
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Dec 4, 2017 at 9:25 AM, dan <
> > > > > > > > > dan.norw...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I would like to start a discussion about
> > > > KIP-234
> > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > > > > > > > > > > 234%3A+add+support+for+
> > > > > > > getting+topic+defaults+from+
> > > > > > > > > > > > AdminClient
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > thanks
> > > > > > > > > > > > > > > > > > > dan
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
