> 2. Part of the usage of the file is to have persistent storage of the
topic
ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
There is some discussion in the KIP about changes to the directory
structure, but I believe directory changes were considered to be out of
scope when the KIP was written.
Yeah, I was hoping to get a better understanding of why it was taken out of
scope. Perhaps Lucas Bradstreet might have more insight about the decision.
Basically my point is that we have to create additional infrastructure here
to support the name/id mapping, so I wanted to understand if we just
consider this a sort of tech debt. It would be useful to cover how we
handle the case when this file gets corrupted. Seems like we just have to
trust that it matches whatever the controller tells us and rewrite it?
Hi Jason, Justine,
My thought process is that we will likely need the metadata file whether we
rename the directories or not.
Linux supports filenames of up to 255 bytes and that would not be enough to
support a directory name
including both the name and topic ID. Given that fact, it seemed
reasonable to add the metadata file
and leave the directory rename until some time in the future (possibly
never).
If the file does get corrupted I think you're right. We would either have
to trust it matches what the controller tells us
or error out and let an administrator resolve it by checking across
replicas for consistency.
Lucas
On Thu, Sep 24, 2020 at 3:41 PM Jason Gustafson <ja...@confluent.io> wrote:
> Thanks Justine. Responses below:
>
> > 1. Yes, the directory will still be based on the topic names.
> LeaderAndIsrRequest is one of the few requests that will still contain the
> topic name. So I think we have this covered. Sorry for confusion.
>
> Ah, you're right. My eyes passed right over the field.
>
> > 2. Part of the usage of the file is to have persistent storage of the
> topic
> ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
> There is some discussion in the KIP about changes to the directory
> structure, but I believe directory changes were considered to be out of
> scope when the KIP was written.
>
> Yeah, I was hoping to get a better understanding of why it was taken out of
> scope. Perhaps Lucas Bradstreet might have more insight about the decision.
> Basically my point is that we have to create additional infrastructure here
> to support the name/id mapping, so I wanted to understand if we just
> consider this a sort of tech debt. It would be useful to cover how we
> handle the case when this file gets corrupted. Seems like we just have to
> trust that it matches whatever the controller tells us and rewrite it?
>
> > 3. I think this is a good point, but I again I wonder about the scope of
> the KIP. Most of the changes mentioned in the KIP are for supporting topic
> deletion and I believe that is why the produce request was listed under
> future work.
>
> That's fair. I brought it up since `Fetch` is already included. If we've
> got `Metadata` and `Fetch`, seems we may as well do `Produce` and save an
> extra kip. No strong objection though if you want to leave it out.
>
>
> -Jason
>
>
> On Thu, Sep 24, 2020 at 3:26 PM Justine Olshan <jols...@confluent.io>
> wrote:
>
> > Hi Jason,
> >
> > Thanks for your comments.
> >
> > 1. Yes, the directory will still be based on the topic names.
> > LeaderAndIsrRequest is one of the few requests that will still contain
> the
> > topic name. So I think we have this covered. Sorry for confusion.
> >
> > 2. Part of the usage of the file is to have persistent storage of the
> topic
> > ID and use it to compare with the ID supplied in the LeaderAndIsr
> Request.
> > There is some discussion in the KIP about changes to the directory
> > structure, but I believe directory changes were considered to be out of
> > scope when the KIP was written.
> >
> > 3. I think this is a good point, but I again I wonder about the scope of
> > the KIP. Most of the changes mentioned in the KIP are for supporting
> topic
> > deletion and I believe that is why the produce request was listed under
> > future work.
> >
> > 4. This sounds like it might be a good solution, but I will need to
> discuss
> > more with KIP-500 folks to get the details right.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Sep 24, 2020 at 12:30 PM Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for picking up this work. I have a few questions/comments:
> > >
> > > 1. It sounds like the directory structure is still going to be based on
> > > topic names. Do I have that right? One complication is that the
> > > LeaderAndIsr request does not include the topic name any longer. This
> > means
> > > that a replica must wait for the UpdateMetadata request to arrive with
> > the
> > > topic name mapping before it can create the directory. I wonder if it
> > would
> > > be simpler to avoid assumptions on the ordering of UpdateMetadata and
> let
> > > LeaderAndIsr include the topic name as well. Feels like we are not
> saving
> > > that much by excluding it.
> > >
> > > 2. On a related note, it seems that the reason we have the partition
> > > metadata file is because we are not changing the directory structure.
> We
> > > need it so that we remember which directories map to which topic id. I
> > > think it would be useful to add some detail about why changing the
> > > directory layout was rejected. Basically trying to understand how much
> > > effort we are saving by skipping this step if we have to implement this
> > > additional bookkeeping. One concern I have for example is that the
> > > partition metadata file gets corrupt and then all bets are off as far
> as
> > > topic consistency.
> > >
> > > 3. Is it worth adding support now for topic ids in the `Produce`
> request
> > > now? Overhead is mentioned as one of the motivations and the best APIs
> to
> > > get savings from are `Produce` and `Fetch`.
> > >
> > > 4. When it comes to bootstrapping the metadata topic with respect to
> > > KIP-500, one suggestion would be to reserve a sentinel topic ID which
> can
> > > be used initially by new brokers. When the first controller is elected,
> > it
> > > can assign a topicId to the metadata topic. This is a bit similar to
> how
> > we
> > > were planning to generate the clusterId.
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Thu, Sep 24, 2020 at 11:10 AM Jun Rao <j...@confluent.io> wrote:
> > >
> > > > Hi, Justine,
> > > >
> > > > Thanks for the updated KIP. A few more comments below.
> > > >
> > > > 30. {"name": "id", "type": "string", "doc": "version id"}}: The doc
> > > should
> > > > say UUID.
> > > >
> > > > 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems
> there
> > > is
> > > > no need to add topic_id at partitions level.
> > > >
> > > > 32. Regarding partition metadata file. Perhaps the key can be a
> single
> > > > word, sth like the following.
> > > > version: 0
> > > > topic_id: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4
> > > >
> > > > 33. Another tricky thing that I realized is how to support the
> metadata
> > > > topic introduced in KIP-595. It's a bit tricky to assign a UUID to
> the
> > > > metadata topic since we have a chicken-and-egg problem. The
> controller
> > > > needs to persist the UUID in the metadata topic in order to assign
> one
> > > > successfully, but the metadata topic is needed to elect a controller
> > > > (KIP-631). So, this probably needs a bit more thought.
> > > >
> > > > Jun
> > > >
> > > > On Thu, Sep 24, 2020 at 3:04 AM Ismael Juma <ism...@juma.me.uk>
> wrote:
> > > >
> > > > > Also, can we provide more details on how the Partition Metadata
> file
> > > will
> > > > > be used?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <ism...@juma.me.uk>
> > wrote:
> > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > I think we need to update the "Rejected Alternatives" section to
> > take
> > > > > into
> > > > > > account that the proposal now removes the topic name from the
> fetch
> > > > > > request. Also, if we are removing it from the Fetch request, does
> > it
> > > > make
> > > > > > sense not to remove it from similar requests like
> > ListOffsetRequest?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Sep 24, 2020 at 2:46 AM David Jacot <dja...@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > >> Hi Justine,
> > > > > >>
> > > > > >> Thanks for the KIP. I finally had time to read it :). It is a
> > great
> > > > > >> improvement.
> > > > > >>
> > > > > >> I have a few comments/questions:
> > > > > >>
> > > > > >> 1. It seems that the schema of the StopReplicaRequest is
> slightly
> > > > > >> outdated.
> > > > > >> We
> > > > > >> did some changes as part of KIP-570. V3 is already organized by
> > > > topics.
> > > > > >>
> > > > > >> 2. I just want to make sure that I understand the reconciliation
> > > > > >> logic correctly. When
> > > > > >> an "INCREMENTAL" LeaderAndIsr Request is received, the broker
> will
> > > > also
> > > > > >> reconcile
> > > > > >> when the local uuid does not match the uuid in the request,
> right?
> > > In
> > > > > this
> > > > > >> case, the
> > > > > >> local log is staged for deletion.
> > > > > >>
> > > > > >> 3. In the documentation of the `delete.stale.topic.delay.ms`
> > > config,
> > > > it
> > > > > >> says "When a
> > > > > >> FULL LeaderAndIsrRequest is received..." but I suppose it
> applies
> > to
> > > > > both
> > > > > >> types.
> > > > > >>
> > > > > >> Best,
> > > > > >> David
> > > > > >>
> > > > > >> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <
> > > jols...@confluent.io>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Jun,
> > > > > >> >
> > > > > >> > Thanks for the comments. I apologize for some of the typos and
> > > > > >> confusion.
> > > > > >> > I’ve updated the KIP to fix some of the issues you mentioned.
> > > > > >> >
> > > > > >> > 20.2 I’ve changed the type to String.
> > > > > >> > 20.1/3 I’ve updated the TopicZNode to fix formatting and
> reflect
> > > the
> > > > > >> latest
> > > > > >> > version before this change.
> > > > > >> >
> > > > > >> > 21. You are correct and I’ve removed this line. I’ve also
> added
> > a
> > > > line
> > > > > >> > mentioning an IBP bump is necessary for migration
> > > > > >> >
> > > > > >> > 22. I think the wording was unclear but your summary is what
> was
> > > > > >> intended
> > > > > >> > by this line. I’ve updated to make this point more clear.
> > “Remove
> > > > > >> deleted
> > > > > >> > topics from replicas by sending StopReplicaRequest V3 before
> the
> > > IBP
> > > > > >> bump
> > > > > >> > using the old logic, and using V4 and the new logic with topic
> > IDs
> > > > > after
> > > > > >> > the IBP bump.”
> > > > > >> >
> > > > > >> > 23. I’ve removed the unspecified type from the KIP and mention
> > > that
> > > > > IBP
> > > > > >> > will be used to handle this request. “IBP will be used to
> > > determine
> > > > > >> whether
> > > > > >> > this new form of the request will be used. For older requests,
> > we
> > > > will
> > > > > >> > ignore this field and default to previous behavior.”
> > > > > >> >
> > > > > >> > 24. I’ve fixed this typo.
> > > > > >> >
> > > > > >> > 25. I've added a topics at a higher level for
> > LeaderAndIsrResponse
> > > > v5,
> > > > > >> > StopReplicaResponse v4. I've also changed StopReplicaRequest
> v4
> > to
> > > > > have
> > > > > >> > topics at a higher level.
> > > > > >> >
> > > > > >> > 26. I’ve updated forgotten_topics_data--added the topic ID and
> > > > removed
> > > > > >> the
> > > > > >> > topic name
> > > > > >> >
> > > > > >> > 27. I’ve decided on plain text, and I’ve added an example.
> > > > > >> >
> > > > > >> > 28. I’ve added this idea to future work.
> > > > > >> >
> > > > > >> > Thanks again for taking a look,
> > > > > >> >
> > > > > >> > Justine
> > > > > >> >
> > > > > >> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <j...@confluent.io>
> > > wrote:
> > > > > >> >
> > > > > >> > > Hi, Justine,
> > > > > >> > >
> > > > > >> > > Thanks for the response. Made another pass. A few more
> > comments
> > > > > below.
> > > > > >> > >
> > > > > >> > > 20. znode schema:
> > > > > >> > > 20.1 It seems that {"name": "version", "type": "int", "id":
> > > > "UUID",
> > > > > >> > "doc":
> > > > > >> > > "version id"} should be {"name": "version", "type": "int"},
> > > > {"name":
> > > > > >> > "id",
> > > > > >> > > "type": "UUID", "doc": "version id"}.
> > > > > >> > > 20.2 The znode format is JSON which doesn't have UUID type.
> So
> > > the
> > > > > >> type
> > > > > >> > > probably should be string?
> > > > > >> > > 20.3 Also, the existing format used seems outdated. It
> should
> > > have
> > > > > the
> > > > > >> > > following format.
> > > > > >> > > Json.encodeAsBytes(Map(
> > > > > >> > > "version" -> 2,
> > > > > >> > > "partitions" -> replicaAssignmentJson.asJava,
> > > > > >> > > "adding_replicas" ->
> > addingReplicasAssignmentJson.asJava,
> > > > > >> > > "removing_replicas" ->
> > > removingReplicasAssignmentJson.asJava
> > > > > >> > > ).asJava)
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > 21. Migration: The KIP says "The migration process can take
> > > place
> > > > > >> without
> > > > > >> > > an inter-broker protocol bump, as the format stored in
> > > > > >> > > /brokers/topics/[topic] will be compatible with older broker
> > > > > >> versions."
> > > > > >> > > However, since we are bumping up the version of inter-broker
> > > > > >> requests, it
> > > > > >> > > seems that we need to use IBP for migration.
> > > > > >> > >
> > > > > >> > > 22. The KIP says "Remove deleted topics from replicas by
> > sending
> > > > > >> > > StopReplicaRequest V3 for any topics which do not contain a
> > > topic
> > > > > ID,
> > > > > >> and
> > > > > >> > > V4 for any topics which do contain a topic ID.". However, if
> > we
> > > > use
> > > > > >> IBP,
> > > > > >> > it
> > > > > >> > > seems that the controller will either send
> StopReplicaRequest
> > V3
> > > > > >> > > or StopReplicaRequest V4, but never mixed V3 and V4 for
> > > different
> > > > > >> topics.
> > > > > >> > > Basically, before the IBP bump, V3 will be used. After the
> IBP
> > > > bump,
> > > > > >> > > topicId will be created and V4 will be used.
> > > > > >> > >
> > > > > >> > > 23. Given that we depend on IBP, do we still need "0
> > > UNSPECIFIED"
> > > > > >> > > in LeaderAndIsr?
> > > > > >> > >
> > > > > >> > > 24. LeaderAndIsrResponse v5 : It still has the topic field.
> > > > > >> > >
> > > > > >> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse v4: Could
> we
> > > use
> > > > > this
> > > > > >> > > opportunity to organize the response in 2 levels, first by
> > > topic,
> > > > > >> then by
> > > > > >> > > partition, as most other requests/responses?
> > > > > >> > >
> > > > > >> > > 26. FetchRequest v13 : Should forgotten_topics_data use
> > topicId
> > > > too?
> > > > > >> > >
> > > > > >> > > 27. "This file can either be plain text (key/value pairs) or
> > > > JSON."
> > > > > >> Have
> > > > > >> > we
> > > > > >> > > decided which one to use? Also, it would be helpful to
> provide
> > > an
> > > > > >> > example.
> > > > > >> > >
> > > > > >> > > 28. Future improvement: Another future improvement
> opportunity
> > > is
> > > > to
> > > > > >> use
> > > > > >> > > topicId in GroupMetadataManager.offsetCommitKey in the
> > > > offset_commit
> > > > > >> > topic.
> > > > > >> > > This may save some space.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Jun
> > > > > >> > >
> > > > > >> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <
> > > > > jols...@confluent.io>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Tom,
> > > > > >> > > >
> > > > > >> > > > Thanks for the comment. I think this is a really good idea
> > and
> > > > it
> > > > > >> has
> > > > > >> > > been
> > > > > >> > > > added to the KIP under the newly added tooling section.
> > > > > >> > > >
> > > > > >> > > > Thanks again,
> > > > > >> > > > Justine
> > > > > >> > > >
> > > > > >> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <
> > > > tbent...@redhat.com>
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Justine,
> > > > > >> > > > >
> > > > > >> > > > > I know you started the vote thread, but on re-reading
> the
> > > KIP
> > > > I
> > > > > >> > noticed
> > > > > >> > > > > that although the topic id is included in the
> > > MetadataResponse
> > > > > >> it's
> > > > > >> > not
> > > > > >> > > > > surfaced in the output from `kafka-topics.sh
> --describe`.
> > > > Maybe
> > > > > >> that
> > > > > >> > > was
> > > > > >> > > > > intentional because ids are intentionally not really
> > > something
> > > > > the
> > > > > >> > user
> > > > > >> > > > > should care deeply about, but it would also make life
> > harder
> > > > for
> > > > > >> > anyone
> > > > > >> > > > > debugging Kafka and this would likely get worse the more
> > > topic
> > > > > ids
> > > > > >> > got
> > > > > >> > > > > rolled out across the protocols, clients etc. It seems
> > > likely
> > > > > that
> > > > > >> > > > > `kafka-topics.sh` will eventually need the ability to
> show
> > > the
> > > > > id
> > > > > >> of
> > > > > >> > a
> > > > > >> > > > > topic and perhaps find a topic name given an id. Is
> there
> > > any
> > > > > >> reason
> > > > > >> > > not
> > > > > >> > > > to
> > > > > >> > > > > implement that in this KIP?
> > > > > >> > > > >
> > > > > >> > > > > Many thanks,
> > > > > >> > > > >
> > > > > >> > > > > Tom
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan <
> > > > > >> jols...@confluent.io
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi all,
> > > > > >> > > > > >
> > > > > >> > > > > > After thinking about it, I've decided to remove the
> > topic
> > > > name
> > > > > >> from
> > > > > >> > > the
> > > > > >> > > > > > Fetch Request and Response after all. Since there are
> so
> > > > many
> > > > > of
> > > > > >> > > these
> > > > > >> > > > > > requests per second, it is worth removing the extra
> > > > > information.
> > > > > >> > I've
> > > > > >> > > > > > updated the KIP to reflect this change.
> > > > > >> > > > > >
> > > > > >> > > > > > Please let me know if there is anything else we should
> > > > discuss
> > > > > >> > before
> > > > > >> > > > > > voting.
> > > > > >> > > > > >
> > > > > >> > > > > > Thank you,
> > > > > >> > > > > > Justine
> > > > > >> > > > > >
> > > > > >> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine Olshan <
> > > > > >> > jols...@confluent.io
> > > > > >> > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Jun,
> > > > > >> > > > > > >
> > > > > >> > > > > > > I see what you are saying. For now we can remove the
> > > extra
> > > > > >> > > > information.
> > > > > >> > > > > > > I'll leave the option to add more fields to the file
> > in
> > > > the
> > > > > >> > future.
> > > > > >> > > > The
> > > > > >> > > > > > KIP
> > > > > >> > > > > > > has been updated to reflect this change.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks,
> > > > > >> > > > > > > Justine
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <
> > > j...@confluent.io
> > > > >
> > > > > >> > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > >> Hi, Justine,
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> Thanks for the reply.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> 13. If the log directory is the source of truth, it
> > > means
> > > > > >> that
> > > > > >> > the
> > > > > >> > > > > > >> redundant info in the metadata file will be
> ignored.
> > > Then
> > > > > the
> > > > > >> > > > question
> > > > > >> > > > > > is
> > > > > >> > > > > > >> why do we need to put the redundant info in the
> > > metadata
> > > > > file
> > > > > >> > now?
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> Thanks,
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> Jun
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine Olshan <
> > > > > >> > > > jols...@confluent.io>
> > > > > >> > > > > > >> wrote:
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> > Hi Jun,
> > > > > >> > > > > > >> > Thanks for the quick response!
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > 12. I've decided to bump up the versions on the
> > > > requests
> > > > > >> and
> > > > > >> > > > updated
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > KIP. I think it's good we thoroughly discussed
> the
> > > > > options
> > > > > >> > here,
> > > > > >> > > > so
> > > > > >> > > > > we
> > > > > >> > > > > > >> know
> > > > > >> > > > > > >> > we made a good choice. :)
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > 13. This is an interesting situation. I think if
> > this
> > > > > does
> > > > > >> > occur
> > > > > >> > > > we
> > > > > >> > > > > > >> should
> > > > > >> > > > > > >> > give a warning. I agree that it's hard to know
> the
> > > > source
> > > > > >> of
> > > > > >> > > truth
> > > > > >> > > > > for
> > > > > >> > > > > > >> sure
> > > > > >> > > > > > >> > since the directory or the file could be manually
> > > > > >> modified. I
> > > > > >> > > > guess
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > directory could be used as the source of truth.
> To
> > be
> > > > > >> honest,
> > > > > >> > > I'm
> > > > > >> > > > > not
> > > > > >> > > > > > >> > really sure what happens in kafka when the log
> > > > directory
> > > > > is
> > > > > >> > > > renamed
> > > > > >> > > > > > >> > manually in such a way. I'm also wondering if the
> > > > > >> situation is
> > > > > >> > > > > > >> recoverable
> > > > > >> > > > > > >> > in this scenario.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > Thanks,
> > > > > >> > > > > > >> > Justine
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao <
> > > > > j...@confluent.io>
> > > > > >> > > wrote:
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > > Hi, Justine,
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Thanks for the reply.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > 12. I don't have a strong preference either.
> > > However,
> > > > > if
> > > > > >> we
> > > > > >> > > need
> > > > > >> > > > > IBP
> > > > > >> > > > > > >> > > anyway, maybe it's easier to just bump up the
> > > version
> > > > > for
> > > > > >> > all
> > > > > >> > > > > inter
> > > > > >> > > > > > >> > broker
> > > > > >> > > > > > >> > > requests and add the topic id field as a
> regular
> > > > > field. A
> > > > > >> > > > regular
> > > > > >> > > > > > >> field
> > > > > >> > > > > > >> > is
> > > > > >> > > > > > >> > > a bit more concise in wire transfer than a
> > flexible
> > > > > >> field.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > 13. The confusion that I was referring to is
> > > between
> > > > > the
> > > > > >> > topic
> > > > > >> > > > > name
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > partition number between the log dir and the
> > > metadata
> > > > > >> file.
> > > > > >> > > For
> > > > > >> > > > > > >> example,
> > > > > >> > > > > > >> > if
> > > > > >> > > > > > >> > > the log dir is topicA-1 and the metadata file
> in
> > it
> > > > has
> > > > > >> > topicB
> > > > > >> > > > and
> > > > > >> > > > > > >> > > partition 0 (say due to a bug or manual
> > > > modification),
> > > > > >> which
> > > > > >> > > one
> > > > > >> > > > > do
> > > > > >> > > > > > we
> > > > > >> > > > > > >> > use
> > > > > >> > > > > > >> > > as the source of truth?
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Jun
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM Justine Olshan
> <
> > > > > >> > > > > > jols...@confluent.io>
> > > > > >> > > > > > >> > > wrote:
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > > Hi Jun,
> > > > > >> > > > > > >> > > > Thanks for the comments.
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest because
> I
> > > > > removed
> > > > > >> the
> > > > > >> > > > topic
> > > > > >> > > > > > >> name
> > > > > >> > > > > > >> > > field
> > > > > >> > > > > > >> > > > in the response. It may be possible to avoid
> > > > bumping
> > > > > >> the
> > > > > >> > > > version
> > > > > >> > > > > > >> > without
> > > > > >> > > > > > >> > > > that change, but I may be missing something.
> > > > > >> > > > > > >> > > > I believe StopReplica is actually on version
> 3
> > > now,
> > > > > but
> > > > > >> > > > because
> > > > > >> > > > > > >> > version 2
> > > > > >> > > > > > >> > > > is flexible, I kept that listed as version 2
> on
> > > the
> > > > > KIP
> > > > > >> > > page.
> > > > > >> > > > > > >> However,
> > > > > >> > > > > > >> > > you
> > > > > >> > > > > > >> > > > may be right in that we may need to bump the
> > > > version
> > > > > on
> > > > > >> > > > > > StopReplica
> > > > > >> > > > > > >> to
> > > > > >> > > > > > >> > > deal
> > > > > >> > > > > > >> > > > with deletion differently as mentioned
> above. I
> > > > don't
> > > > > >> know
> > > > > >> > > if
> > > > > >> > > > I
> > > > > >> > > > > > >> have a
> > > > > >> > > > > > >> > > big
> > > > > >> > > > > > >> > > > preference over used tagged fields or not.
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > 13. I was thinking that in the case where the
> > > file
> > > > > and
> > > > > >> the
> > > > > >> > > > > request
> > > > > >> > > > > > >> > topic
> > > > > >> > > > > > >> > > > ids don't match, it means that the broker's
> > > > topic/the
> > > > > >> one
> > > > > >> > in
> > > > > >> > > > the
> > > > > >> > > > > > >> file
> > > > > >> > > > > > >> > has
> > > > > >> > > > > > >> > > > been deleted. In that case, we would need to
> > > delete
> > > > > the
> > > > > >> > old
> > > > > >> > > > > topic
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > start
> > > > > >> > > > > > >> > > > receiving the new version. If the topic name
> > were
> > > > to
> > > > > >> > change,
> > > > > >> > > > but
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > ids
> > > > > >> > > > > > >> > > > still matched, the file would also need to
> > > update.
> > > > > Am I
> > > > > >> > > > missing
> > > > > >> > > > > a
> > > > > >> > > > > > >> case
> > > > > >> > > > > > >> > > > where the file would be correct and not the
> > > > request?
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > Thanks,
> > > > > >> > > > > > >> > > > Justine
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun Rao <
> > > > > >> j...@confluent.io
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > > Hi, Justine,
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Thanks for the reply. A couple of more
> > comments
> > > > > >> below.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > 12. ListOffset and OffsetForLeader
> currently
> > > > don't
> > > > > >> > support
> > > > > >> > > > > > >> flexible
> > > > > >> > > > > > >> > > > fields.
> > > > > >> > > > > > >> > > > > So, we have to bump up the version number
> and
> > > use
> > > > > >> IBP at
> > > > > >> > > > least
> > > > > >> > > > > > for
> > > > > >> > > > > > >> > > these
> > > > > >> > > > > > >> > > > > two requests. Note that it seems 2.7.0 will
> > > > require
> > > > > >> IBP
> > > > > >> > > > anyway
> > > > > >> > > > > > >> > because
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > changes in KAFKA-10435. Also, it seems that
> > the
> > > > > >> version
> > > > > >> > > for
> > > > > >> > > > > > >> > > > > LeaderAndIsrRequest and StopReplica are
> > bumped
> > > > even
> > > > > >> > though
> > > > > >> > > > we
> > > > > >> > > > > > only
> > > > > >> > > > > > >> > > added
> > > > > >> > > > > > >> > > > a
> > > > > >> > > > > > >> > > > > tagged field. But since IBP is needed
> anyway,
> > > we
> > > > > may
> > > > > >> > want
> > > > > >> > > to
> > > > > >> > > > > > >> revisit
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > overall tagged field choice.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > 13. The only downside is the potential
> > > confusion
> > > > on
> > > > > >> > which
> > > > > >> > > > one
> > > > > >> > > > > is
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > > source
> > > > > >> > > > > > >> > > > > of truth if they don't match. Another
> option
> > is
> > > > to
> > > > > >> > include
> > > > > >> > > > > those
> > > > > >> > > > > > >> > fields
> > > > > >> > > > > > >> > > > in
> > > > > >> > > > > > >> > > > > the metadata file when we actually change
> the
> > > > > >> directory
> > > > > >> > > > > > structure.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Thanks,
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Jun
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM Justine
> > Olshan
> > > <
> > > > > >> > > > > > >> jols...@confluent.io
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > > > wrote:
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > Hello all,
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > I've thought some more about removing the
> > > topic
> > > > > >> name
> > > > > >> > > field
> > > > > >> > > > > > from
> > > > > >> > > > > > >> > some
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > requests. On closer inspection of the
> > > > > >> > > requests/responses,
> > > > > >> > > > it
> > > > > >> > > > > > >> seems
> > > > > >> > > > > > >> > > that
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > internal changes would be much larger
> than
> > I
> > > > > >> expected.
> > > > > >> > > > Some
> > > > > >> > > > > > >> > protocols
> > > > > >> > > > > > >> > > > > > involve clients, so they would require
> > > changes
> > > > > too.
> > > > > >> > I'm
> > > > > >> > > > > > thinking
> > > > > >> > > > > > >> > that
> > > > > >> > > > > > >> > > > for
> > > > > >> > > > > > >> > > > > > now, removing the topic name from these
> > > > requests
> > > > > >> and
> > > > > >> > > > > responses
> > > > > >> > > > > > >> are
> > > > > >> > > > > > >> > > out
> > > > > >> > > > > > >> > > > of
> > > > > >> > > > > > >> > > > > > scope.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > I have decided to just keep the change
> > > > > >> > > > LeaderAndIsrResponse
> > > > > >> > > > > to
> > > > > >> > > > > > >> > remove
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > topic name, and have updated the KIP to
> > > reflect
> > > > > >> this
> > > > > >> > > > > change. I
> > > > > >> > > > > > >> have
> > > > > >> > > > > > >> > > > also
> > > > > >> > > > > > >> > > > > > mentioned the other requests and
> responses
> > in
> > > > > >> future
> > > > > >> > > work.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > I'm hoping to start the voting process
> > soon,
> > > so
> > > > > >> let me
> > > > > >> > > > know
> > > > > >> > > > > if
> > > > > >> > > > > > >> > there
> > > > > >> > > > > > >> > > is
> > > > > >> > > > > > >> > > > > > anything else we should discuss.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > Thank you,
> > > > > >> > > > > > >> > > > > > Justine
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM Justine
> > > Olshan
> > > > <
> > > > > >> > > > > > >> > jols...@confluent.io
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > > > wrote:
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > > Hello again,
> > > > > >> > > > > > >> > > > > > > To follow up on some of the other
> > comments:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > 10/11) We can remove the topic name
> from
> > > > these
> > > > > >> > > > > > >> > requests/responses,
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > > that means we will just have to make a
> > few
> > > > > >> internal
> > > > > >> > > > > changes
> > > > > >> > > > > > to
> > > > > >> > > > > > >> > make
> > > > > >> > > > > > >> > > > > > > partitions accessible by topic id and
> > > > > partition.
> > > > > >> I
> > > > > >> > can
> > > > > >> > > > > > update
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > KIP
> > > > > >> > > > > > >> > > > > to
> > > > > >> > > > > > >> > > > > > > remove them unless anyone thinks they
> > > should
> > > > > >> stay.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > 12) Addressed in the previous email.
> I've
> > > > > updated
> > > > > >> > the
> > > > > >> > > > KIP
> > > > > >> > > > > to
> > > > > >> > > > > > >> > > include
> > > > > >> > > > > > >> > > > > > > tagged fields for the requests and
> > > responses.
> > > > > >> (More
> > > > > >> > on
> > > > > >> > > > > that
> > > > > >> > > > > > >> > below)
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > 13) I think part of the idea for
> > including
> > > > this
> > > > > >> > > > > information
> > > > > >> > > > > > >> is to
> > > > > >> > > > > > >> > > > > prepare
> > > > > >> > > > > > >> > > > > > > for future changes. Perhaps the
> directory
> > > > > >> structure
> > > > > >> > > > might
> > > > > >> > > > > > >> change
> > > > > >> > > > > > >> > > from
> > > > > >> > > > > > >> > > > > > > topicName_partitionNumber to something
> > like
> > > > > >> > > > > > >> > > topicID_partitionNumber.
> > > > > >> > > > > > >> > > > > Then
> > > > > >> > > > > > >> > > > > > > it would be useful to have the topic
> name
> > > in
> > > > > the
> > > > > >> > file
> > > > > >> > > > > since
> > > > > >> > > > > > it
> > > > > >> > > > > > >> > > would
> > > > > >> > > > > > >> > > > > not
> > > > > >> > > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > > > > in the directory structure. Supporting
> > > topic
> > > > > >> renames
> > > > > >> > > > might
> > > > > >> > > > > > be
> > > > > >> > > > > > >> > > easier
> > > > > >> > > > > > >> > > > if
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > other fields are included. Would there
> be
> > > any
> > > > > >> > > downsides
> > > > > >> > > > to
> > > > > >> > > > > > >> > > including
> > > > > >> > > > > > >> > > > > this
> > > > > >> > > > > > >> > > > > > > information?
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > 14) Yes, we would need to copy the
> > > partition
> > > > > >> > metadata
> > > > > >> > > > > file
> > > > > >> > > > > > in
> > > > > >> > > > > > >> > this
> > > > > >> > > > > > >> > > > > > > process. I've updated the KIP to
> include
> > > > this.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > 15) I believe Lucas meant v1 and v2
> here.
> > > He
> > > > > was
> > > > > >> > > > referring
> > > > > >> > > > > > to
> > > > > >> > > > > > >> how
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > > requests would fall under different IBP
> > and
> > > > > meant
> > > > > >> > that
> > > > > >> > > > > older
> > > > > >> > > > > > >> > > brokers
> > > > > >> > > > > > >> > > > > > would
> > > > > >> > > > > > >> > > > > > > have to use the older version of the
> > > request
> > > > > and
> > > > > >> the
> > > > > >> > > > > > existing
> > > > > >> > > > > > >> > topic
> > > > > >> > > > > > >> > > > > > > deletion process. At first, it seemed
> > like
> > > > > tagged
> > > > > >> > > fields
> > > > > >> > > > > > would
> > > > > >> > > > > > >> > > > resolve
> > > > > >> > > > > > >> > > > > > > the IBP issue. However, we may need IBP
> > for
> > > > > this
> > > > > >> > > request
> > > > > >> > > > > > after
> > > > > >> > > > > > >> > all
> > > > > >> > > > > > >> > > > > since
> > > > > >> > > > > > >> > > > > > > the controller handles the topic
> deletion
> > > > > >> > differently
> > > > > >> > > > > > >> depending
> > > > > >> > > > > > >> > on
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > IBP
> > > > > >> > > > > > >> > > > > > > version. In an older version, we can't
> > just
> > > > > send
> > > > > >> a
> > > > > >> > > > > > StopReplica
> > > > > >> > > > > > >> > > delete
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > > topic immediately like we'd want to for
> > > this
> > > > > KIP.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > This makes me wonder if we want tagged
> > > fields
> > > > > on
> > > > > >> all
> > > > > >> > > the
> > > > > >> > > > > > >> requests
> > > > > >> > > > > > >> > > > after
> > > > > >> > > > > > >> > > > > > > all. Let me know your thoughts!
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > Justine
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03 PM Justine
> > > > Olshan
> > > > > <
> > > > > >> > > > > > >> > > jols...@confluent.io
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > >> Hi all,
> > > > > >> > > > > > >> > > > > > >> Jun brought up a good point in his
> last
> > > > email
> > > > > >> about
> > > > > >> > > > > tagged
> > > > > >> > > > > > >> > fields,
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > >> I've updated the KIP to reflect that
> the
> > > > > >> changes to
> > > > > >> > > > > > requests
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > > > responses
> > > > > >> > > > > > >> > > > > > >> will be in the form of tagged fields
> to
> > > > avoid
> > > > > >> > > changing
> > > > > >> > > > > IBP.
> > > > > >> > > > > > >> > > > > > >>
> > > > > >> > > > > > >> > > > > > >> Jun: I plan on sending a followup
> email
> > to
> > > > > >> address
> > > > > >> > > some
> > > > > >> > > > > of
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > other
> > > > > >> > > > > > >> > > > > > >> points.
> > > > > >> > > > > > >> > > > > > >>
> > > > > >> > > > > > >> > > > > > >> Thanks,
> > > > > >> > > > > > >> > > > > > >> Justine
> > > > > >> > > > > > >> > > > > > >>
> > > > > >> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25 PM Jun
> Rao
> > <
> > > > > >> > > > > j...@confluent.io>
> > > > > >> > > > > > >> > wrote:
> > > > > >> > > > > > >> > > > > > >>
> > > > > >> > > > > > >> > > > > > >>> Hi, Justine,
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> Thanks for the updated KIP. A few
> > > comments
> > > > > >> below.
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response: Do we need
> > the
> > > > > topic
> > > > > >> > > name?
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 11. For the changed request/response,
> > > other
> > > > > >> than
> > > > > >> > > > > > >> LeaderAndIsr,
> > > > > >> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do we need
> to
> > > > > include
> > > > > >> > the
> > > > > >> > > > > topic
> > > > > >> > > > > > >> name?
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 12. It seems that upgrades don't
> > require
> > > > IBP.
> > > > > >> Does
> > > > > >> > > > that
> > > > > >> > > > > > mean
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > new
> > > > > >> > > > > > >> > > > > > >>> fields
> > > > > >> > > > > > >> > > > > > >>> in all the request/response are added
> > as
> > > > > tagged
> > > > > >> > > fields
> > > > > >> > > > > > >> without
> > > > > >> > > > > > >> > > > > bumping
> > > > > >> > > > > > >> > > > > > up
> > > > > >> > > > > > >> > > > > > >>> the request version? It would be
> useful
> > > to
> > > > > make
> > > > > >> > that
> > > > > >> > > > > > clear.
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 13. Partition Metadata file: Do we
> need
> > > to
> > > > > >> include
> > > > > >> > > the
> > > > > >> > > > > > topic
> > > > > >> > > > > > >> > name
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > >>> partition id since they are implied
> in
> > > the
> > > > > >> > directory
> > > > > >> > > > > name?
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 14. In the JBOD mode, we support
> > moving a
> > > > > >> > > partition's
> > > > > >> > > > > data
> > > > > >> > > > > > >> from
> > > > > >> > > > > > >> > > one
> > > > > >> > > > > > >> > > > > > disk
> > > > > >> > > > > > >> > > > > > >>> to
> > > > > >> > > > > > >> > > > > > >>> another. Will the new partition
> > metadata
> > > > file
> > > > > >> be
> > > > > >> > > > copied
> > > > > >> > > > > > >> during
> > > > > >> > > > > > >> > > that
> > > > > >> > > > > > >> > > > > > >>> process?
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> 15. The KIP says "Remove deleted
> topics
> > > > from
> > > > > >> > > replicas
> > > > > >> > > > by
> > > > > >> > > > > > >> > sending
> > > > > >> > > > > > >> > > > > > >>> StopReplicaRequest V2 for any topics
> > > which
> > > > do
> > > > > >> not
> > > > > >> > > > > contain
> > > > > >> > > > > > a
> > > > > >> > > > > > >> > topic
> > > > > >> > > > > > >> > > > ID,
> > > > > >> > > > > > >> > > > > > and
> > > > > >> > > > > > >> > > > > > >>> V3 for any topics which do contain a
> > > topic
> > > > > >> ID.".
> > > > > >> > > > > However,
> > > > > >> > > > > > it
> > > > > >> > > > > > >> > > seems
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > >>> updated controller will create all
> > > missing
> > > > > >> topic
> > > > > >> > IDs
> > > > > >> > > > > first
> > > > > >> > > > > > >> > before
> > > > > >> > > > > > >> > > > > doing
> > > > > >> > > > > > >> > > > > > >>> other actions. So, is
> > StopReplicaRequest
> > > V2
> > > > > >> > needed?
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> Jun
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at 10:31 AM John
> > > > > Roesler <
> > > > > >> > > > > > >> > > vvcep...@apache.org
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > >>> wrote:
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>> > Thanks, Justine!
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>> > Your response seems compelling to
> me.
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>> > -John
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at 10:17 -0700,
> > > > Justine
> > > > > >> > Olshan
> > > > > >> > > > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > Hello all,
> > > > > >> > > > > > >> > > > > > >>> > > Thanks for continuing the
> > > discussion! I
> > > > > >> have a
> > > > > >> > > few
> > > > > >> > > > > > >> > responses
> > > > > >> > > > > > >> > > to
> > > > > >> > > > > > >> > > > > > your
> > > > > >> > > > > > >> > > > > > >>> > points.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > Tom: You are correct in that this
> > KIP
> > > > has
> > > > > >> not
> > > > > >> > > > > > mentioned
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I think that
> > > this
> > > > > >> would
> > > > > >> > be
> > > > > >> > > > out
> > > > > >> > > > > of
> > > > > >> > > > > > >> > scope
> > > > > >> > > > > > >> > > > for
> > > > > >> > > > > > >> > > > > > >>> now, but
> > > > > >> > > > > > >> > > > > > >>> > > may be something worth adding in
> > the
> > > > > >> future.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > John: We did consider sequence
> ids,
> > > but
> > > > > >> there
> > > > > >> > > are
> > > > > >> > > > a
> > > > > >> > > > > > few
> > > > > >> > > > > > >> > > reasons
> > > > > >> > > > > > >> > > > > to
> > > > > >> > > > > > >> > > > > > >>> favor
> > > > > >> > > > > > >> > > > > > >>> > > UUIDs. There are several cases
> > where
> > > > > topics
> > > > > >> > from
> > > > > >> > > > > > >> different
> > > > > >> > > > > > >> > > > > clusters
> > > > > >> > > > > > >> > > > > > >>> may
> > > > > >> > > > > > >> > > > > > >>> > > interact now and in the future.
> For
> > > > > >> example,
> > > > > >> > > > Mirror
> > > > > >> > > > > > >> Maker 2
> > > > > >> > > > > > >> > > may
> > > > > >> > > > > > >> > > > > > >>> benefit
> > > > > >> > > > > > >> > > > > > >>> > > from being able to detect when a
> > > > cluster
> > > > > >> being
> > > > > >> > > > > > mirrored
> > > > > >> > > > > > >> is
> > > > > >> > > > > > >> > > > > deleted
> > > > > >> > > > > > >> > > > > > >>> and
> > > > > >> > > > > > >> > > > > > >>> > > recreated and globally unique
> > > > identifiers
> > > > > >> > would
> > > > > >> > > > make
> > > > > >> > > > > > >> > > resolving
> > > > > >> > > > > > >> > > > > > issues
> > > > > >> > > > > > >> > > > > > >>> > > easier than sequence IDs which
> may
> > > > > collide
> > > > > >> > > between
> > > > > >> > > > > > >> > clusters.
> > > > > >> > > > > > >> > > > > > KIP-405
> > > > > >> > > > > > >> > > > > > >>> > > (tiered storage) will also
> benefit
> > > from
> > > > > >> > globally
> > > > > >> > > > > > unique
> > > > > >> > > > > > >> IDs
> > > > > >> > > > > > >> > > as
> > > > > >> > > > > > >> > > > > > shared
> > > > > >> > > > > > >> > > > > > >>> > > buckets may be used between
> > clusters.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > Globally unique IDs would also
> make
> > > > > >> > > functionality
> > > > > >> > > > > like
> > > > > >> > > > > > >> > moving
> > > > > >> > > > > > >> > > > > > topics
> > > > > >> > > > > > >> > > > > > >>> > > between disparate clusters easier
> > in
> > > > the
> > > > > >> > future,
> > > > > >> > > > > > >> simplify
> > > > > >> > > > > > >> > any
> > > > > >> > > > > > >> > > > > > future
> > > > > >> > > > > > >> > > > > > >>> > > implementations of backups and
> > > > restores,
> > > > > >> and
> > > > > >> > > more.
> > > > > >> > > > > In
> > > > > >> > > > > > >> > > general,
> > > > > >> > > > > > >> > > > > > >>> unique IDs
> > > > > >> > > > > > >> > > > > > >>> > > would ensure that the source
> > cluster
> > > > > >> topics do
> > > > > >> > > not
> > > > > >> > > > > > >> conflict
> > > > > >> > > > > > >> > > > with
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > >>> > > destination cluster topics.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > If we were to use sequence ids,
> we
> > > > would
> > > > > >> need
> > > > > >> > > > > > >> sufficiently
> > > > > >> > > > > > >> > > > large
> > > > > >> > > > > > >> > > > > > >>> cluster
> > > > > >> > > > > > >> > > > > > >>> > > ids to be stored with the topic
> > > > > >> identifiers or
> > > > > >> > > we
> > > > > >> > > > > run
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > risk
> > > > > >> > > > > > >> > > > of
> > > > > >> > > > > > >> > > > > > >>> > > collisions. This will give up any
> > > > > >> advantage in
> > > > > >> > > > > > >> compactness
> > > > > >> > > > > > >> > > that
> > > > > >> > > > > > >> > > > > > >>> sequence
> > > > > >> > > > > > >> > > > > > >>> > > numbers may bring. Given these
> > > > > advantages I
> > > > > >> > > think
> > > > > >> > > > it
> > > > > >> > > > > > >> makes
> > > > > >> > > > > > >> > > > sense
> > > > > >> > > > > > >> > > > > to
> > > > > >> > > > > > >> > > > > > >>> use
> > > > > >> > > > > > >> > > > > > >>> > > UUIDs.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > Gokul: This is an interesting
> idea,
> > > but
> > > > > >> this
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > >> breaking
> > > > > >> > > > > > >> > > > > change.
> > > > > >> > > > > > >> > > > > > >>> Out of
> > > > > >> > > > > > >> > > > > > >>> > > scope for now, but maybe worth
> > > > discussing
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > future.
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > Hope this explains some of the
> > > > decisions,
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > Justine
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at 8:27 AM
> > Gokul
> > > > > >> Ramanan
> > > > > >> > > > > > >> Subramanian <
> > > > > >> > > > > > >> > > > > > >>> > > gokul24...@gmail.com> wrote:
> > > > > >> > > > > > >> > > > > > >>> > >
> > > > > >> > > > > > >> > > > > > >>> > > > Hi.
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > Have you thought about whether
> it
> > > > makes
> > > > > >> > sense
> > > > > >> > > to
> > > > > >> > > > > > >> support
> > > > > >> > > > > > >> > > > > > >>> authorizing a
> > > > > >> > > > > > >> > > > > > >>> > > > principal for a topic ID rather
> > > than
> > > > a
> > > > > >> topic
> > > > > >> > > > name
> > > > > >> > > > > to
> > > > > >> > > > > > >> > > achieve
> > > > > >> > > > > > >> > > > > > >>> tighter
> > > > > >> > > > > > >> > > > > > >>> > > > security?
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > Or is the topic ID
> fundamentally
> > an
> > > > > >> internal
> > > > > >> > > > > detail
> > > > > >> > > > > > >> > similar
> > > > > >> > > > > > >> > > > to
> > > > > >> > > > > > >> > > > > > >>> epochs
> > > > > >> > > > > > >> > > > > > >>> > used
> > > > > >> > > > > > >> > > > > > >>> > > > in a bunch of other places in
> > > Kafka?
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > Thanks.
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020 at 4:06 PM
> > > John
> > > > > >> > Roesler <
> > > > > >> > > > > > >> > > > > > vvcep...@apache.org>
> > > > > >> > > > > > >> > > > > > >>> > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Hello Justine,
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > I happen to have been
> > confronted
> > > > > >> recently
> > > > > >> > > with
> > > > > >> > > > > the
> > > > > >> > > > > > >> need
> > > > > >> > > > > > >> > > to
> > > > > >> > > > > > >> > > > > keep
> > > > > >> > > > > > >> > > > > > >>> > track of
> > > > > >> > > > > > >> > > > > > >>> > > > a
> > > > > >> > > > > > >> > > > > > >>> > > > > large number of topics as
> > > compactly
> > > > > as
> > > > > >> > > > > possible. I
> > > > > >> > > > > > >> was
> > > > > >> > > > > > >> > > > going
> > > > > >> > > > > > >> > > > > to
> > > > > >> > > > > > >> > > > > > >>> come
> > > > > >> > > > > > >> > > > > > >>> > up
> > > > > >> > > > > > >> > > > > > >>> > > > > with some way to dictionary
> > > encode
> > > > > the
> > > > > >> > topic
> > > > > >> > > > > names
> > > > > >> > > > > > >> as
> > > > > >> > > > > > >> > > > > integers,
> > > > > >> > > > > > >> > > > > > >>> but
> > > > > >> > > > > > >> > > > > > >>> > this
> > > > > >> > > > > > >> > > > > > >>> > > > > seems much better!
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Apologies if this has been
> > raised
> > > > > >> before,
> > > > > >> > > but
> > > > > >> > > > > I’m
> > > > > >> > > > > > >> > > wondering
> > > > > >> > > > > > >> > > > > > >>> about the
> > > > > >> > > > > > >> > > > > > >>> > > > > choice of UUID vs sequence
> > number
> > > > for
> > > > > >> the
> > > > > >> > > ids.
> > > > > >> > > > > > >> > Typically,
> > > > > >> > > > > > >> > > > > I’ve
> > > > > >> > > > > > >> > > > > > >>> seen
> > > > > >> > > > > > >> > > > > > >>> > UUIDs
> > > > > >> > > > > > >> > > > > > >>> > > > > in two situations:
> > > > > >> > > > > > >> > > > > > >>> > > > > 1. When processes need to
> > > generate
> > > > > >> > > > non-colliding
> > > > > >> > > > > > >> > > > identifiers
> > > > > >> > > > > > >> > > > > > >>> without
> > > > > >> > > > > > >> > > > > > >>> > > > > coordination.
> > > > > >> > > > > > >> > > > > > >>> > > > > 2. When the identifier needs
> to
> > > be
> > > > > >> > > > “universally
> > > > > >> > > > > > >> > unique”;
> > > > > >> > > > > > >> > > > > I.e.,
> > > > > >> > > > > > >> > > > > > >>> the
> > > > > >> > > > > > >> > > > > > >>> > > > > identifier needs to
> distinguish
> > > the
> > > > > >> entity
> > > > > >> > > > from
> > > > > >> > > > > > all
> > > > > >> > > > > > >> > other
> > > > > >> > > > > > >> > > > > > >>> entities
> > > > > >> > > > > > >> > > > > > >>> > that
> > > > > >> > > > > > >> > > > > > >>> > > > > could ever exist. This is
> > useful
> > > in
> > > > > >> cases
> > > > > >> > > > where
> > > > > >> > > > > > >> > entities
> > > > > >> > > > > > >> > > > from
> > > > > >> > > > > > >> > > > > > all
> > > > > >> > > > > > >> > > > > > >>> > kinds
> > > > > >> > > > > > >> > > > > > >>> > > > of
> > > > > >> > > > > > >> > > > > > >>> > > > > systems get mixed together,
> > such
> > > as
> > > > > >> when
> > > > > >> > > > dumping
> > > > > >> > > > > > >> logs
> > > > > >> > > > > > >> > > from
> > > > > >> > > > > > >> > > > > all
> > > > > >> > > > > > >> > > > > > >>> > processes
> > > > > >> > > > > > >> > > > > > >>> > > > in
> > > > > >> > > > > > >> > > > > > >>> > > > > a company into a common
> system.
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Maybe I’m being
> short-sighted,
> > > but
> > > > it
> > > > > >> > > doesn’t
> > > > > >> > > > > seem
> > > > > >> > > > > > >> like
> > > > > >> > > > > > >> > > > > either
> > > > > >> > > > > > >> > > > > > >>> really
> > > > > >> > > > > > >> > > > > > >>> > > > > applies here. It seems like
> the
> > > > > brokers
> > > > > >> > > could
> > > > > >> > > > > and
> > > > > >> > > > > > >> would
> > > > > >> > > > > > >> > > > > achieve
> > > > > >> > > > > > >> > > > > > >>> > consensus
> > > > > >> > > > > > >> > > > > > >>> > > > > when creating a topic anyway,
> > > which
> > > > > is
> > > > > >> all
> > > > > >> > > > > that’s
> > > > > >> > > > > > >> > > required
> > > > > >> > > > > > >> > > > to
> > > > > >> > > > > > >> > > > > > >>> > generate
> > > > > >> > > > > > >> > > > > > >>> > > > > non-colliding sequence ids.
> For
> > > the
> > > > > >> > second,
> > > > > >> > > as
> > > > > >> > > > > you
> > > > > >> > > > > > >> > > mention,
> > > > > >> > > > > > >> > > > > we
> > > > > >> > > > > > >> > > > > > >>> could
> > > > > >> > > > > > >> > > > > > >>> > > > assign
> > > > > >> > > > > > >> > > > > > >>> > > > > a UUID to the cluster as a
> > whole,
> > > > > which
> > > > > >> > > would
> > > > > >> > > > > > render
> > > > > >> > > > > > >> > any
> > > > > >> > > > > > >> > > > > > resource
> > > > > >> > > > > > >> > > > > > >>> > scoped
> > > > > >> > > > > > >> > > > > > >>> > > > to
> > > > > >> > > > > > >> > > > > > >>> > > > > the broker universally unique
> > as
> > > > > well.
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > The reason I mention this is
> > > that,
> > > > > >> > although
> > > > > >> > > a
> > > > > >> > > > > UUID
> > > > > >> > > > > > >> is
> > > > > >> > > > > > >> > way
> > > > > >> > > > > > >> > > > > more
> > > > > >> > > > > > >> > > > > > >>> > compact
> > > > > >> > > > > > >> > > > > > >>> > > > > than topic names, it’s still
> 16
> > > > > bytes.
> > > > > >> In
> > > > > >> > > > > > contrast,
> > > > > >> > > > > > >> a
> > > > > >> > > > > > >> > > > 4-byte
> > > > > >> > > > > > >> > > > > > >>> integer
> > > > > >> > > > > > >> > > > > > >>> > > > > sequence id would give us 4
> > > billion
> > > > > >> unique
> > > > > >> > > > > topics
> > > > > >> > > > > > >> per
> > > > > >> > > > > > >> > > > > cluster,
> > > > > >> > > > > > >> > > > > > >>> which
> > > > > >> > > > > > >> > > > > > >>> > > > seems
> > > > > >> > > > > > >> > > > > > >>> > > > > like enough ;)
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Considering the number of
> > > different
> > > > > >> times
> > > > > >> > > > these
> > > > > >> > > > > > >> topic
> > > > > >> > > > > > >> > > > > > >>> identifiers are
> > > > > >> > > > > > >> > > > > > >>> > > > sent
> > > > > >> > > > > > >> > > > > > >>> > > > > over the wire or stored in
> > > memory,
> > > > it
> > > > > >> > seems
> > > > > >> > > > like
> > > > > >> > > > > > it
> > > > > >> > > > > > >> > might
> > > > > >> > > > > > >> > > > be
> > > > > >> > > > > > >> > > > > > >>> worth
> > > > > >> > > > > > >> > > > > > >>> > the
> > > > > >> > > > > > >> > > > > > >>> > > > > additional 4x space savings.
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > What do you think about this?
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > Thanks,
> > > > > >> > > > > > >> > > > > > >>> > > > > John
> > > > > >> > > > > > >> > > > > > >>> > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11, 2020, at
> 03:20,
> > > Tom
> > > > > >> > Bentley
> > > > > >> > > > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > > Hi Justine,
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > This looks like a very
> > welcome
> > > > > >> > > improvement.
> > > > > >> > > > > > >> Thanks!
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > Maybe I missed it, but the
> > KIP
> > > > > >> doesn't
> > > > > >> > > seem
> > > > > >> > > > to
> > > > > >> > > > > > >> > mention
> > > > > >> > > > > > >> > > > > > changing
> > > > > >> > > > > > >> > > > > > >>> > > > > > DeleteTopicsRequest to
> > identify
> > > > the
> > > > > >> > topic
> > > > > >> > > > > using
> > > > > >> > > > > > an
> > > > > >> > > > > > >> > id.
> > > > > >> > > > > > >> > > > > Maybe
> > > > > >> > > > > > >> > > > > > >>> > that's out
> > > > > >> > > > > > >> > > > > > >>> > > > > of
> > > > > >> > > > > > >> > > > > > >>> > > > > > scope, but
> > DeleteTopicsRequest
> > > is
> > > > > not
> > > > > >> > > listed
> > > > > >> > > > > > among
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > > Future
> > > > > >> > > > > > >> > > > > > >>> Work
> > > > > >> > > > > > >> > > > > > >>> > APIs
> > > > > >> > > > > > >> > > > > > >>> > > > > > either.
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > Kind regards,
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > Tom
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10, 2020 at
> 3:59
> > PM
> > > > > >> Satish
> > > > > >> > > > > Duggana <
> > > > > >> > > > > > >> > > > > > >>> > > > satish.dugg...@gmail.com
> > > > > >> > > > > > >> > > > > > >>> > > > > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > Thanks Lucas/Justine for
> > the
> > > > nice
> > > > > >> KIP.
> > > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > It has several benefits
> > which
> > > > > also
> > > > > >> > > include
> > > > > >> > > > > > >> > > simplifying
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > >>> topic
> > > > > >> > > > > > >> > > > > > >>> > > > > > > deletion process by
> > > controller
> > > > > and
> > > > > >> > logs
> > > > > >> > > > > > cleanup
> > > > > >> > > > > > >> by
> > > > > >> > > > > > >> > > > > brokers
> > > > > >> > > > > > >> > > > > > in
> > > > > >> > > > > > >> > > > > > >>> > corner
> > > > > >> > > > > > >> > > > > > >>> > > > > > > cases.
> > > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > Best,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > Satish.
> > > > > >> > > > > > >> > > > > > >>> > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9, 2020 at
> > 10:07
> > > PM
> > > > > >> > Justine
> > > > > >> > > > > > Olshan <
> > > > > >> > > > > > >> > > > > > >>> > jols...@confluent.io
> > > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > Hello all, it's been
> > > almost a
> > > > > >> year!
> > > > > >> > > I've
> > > > > >> > > > > > made
> > > > > >> > > > > > >> > some
> > > > > >> > > > > > >> > > > > > changes
> > > > > >> > > > > > >> > > > > > >>> to
> > > > > >> > > > > > >> > > > > > >>> > this
> > > > > >> > > > > > >> > > > > > >>> > > > > KIP
> > > > > >> > > > > > >> > > > > > >>> > > > > > > and hope to continue the
> > > > > >> discussion.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > One of the main changes
> > > I've
> > > > > >> added
> > > > > >> > is
> > > > > >> > > > now
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > metadata
> > > > > >> > > > > > >> > > > > > >>> response
> > > > > >> > > > > > >> > > > > > >>> > > > will
> > > > > >> > > > > > >> > > > > > >>> > > > > > > include the topic ID (as
> > > Colin
> > > > > >> > > suggested).
> > > > > >> > > > > > >> Clients
> > > > > >> > > > > > >> > > can
> > > > > >> > > > > > >> > > > > > >>> obtain the
> > > > > >> > > > > > >> > > > > > >>> > > > > topicID
> > > > > >> > > > > > >> > > > > > >>> > > > > > > of a given topic through
> a
> > > > > >> > > > TopicDescription.
> > > > > >> > > > > > The
> > > > > >> > > > > > >> > > > topicId
> > > > > >> > > > > > >> > > > > > will
> > > > > >> > > > > > >> > > > > > >>> > also be
> > > > > >> > > > > > >> > > > > > >>> > > > > > > included with the
> > > > UpdateMetadata
> > > > > >> > > request.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > Let me know what you
> all
> > > > think.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > Thank you,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > Justine
> > > > > >> > > > > > >> > > > > > >>> > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13 16:38:26,
> > > > "Colin
> > > > > >> > > McCabe" <
> > > > > >> > > > > > >> > > > > > cmcc...@apache.org
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>> > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Thanks for tackling
> > this.
> > > > > >> Topic
> > > > > >> > IDs
> > > > > >> > > > > are a
> > > > > >> > > > > > >> > great
> > > > > >> > > > > > >> > > > > idea,
> > > > > >> > > > > > >> > > > > > >>> and
> > > > > >> > > > > > >> > > > > > >>> > this
> > > > > >> > > > > > >> > > > > > >>> > > > is
> > > > > >> > > > > > >> > > > > > >>> > > > > a
> > > > > >> > > > > > >> > > > > > >>> > > > > > > really good writeup.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > For
> > > > /brokers/topics/[topic],
> > > > > >> the
> > > > > >> > > > schema
> > > > > >> > > > > > >> version
> > > > > >> > > > > > >> > > > > should
> > > > > >> > > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > > > >>> > bumped
> > > > > >> > > > > > >> > > > > > >>> > > > to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > version 3, rather than 2.
> > > > > KIP-455
> > > > > >> > > bumped
> > > > > >> > > > > the
> > > > > >> > > > > > >> > version
> > > > > >> > > > > > >> > > > of
> > > > > >> > > > > > >> > > > > > this
> > > > > >> > > > > > >> > > > > > >>> > znode
> > > > > >> > > > > > >> > > > > > >>> > > > to
> > > > > >> > > > > > >> > > > > > >>> > > > > 2
> > > > > >> > > > > > >> > > > > > >>> > > > > > > already :)
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Given that we're
> going
> > to
> > > > be
> > > > > >> > seeing
> > > > > >> > > > > these
> > > > > >> > > > > > >> > things
> > > > > >> > > > > > >> > > as
> > > > > >> > > > > > >> > > > > > >>> strings
> > > > > >> > > > > > >> > > > > > >>> > as
> > > > > >> > > > > > >> > > > > > >>> > > > lot
> > > > > >> > > > > > >> > > > > > >>> > > > > (in
> > > > > >> > > > > > >> > > > > > >>> > > > > > > logs, in ZooKeeper, on
> the
> > > > > >> > command-line,
> > > > > >> > > > > > etc.),
> > > > > >> > > > > > >> > does
> > > > > >> > > > > > >> > > it
> > > > > >> > > > > > >> > > > > > make
> > > > > >> > > > > > >> > > > > > >>> > sense to
> > > > > >> > > > > > >> > > > > > >>> > > > > use
> > > > > >> > > > > > >> > > > > > >>> > > > > > > base64 when converting
> them
> > > to
> > > > > >> > strings?
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Here is an example of
> > the
> > > > hex
> > > > > >> > > > > > >> representation:
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > And here is an
> example
> > in
> > > > > >> base64.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> b8tRS7h4TJ2Vt43Dp85v2A
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > The base64 version
> > saves
> > > 15
> > > > > >> > letters
> > > > > >> > > > (to
> > > > > >> > > > > be
> > > > > >> > > > > > >> > fair,
> > > > > >> > > > > > >> > > 4
> > > > > >> > > > > > >> > > > of
> > > > > >> > > > > > >> > > > > > >>> those
> > > > > >> > > > > > >> > > > > > >>> > were
> > > > > >> > > > > > >> > > > > > >>> > > > > > > dashes that we could have
> > > > elided
> > > > > in
> > > > > >> > the
> > > > > >> > > > hex
> > > > > >> > > > > > >> > > > > > representation.)
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Another thing to
> > consider
> > > > is
> > > > > >> that
> > > > > >> > we
> > > > > >> > > > > > should
> > > > > >> > > > > > >> > > specify
> > > > > >> > > > > > >> > > > > > that
> > > > > >> > > > > > >> > > > > > >>> the
> > > > > >> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID is not a
> > > valid
> > > > > >> topic
> > > > > >> > > UUID.
> > > > > >> > > > > > We
> > > > > >> > > > > > >> > can't
> > > > > >> > > > > > >> > > > use
> > > > > >> > > > > > >> > > > > > >>> null
> > > > > >> > > > > > >> > > > > > >>> > for
> > > > > >> > > > > > >> > > > > > >>> > > > this
> > > > > >> > > > > > >> > > > > > >>> > > > > > > because we can't pass a
> > null
> > > > UUID
> > > > > >> over
> > > > > >> > > the
> > > > > >> > > > > RPC
> > > > > >> > > > > > >> > > protocol
> > > > > >> > > > > > >> > > > > > >>> (there
> > > > > >> > > > > > >> > > > > > >>> > is no
> > > > > >> > > > > > >> > > > > > >>> > > > > > > special pattern for null,
> > nor
> > > > do
> > > > > we
> > > > > >> > want
> > > > > >> > > > to
> > > > > >> > > > > > >> waste
> > > > > >> > > > > > >> > > space
> > > > > >> > > > > > >> > > > > > >>> reserving
> > > > > >> > > > > > >> > > > > > >>> > > > such
> > > > > >> > > > > > >> > > > > > >>> > > > > a
> > > > > >> > > > > > >> > > > > > >>> > > > > > > pattern.)
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Maybe I missed it,
> but
> > > did
> > > > > you
> > > > > >> > > > describe
> > > > > >> > > > > > >> > > "migration
> > > > > >> > > > > > >> > > > > > of...
> > > > > >> > > > > > >> > > > > > >>> > existing
> > > > > >> > > > > > >> > > > > > >>> > > > > > > topic[s] without topic
> IDs"
> > > in
> > > > > >> detail
> > > > > >> > in
> > > > > >> > > > any
> > > > > >> > > > > > >> > section?
> > > > > >> > > > > > >> > > > It
> > > > > >> > > > > > >> > > > > > >>> seems
> > > > > >> > > > > > >> > > > > > >>> > like
> > > > > >> > > > > > >> > > > > > >>> > > > > when
> > > > > >> > > > > > >> > > > > > >>> > > > > > > the new controller
> becomes
> > > > > active,
> > > > > >> it
> > > > > >> > > > should
> > > > > >> > > > > > >> just
> > > > > >> > > > > > >> > > > > generate
> > > > > >> > > > > > >> > > > > > >>> random
> > > > > >> > > > > > >> > > > > > >>> > > > > UUIDs for
> > > > > >> > > > > > >> > > > > > >>> > > > > > > these, and write the
> random
> > > > UUIDs
> > > > > >> back
> > > > > >> > > to
> > > > > >> > > > > > >> > ZooKeeper.
> > > > > >> > > > > > >> > > > It
> > > > > >> > > > > > >> > > > > > >>> would be
> > > > > >> > > > > > >> > > > > > >>> > > > good
> > > > > >> > > > > > >> > > > > > >>> > > > > to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > spell that out. We
> should
> > > make
> > > > > it
> > > > > >> > clear
> > > > > >> > > > > that
> > > > > >> > > > > > >> this
> > > > > >> > > > > > >> > > > > happens
> > > > > >> > > > > > >> > > > > > >>> > regardless
> > > > > >> > > > > > >> > > > > > >>> > > > > of
> > > > > >> > > > > > >> > > > > > >>> > > > > > > the inter-broker protocol
> > > > version
> > > > > >> > (it's
> > > > > >> > > a
> > > > > >> > > > > > >> > compatible
> > > > > >> > > > > > >> > > > > > change).
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > "LeaderAndIsrRequests
> > > > > >> including an
> > > > > >> > > > > > >> > > > is_every_partition
> > > > > >> > > > > > >> > > > > > >>> flag"
> > > > > >> > > > > > >> > > > > > >>> > > > seems a
> > > > > >> > > > > > >> > > > > > >>> > > > > > > bit wordy. Can we just
> > call
> > > > > these
> > > > > >> > "full
> > > > > >> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
> > > > > >> > > > > > >> > > > > > >>> > Then
> > > > > >> > > > > > >> > > > > > >>> > > > > the
> > > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field could be named
> > > > "full".
> > > > > >> > Also,
> > > > > >> > > it
> > > > > >> > > > > > would
> > > > > >> > > > > > >> > > > probably
> > > > > >> > > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > > > >>> > better
> > > > > >> > > > > > >> > > > > > >>> > > > > for the
> > > > > >> > > > > > >> > > > > > >>> > > > > > > RPC field to be an enum
> of
> > {
> > > > > >> > > UNSPECIFIED,
> > > > > >> > > > > > >> > > INCREMENTAL,
> > > > > >> > > > > > >> > > > > FULL
> > > > > >> > > > > > >> > > > > > >>> }, so
> > > > > >> > > > > > >> > > > > > >>> > > > that
> > > > > >> > > > > > >> > > > > > >>> > > > > we
> > > > > >> > > > > > >> > > > > > >>> > > > > > > can cleanly handle old
> > > versions
> > > > > (by
> > > > > >> > > > treating
> > > > > >> > > > > > >> them
> > > > > >> > > > > > >> > as
> > > > > >> > > > > > >> > > > > > >>> UNSPECIFIED)
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > In the
> > > LeaderAndIsrRequest
> > > > > >> > section,
> > > > > >> > > > you
> > > > > >> > > > > > >> write
> > > > > >> > > > > > >> > "A
> > > > > >> > > > > > >> > > > > final
> > > > > >> > > > > > >> > > > > > >>> > deletion
> > > > > >> > > > > > >> > > > > > >>> > > > > event
> > > > > >> > > > > > >> > > > > > >>> > > > > > > will be secheduled for X
> ms
> > > > after
> > > > > >> the
> > > > > >> > > > > > >> > > > LeaderAndIsrRequest
> > > > > >> > > > > > >> > > > > > was
> > > > > >> > > > > > >> > > > > > >>> > first
> > > > > >> > > > > > >> > > > > > >>> > > > > > > received..." I guess
> the X
> > > > was a
> > > > > >> > > > > placeholder
> > > > > >> > > > > > >> that
> > > > > >> > > > > > >> > > you
> > > > > >> > > > > > >> > > > > > >>> intended
> > > > > >> > > > > > >> > > > > > >>> > to
> > > > > >> > > > > > >> > > > > > >>> > > > > replace
> > > > > >> > > > > > >> > > > > > >>> > > > > > > before posting? :) In
> any
> > > > case,
> > > > > >> this
> > > > > >> > > > seems
> > > > > >> > > > > > like
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > kind
> > > > > >> > > > > > >> > > > > > of
> > > > > >> > > > > > >> > > > > > >>> > thing
> > > > > >> > > > > > >> > > > > > >>> > > > we'd
> > > > > >> > > > > > >> > > > > > >>> > > > > > > want a configuration for.
> > > > Let's
> > > > > >> > > describe
> > > > > >> > > > > that
> > > > > >> > > > > > >> > > > > > configuration
> > > > > >> > > > > > >> > > > > > >>> key
> > > > > >> > > > > > >> > > > > > >>> > > > > somewhere
> > > > > >> > > > > > >> > > > > > >>> > > > > > > in this KIP, including
> what
> > > its
> > > > > >> > default
> > > > > >> > > > > value
> > > > > >> > > > > > >> is.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > We should probably
> also
> > > > log a
> > > > > >> > bunch
> > > > > >> > > of
> > > > > >> > > > > > >> messages
> > > > > >> > > > > > >> > > at
> > > > > >> > > > > > >> > > > > WARN
> > > > > >> > > > > > >> > > > > > >>> level
> > > > > >> > > > > > >> > > > > > >>> > > > when
> > > > > >> > > > > > >> > > > > > >>> > > > > > > something is scheduled
> for
> > > > > >> deletion,
> > > > > >> > as
> > > > > >> > > > > well.
> > > > > >> > > > > > >> > (Maybe
> > > > > >> > > > > > >> > > > > this
> > > > > >> > > > > > >> > > > > > >>> was
> > > > > >> > > > > > >> > > > > > >>> > > > > assumed, but
> > > > > >> > > > > > >> > > > > > >>> > > > > > > it would be good to
> mention
> > > > it).
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > I feel like there
> are a
> > > few
> > > > > >> > sections
> > > > > >> > > > > that
> > > > > >> > > > > > >> > should
> > > > > >> > > > > > >> > > be
> > > > > >> > > > > > >> > > > > > >>> moved to
> > > > > >> > > > > > >> > > > > > >>> > > > > "rejected
> > > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives." For
> > example,
> > > in
> > > > > the
> > > > > >> > > > > > DeleteTopics
> > > > > >> > > > > > >> > > > section,
> > > > > >> > > > > > >> > > > > > >>> since
> > > > > >> > > > > > >> > > > > > >>> > we're
> > > > > >> > > > > > >> > > > > > >>> > > > > not
> > > > > >> > > > > > >> > > > > > >>> > > > > > > going to do option 1 or
> 2,
> > > > these
> > > > > >> > should
> > > > > >> > > be
> > > > > >> > > > > > moved
> > > > > >> > > > > > >> > into
> > > > > >> > > > > > >> > > > > > >>> "rejected
> > > > > >> > > > > > >> > > > > > >>> > > > > > > alternatives," rather
> than
> > > > > >> appearing
> > > > > >> > > > > inline.
> > > > > >> > > > > > >> > > Another
> > > > > >> > > > > > >> > > > > case
> > > > > >> > > > > > >> > > > > > >>> is
> > > > > >> > > > > > >> > > > > > >>> > the
> > > > > >> > > > > > >> > > > > > >>> > > > > "Should
> > > > > >> > > > > > >> > > > > > >>> > > > > > > we remove topic name from
> > the
> > > > > >> protocol
> > > > > >> > > > where
> > > > > >> > > > > > >> > > possible"
> > > > > >> > > > > > >> > > > > > >>> section.
> > > > > >> > > > > > >> > > > > > >>> > This
> > > > > >> > > > > > >> > > > > > >>> > > > > is
> > > > > >> > > > > > >> > > > > > >>> > > > > > > clearly discussing a
> design
> > > > > >> > alternative
> > > > > >> > > > that
> > > > > >> > > > > > >> we're
> > > > > >> > > > > > >> > > not
> > > > > >> > > > > > >> > > > > > >>> proposing
> > > > > >> > > > > > >> > > > > > >>> > to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > implement: removing the
> > topic
> > > > > name
> > > > > >> > from
> > > > > >> > > > > those
> > > > > >> > > > > > >> > > > protocols.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Is it really
> necessary
> > to
> > > > > have
> > > > > >> a
> > > > > >> > new
> > > > > >> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
> > > > > >> > > > > > >> > > > > > >>> > > > > path
> > > > > >> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper? It seems
> > like
> > > we
> > > > > >> don't
> > > > > >> > > > really
> > > > > >> > > > > > >> need
> > > > > >> > > > > > >> > > this.
> > > > > >> > > > > > >> > > > > > >>> Whenever
> > > > > >> > > > > > >> > > > > > >>> > > > > there is
> > > > > >> > > > > > >> > > > > > >>> > > > > > > a new controller, we'll
> > send
> > > > out
> > > > > >> full
> > > > > >> > > > > > >> > > > > LeaderAndIsrRequests
> > > > > >> > > > > > >> > > > > > >>> which
> > > > > >> > > > > > >> > > > > > >>> > will
> > > > > >> > > > > > >> > > > > > >>> > > > > > > trigger the stale topics
> to
> > > be
> > > > > >> cleaned
> > > > > >> > > up.
> > > > > >> > > > > > The
> > > > > >> > > > > > >> > > active
> > > > > >> > > > > > >> > > > > > >>> > controller
> > > > > >> > > > > > >> > > > > > >>> > > > will
> > > > > >> > > > > > >> > > > > > >>> > > > > > > also send the full
> > > > > >> LeaderAndIsrRequest
> > > > > >> > > to
> > > > > >> > > > > > >> brokers
> > > > > >> > > > > > >> > > that
> > > > > >> > > > > > >> > > > > are
> > > > > >> > > > > > >> > > > > > >>> just
> > > > > >> > > > > > >> > > > > > >>> > > > > starting
> > > > > >> > > > > > >> > > > > > >>> > > > > > > up. So we don't really
> > > need
> > > > > this
> > > > > >> > kind
> > > > > >> > > > of
> > > > > >> > > > > > >> > two-phase
> > > > > >> > > > > > >> > > > > > commit
> > > > > >> > > > > > >> > > > > > >>> > (send
> > > > > >> > > > > > >> > > > > > >>> > > > out
> > > > > >> > > > > > >> > > > > > >>> > > > > > > StopReplicasRequest, get
> > ACKs
> > > > > from
> > > > > >> all
> > > > > >> > > > > nodes,
> > > > > >> > > > > > >> > commit
> > > > > >> > > > > > >> > > by
> > > > > >> > > > > > >> > > > > > >>> removing
> > > > > >> > > > > > >> > > > > > >>> > > > > > > /admin/delete_topics
> node)
> > > any
> > > > > >> more.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > You mention that
> > > > FetchRequest
> > > > > >> will
> > > > > >> > > now
> > > > > >> > > > > > >> include
> > > > > >> > > > > > >> > > UUID
> > > > > >> > > > > > >> > > > > to
> > > > > >> > > > > > >> > > > > > >>> avoid
> > > > > >> > > > > > >> > > > > > >>> > > > issues
> > > > > >> > > > > > >> > > > > > >>> > > > > > > where requests are made
> to
> > > > stale
> > > > > >> > > > partitions.
> > > > > >> > > > > > >> > > However,
> > > > > >> > > > > > >> > > > > > >>> adding a
> > > > > >> > > > > > >> > > > > > >>> > UUID
> > > > > >> > > > > > >> > > > > > >>> > > > to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataRequest is listed
> > as
> > > > > future
> > > > > >> > > work,
> > > > > >> > > > > out
> > > > > >> > > > > > of
> > > > > >> > > > > > >> > > scope
> > > > > >> > > > > > >> > > > > for
> > > > > >> > > > > > >> > > > > > >>> this
> > > > > >> > > > > > >> > > > > > >>> > KIP.
> > > > > >> > > > > > >> > > > > > >>> > > > > How
> > > > > >> > > > > > >> > > > > > >>> > > > > > > will the client learn
> what
> > > the
> > > > > >> topic
> > > > > >> > > UUID
> > > > > >> > > > > is,
> > > > > >> > > > > > if
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > > > metadata
> > > > > >> > > > > > >> > > > > > >>> > > > response
> > > > > >> > > > > > >> > > > > > >>> > > > > > > doesn't include that
> > > > information?
> > > > > >> It
> > > > > >> > > > seems
> > > > > >> > > > > > like
> > > > > >> > > > > > >> > > adding
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > >>> UUID
> > > > > >> > > > > > >> > > > > > >>> > to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > MetadataResponse would be
> > an
> > > > > >> > improvement
> > > > > >> > > > > here
> > > > > >> > > > > > >> that
> > > > > >> > > > > > >> > > > might
> > > > > >> > > > > > >> > > > > > not
> > > > > >> > > > > > >> > > > > > >>> be
> > > > > >> > > > > > >> > > > > > >>> > too
> > > > > >> > > > > > >> > > > > > >>> > > > > hard to
> > > > > >> > > > > > >> > > > > > >>> > > > > > > make.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > best,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > Colin
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep 9, 2019,
> at
> > > > > 17:48,
> > > > > >> > > Ryanne
> > > > > >> > > > > > Dolan
> > > > > >> > > > > > >> > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Lucas, this would
> be
> > > > great.
> > > > > >> I've
> > > > > >> > > run
> > > > > >> > > > > > into
> > > > > >> > > > > > >> > > issues
> > > > > >> > > > > > >> > > > > with
> > > > > >> > > > > > >> > > > > > >>> > topics
> > > > > >> > > > > > >> > > > > > >>> > > > > being
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > resurrected
> > > accidentally,
> > > > > >> since
> > > > > >> > a
> > > > > >> > > > > client
> > > > > >> > > > > > >> > cannot
> > > > > >> > > > > > >> > > > > > easily
> > > > > >> > > > > > >> > > > > > >>> > > > > distinguish
> > > > > >> > > > > > >> > > > > > >>> > > > > > > between
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > a deleted topic
> and a
> > > new
> > > > > >> topic
> > > > > >> > > with
> > > > > >> > > > > the
> > > > > >> > > > > > >> same
> > > > > >> > > > > > >> > > > name.
> > > > > >> > > > > > >> > > > > > I'd
> > > > > >> > > > > > >> > > > > > >>> > need
> > > > > >> > > > > > >> > > > > > >>> > > > the
> > > > > >> > > > > > >> > > > > > >>> > > > > ID
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > accessible from the
> > > > client
> > > > > to
> > > > > >> > > solve
> > > > > >> > > > > that
> > > > > >> > > > > > >> > issue,
> > > > > >> > > > > > >> > > > but
> > > > > >> > > > > > >> > > > > > >>> this
> > > > > >> > > > > > >> > > > > > >>> > is a
> > > > > >> > > > > > >> > > > > > >>> > > > > good
> > > > > >> > > > > > >> > > > > > >>> > > > > > > first
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > step.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > On Wed, Sep 4, 2019
> > at
> > > > 1:41
> > > > > >> PM
> > > > > >> > > Lucas
> > > > > >> > > > > > >> > > Bradstreet <
> > > > > >> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
> > > > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > I would like to
> > kick
> > > > off
> > > > > >> > > > discussion
> > > > > >> > > > > of
> > > > > >> > > > > > >> > > KIP-516,
> > > > > >> > > > > > >> > > > > an
> > > > > >> > > > > > >> > > > > > >>> > > > > implementation
> > > > > >> > > > > > >> > > > > > >>> > > > > > > of topic
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for Kafka.
> > Topic
> > > > IDs
> > > > > >> aim
> > > > > >> > to
> > > > > >> > > > > solve
> > > > > >> > > > > > >> topic
> > > > > >> > > > > > >> > > > > > >>> uniqueness
> > > > > >> > > > > > >> > > > > > >>> > > > > problems in
> > > > > >> > > > > > >> > > > > > >>> > > > > > > Kafka,
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > where referring
> to
> > a
> > > > > topic
> > > > > >> by
> > > > > >> > > name
> > > > > >> > > > > > >> alone is
> > > > > >> > > > > > >> > > > > > >>> insufficient.
> > > > > >> > > > > > >> > > > > > >>> > > > Such
> > > > > >> > > > > > >> > > > > > >>> > > > > > > cases
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > include when a
> > topic
> > > > has
> > > > > >> been
> > > > > >> > > > > deleted
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > > recreated
> > > > > >> > > > > > >> > > > > > >>> with
> > > > > >> > > > > > >> > > > > > >>> > the
> > > > > >> > > > > > >> > > > > > >>> > > > > same
> > > > > >> > > > > > >> > > > > > >>> > > > > > > name.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Unique
> identifiers
> > > will
> > > > > >> help
> > > > > >> > > > > simplify
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > improve
> > > > > >> > > > > > >> > > > > > >>> Kafka's
> > > > > >> > > > > > >> > > > > > >>> > > > topic
> > > > > >> > > > > > >> > > > > > >>> > > > > > > deletion
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > process, as well
> as
> > > > > prevent
> > > > > >> > > cases
> > > > > >> > > > > > where
> > > > > >> > > > > > >> > > brokers
> > > > > >> > > > > > >> > > > > may
> > > > > >> > > > > > >> > > > > > >>> > > > incorrectly
> > > > > >> > > > > > >> > > > > > >>> > > > > > > interact
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > with stale
> versions
> > > of
> > > > > >> topics.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > >
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Looking forward
> to
> > > your
> > > > > >> > > thoughts.
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
> > > > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>> >
> > > > > >> > > > > > >> > > > > > >>>
> > > > > >> > > > > > >> > > > > > >>
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
