Also, can we provide more details on how the Partition Metadata file will
be used?
Ismael
On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <ism...@juma.me.uk> wrote:
> Hi Justine,
>
> I think we need to update the "Rejected Alternatives" section to take into
> account that the proposal now removes the topic name from the fetch
> request. Also, if we are removing it from the Fetch request, does it make
> sense not to remove it from similar requests like ListOffsetRequest?
>
> Ismael
>
> On Thu, Sep 24, 2020 at 2:46 AM David Jacot <dja...@confluent.io> wrote:
>
>> Hi Justine,
>>
>> Thanks for the KIP. I finally had time to read it :). It is a great
>> improvement.
>>
>> I have a few comments/questions:
>>
>> 1. It seems that the schema of the StopReplicaRequest is slightly
>> outdated.
>> We
>> did some changes as part of KIP-570. V3 is already organized by topics.
>>
>> 2. I just want to make sure that I understand the reconciliation
>> logic correctly. When
>> an "INCREMENTAL" LeaderAndIsr Request is received, the broker will also
>> reconcile
>> when the local uuid does not match the uuid in the request, right? In this
>> case, the
>> local log is staged for deletion.
>>
>> 3. In the documentation of the `delete.stale.topic.delay.ms` config, it
>> says "When a
>> FULL LeaderAndIsrRequest is received..." but I suppose it applies to both
>> types.
>>
>> Best,
>> David
>>
>> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <jols...@confluent.io>
>> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the comments. I apologize for some of the typos and
>> confusion.
>> > I’ve updated the KIP to fix some of the issues you mentioned.
>> >
>> > 20.2 I’ve changed the type to String.
>> > 20.1/3 I’ve updated the TopicZNode to fix formatting and reflect the
>> latest
>> > version before this change.
>> >
>> > 21. You are correct and I’ve removed this line. I’ve also added a line
>> > mentioning an IBP bump is necessary for migration
>> >
>> > 22. I think the wording was unclear but your summary is what was
>> intended
>> > by this line. I’ve updated to make this point more clear. “Remove
>> deleted
>> > topics from replicas by sending StopReplicaRequest V3 before the IBP
>> bump
>> > using the old logic, and using V4 and the new logic with topic IDs after
>> > the IBP bump.”
>> >
>> > 23. I’ve removed the unspecified type from the KIP and mention that IBP
>> > will be used to handle this request. “IBP will be used to determine
>> whether
>> > this new form of the request will be used. For older requests, we will
>> > ignore this field and default to previous behavior.”
>> >
>> > 24. I’ve fixed this typo.
>> >
>> > 25. I've added a topics at a higher level for LeaderAndIsrResponse v5,
>> > StopReplicaResponse v4. I've also changed StopReplicaRequest v4 to have
>> > topics at a higher level.
>> >
>> > 26. I’ve updated forgotten_topics_data--added the topic ID and removed
>> the
>> > topic name
>> >
>> > 27. I’ve decided on plain text, and I’ve added an example.
>> >
>> > 28. I’ve added this idea to future work.
>> >
>> > Thanks again for taking a look,
>> >
>> > Justine
>> >
>> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <j...@confluent.io> wrote:
>> >
>> > > Hi, Justine,
>> > >
>> > > Thanks for the response. Made another pass. A few more comments below.
>> > >
>> > > 20. znode schema:
>> > > 20.1 It seems that {"name": "version", "type": "int", "id": "UUID",
>> > "doc":
>> > > "version id"} should be {"name": "version", "type": "int"}, {"name":
>> > "id",
>> > > "type": "UUID", "doc": "version id"}.
>> > > 20.2 The znode format is JSON which doesn't have UUID type. So the
>> type
>> > > probably should be string?
>> > > 20.3 Also, the existing format used seems outdated. It should have the
>> > > following format.
>> > > Json.encodeAsBytes(Map(
>> > > "version" -> 2,
>> > > "partitions" -> replicaAssignmentJson.asJava,
>> > > "adding_replicas" -> addingReplicasAssignmentJson.asJava,
>> > > "removing_replicas" -> removingReplicasAssignmentJson.asJava
>> > > ).asJava)
>> > > }
>> > >
>> > > 21. Migration: The KIP says "The migration process can take place
>> without
>> > > an inter-broker protocol bump, as the format stored in
>> > > /brokers/topics/[topic] will be compatible with older broker
>> versions."
>> > > However, since we are bumping up the version of inter-broker
>> requests, it
>> > > seems that we need to use IBP for migration.
>> > >
>> > > 22. The KIP says "Remove deleted topics from replicas by sending
>> > > StopReplicaRequest V3 for any topics which do not contain a topic ID,
>> and
>> > > V4 for any topics which do contain a topic ID.". However, if we use
>> IBP,
>> > it
>> > > seems that the controller will either send StopReplicaRequest V3
>> > > or StopReplicaRequest V4, but never mixed V3 and V4 for different
>> topics.
>> > > Basically, before the IBP bump, V3 will be used. After the IBP bump,
>> > > topicId will be created and V4 will be used.
>> > >
>> > > 23. Given that we depend on IBP, do we still need "0 UNSPECIFIED"
>> > > in LeaderAndIsr?
>> > >
>> > > 24. LeaderAndIsrResponse v5 : It still has the topic field.
>> > >
>> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse v4: Could we use this
>> > > opportunity to organize the response in 2 levels, first by topic,
>> then by
>> > > partition, as most other requests/responses?
>> > >
>> > > 26. FetchRequest v13 : Should forgotten_topics_data use topicId too?
>> > >
>> > > 27. "This file can either be plain text (key/value pairs) or JSON."
>> Have
>> > we
>> > > decided which one to use? Also, it would be helpful to provide an
>> > example.
>> > >
>> > > 28. Future improvement: Another future improvement opportunity is to
>> use
>> > > topicId in GroupMetadataManager.offsetCommitKey in the offset_commit
>> > topic.
>> > > This may save some space.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <jols...@confluent.io>
>> > > wrote:
>> > >
>> > > > Hi Tom,
>> > > >
>> > > > Thanks for the comment. I think this is a really good idea and it
>> has
>> > > been
>> > > > added to the KIP under the newly added tooling section.
>> > > >
>> > > > Thanks again,
>> > > > Justine
>> > > >
>> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <tbent...@redhat.com>
>> > wrote:
>> > > >
>> > > > > Hi Justine,
>> > > > >
>> > > > > I know you started the vote thread, but on re-reading the KIP I
>> > noticed
>> > > > > that although the topic id is included in the MetadataResponse
>> it's
>> > not
>> > > > > surfaced in the output from `kafka-topics.sh --describe`. Maybe
>> that
>> > > was
>> > > > > intentional because ids are intentionally not really something the
>> > user
>> > > > > should care deeply about, but it would also make life harder for
>> > anyone
>> > > > > debugging Kafka and this would likely get worse the more topic ids
>> > got
>> > > > > rolled out across the protocols, clients etc. It seems likely that
>> > > > > `kafka-topics.sh` will eventually need the ability to show the id
>> of
>> > a
>> > > > > topic and perhaps find a topic name given an id. Is there any
>> reason
>> > > not
>> > > > to
>> > > > > implement that in this KIP?
>> > > > >
>> > > > > Many thanks,
>> > > > >
>> > > > > Tom
>> > > > >
>> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan <
>> jols...@confluent.io
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > After thinking about it, I've decided to remove the topic name
>> from
>> > > the
>> > > > > > Fetch Request and Response after all. Since there are so many of
>> > > these
>> > > > > > requests per second, it is worth removing the extra information.
>> > I've
>> > > > > > updated the KIP to reflect this change.
>> > > > > >
>> > > > > > Please let me know if there is anything else we should discuss
>> > before
>> > > > > > voting.
>> > > > > >
>> > > > > > Thank you,
>> > > > > > Justine
>> > > > > >
>> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine Olshan <
>> > jols...@confluent.io
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Jun,
>> > > > > > >
>> > > > > > > I see what you are saying. For now we can remove the extra
>> > > > information.
>> > > > > > > I'll leave the option to add more fields to the file in the
>> > future.
>> > > > The
>> > > > > > KIP
>> > > > > > > has been updated to reflect this change.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Justine
>> > > > > > >
>> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <j...@confluent.io>
>> > wrote:
>> > > > > > >
>> > > > > > >> Hi, Justine,
>> > > > > > >>
>> > > > > > >> Thanks for the reply.
>> > > > > > >>
>> > > > > > >> 13. If the log directory is the source of truth, it means
>> that
>> > the
>> > > > > > >> redundant info in the metadata file will be ignored. Then the
>> > > > question
>> > > > > > is
>> > > > > > >> why do we need to put the redundant info in the metadata file
>> > now?
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >>
>> > > > > > >> Jun
>> > > > > > >>
>> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine Olshan <
>> > > > jols...@confluent.io>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Hi Jun,
>> > > > > > >> > Thanks for the quick response!
>> > > > > > >> >
>> > > > > > >> > 12. I've decided to bump up the versions on the requests
>> and
>> > > > updated
>> > > > > > the
>> > > > > > >> > KIP. I think it's good we thoroughly discussed the options
>> > here,
>> > > > so
>> > > > > we
>> > > > > > >> know
>> > > > > > >> > we made a good choice. :)
>> > > > > > >> >
>> > > > > > >> > 13. This is an interesting situation. I think if this does
>> > occur
>> > > > we
>> > > > > > >> should
>> > > > > > >> > give a warning. I agree that it's hard to know the source
>> of
>> > > truth
>> > > > > for
>> > > > > > >> sure
>> > > > > > >> > since the directory or the file could be manually
>> modified. I
>> > > > guess
>> > > > > > the
>> > > > > > >> > directory could be used as the source of truth. To be
>> honest,
>> > > I'm
>> > > > > not
>> > > > > > >> > really sure what happens in kafka when the log directory is
>> > > > renamed
>> > > > > > >> > manually in such a way. I'm also wondering if the
>> situation is
>> > > > > > >> recoverable
>> > > > > > >> > in this scenario.
>> > > > > > >> >
>> > > > > > >> > Thanks,
>> > > > > > >> > Justine
>> > > > > > >> >
>> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao <j...@confluent.io>
>> > > wrote:
>> > > > > > >> >
>> > > > > > >> > > Hi, Justine,
>> > > > > > >> > >
>> > > > > > >> > > Thanks for the reply.
>> > > > > > >> > >
>> > > > > > >> > > 12. I don't have a strong preference either. However, if
>> we
>> > > need
>> > > > > IBP
>> > > > > > >> > > anyway, maybe it's easier to just bump up the version for
>> > all
>> > > > > inter
>> > > > > > >> > broker
>> > > > > > >> > > requests and add the topic id field as a regular field. A
>> > > > regular
>> > > > > > >> field
>> > > > > > >> > is
>> > > > > > >> > > a bit more concise in wire transfer than a flexible
>> field.
>> > > > > > >> > >
>> > > > > > >> > > 13. The confusion that I was referring to is between the
>> > topic
>> > > > > name
>> > > > > > >> and
>> > > > > > >> > > partition number between the log dir and the metadata
>> file.
>> > > For
>> > > > > > >> example,
>> > > > > > >> > if
>> > > > > > >> > > the log dir is topicA-1 and the metadata file in it has
>> > topicB
>> > > > and
>> > > > > > >> > > partition 0 (say due to a bug or manual modification),
>> which
>> > > one
>> > > > > do
>> > > > > > we
>> > > > > > >> > use
>> > > > > > >> > > as the source of truth?
>> > > > > > >> > >
>> > > > > > >> > > Jun
>> > > > > > >> > >
>> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM Justine Olshan <
>> > > > > > jols...@confluent.io>
>> > > > > > >> > > wrote:
>> > > > > > >> > >
>> > > > > > >> > > > Hi Jun,
>> > > > > > >> > > > Thanks for the comments.
>> > > > > > >> > > >
>> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest because I removed
>> the
>> > > > topic
>> > > > > > >> name
>> > > > > > >> > > field
>> > > > > > >> > > > in the response. It may be possible to avoid bumping
>> the
>> > > > version
>> > > > > > >> > without
>> > > > > > >> > > > that change, but I may be missing something.
>> > > > > > >> > > > I believe StopReplica is actually on version 3 now, but
>> > > > because
>> > > > > > >> > version 2
>> > > > > > >> > > > is flexible, I kept that listed as version 2 on the KIP
>> > > page.
>> > > > > > >> However,
>> > > > > > >> > > you
>> > > > > > >> > > > may be right in that we may need to bump the version on
>> > > > > > StopReplica
>> > > > > > >> to
>> > > > > > >> > > deal
>> > > > > > >> > > > with deletion differently as mentioned above. I don't
>> know
>> > > if
>> > > > I
>> > > > > > >> have a
>> > > > > > >> > > big
>> > > > > > >> > > > preference over used tagged fields or not.
>> > > > > > >> > > >
>> > > > > > >> > > > 13. I was thinking that in the case where the file and
>> the
>> > > > > request
>> > > > > > >> > topic
>> > > > > > >> > > > ids don't match, it means that the broker's topic/the
>> one
>> > in
>> > > > the
>> > > > > > >> file
>> > > > > > >> > has
>> > > > > > >> > > > been deleted. In that case, we would need to delete the
>> > old
>> > > > > topic
>> > > > > > >> and
>> > > > > > >> > > start
>> > > > > > >> > > > receiving the new version. If the topic name were to
>> > change,
>> > > > but
>> > > > > > the
>> > > > > > >> > ids
>> > > > > > >> > > > still matched, the file would also need to update. Am I
>> > > > missing
>> > > > > a
>> > > > > > >> case
>> > > > > > >> > > > where the file would be correct and not the request?
>> > > > > > >> > > >
>> > > > > > >> > > > Thanks,
>> > > > > > >> > > > Justine
>> > > > > > >> > > >
>> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun Rao <
>> j...@confluent.io
>> > >
>> > > > > wrote:
>> > > > > > >> > > >
>> > > > > > >> > > > > Hi, Justine,
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks for the reply. A couple of more comments
>> below.
>> > > > > > >> > > > >
>> > > > > > >> > > > > 12. ListOffset and OffsetForLeader currently don't
>> > support
>> > > > > > >> flexible
>> > > > > > >> > > > fields.
>> > > > > > >> > > > > So, we have to bump up the version number and use
>> IBP at
>> > > > least
>> > > > > > for
>> > > > > > >> > > these
>> > > > > > >> > > > > two requests. Note that it seems 2.7.0 will require
>> IBP
>> > > > anyway
>> > > > > > >> > because
>> > > > > > >> > > of
>> > > > > > >> > > > > changes in KAFKA-10435. Also, it seems that the
>> version
>> > > for
>> > > > > > >> > > > > LeaderAndIsrRequest and StopReplica are bumped even
>> > though
>> > > > we
>> > > > > > only
>> > > > > > >> > > added
>> > > > > > >> > > > a
>> > > > > > >> > > > > tagged field. But since IBP is needed anyway, we may
>> > want
>> > > to
>> > > > > > >> revisit
>> > > > > > >> > > the
>> > > > > > >> > > > > overall tagged field choice.
>> > > > > > >> > > > >
>> > > > > > >> > > > > 13. The only downside is the potential confusion on
>> > which
>> > > > one
>> > > > > is
>> > > > > > >> the
>> > > > > > >> > > > source
>> > > > > > >> > > > > of truth if they don't match. Another option is to
>> > include
>> > > > > those
>> > > > > > >> > fields
>> > > > > > >> > > > in
>> > > > > > >> > > > > the metadata file when we actually change the
>> directory
>> > > > > > structure.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks,
>> > > > > > >> > > > >
>> > > > > > >> > > > > Jun
>> > > > > > >> > > > >
>> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM Justine Olshan <
>> > > > > > >> jols...@confluent.io
>> > > > > > >> > >
>> > > > > > >> > > > > wrote:
>> > > > > > >> > > > >
>> > > > > > >> > > > > > Hello all,
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > I've thought some more about removing the topic
>> name
>> > > field
>> > > > > > from
>> > > > > > >> > some
>> > > > > > >> > > of
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > requests. On closer inspection of the
>> > > requests/responses,
>> > > > it
>> > > > > > >> seems
>> > > > > > >> > > that
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > internal changes would be much larger than I
>> expected.
>> > > > Some
>> > > > > > >> > protocols
>> > > > > > >> > > > > > involve clients, so they would require changes too.
>> > I'm
>> > > > > > thinking
>> > > > > > >> > that
>> > > > > > >> > > > for
>> > > > > > >> > > > > > now, removing the topic name from these requests
>> and
>> > > > > responses
>> > > > > > >> are
>> > > > > > >> > > out
>> > > > > > >> > > > of
>> > > > > > >> > > > > > scope.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > I have decided to just keep the change
>> > > > LeaderAndIsrResponse
>> > > > > to
>> > > > > > >> > remove
>> > > > > > >> > > > the
>> > > > > > >> > > > > > topic name, and have updated the KIP to reflect
>> this
>> > > > > change. I
>> > > > > > >> have
>> > > > > > >> > > > also
>> > > > > > >> > > > > > mentioned the other requests and responses in
>> future
>> > > work.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > I'm hoping to start the voting process soon, so
>> let me
>> > > > know
>> > > > > if
>> > > > > > >> > there
>> > > > > > >> > > is
>> > > > > > >> > > > > > anything else we should discuss.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > Thank you,
>> > > > > > >> > > > > > Justine
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM Justine Olshan <
>> > > > > > >> > jols...@confluent.io
>> > > > > > >> > > >
>> > > > > > >> > > > > > wrote:
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > > Hello again,
>> > > > > > >> > > > > > > To follow up on some of the other comments:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > 10/11) We can remove the topic name from these
>> > > > > > >> > requests/responses,
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > that means we will just have to make a few
>> internal
>> > > > > changes
>> > > > > > to
>> > > > > > >> > make
>> > > > > > >> > > > > > > partitions accessible by topic id and partition.
>> I
>> > can
>> > > > > > update
>> > > > > > >> the
>> > > > > > >> > > KIP
>> > > > > > >> > > > > to
>> > > > > > >> > > > > > > remove them unless anyone thinks they should
>> stay.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > 12) Addressed in the previous email. I've updated
>> > the
>> > > > KIP
>> > > > > to
>> > > > > > >> > > include
>> > > > > > >> > > > > > > tagged fields for the requests and responses.
>> (More
>> > on
>> > > > > that
>> > > > > > >> > below)
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > 13) I think part of the idea for including this
>> > > > > information
>> > > > > > >> is to
>> > > > > > >> > > > > prepare
>> > > > > > >> > > > > > > for future changes. Perhaps the directory
>> structure
>> > > > might
>> > > > > > >> change
>> > > > > > >> > > from
>> > > > > > >> > > > > > > topicName_partitionNumber to something like
>> > > > > > >> > > topicID_partitionNumber.
>> > > > > > >> > > > > Then
>> > > > > > >> > > > > > > it would be useful to have the topic name in the
>> > file
>> > > > > since
>> > > > > > it
>> > > > > > >> > > would
>> > > > > > >> > > > > not
>> > > > > > >> > > > > > be
>> > > > > > >> > > > > > > in the directory structure. Supporting topic
>> renames
>> > > > might
>> > > > > > be
>> > > > > > >> > > easier
>> > > > > > >> > > > if
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > other fields are included. Would there be any
>> > > downsides
>> > > > to
>> > > > > > >> > > including
>> > > > > > >> > > > > this
>> > > > > > >> > > > > > > information?
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > 14) Yes, we would need to copy the partition
>> > metadata
>> > > > > file
>> > > > > > in
>> > > > > > >> > this
>> > > > > > >> > > > > > > process. I've updated the KIP to include this.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > 15) I believe Lucas meant v1 and v2 here. He was
>> > > > referring
>> > > > > > to
>> > > > > > >> how
>> > > > > > >> > > the
>> > > > > > >> > > > > > > requests would fall under different IBP and meant
>> > that
>> > > > > older
>> > > > > > >> > > brokers
>> > > > > > >> > > > > > would
>> > > > > > >> > > > > > > have to use the older version of the request and
>> the
>> > > > > > existing
>> > > > > > >> > topic
>> > > > > > >> > > > > > > deletion process. At first, it seemed like tagged
>> > > fields
>> > > > > > would
>> > > > > > >> > > > resolve
>> > > > > > >> > > > > > > the IBP issue. However, we may need IBP for this
>> > > request
>> > > > > > after
>> > > > > > >> > all
>> > > > > > >> > > > > since
>> > > > > > >> > > > > > > the controller handles the topic deletion
>> > differently
>> > > > > > >> depending
>> > > > > > >> > on
>> > > > > > >> > > > the
>> > > > > > >> > > > > > IBP
>> > > > > > >> > > > > > > version. In an older version, we can't just send
>> a
>> > > > > > StopReplica
>> > > > > > >> > > delete
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > > topic immediately like we'd want to for this KIP.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > This makes me wonder if we want tagged fields on
>> all
>> > > the
>> > > > > > >> requests
>> > > > > > >> > > > after
>> > > > > > >> > > > > > > all. Let me know your thoughts!
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > Justine
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03 PM Justine Olshan <
>> > > > > > >> > > jols...@confluent.io
>> > > > > > >> > > > >
>> > > > > > >> > > > > > > wrote:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >> Hi all,
>> > > > > > >> > > > > > >> Jun brought up a good point in his last email
>> about
>> > > > > tagged
>> > > > > > >> > fields,
>> > > > > > >> > > > and
>> > > > > > >> > > > > > >> I've updated the KIP to reflect that the
>> changes to
>> > > > > > requests
>> > > > > > >> and
>> > > > > > >> > > > > > responses
>> > > > > > >> > > > > > >> will be in the form of tagged fields to avoid
>> > > changing
>> > > > > IBP.
>> > > > > > >> > > > > > >>
>> > > > > > >> > > > > > >> Jun: I plan on sending a followup email to
>> address
>> > > some
>> > > > > of
>> > > > > > >> the
>> > > > > > >> > > other
>> > > > > > >> > > > > > >> points.
>> > > > > > >> > > > > > >>
>> > > > > > >> > > > > > >> Thanks,
>> > > > > > >> > > > > > >> Justine
>> > > > > > >> > > > > > >>
>> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25 PM Jun Rao <
>> > > > > j...@confluent.io>
>> > > > > > >> > wrote:
>> > > > > > >> > > > > > >>
>> > > > > > >> > > > > > >>> Hi, Justine,
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> Thanks for the updated KIP. A few comments
>> below.
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response: Do we need the topic
>> > > name?
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 11. For the changed request/response, other
>> than
>> > > > > > >> LeaderAndIsr,
>> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do we need to include
>> > the
>> > > > > topic
>> > > > > > >> name?
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 12. It seems that upgrades don't require IBP.
>> Does
>> > > > that
>> > > > > > mean
>> > > > > > >> > the
>> > > > > > >> > > > new
>> > > > > > >> > > > > > >>> fields
>> > > > > > >> > > > > > >>> in all the request/response are added as tagged
>> > > fields
>> > > > > > >> without
>> > > > > > >> > > > > bumping
>> > > > > > >> > > > > > up
>> > > > > > >> > > > > > >>> the request version? It would be useful to make
>> > that
>> > > > > > clear.
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 13. Partition Metadata file: Do we need to
>> include
>> > > the
>> > > > > > topic
>> > > > > > >> > name
>> > > > > > >> > > > and
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > >>> partition id since they are implied in the
>> > directory
>> > > > > name?
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 14. In the JBOD mode, we support moving a
>> > > partition's
>> > > > > data
>> > > > > > >> from
>> > > > > > >> > > one
>> > > > > > >> > > > > > disk
>> > > > > > >> > > > > > >>> to
>> > > > > > >> > > > > > >>> another. Will the new partition metadata file
>> be
>> > > > copied
>> > > > > > >> during
>> > > > > > >> > > that
>> > > > > > >> > > > > > >>> process?
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> 15. The KIP says "Remove deleted topics from
>> > > replicas
>> > > > by
>> > > > > > >> > sending
>> > > > > > >> > > > > > >>> StopReplicaRequest V2 for any topics which do
>> not
>> > > > > contain
>> > > > > > a
>> > > > > > >> > topic
>> > > > > > >> > > > ID,
>> > > > > > >> > > > > > and
>> > > > > > >> > > > > > >>> V3 for any topics which do contain a topic
>> ID.".
>> > > > > However,
>> > > > > > it
>> > > > > > >> > > seems
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > >>> updated controller will create all missing
>> topic
>> > IDs
>> > > > > first
>> > > > > > >> > before
>> > > > > > >> > > > > doing
>> > > > > > >> > > > > > >>> other actions. So, is StopReplicaRequest V2
>> > needed?
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> Jun
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at 10:31 AM John Roesler <
>> > > > > > >> > > vvcep...@apache.org
>> > > > > > >> > > > >
>> > > > > > >> > > > > > >>> wrote:
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>> > Thanks, Justine!
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>> > Your response seems compelling to me.
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>> > -John
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at 10:17 -0700, Justine
>> > Olshan
>> > > > > wrote:
>> > > > > > >> > > > > > >>> > > Hello all,
>> > > > > > >> > > > > > >>> > > Thanks for continuing the discussion! I
>> have a
>> > > few
>> > > > > > >> > responses
>> > > > > > >> > > to
>> > > > > > >> > > > > > your
>> > > > > > >> > > > > > >>> > points.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > Tom: You are correct in that this KIP has
>> not
>> > > > > > mentioned
>> > > > > > >> the
>> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I think that this
>> would
>> > be
>> > > > out
>> > > > > of
>> > > > > > >> > scope
>> > > > > > >> > > > for
>> > > > > > >> > > > > > >>> now, but
>> > > > > > >> > > > > > >>> > > may be something worth adding in the
>> future.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > John: We did consider sequence ids, but
>> there
>> > > are
>> > > > a
>> > > > > > few
>> > > > > > >> > > reasons
>> > > > > > >> > > > > to
>> > > > > > >> > > > > > >>> favor
>> > > > > > >> > > > > > >>> > > UUIDs. There are several cases where topics
>> > from
>> > > > > > >> different
>> > > > > > >> > > > > clusters
>> > > > > > >> > > > > > >>> may
>> > > > > > >> > > > > > >>> > > interact now and in the future. For
>> example,
>> > > > Mirror
>> > > > > > >> Maker 2
>> > > > > > >> > > may
>> > > > > > >> > > > > > >>> benefit
>> > > > > > >> > > > > > >>> > > from being able to detect when a cluster
>> being
>> > > > > > mirrored
>> > > > > > >> is
>> > > > > > >> > > > > deleted
>> > > > > > >> > > > > > >>> and
>> > > > > > >> > > > > > >>> > > recreated and globally unique identifiers
>> > would
>> > > > make
>> > > > > > >> > > resolving
>> > > > > > >> > > > > > issues
>> > > > > > >> > > > > > >>> > > easier than sequence IDs which may collide
>> > > between
>> > > > > > >> > clusters.
>> > > > > > >> > > > > > KIP-405
>> > > > > > >> > > > > > >>> > > (tiered storage) will also benefit from
>> > globally
>> > > > > > unique
>> > > > > > >> IDs
>> > > > > > >> > > as
>> > > > > > >> > > > > > shared
>> > > > > > >> > > > > > >>> > > buckets may be used between clusters.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > Globally unique IDs would also make
>> > > functionality
>> > > > > like
>> > > > > > >> > moving
>> > > > > > >> > > > > > topics
>> > > > > > >> > > > > > >>> > > between disparate clusters easier in the
>> > future,
>> > > > > > >> simplify
>> > > > > > >> > any
>> > > > > > >> > > > > > future
>> > > > > > >> > > > > > >>> > > implementations of backups and restores,
>> and
>> > > more.
>> > > > > In
>> > > > > > >> > > general,
>> > > > > > >> > > > > > >>> unique IDs
>> > > > > > >> > > > > > >>> > > would ensure that the source cluster
>> topics do
>> > > not
>> > > > > > >> conflict
>> > > > > > >> > > > with
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > >>> > > destination cluster topics.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > If we were to use sequence ids, we would
>> need
>> > > > > > >> sufficiently
>> > > > > > >> > > > large
>> > > > > > >> > > > > > >>> cluster
>> > > > > > >> > > > > > >>> > > ids to be stored with the topic
>> identifiers or
>> > > we
>> > > > > run
>> > > > > > >> the
>> > > > > > >> > > risk
>> > > > > > >> > > > of
>> > > > > > >> > > > > > >>> > > collisions. This will give up any
>> advantage in
>> > > > > > >> compactness
>> > > > > > >> > > that
>> > > > > > >> > > > > > >>> sequence
>> > > > > > >> > > > > > >>> > > numbers may bring. Given these advantages I
>> > > think
>> > > > it
>> > > > > > >> makes
>> > > > > > >> > > > sense
>> > > > > > >> > > > > to
>> > > > > > >> > > > > > >>> use
>> > > > > > >> > > > > > >>> > > UUIDs.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > Gokul: This is an interesting idea, but
>> this
>> > is
>> > > a
>> > > > > > >> breaking
>> > > > > > >> > > > > change.
>> > > > > > >> > > > > > >>> Out of
>> > > > > > >> > > > > > >>> > > scope for now, but maybe worth discussing
>> in
>> > the
>> > > > > > future.
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > Hope this explains some of the decisions,
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > Justine
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at 8:27 AM Gokul
>> Ramanan
>> > > > > > >> Subramanian <
>> > > > > > >> > > > > > >>> > > gokul24...@gmail.com> wrote:
>> > > > > > >> > > > > > >>> > >
>> > > > > > >> > > > > > >>> > > > Hi.
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > Have you thought about whether it makes
>> > sense
>> > > to
>> > > > > > >> support
>> > > > > > >> > > > > > >>> authorizing a
>> > > > > > >> > > > > > >>> > > > principal for a topic ID rather than a
>> topic
>> > > > name
>> > > > > to
>> > > > > > >> > > achieve
>> > > > > > >> > > > > > >>> tighter
>> > > > > > >> > > > > > >>> > > > security?
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > Or is the topic ID fundamentally an
>> internal
>> > > > > detail
>> > > > > > >> > similar
>> > > > > > >> > > > to
>> > > > > > >> > > > > > >>> epochs
>> > > > > > >> > > > > > >>> > used
>> > > > > > >> > > > > > >>> > > > in a bunch of other places in Kafka?
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > Thanks.
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020 at 4:06 PM John
>> > Roesler <
>> > > > > > >> > > > > > vvcep...@apache.org>
>> > > > > > >> > > > > > >>> > wrote:
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> > > > > Hello Justine,
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > I happen to have been confronted
>> recently
>> > > with
>> > > > > the
>> > > > > > >> need
>> > > > > > >> > > to
>> > > > > > >> > > > > keep
>> > > > > > >> > > > > > >>> > track of
>> > > > > > >> > > > > > >>> > > > a
>> > > > > > >> > > > > > >>> > > > > large number of topics as compactly as
>> > > > > possible. I
>> > > > > > >> was
>> > > > > > >> > > > going
>> > > > > > >> > > > > to
>> > > > > > >> > > > > > >>> come
>> > > > > > >> > > > > > >>> > up
>> > > > > > >> > > > > > >>> > > > > with some way to dictionary encode the
>> > topic
>> > > > > names
>> > > > > > >> as
>> > > > > > >> > > > > integers,
>> > > > > > >> > > > > > >>> but
>> > > > > > >> > > > > > >>> > this
>> > > > > > >> > > > > > >>> > > > > seems much better!
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > Apologies if this has been raised
>> before,
>> > > but
>> > > > > I’m
>> > > > > > >> > > wondering
>> > > > > > >> > > > > > >>> about the
>> > > > > > >> > > > > > >>> > > > > choice of UUID vs sequence number for
>> the
>> > > ids.
>> > > > > > >> > Typically,
>> > > > > > >> > > > > I’ve
>> > > > > > >> > > > > > >>> seen
>> > > > > > >> > > > > > >>> > UUIDs
>> > > > > > >> > > > > > >>> > > > > in two situations:
>> > > > > > >> > > > > > >>> > > > > 1. When processes need to generate
>> > > > non-colliding
>> > > > > > >> > > > identifiers
>> > > > > > >> > > > > > >>> without
>> > > > > > >> > > > > > >>> > > > > coordination.
>> > > > > > >> > > > > > >>> > > > > 2. When the identifier needs to be
>> > > > “universally
>> > > > > > >> > unique”;
>> > > > > > >> > > > > I.e.,
>> > > > > > >> > > > > > >>> the
>> > > > > > >> > > > > > >>> > > > > identifier needs to distinguish the
>> entity
>> > > > from
>> > > > > > all
>> > > > > > >> > other
>> > > > > > >> > > > > > >>> entities
>> > > > > > >> > > > > > >>> > that
>> > > > > > >> > > > > > >>> > > > > could ever exist. This is useful in
>> cases
>> > > > where
>> > > > > > >> > entities
>> > > > > > >> > > > from
>> > > > > > >> > > > > > all
>> > > > > > >> > > > > > >>> > kinds
>> > > > > > >> > > > > > >>> > > > of
>> > > > > > >> > > > > > >>> > > > > systems get mixed together, such as
>> when
>> > > > dumping
>> > > > > > >> logs
>> > > > > > >> > > from
>> > > > > > >> > > > > all
>> > > > > > >> > > > > > >>> > processes
>> > > > > > >> > > > > > >>> > > > in
>> > > > > > >> > > > > > >>> > > > > a company into a common system.
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > Maybe I’m being short-sighted, but it
>> > > doesn’t
>> > > > > seem
>> > > > > > >> like
>> > > > > > >> > > > > either
>> > > > > > >> > > > > > >>> really
>> > > > > > >> > > > > > >>> > > > > applies here. It seems like the brokers
>> > > could
>> > > > > and
>> > > > > > >> would
>> > > > > > >> > > > > achieve
>> > > > > > >> > > > > > >>> > consensus
>> > > > > > >> > > > > > >>> > > > > when creating a topic anyway, which is
>> all
>> > > > > that’s
>> > > > > > >> > > required
>> > > > > > >> > > > to
>> > > > > > >> > > > > > >>> > generate
>> > > > > > >> > > > > > >>> > > > > non-colliding sequence ids. For the
>> > second,
>> > > as
>> > > > > you
>> > > > > > >> > > mention,
>> > > > > > >> > > > > we
>> > > > > > >> > > > > > >>> could
>> > > > > > >> > > > > > >>> > > > assign
>> > > > > > >> > > > > > >>> > > > > a UUID to the cluster as a whole, which
>> > > would
>> > > > > > render
>> > > > > > >> > any
>> > > > > > >> > > > > > resource
>> > > > > > >> > > > > > >>> > scoped
>> > > > > > >> > > > > > >>> > > > to
>> > > > > > >> > > > > > >>> > > > > the broker universally unique as well.
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > The reason I mention this is that,
>> > although
>> > > a
>> > > > > UUID
>> > > > > > >> is
>> > > > > > >> > way
>> > > > > > >> > > > > more
>> > > > > > >> > > > > > >>> > compact
>> > > > > > >> > > > > > >>> > > > > than topic names, it’s still 16 bytes.
>> In
>> > > > > > contrast,
>> > > > > > >> a
>> > > > > > >> > > > 4-byte
>> > > > > > >> > > > > > >>> integer
>> > > > > > >> > > > > > >>> > > > > sequence id would give us 4 billion
>> unique
>> > > > > topics
>> > > > > > >> per
>> > > > > > >> > > > > cluster,
>> > > > > > >> > > > > > >>> which
>> > > > > > >> > > > > > >>> > > > seems
>> > > > > > >> > > > > > >>> > > > > like enough ;)
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > Considering the number of different
>> times
>> > > > these
>> > > > > > >> topic
>> > > > > > >> > > > > > >>> identifiers are
>> > > > > > >> > > > > > >>> > > > sent
>> > > > > > >> > > > > > >>> > > > > over the wire or stored in memory, it
>> > seems
>> > > > like
>> > > > > > it
>> > > > > > >> > might
>> > > > > > >> > > > be
>> > > > > > >> > > > > > >>> worth
>> > > > > > >> > > > > > >>> > the
>> > > > > > >> > > > > > >>> > > > > additional 4x space savings.
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > What do you think about this?
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > Thanks,
>> > > > > > >> > > > > > >>> > > > > John
>> > > > > > >> > > > > > >>> > > > >
>> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11, 2020, at 03:20, Tom
>> > Bentley
>> > > > > wrote:
>> > > > > > >> > > > > > >>> > > > > > Hi Justine,
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > This looks like a very welcome
>> > > improvement.
>> > > > > > >> Thanks!
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > Maybe I missed it, but the KIP
>> doesn't
>> > > seem
>> > > > to
>> > > > > > >> > mention
>> > > > > > >> > > > > > changing
>> > > > > > >> > > > > > >>> > > > > > DeleteTopicsRequest to identify the
>> > topic
>> > > > > using
>> > > > > > an
>> > > > > > >> > id.
>> > > > > > >> > > > > Maybe
>> > > > > > >> > > > > > >>> > that's out
>> > > > > > >> > > > > > >>> > > > > of
>> > > > > > >> > > > > > >>> > > > > > scope, but DeleteTopicsRequest is not
>> > > listed
>> > > > > > among
>> > > > > > >> > the
>> > > > > > >> > > > > Future
>> > > > > > >> > > > > > >>> Work
>> > > > > > >> > > > > > >>> > APIs
>> > > > > > >> > > > > > >>> > > > > > either.
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > Kind regards,
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > Tom
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10, 2020 at 3:59 PM
>> Satish
>> > > > > Duggana <
>> > > > > > >> > > > > > >>> > > > satish.dugg...@gmail.com
>> > > > > > >> > > > > > >>> > > > > > wrote:
>> > > > > > >> > > > > > >>> > > > > >
>> > > > > > >> > > > > > >>> > > > > > > Thanks Lucas/Justine for the nice
>> KIP.
>> > > > > > >> > > > > > >>> > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > It has several benefits which also
>> > > include
>> > > > > > >> > > simplifying
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > >>> topic
>> > > > > > >> > > > > > >>> > > > > > > deletion process by controller and
>> > logs
>> > > > > > cleanup
>> > > > > > >> by
>> > > > > > >> > > > > brokers
>> > > > > > >> > > > > > in
>> > > > > > >> > > > > > >>> > corner
>> > > > > > >> > > > > > >>> > > > > > > cases.
>> > > > > > >> > > > > > >>> > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > Best,
>> > > > > > >> > > > > > >>> > > > > > > Satish.
>> > > > > > >> > > > > > >>> > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9, 2020 at 10:07 PM
>> > Justine
>> > > > > > Olshan <
>> > > > > > >> > > > > > >>> > jols...@confluent.io
>> > > > > > >> > > > > > >>> > > > > > > wrote:
>> > > > > > >> > > > > > >>> > > > > > > > Hello all, it's been almost a
>> year!
>> > > I've
>> > > > > > made
>> > > > > > >> > some
>> > > > > > >> > > > > > changes
>> > > > > > >> > > > > > >>> to
>> > > > > > >> > > > > > >>> > this
>> > > > > > >> > > > > > >>> > > > > KIP
>> > > > > > >> > > > > > >>> > > > > > > and hope to continue the
>> discussion.
>> > > > > > >> > > > > > >>> > > > > > > > One of the main changes I've
>> added
>> > is
>> > > > now
>> > > > > > the
>> > > > > > >> > > > metadata
>> > > > > > >> > > > > > >>> response
>> > > > > > >> > > > > > >>> > > > will
>> > > > > > >> > > > > > >>> > > > > > > include the topic ID (as Colin
>> > > suggested).
>> > > > > > >> Clients
>> > > > > > >> > > can
>> > > > > > >> > > > > > >>> obtain the
>> > > > > > >> > > > > > >>> > > > > topicID
>> > > > > > >> > > > > > >>> > > > > > > of a given topic through a
>> > > > TopicDescription.
>> > > > > > The
>> > > > > > >> > > > topicId
>> > > > > > >> > > > > > will
>> > > > > > >> > > > > > >>> > also be
>> > > > > > >> > > > > > >>> > > > > > > included with the UpdateMetadata
>> > > request.
>> > > > > > >> > > > > > >>> > > > > > > > Let me know what you all think.
>> > > > > > >> > > > > > >>> > > > > > > > Thank you,
>> > > > > > >> > > > > > >>> > > > > > > > Justine
>> > > > > > >> > > > > > >>> > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13 16:38:26, "Colin
>> > > McCabe" <
>> > > > > > >> > > > > > cmcc...@apache.org
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>> > wrote:
>> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > Thanks for tackling this.
>> Topic
>> > IDs
>> > > > > are a
>> > > > > > >> > great
>> > > > > > >> > > > > idea,
>> > > > > > >> > > > > > >>> and
>> > > > > > >> > > > > > >>> > this
>> > > > > > >> > > > > > >>> > > > is
>> > > > > > >> > > > > > >>> > > > > a
>> > > > > > >> > > > > > >>> > > > > > > really good writeup.
>> > > > > > >> > > > > > >>> > > > > > > > > For /brokers/topics/[topic],
>> the
>> > > > schema
>> > > > > > >> version
>> > > > > > >> > > > > should
>> > > > > > >> > > > > > be
>> > > > > > >> > > > > > >>> > bumped
>> > > > > > >> > > > > > >>> > > > to
>> > > > > > >> > > > > > >>> > > > > > > version 3, rather than 2. KIP-455
>> > > bumped
>> > > > > the
>> > > > > > >> > version
>> > > > > > >> > > > of
>> > > > > > >> > > > > > this
>> > > > > > >> > > > > > >>> > znode
>> > > > > > >> > > > > > >>> > > > to
>> > > > > > >> > > > > > >>> > > > > 2
>> > > > > > >> > > > > > >>> > > > > > > already :)
>> > > > > > >> > > > > > >>> > > > > > > > > Given that we're going to be
>> > seeing
>> > > > > these
>> > > > > > >> > things
>> > > > > > >> > > as
>> > > > > > >> > > > > > >>> strings
>> > > > > > >> > > > > > >>> > as
>> > > > > > >> > > > > > >>> > > > lot
>> > > > > > >> > > > > > >>> > > > > (in
>> > > > > > >> > > > > > >>> > > > > > > logs, in ZooKeeper, on the
>> > command-line,
>> > > > > > etc.),
>> > > > > > >> > does
>> > > > > > >> > > it
>> > > > > > >> > > > > > make
>> > > > > > >> > > > > > >>> > sense to
>> > > > > > >> > > > > > >>> > > > > use
>> > > > > > >> > > > > > >>> > > > > > > base64 when converting them to
>> > strings?
>> > > > > > >> > > > > > >>> > > > > > > > > Here is an example of the hex
>> > > > > > >> representation:
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > And here is an example in
>> base64.
>> > > > > > >> > > > > > >>> > > > > > > > > b8tRS7h4TJ2Vt43Dp85v2A
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > The base64 version saves 15
>> > letters
>> > > > (to
>> > > > > be
>> > > > > > >> > fair,
>> > > > > > >> > > 4
>> > > > > > >> > > > of
>> > > > > > >> > > > > > >>> those
>> > > > > > >> > > > > > >>> > were
>> > > > > > >> > > > > > >>> > > > > > > dashes that we could have elided in
>> > the
>> > > > hex
>> > > > > > >> > > > > > representation.)
>> > > > > > >> > > > > > >>> > > > > > > > > Another thing to consider is
>> that
>> > we
>> > > > > > should
>> > > > > > >> > > specify
>> > > > > > >> > > > > > that
>> > > > > > >> > > > > > >>> the
>> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID is not a valid
>> topic
>> > > UUID.
>> > > > > > We
>> > > > > > >> > can't
>> > > > > > >> > > > use
>> > > > > > >> > > > > > >>> null
>> > > > > > >> > > > > > >>> > for
>> > > > > > >> > > > > > >>> > > > this
>> > > > > > >> > > > > > >>> > > > > > > because we can't pass a null UUID
>> over
>> > > the
>> > > > > RPC
>> > > > > > >> > > protocol
>> > > > > > >> > > > > > >>> (there
>> > > > > > >> > > > > > >>> > is no
>> > > > > > >> > > > > > >>> > > > > > > special pattern for null, nor do we
>> > want
>> > > > to
>> > > > > > >> waste
>> > > > > > >> > > space
>> > > > > > >> > > > > > >>> reserving
>> > > > > > >> > > > > > >>> > > > such
>> > > > > > >> > > > > > >>> > > > > a
>> > > > > > >> > > > > > >>> > > > > > > pattern.)
>> > > > > > >> > > > > > >>> > > > > > > > > Maybe I missed it, but did you
>> > > > describe
>> > > > > > >> > > "migration
>> > > > > > >> > > > > > of...
>> > > > > > >> > > > > > >>> > existing
>> > > > > > >> > > > > > >>> > > > > > > topic[s] without topic IDs" in
>> detail
>> > in
>> > > > any
>> > > > > > >> > section?
>> > > > > > >> > > > It
>> > > > > > >> > > > > > >>> seems
>> > > > > > >> > > > > > >>> > like
>> > > > > > >> > > > > > >>> > > > > when
>> > > > > > >> > > > > > >>> > > > > > > the new controller becomes active,
>> it
>> > > > should
>> > > > > > >> just
>> > > > > > >> > > > > generate
>> > > > > > >> > > > > > >>> random
>> > > > > > >> > > > > > >>> > > > > UUIDs for
>> > > > > > >> > > > > > >>> > > > > > > these, and write the random UUIDs
>> back
>> > > to
>> > > > > > >> > ZooKeeper.
>> > > > > > >> > > > It
>> > > > > > >> > > > > > >>> would be
>> > > > > > >> > > > > > >>> > > > good
>> > > > > > >> > > > > > >>> > > > > to
>> > > > > > >> > > > > > >>> > > > > > > spell that out. We should make it
>> > clear
>> > > > > that
>> > > > > > >> this
>> > > > > > >> > > > > happens
>> > > > > > >> > > > > > >>> > regardless
>> > > > > > >> > > > > > >>> > > > > of
>> > > > > > >> > > > > > >>> > > > > > > the inter-broker protocol version
>> > (it's
>> > > a
>> > > > > > >> > compatible
>> > > > > > >> > > > > > change).
>> > > > > > >> > > > > > >>> > > > > > > > > "LeaderAndIsrRequests
>> including an
>> > > > > > >> > > > is_every_partition
>> > > > > > >> > > > > > >>> flag"
>> > > > > > >> > > > > > >>> > > > seems a
>> > > > > > >> > > > > > >>> > > > > > > bit wordy. Can we just call these
>> > "full
>> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
>> > > > > > >> > > > > > >>> > Then
>> > > > > > >> > > > > > >>> > > > > the
>> > > > > > >> > > > > > >>> > > > > > > RPC field could be named "full".
>> > Also,
>> > > it
>> > > > > > would
>> > > > > > >> > > > probably
>> > > > > > >> > > > > > be
>> > > > > > >> > > > > > >>> > better
>> > > > > > >> > > > > > >>> > > > > for the
>> > > > > > >> > > > > > >>> > > > > > > RPC field to be an enum of {
>> > > UNSPECIFIED,
>> > > > > > >> > > INCREMENTAL,
>> > > > > > >> > > > > FULL
>> > > > > > >> > > > > > >>> }, so
>> > > > > > >> > > > > > >>> > > > that
>> > > > > > >> > > > > > >>> > > > > we
>> > > > > > >> > > > > > >>> > > > > > > can cleanly handle old versions (by
>> > > > treating
>> > > > > > >> them
>> > > > > > >> > as
>> > > > > > >> > > > > > >>> UNSPECIFIED)
>> > > > > > >> > > > > > >>> > > > > > > > > In the LeaderAndIsrRequest
>> > section,
>> > > > you
>> > > > > > >> write
>> > > > > > >> > "A
>> > > > > > >> > > > > final
>> > > > > > >> > > > > > >>> > deletion
>> > > > > > >> > > > > > >>> > > > > event
>> > > > > > >> > > > > > >>> > > > > > > will be secheduled for X ms after
>> the
>> > > > > > >> > > > LeaderAndIsrRequest
>> > > > > > >> > > > > > was
>> > > > > > >> > > > > > >>> > first
>> > > > > > >> > > > > > >>> > > > > > > received..." I guess the X was a
>> > > > > placeholder
>> > > > > > >> that
>> > > > > > >> > > you
>> > > > > > >> > > > > > >>> intended
>> > > > > > >> > > > > > >>> > to
>> > > > > > >> > > > > > >>> > > > > replace
>> > > > > > >> > > > > > >>> > > > > > > before posting? :) In any case,
>> this
>> > > > seems
>> > > > > > like
>> > > > > > >> > the
>> > > > > > >> > > > kind
>> > > > > > >> > > > > > of
>> > > > > > >> > > > > > >>> > thing
>> > > > > > >> > > > > > >>> > > > we'd
>> > > > > > >> > > > > > >>> > > > > > > want a configuration for. Let's
>> > > describe
>> > > > > that
>> > > > > > >> > > > > > configuration
>> > > > > > >> > > > > > >>> key
>> > > > > > >> > > > > > >>> > > > > somewhere
>> > > > > > >> > > > > > >>> > > > > > > in this KIP, including what its
>> > default
>> > > > > value
>> > > > > > >> is.
>> > > > > > >> > > > > > >>> > > > > > > > > We should probably also log a
>> > bunch
>> > > of
>> > > > > > >> messages
>> > > > > > >> > > at
>> > > > > > >> > > > > WARN
>> > > > > > >> > > > > > >>> level
>> > > > > > >> > > > > > >>> > > > when
>> > > > > > >> > > > > > >>> > > > > > > something is scheduled for
>> deletion,
>> > as
>> > > > > well.
>> > > > > > >> > (Maybe
>> > > > > > >> > > > > this
>> > > > > > >> > > > > > >>> was
>> > > > > > >> > > > > > >>> > > > > assumed, but
>> > > > > > >> > > > > > >>> > > > > > > it would be good to mention it).
>> > > > > > >> > > > > > >>> > > > > > > > > I feel like there are a few
>> > sections
>> > > > > that
>> > > > > > >> > should
>> > > > > > >> > > be
>> > > > > > >> > > > > > >>> moved to
>> > > > > > >> > > > > > >>> > > > > "rejected
>> > > > > > >> > > > > > >>> > > > > > > alternatives." For example, in the
>> > > > > > DeleteTopics
>> > > > > > >> > > > section,
>> > > > > > >> > > > > > >>> since
>> > > > > > >> > > > > > >>> > we're
>> > > > > > >> > > > > > >>> > > > > not
>> > > > > > >> > > > > > >>> > > > > > > going to do option 1 or 2, these
>> > should
>> > > be
>> > > > > > moved
>> > > > > > >> > into
>> > > > > > >> > > > > > >>> "rejected
>> > > > > > >> > > > > > >>> > > > > > > alternatives," rather than
>> appearing
>> > > > > inline.
>> > > > > > >> > > Another
>> > > > > > >> > > > > case
>> > > > > > >> > > > > > >>> is
>> > > > > > >> > > > > > >>> > the
>> > > > > > >> > > > > > >>> > > > > "Should
>> > > > > > >> > > > > > >>> > > > > > > we remove topic name from the
>> protocol
>> > > > where
>> > > > > > >> > > possible"
>> > > > > > >> > > > > > >>> section.
>> > > > > > >> > > > > > >>> > This
>> > > > > > >> > > > > > >>> > > > > is
>> > > > > > >> > > > > > >>> > > > > > > clearly discussing a design
>> > alternative
>> > > > that
>> > > > > > >> we're
>> > > > > > >> > > not
>> > > > > > >> > > > > > >>> proposing
>> > > > > > >> > > > > > >>> > to
>> > > > > > >> > > > > > >>> > > > > > > implement: removing the topic name
>> > from
>> > > > > those
>> > > > > > >> > > > protocols.
>> > > > > > >> > > > > > >>> > > > > > > > > Is it really necessary to have
>> a
>> > new
>> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
>> > > > > > >> > > > > > >>> > > > > path
>> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper? It seems like we
>> don't
>> > > > really
>> > > > > > >> need
>> > > > > > >> > > this.
>> > > > > > >> > > > > > >>> Whenever
>> > > > > > >> > > > > > >>> > > > > there is
>> > > > > > >> > > > > > >>> > > > > > > a new controller, we'll send out
>> full
>> > > > > > >> > > > > LeaderAndIsrRequests
>> > > > > > >> > > > > > >>> which
>> > > > > > >> > > > > > >>> > will
>> > > > > > >> > > > > > >>> > > > > > > trigger the stale topics to be
>> cleaned
>> > > up.
>> > > > > > The
>> > > > > > >> > > active
>> > > > > > >> > > > > > >>> > controller
>> > > > > > >> > > > > > >>> > > > will
>> > > > > > >> > > > > > >>> > > > > > > also send the full
>> LeaderAndIsrRequest
>> > > to
>> > > > > > >> brokers
>> > > > > > >> > > that
>> > > > > > >> > > > > are
>> > > > > > >> > > > > > >>> just
>> > > > > > >> > > > > > >>> > > > > starting
>> > > > > > >> > > > > > >>> > > > > > > up. So we don't really need this
>> > kind
>> > > > of
>> > > > > > >> > two-phase
>> > > > > > >> > > > > > commit
>> > > > > > >> > > > > > >>> > (send
>> > > > > > >> > > > > > >>> > > > out
>> > > > > > >> > > > > > >>> > > > > > > StopReplicasRequest, get ACKs from
>> all
>> > > > > nodes,
>> > > > > > >> > commit
>> > > > > > >> > > by
>> > > > > > >> > > > > > >>> removing
>> > > > > > >> > > > > > >>> > > > > > > /admin/delete_topics node) any
>> more.
>> > > > > > >> > > > > > >>> > > > > > > > > You mention that FetchRequest
>> will
>> > > now
>> > > > > > >> include
>> > > > > > >> > > UUID
>> > > > > > >> > > > > to
>> > > > > > >> > > > > > >>> avoid
>> > > > > > >> > > > > > >>> > > > issues
>> > > > > > >> > > > > > >>> > > > > > > where requests are made to stale
>> > > > partitions.
>> > > > > > >> > > However,
>> > > > > > >> > > > > > >>> adding a
>> > > > > > >> > > > > > >>> > UUID
>> > > > > > >> > > > > > >>> > > > to
>> > > > > > >> > > > > > >>> > > > > > > MetadataRequest is listed as future
>> > > work,
>> > > > > out
>> > > > > > of
>> > > > > > >> > > scope
>> > > > > > >> > > > > for
>> > > > > > >> > > > > > >>> this
>> > > > > > >> > > > > > >>> > KIP.
>> > > > > > >> > > > > > >>> > > > > How
>> > > > > > >> > > > > > >>> > > > > > > will the client learn what the
>> topic
>> > > UUID
>> > > > > is,
>> > > > > > if
>> > > > > > >> > the
>> > > > > > >> > > > > > metadata
>> > > > > > >> > > > > > >>> > > > response
>> > > > > > >> > > > > > >>> > > > > > > doesn't include that information?
>> It
>> > > > seems
>> > > > > > like
>> > > > > > >> > > adding
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > >>> UUID
>> > > > > > >> > > > > > >>> > to
>> > > > > > >> > > > > > >>> > > > > > > MetadataResponse would be an
>> > improvement
>> > > > > here
>> > > > > > >> that
>> > > > > > >> > > > might
>> > > > > > >> > > > > > not
>> > > > > > >> > > > > > >>> be
>> > > > > > >> > > > > > >>> > too
>> > > > > > >> > > > > > >>> > > > > hard to
>> > > > > > >> > > > > > >>> > > > > > > make.
>> > > > > > >> > > > > > >>> > > > > > > > > best,
>> > > > > > >> > > > > > >>> > > > > > > > > Colin
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep 9, 2019, at 17:48,
>> > > Ryanne
>> > > > > > Dolan
>> > > > > > >> > > wrote:
>> > > > > > >> > > > > > >>> > > > > > > > > > Lucas, this would be great.
>> I've
>> > > run
>> > > > > > into
>> > > > > > >> > > issues
>> > > > > > >> > > > > with
>> > > > > > >> > > > > > >>> > topics
>> > > > > > >> > > > > > >>> > > > > being
>> > > > > > >> > > > > > >>> > > > > > > > > > resurrected accidentally,
>> since
>> > a
>> > > > > client
>> > > > > > >> > cannot
>> > > > > > >> > > > > > easily
>> > > > > > >> > > > > > >>> > > > > distinguish
>> > > > > > >> > > > > > >>> > > > > > > between
>> > > > > > >> > > > > > >>> > > > > > > > > > a deleted topic and a new
>> topic
>> > > with
>> > > > > the
>> > > > > > >> same
>> > > > > > >> > > > name.
>> > > > > > >> > > > > > I'd
>> > > > > > >> > > > > > >>> > need
>> > > > > > >> > > > > > >>> > > > the
>> > > > > > >> > > > > > >>> > > > > ID
>> > > > > > >> > > > > > >>> > > > > > > > > > accessible from the client to
>> > > solve
>> > > > > that
>> > > > > > >> > issue,
>> > > > > > >> > > > but
>> > > > > > >> > > > > > >>> this
>> > > > > > >> > > > > > >>> > is a
>> > > > > > >> > > > > > >>> > > > > good
>> > > > > > >> > > > > > >>> > > > > > > first
>> > > > > > >> > > > > > >>> > > > > > > > > > step.
>> > > > > > >> > > > > > >>> > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
>> > > > > > >> > > > > > >>> > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > On Wed, Sep 4, 2019 at 1:41
>> PM
>> > > Lucas
>> > > > > > >> > > Bradstreet <
>> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
>> > > > > > >> > > > > > >>> > > > > > > wrote:
>> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > > I would like to kick off
>> > > > discussion
>> > > > > of
>> > > > > > >> > > KIP-516,
>> > > > > > >> > > > > an
>> > > > > > >> > > > > > >>> > > > > implementation
>> > > > > > >> > > > > > >>> > > > > > > of topic
>> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for Kafka. Topic IDs
>> aim
>> > to
>> > > > > solve
>> > > > > > >> topic
>> > > > > > >> > > > > > >>> uniqueness
>> > > > > > >> > > > > > >>> > > > > problems in
>> > > > > > >> > > > > > >>> > > > > > > Kafka,
>> > > > > > >> > > > > > >>> > > > > > > > > > > where referring to a topic
>> by
>> > > name
>> > > > > > >> alone is
>> > > > > > >> > > > > > >>> insufficient.
>> > > > > > >> > > > > > >>> > > > Such
>> > > > > > >> > > > > > >>> > > > > > > cases
>> > > > > > >> > > > > > >>> > > > > > > > > > > include when a topic has
>> been
>> > > > > deleted
>> > > > > > >> and
>> > > > > > >> > > > > recreated
>> > > > > > >> > > > > > >>> with
>> > > > > > >> > > > > > >>> > the
>> > > > > > >> > > > > > >>> > > > > same
>> > > > > > >> > > > > > >>> > > > > > > name.
>> > > > > > >> > > > > > >>> > > > > > > > > > > Unique identifiers will
>> help
>> > > > > simplify
>> > > > > > >> and
>> > > > > > >> > > > improve
>> > > > > > >> > > > > > >>> Kafka's
>> > > > > > >> > > > > > >>> > > > topic
>> > > > > > >> > > > > > >>> > > > > > > deletion
>> > > > > > >> > > > > > >>> > > > > > > > > > > process, as well as prevent
>> > > cases
>> > > > > > where
>> > > > > > >> > > brokers
>> > > > > > >> > > > > may
>> > > > > > >> > > > > > >>> > > > incorrectly
>> > > > > > >> > > > > > >>> > > > > > > interact
>> > > > > > >> > > > > > >>> > > > > > > > > > > with stale versions of
>> topics.
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> > > >
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
>> > > > > > >> > > > > > >>> > > > > > > > > > > Looking forward to your
>> > > thoughts.
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
>> > > > > > >> > > > > > >>> > > > > > > > > > >
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>> >
>> > > > > > >> > > > > > >>>
>> > > > > > >> > > > > > >>
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
