Hi.
Thanks for the KIP.
Have you thought about whether it makes sense to support authorizing a
principal for a topic ID rather than a topic name to achieve tighter
security?
Or is the topic ID fundamentally an internal detail similar to epochs used
in a bunch of other places in Kafka?
Thanks.
On Fri, Sep 11, 2020 at 4:06 PM John Roesler <vvcep...@apache.org> wrote:
> Hello Justine,
>
> Thanks for the KIP!
>
> I happen to have been confronted recently with the need to keep track of a
> large number of topics as compactly as possible. I was going to come up
> with some way to dictionary encode the topic names as integers, but this
> seems much better!
>
> Apologies if this has been raised before, but I’m wondering about the
> choice of UUID vs sequence number for the ids. Typically, I’ve seen UUIDs
> in two situations:
> 1. When processes need to generate non-colliding identifiers without
> coordination.
> 2. When the identifier needs to be “universally unique”; I.e., the
> identifier needs to distinguish the entity from all other entities that
> could ever exist. This is useful in cases where entities from all kinds of
> systems get mixed together, such as when dumping logs from all processes in
> a company into a common system.
>
> Maybe I’m being short-sighted, but it doesn’t seem like either really
> applies here. It seems like the brokers could and would achieve consensus
> when creating a topic anyway, which is all that’s required to generate
> non-colliding sequence ids. For the second, as you mention, we could assign
> a UUID to the cluster as a whole, which would render any resource scoped to
> the broker universally unique as well.
>
> The reason I mention this is that, although a UUID is way more compact
> than topic names, it’s still 16 bytes. In contrast, a 4-byte integer
> sequence id would give us 4 billion unique topics per cluster, which seems
> like enough ;)
>
> Considering the number of different times these topic identifiers are sent
> over the wire or stored in memory, it seems like it might be worth the
> additional 4x space savings.
>
> What do you think about this?
>
> Thanks,
> John
>
> On Fri, Sep 11, 2020, at 03:20, Tom Bentley wrote:
> > Hi Justine,
> >
> > This looks like a very welcome improvement. Thanks!
> >
> > Maybe I missed it, but the KIP doesn't seem to mention changing
> > DeleteTopicsRequest to identify the topic using an id. Maybe that's out
> of
> > scope, but DeleteTopicsRequest is not listed among the Future Work APIs
> > either.
> >
> > Kind regards,
> >
> > Tom
> >
> > On Thu, Sep 10, 2020 at 3:59 PM Satish Duggana <satish.dugg...@gmail.com
> >
> > wrote:
> >
> > > Thanks Lucas/Justine for the nice KIP.
> > >
> > > It has several benefits which also include simplifying the topic
> > > deletion process by controller and logs cleanup by brokers in corner
> > > cases.
> > >
> > > Best,
> > > Satish.
> > >
> > > On Wed, Sep 9, 2020 at 10:07 PM Justine Olshan <jols...@confluent.io>
> > > wrote:
> > > >
> > > > Hello all, it's been almost a year! I've made some changes to this
> KIP
> > > and hope to continue the discussion.
> > > >
> > > > One of the main changes I've added is now the metadata response will
> > > include the topic ID (as Colin suggested). Clients can obtain the
> topicID
> > > of a given topic through a TopicDescription. The topicId will also be
> > > included with the UpdateMetadata request.
> > > >
> > > > Let me know what you all think.
> > > > Thank you,
> > > > Justine
> > > >
> > > > On 2019/09/13 16:38:26, "Colin McCabe" <cmcc...@apache.org> wrote:
> > > > > Hi Lucas,
> > > > >
> > > > > Thanks for tackling this. Topic IDs are a great idea, and this is
> a
> > > really good writeup.
> > > > >
> > > > > For /brokers/topics/[topic], the schema version should be bumped to
> > > version 3, rather than 2. KIP-455 bumped the version of this znode to
> 2
> > > already :)
> > > > >
> > > > > Given that we're going to be seeing these things as strings as lot
> (in
> > > logs, in ZooKeeper, on the command-line, etc.), does it make sense to
> use
> > > base64 when converting them to strings?
> > > > >
> > > > > Here is an example of the hex representation:
> > > > > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > > > >
> > > > > And here is an example in base64.
> > > > > b8tRS7h4TJ2Vt43Dp85v2A
> > > > >
> > > > > The base64 version saves 15 letters (to be fair, 4 of those were
> > > dashes that we could have elided in the hex representation.)
> > > > >
> > > > > Another thing to consider is that we should specify that the
> > > all-zeroes UUID is not a valid topic UUID. We can't use null for this
> > > because we can't pass a null UUID over the RPC protocol (there is no
> > > special pattern for null, nor do we want to waste space reserving such
> a
> > > pattern.)
> > > > >
> > > > > Maybe I missed it, but did you describe "migration of... existing
> > > topic[s] without topic IDs" in detail in any section? It seems like
> when
> > > the new controller becomes active, it should just generate random
> UUIDs for
> > > these, and write the random UUIDs back to ZooKeeper. It would be good
> to
> > > spell that out. We should make it clear that this happens regardless
> of
> > > the inter-broker protocol version (it's a compatible change).
> > > > >
> > > > > "LeaderAndIsrRequests including an is_every_partition flag" seems a
> > > bit wordy. Can we just call these "full LeaderAndIsrRequests"? Then
> the
> > > RPC field could be named "full". Also, it would probably be better
> for the
> > > RPC field to be an enum of { UNSPECIFIED, INCREMENTAL, FULL }, so that
> we
> > > can cleanly handle old versions (by treating them as UNSPECIFIED)
> > > > >
> > > > > In the LeaderAndIsrRequest section, you write "A final deletion
> event
> > > will be secheduled for X ms after the LeaderAndIsrRequest was first
> > > received..." I guess the X was a placeholder that you intended to
> replace
> > > before posting? :) In any case, this seems like the kind of thing we'd
> > > want a configuration for. Let's describe that configuration key
> somewhere
> > > in this KIP, including what its default value is.
> > > > >
> > > > > We should probably also log a bunch of messages at WARN level when
> > > something is scheduled for deletion, as well. (Maybe this was
> assumed, but
> > > it would be good to mention it).
> > > > >
> > > > > I feel like there are a few sections that should be moved to
> "rejected
> > > alternatives." For example, in the DeleteTopics section, since we're
> not
> > > going to do option 1 or 2, these should be moved into "rejected
> > > alternatives," rather than appearing inline. Another case is the
> "Should
> > > we remove topic name from the protocol where possible" section. This
> is
> > > clearly discussing a design alternative that we're not proposing to
> > > implement: removing the topic name from those protocols.
> > > > >
> > > > > Is it really necessary to have a new /admin/delete_topics_by_id
> path
> > > in ZooKeeper? It seems like we don't really need this. Whenever
> there is
> > > a new controller, we'll send out full LeaderAndIsrRequests which will
> > > trigger the stale topics to be cleaned up. The active controller will
> > > also send the full LeaderAndIsrRequest to brokers that are just
> starting
> > > up. So we don't really need this kind of two-phase commit (send out
> > > StopReplicasRequest, get ACKs from all nodes, commit by removing
> > > /admin/delete_topics node) any more.
> > > > >
> > > > > You mention that FetchRequest will now include UUID to avoid issues
> > > where requests are made to stale partitions. However, adding a UUID to
> > > MetadataRequest is listed as future work, out of scope for this KIP.
> How
> > > will the client learn what the topic UUID is, if the metadata response
> > > doesn't include that information? It seems like adding the UUID to
> > > MetadataResponse would be an improvement here that might not be too
> hard to
> > > make.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Mon, Sep 9, 2019, at 17:48, Ryanne Dolan wrote:
> > > > > > Lucas, this would be great. I've run into issues with topics
> being
> > > > > > resurrected accidentally, since a client cannot easily
> distinguish
> > > between
> > > > > > a deleted topic and a new topic with the same name. I'd need the
> ID
> > > > > > accessible from the client to solve that issue, but this is a
> good
> > > first
> > > > > > step.
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > > > On Wed, Sep 4, 2019 at 1:41 PM Lucas Bradstreet <
> lu...@confluent.io>
> > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to kick off discussion of KIP-516, an
> implementation
> > > of topic
> > > > > > > IDs for Kafka. Topic IDs aim to solve topic uniqueness
> problems in
> > > Kafka,
> > > > > > > where referring to a topic by name alone is insufficient. Such
> > > cases
> > > > > > > include when a topic has been deleted and recreated with the
> same
> > > name.
> > > > > > >
> > > > > > > Unique identifiers will help simplify and improve Kafka's topic
> > > deletion
> > > > > > > process, as well as prevent cases where brokers may incorrectly
> > > interact
> > > > > > > with stale versions of topics.
> > > > > > >
> > > > > > >
> > > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > > > > > >
> > > > > > > Looking forward to your thoughts.
> > > > > > >
> > > > > > > Lucas
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>
