[
https://issues.apache.org/jira/browse/KAFKA-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384526#comment-17384526
]
Jason Gustafson commented on KAFKA-13111:
-----------------------------------------
Thanks, this makes sense to me. At a high level, we want to keep the protocol
simple even if it costs a little more complexity in the implementation. I think
it is indeed simpler if topic IDs are handled similarly to topic names. When a
topic ID is unknown, we nevertheless store it in the session and keep sending
the client UNKNOWN_TOPIC_ID until it gets removed through the forgotten topic
list or the broker learns the topic name mapping from the controller.
> Re-evaluate Fetch Sessions when using topic IDs
> -----------------------------------------------
>
> Key: KAFKA-13111
> URL: https://issues.apache.org/jira/browse/KAFKA-13111
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: Justine Olshan
> Assignee: Justine Olshan
> Priority: Major
>
> For fetch request version 13 we have the current method of handling unknown
> topic IDs.
> * When the receiving broker sees an unknown topic ID in a request or
> encounters an inconsistent (mismatched) ID in the logs, it sends a top-level
> error back, delays *all* partitions (in fetcher thread), and closes the
> session
> Ideally, we want to handle the same way as unknown topic names. We hold the
> topic partition in the session and try to resolve on a future fetch request.
> However, there are a few complications with this approach and this is why we
> opted to simply close the session. One is that many objects in the fetch
> session are keyed using topic name+partition. We also still need the topic
> name for a few other checks in the fetch path.
> Still, upon further inspection, closing the session on any new topics (when
> using topic names, we often see a few unknown topic or partition exceptions)
> is not ideal.
> One way to see similar behavior in the topic ID case is to store unresolved
> IDs in the session. For compatibility reasons, we will need to add and/or
> generify a few classes. The general idea is that for sessions using version
> 13+ we will use a structure that keys using topic ID and partition with an
> optional name. That way, we can send partition level errors for unknown topic
> IDs and handle them later in the same session. We can also remove partitions
> using topic ID more easily if the unknown topic ID was due to stale metadata.
> Then the new method will be as follows:
> * When the receiving broker sees an unknown topic ID in a request or
> encounters an inconsistent (mismatched) ID in the logs, it sends an error on
> the unknown partition, delay *only those* partitions (in fetcher thread), and
> keep the session open to try to resolve in the future
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
