Hi Matthias, Thanks for your response. I have change the version property for introducing client instance ID into the request headers to be `clientInstanceIdVersions` which is now an open-ended range, such as "3+", more closely matching what we have for flexible versions.
I also added a change to the RebalanceConsumer.clientInstanceId method introduced in KIP-1306. Thanks, Andrew On 2026/06/03 18:31:06 "Matthias J. Sax" wrote: > Thanks Andrew. > > From a KIP-714 perspective, this is a great change, as it make the new > RPCs and client/broker cross-version compatibility clean. > > One question about your `DeleteGroups` RPC example. Should the new > version of the RPC say > > > "clientInstanceIdVersion": "3+", > > ie, 3+ instead of 3? Similar for other request, which don't use `+` on > `clientInstanceIdVersion` field. > > > > -Matthias > > > > On 6/2/26 7:42 AM, Andrew Schofield wrote: > > Hi, > > Following on from the discussion in the community, and the difficulty of > > trying to accommodate the client instance ID in a tagged field with > > acceptable complexity, I have made a significant change to the KIP. Now, it > > introduces v3 of the request header containing the client instance ID as a > > proper field. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313%3A+Client+instance+ID+in+all+request+headers > > > > Please review and let me know what you think. > > > > Thanks, > > Andrew > > > > On 2026/05/28 21:51:01 "Matthias J. Sax" wrote: > >> Thanks Andrew. I agree to the issue that tagged fields are optional. > >> > >> Nevertheless, I find the proposed compatibility protocol too complex and > >> unnecessary, and would still prefer a version bump for > >> `GetTelemetrySubscriptionsRequest/Reponse` to clean it all up. -- For > >> `PushTelemetryRequest` the argument from above applies, so leaving it > >> untouched is ok. > >> > >> I don't see why `GetTelemetrySubscriptionsRequest` would need to encode > >> the `clientInstandId` in it's request body. Given that the > >> `clientInstanceId` is a random UUID, the returned subscription cannot > >> really depend on it. So why would we need to send it to the broker? > >> Right now, it's set to ZERO only for the purpose to get the broker to > >> assign a new UUID. But it has nothing to do with the actual "what > >> metrics should I send" question. > >> > >> To be fair, KIP-714 lists `client_instance_id` as a field that can be > >> used to define a subscription. But it seems rather useless in practice > >> to me? I a user defines a subscription, they cannot know the UUID. Thus, > >> I think we should actually drop supporting `client_instance_id` as a > >> "subscription matching parameter"? Of course, there is a backward > >> compatibility question, but I think we can address this: if brokers are > >> upgraded and they have an existing `client_instance_id` based > >> subscription defined, they could advertise to only support v0, and they > >> should log a warning that this feature is deprecated. New brokers would > >> also not allow to use `client_instance_id` any longer to define a > >> subscription. > >> > >> Similarly, for `GetTelemetrySubscriptionsResponse`, we only need > >> `clientInstanceId` if the broker computes the UUID. But if we let the > >> client compute it, this field is not needed any longer. > >> > >> > >> -Matthias > >> > >> > >> > >> On 5/28/26 1:21 AM, Andrew Schofield wrote: > >>> Hi Jun, > >>> Thanks for your response. > >>> > >>> JR24: I'll update the KIP shortly. Here's a summary. The various modern > >>> group protocol heartbeat requests encode the member ID as a string for > >>> historical reasons, even though it's in practice a UUID. The Apache Kafka > >>> Java client happens to use org.apache.kafka.common.Uuid to create the > >>> member ID and convert it into a string. This means that when the > >>> clientInstanceId is converted into a string, its encoding matches the > >>> member ID and it all lines up. However, there are other ways to encode a > >>> UUID into a string and I know for a fact that the librdkafka client does > >>> it differently. > >>> > >>> Thanks, > >>> Andrew > >>> > >>> On 2026/05/27 17:11:25 Jun Rao via dev wrote: > >>>> Hi, Andrew, > >>>> > >>>> Thanks for the reply. One more comment. > >>>> > >>>> JR24. "The alignment of other identifiers is by convention (and the Java > >>>> client will follow the convention) rather than mandate." Could you > >>>> describe > >>>> the convention to convert a clientInstanceId (UUID) to a memberId > >>>> (String)? > >>>> > >>>> Jun > >>>> > >>>> > >>>> On Tue, May 19, 2026 at 2:36 AM Andrew Schofield <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi Jun, > >>>>> Thanks for your response. > >>>>> > >>>>> JR23: You are absolutely correct. It seems to me that not sending a > >>>>> clientInstanceId in the header and explicitly sending a zero UUID as the > >>>>> clientInstanceId in the header can be treated as semantically > >>>>> equivalent. > >>>>> I've tweaked the words slightly. > >>>>> > >>>>> Thanks, > >>>>> Andrew > >>>>> > >>>>> On 2026/05/19 03:42:16 Jun Rao via dev wrote: > >>>>>> Hi, Andrew, > >>>>>> > >>>>>> Thanks for the reply. > >>>>>> > >>>>>> JR23. Our message protocol doc says "Any fields in the message object > >>>>> that > >>>>>> are not present in the version that you are deserializing will be reset > >>>>> to > >>>>>> default values. Unless a custom default has been set:". Uuid fields > >>>>>> default to zero uuid. > >>>>>> So if the server gets header.clientInstanceId=0 in the deserialized > >>>>> header, > >>>>>> could it distinguish between the ID not being present (since client is > >>>>> old) > >>>>>> and the ID being explicitly set to 0 by the client? > >>>>>> > >>>>>> Jun > >>>>>> > >>>>>> > >>>>>> On Mon, May 18, 2026 at 7:45 PM Andrew Schofield > >>>>>> <[email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Jun, > >>>>>>> Thanks for your reply. It's tricky squaring a circle. > >>>>>>> > >>>>>>> JR23: For GetTelemetrySubscriptions, I have changed it so that a > >>>>>>> client > >>>>>>> which omits the ClientInstanceId from the request header is permitted > >>>>> to > >>>>>>> specify a zero ClientInstanceId in the request body, following > >>>>>>> original > >>>>>>> KIP-714 precedent. However, a client which specifies a > >>>>> ClientInstanceId in > >>>>>>> the request header MUST specify the same ClientInstanceId in the > >>>>> request > >>>>>>> body. This ensures that the header and telemetry UUIDs are the same. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Andrew > >>>>>>> > >>>>>>> On 2026/05/12 17:48:23 Andrew Schofield wrote: > >>>>>>>> Hi Jun, > >>>>>>>> Thanks for the reply and digging into the details. > >>>>>>>> > >>>>>>>> JR23: Correct. The client telemetry component will use UUID-B as the > >>>>>>> client instance ID. > >>>>>>>> > >>>>>>>> JR23.1: Yes, I agree. It's not ideal. When I was drawing up the > >>>>> tables, > >>>>>>> I was thinking that this might be a possibility, but I'm less > >>>>>>> convinced > >>>>>>> now. I think that I should mandate that if a client specifies > >>>>>>> header.ClientInstanceId on GetTelemetrySubscriptions request, then > >>>>>>> request.ClientInstanceId must either be zero or equal to > >>>>>>> header.ClientInstanceId. > >>>>>>>> > >>>>>>>> JR23.2: This is perhaps the interesting one. From its original > >>>>> intent, > >>>>>>> it should be UUID-B (the telemetry UUID), but then that contradicts > >>>>>>> the > >>>>>>> change in signature to remove the timeout. Unless I make the change > >>>>> above, > >>>>>>> in which case it will be UUID-H. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Andrew > >>>>>>>> > >>>>>>>> On 2026/05/12 17:23:58 Jun Rao via dev wrote: > >>>>>>>>> Hi, Andrew, > >>>>>>>>> > >>>>>>>>> Thanks for the reply. > >>>>>>>>> > >>>>>>>>> JR23. In the new client -> old broker case, we have > >>>>>>>>> header.ClientInstanceId=UUID-H > >>>>>>>>> request.ClientInstanceId=UUID-B > >>>>>>>>> response.ClientInstanceId=0 > >>>>>>>>> > >>>>>>>>> On the server side, I guess the telemetry component will use > >>>>> UUID-B as > >>>>>>> the > >>>>>>>>> clientInstanceId? This has a couple of implications. > >>>>>>>>> JR23.1 On the server side, we have two different clientInstanceIds > >>>>>>> used in > >>>>>>>>> different places, UUID-H for request logging and UUID-B in > >>>>> telemetry. > >>>>>>> This > >>>>>>>>> seems confusing since we can't uniquely identify a client on the > >>>>> server > >>>>>>>>> side. > >>>>>>>>> JR23.2 On the client side. what uuid does clientInstanceId(Duration > >>>>>>>>> timeout) return? If it returns UUID-H, it will be confusing since > >>>>> it > >>>>>>>>> doesn't match the ID used for telemetry on the server. > >>>>>>>>> > >>>>>>>>> Jun > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, May 12, 2026 at 12:58 AM Andrew Schofield < > >>>>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Jun, > >>>>>>>>>> Thanks for your response. > >>>>>>>>>> > >>>>>>>>>> JR20: I have improved (I hope) the wording. The client sends > >>>>>>>>>> request.clientInstanceId = 0 and header.clientInstanceId = > >>>>> UUID-H, > >>>>>>> and the > >>>>>>>>>> broker responds response.clientInstanceId=UUID-H. In this way, > >>>>> the > >>>>>>> broker > >>>>>>>>>> will have taken the UUID-H from the header, and told the client > >>>>> to > >>>>>>> use it > >>>>>>>>>> for client telemetry also. > >>>>>>>>>> > >>>>>>>>>> JR21: Done. Look for "henceforth". > >>>>>>>>>> > >>>>>>>>>> JR22: Summary table added. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Andrew > >>>>>>>>>> > >>>>>>>>>> On 2026/05/11 19:18:24 Jun Rao via dev wrote: > >>>>>>>>>>> Hi, Andrew, > >>>>>>>>>>> > >>>>>>>>>>> Thanks for the reply. > >>>>>>>>>>> > >>>>>>>>>>> JR20. "If the client requests a new client instance ID on its > >>>>>>> initial > >>>>>>>>>>> GetTelemetrySubscriptions request and it sends a client > >>>>> instance > >>>>>>> ID in > >>>>>>>>>> the > >>>>>>>>>>> request header, the broker will send back that client instance > >>>>> ID > >>>>>>> rather > >>>>>>>>>>> than generating a new UUID. This will automatically align the > >>>>> UUID > >>>>>>> in the > >>>>>>>>>>> request headers and client telemetry." > >>>>>>>>>>> > >>>>>>>>>>> This seems inconsistent with what's in the table. In the > >>>>> table, for > >>>>>>>>>>> example, if the client has the following: > >>>>>>>>>>> GetTelemetrySubscriptions v0 > >>>>>>>>>>> header.ClientInstanceId = UUID-H > >>>>>>>>>>> request.ClientInstanceId = UUID-H > >>>>>>>>>>> > >>>>>>>>>>> or > >>>>>>>>>>> > >>>>>>>>>>> GetTelemetrySubscriptions v0 > >>>>>>>>>>> header.ClientInstanceId = UUID-H > >>>>>>>>>>> request.ClientInstanceId = UUID-R > >>>>>>>>>>> > >>>>>>>>>>> the broker returns > >>>>>>>>>>> response.ClientInstanceId = 0. > >>>>>>>>>>> > >>>>>>>>>>> JR21. It will be useful to document what the new client does > >>>>> with > >>>>>>> the > >>>>>>>>>>> returned response.ClientInstanceId. Note that return value may > >>>>> or > >>>>>>> may not > >>>>>>>>>>> be 0. > >>>>>>>>>>> > >>>>>>>>>>> JR22. It's probably clearer if we could populate the table > >>>>> with 4 > >>>>>>>>>>> combinations: old/new clients with old/new brokers. > >>>>>>>>>>> > >>>>>>>>>>> Jun > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Fri, May 8, 2026 at 2:49 AM Andrew Schofield < > >>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Jun and Chia-Ping, > >>>>>>>>>>>> I've overhauled part of the KIP to do with alignment of the > >>>>>>> request > >>>>>>>>>> header > >>>>>>>>>>>> client instance ID, client telemetry client instance ID and > >>>>> group > >>>>>>>>>> protocol > >>>>>>>>>>>> member IDs. The alignment is by convention, not mandate > >>>>> (SHOULD > >>>>>>> not > >>>>>>>>>> MUST). > >>>>>>>>>>>> > >>>>>>>>>>>> It would be possible to go around the existing RPCs such as > >>>>>>>>>>>> ConsumerGroupHeartbeat and GetTelemetrySubscriptions, and > >>>>> remove > >>>>>>> the > >>>>>>>>>> fields > >>>>>>>>>>>> containing the existing identifiers which are intended to be > >>>>>>> aligned. > >>>>>>>>>> Doing > >>>>>>>>>>>> so would be a bad idea though, because we would then have RPC > >>>>>>> versions > >>>>>>>>>>>> which essentially depend upon the presence of a tagged field > >>>>> in > >>>>>>> the > >>>>>>>>>> request > >>>>>>>>>>>> header. This is a protocol-compatibility nightmare. > >>>>>>>>>>>> > >>>>>>>>>>>> I have removed the new versions of GetTelemetrySubscriptions > >>>>> and > >>>>>>>>>>>> PushTelemetry. I have also explained the behavior of > >>>>>>>>>>>> GetTelemetrySubscriptions in the presence and absence of a > >>>>> client > >>>>>>>>>> instance > >>>>>>>>>>>> ID in the request header. > >>>>>>>>>>>> > >>>>>>>>>>>> Let me know what you think. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Andrew > >>>>>>>>>>>> > >>>>>>>>>>>> On 2026/05/07 15:09:31 Andrew Schofield wrote: > >>>>>>>>>>>>> Hi Jun and Chia-Ping, > >>>>>>>>>>>>> I've been thinking and discussing the changes to the > >>>>> KIP-714 > >>>>>>> RPCs. > >>>>>>>>>> There > >>>>>>>>>>>> are too many combinations for my liking at the moment. I > >>>>> want to > >>>>>>> take > >>>>>>>>>>>> another pass at this area and will make an update in a few > >>>>> days. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I intend to start a new vote once we have consensus because > >>>>>>> the spec > >>>>>>>>>> has > >>>>>>>>>>>> changed somewhat since the earliest votes. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Andrew > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 2026/05/06 17:28:27 Chia-Ping Tsai wrote: > >>>>>>>>>>>>>> hi Andrew > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> chia_0: If the consensus is to remove the "duplicate" > >>>>> field > >>>>>>> from > >>>>>>>>>> the > >>>>>>>>>>>> RPC payloads, the tagged field in the header will essentially > >>>>>>> become a > >>>>>>>>>>>> required field. This means the broker needs to handle the > >>>>> edge > >>>>>>> case > >>>>>>>>>> where > >>>>>>>>>>>> both the header and the request body have no > >>>>> ClientInstanceId, > >>>>>>> right? > >>>>>>>>>> If > >>>>>>>>>>>> so, would you mind clarifying the expected broker behavior in > >>>>>>> the KIP? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2026/04/03 16:17:37 Andrew Schofield wrote: > >>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>> I would like to start the discussion on KIP-1313. This > >>>>>>> adds a > >>>>>>>>>> unique > >>>>>>>>>>>> client instance ID to the request header of all Kafka > >>>>> protocol > >>>>>>>>>> requests to > >>>>>>>>>>>> give a unique identifier which can be used to correlate the > >>>>>>> requests > >>>>>>>>>> from > >>>>>>>>>>>> each client for the purposes of problem determination. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$ > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > >> > >
