Thanks Andrew. I agree to the issue that tagged fields are optional.
Nevertheless, I find the proposed compatibility protocol too complex and
unnecessary, and would still prefer a version bump for
`GetTelemetrySubscriptionsRequest/Reponse` to clean it all up. -- For
`PushTelemetryRequest` the argument from above applies, so leaving it
untouched is ok.
I don't see why `GetTelemetrySubscriptionsRequest` would need to encode
the `clientInstandId` in it's request body. Given that the
`clientInstanceId` is a random UUID, the returned subscription cannot
really depend on it. So why would we need to send it to the broker?
Right now, it's set to ZERO only for the purpose to get the broker to
assign a new UUID. But it has nothing to do with the actual "what
metrics should I send" question.
To be fair, KIP-714 lists `client_instance_id` as a field that can be
used to define a subscription. But it seems rather useless in practice
to me? I a user defines a subscription, they cannot know the UUID. Thus,
I think we should actually drop supporting `client_instance_id` as a
"subscription matching parameter"? Of course, there is a backward
compatibility question, but I think we can address this: if brokers are
upgraded and they have an existing `client_instance_id` based
subscription defined, they could advertise to only support v0, and they
should log a warning that this feature is deprecated. New brokers would
also not allow to use `client_instance_id` any longer to define a
subscription.
Similarly, for `GetTelemetrySubscriptionsResponse`, we only need
`clientInstanceId` if the broker computes the UUID. But if we let the
client compute it, this field is not needed any longer.
-Matthias
On 5/28/26 1:21 AM, Andrew Schofield wrote:
Hi Jun,
Thanks for your response.
JR24: I'll update the KIP shortly. Here's a summary. The various modern group
protocol heartbeat requests encode the member ID as a string for historical
reasons, even though it's in practice a UUID. The Apache Kafka Java client
happens to use org.apache.kafka.common.Uuid to create the member ID and convert
it into a string. This means that when the clientInstanceId is converted into a
string, its encoding matches the member ID and it all lines up. However, there
are other ways to encode a UUID into a string and I know for a fact that the
librdkafka client does it differently.
Thanks,
Andrew
On 2026/05/27 17:11:25 Jun Rao via dev wrote:
Hi, Andrew,
Thanks for the reply. One more comment.
JR24. "The alignment of other identifiers is by convention (and the Java
client will follow the convention) rather than mandate." Could you describe
the convention to convert a clientInstanceId (UUID) to a memberId (String)?
Jun
On Tue, May 19, 2026 at 2:36 AM Andrew Schofield <[email protected]>
wrote:
Hi Jun,
Thanks for your response.
JR23: You are absolutely correct. It seems to me that not sending a
clientInstanceId in the header and explicitly sending a zero UUID as the
clientInstanceId in the header can be treated as semantically equivalent.
I've tweaked the words slightly.
Thanks,
Andrew
On 2026/05/19 03:42:16 Jun Rao via dev wrote:
Hi, Andrew,
Thanks for the reply.
JR23. Our message protocol doc says "Any fields in the message object
that
are not present in the version that you are deserializing will be reset
to
default values. Unless a custom default has been set:". Uuid fields
default to zero uuid.
So if the server gets header.clientInstanceId=0 in the deserialized
header,
could it distinguish between the ID not being present (since client is
old)
and the ID being explicitly set to 0 by the client?
Jun
On Mon, May 18, 2026 at 7:45 PM Andrew Schofield <[email protected]>
wrote:
Hi Jun,
Thanks for your reply. It's tricky squaring a circle.
JR23: For GetTelemetrySubscriptions, I have changed it so that a client
which omits the ClientInstanceId from the request header is permitted
to
specify a zero ClientInstanceId in the request body, following original
KIP-714 precedent. However, a client which specifies a
ClientInstanceId in
the request header MUST specify the same ClientInstanceId in the
request
body. This ensures that the header and telemetry UUIDs are the same.
Thanks,
Andrew
On 2026/05/12 17:48:23 Andrew Schofield wrote:
Hi Jun,
Thanks for the reply and digging into the details.
JR23: Correct. The client telemetry component will use UUID-B as the
client instance ID.
JR23.1: Yes, I agree. It's not ideal. When I was drawing up the
tables,
I was thinking that this might be a possibility, but I'm less convinced
now. I think that I should mandate that if a client specifies
header.ClientInstanceId on GetTelemetrySubscriptions request, then
request.ClientInstanceId must either be zero or equal to
header.ClientInstanceId.
JR23.2: This is perhaps the interesting one. From its original
intent,
it should be UUID-B (the telemetry UUID), but then that contradicts the
change in signature to remove the timeout. Unless I make the change
above,
in which case it will be UUID-H.
Thanks,
Andrew
On 2026/05/12 17:23:58 Jun Rao via dev wrote:
Hi, Andrew,
Thanks for the reply.
JR23. In the new client -> old broker case, we have
header.ClientInstanceId=UUID-H
request.ClientInstanceId=UUID-B
response.ClientInstanceId=0
On the server side, I guess the telemetry component will use
UUID-B as
the
clientInstanceId? This has a couple of implications.
JR23.1 On the server side, we have two different clientInstanceIds
used in
different places, UUID-H for request logging and UUID-B in
telemetry.
This
seems confusing since we can't uniquely identify a client on the
server
side.
JR23.2 On the client side. what uuid does clientInstanceId(Duration
timeout) return? If it returns UUID-H, it will be confusing since
it
doesn't match the ID used for telemetry on the server.
Jun
On Tue, May 12, 2026 at 12:58 AM Andrew Schofield <
[email protected]>
wrote:
Hi Jun,
Thanks for your response.
JR20: I have improved (I hope) the wording. The client sends
request.clientInstanceId = 0 and header.clientInstanceId =
UUID-H,
and the
broker responds response.clientInstanceId=UUID-H. In this way,
the
broker
will have taken the UUID-H from the header, and told the client
to
use it
for client telemetry also.
JR21: Done. Look for "henceforth".
JR22: Summary table added.
Thanks,
Andrew
On 2026/05/11 19:18:24 Jun Rao via dev wrote:
Hi, Andrew,
Thanks for the reply.
JR20. "If the client requests a new client instance ID on its
initial
GetTelemetrySubscriptions request and it sends a client
instance
ID in
the
request header, the broker will send back that client instance
ID
rather
than generating a new UUID. This will automatically align the
UUID
in the
request headers and client telemetry."
This seems inconsistent with what's in the table. In the
table, for
example, if the client has the following:
GetTelemetrySubscriptions v0
header.ClientInstanceId = UUID-H
request.ClientInstanceId = UUID-H
or
GetTelemetrySubscriptions v0
header.ClientInstanceId = UUID-H
request.ClientInstanceId = UUID-R
the broker returns
response.ClientInstanceId = 0.
JR21. It will be useful to document what the new client does
with
the
returned response.ClientInstanceId. Note that return value may
or
may not
be 0.
JR22. It's probably clearer if we could populate the table
with 4
combinations: old/new clients with old/new brokers.
Jun
On Fri, May 8, 2026 at 2:49 AM Andrew Schofield <
[email protected]>
wrote:
Hi Jun and Chia-Ping,
I've overhauled part of the KIP to do with alignment of the
request
header
client instance ID, client telemetry client instance ID and
group
protocol
member IDs. The alignment is by convention, not mandate
(SHOULD
not
MUST).
It would be possible to go around the existing RPCs such as
ConsumerGroupHeartbeat and GetTelemetrySubscriptions, and
remove
the
fields
containing the existing identifiers which are intended to be
aligned.
Doing
so would be a bad idea though, because we would then have RPC
versions
which essentially depend upon the presence of a tagged field
in
the
request
header. This is a protocol-compatibility nightmare.
I have removed the new versions of GetTelemetrySubscriptions
and
PushTelemetry. I have also explained the behavior of
GetTelemetrySubscriptions in the presence and absence of a
client
instance
ID in the request header.
Let me know what you think.
Thanks,
Andrew
On 2026/05/07 15:09:31 Andrew Schofield wrote:
Hi Jun and Chia-Ping,
I've been thinking and discussing the changes to the
KIP-714
RPCs.
There
are too many combinations for my liking at the moment. I
want to
take
another pass at this area and will make an update in a few
days.
I intend to start a new vote once we have consensus because
the spec
has
changed somewhat since the earliest votes.
Thanks,
Andrew
On 2026/05/06 17:28:27 Chia-Ping Tsai wrote:
hi Andrew
chia_0: If the consensus is to remove the "duplicate"
field
from
the
RPC payloads, the tagged field in the header will essentially
become a
required field. This means the broker needs to handle the
edge
case
where
both the header and the request body have no
ClientInstanceId,
right?
If
so, would you mind clarifying the expected broker behavior in
the KIP?
Best,
Chia-Ping
On 2026/04/03 16:17:37 Andrew Schofield wrote:
Hi,
I would like to start the discussion on KIP-1313. This
adds a
unique
client instance ID to the request header of all Kafka
protocol
requests to
give a unique identifier which can be used to correlate the
requests
from
each client for the purposes of problem determination.
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$
Thanks,
Andrew
