Re: [DISCUSS] KIP-1368: Client framework name and version

[Matthias J. Sax]

Hello Andrew,

seems the KIP did make significant progress. Nice! -- Overall LGTM, even 
if it seems there is still some stuff we need to work our for Connect; I 
am not super familiar with Connect details but it seems Chris might have 
a point about worker IDs? Not sure about connect tasks though, which 
seems to be too low level? Chris, correct me if I am wrong, but a single 
consumer/producer would be shared across multiple sink/source tasks, 
right? If there is not a 1:1 mapping between task and client it seems 
task information would not be something we want to add (KS also has 
task, and a 1:N mapping to clients, and I don't see any advantage to add 
KS-task information to a connection -- the assigned tasks can also 
change during the live time of a connect for KS, and I would assume it's 
the same for Connect?)



Bunch of nits:


The broker limits the length (249 characters)

This makes sense, but could imply that long package names cannot be 
fully encoded. While it's more of an framework impl detail, I am 
wondering if we should also talk about abbreviations, like 
o.a.k.connect.<someMoreNestePackagesWithLongNames>

Just bringing it up, as the KIP also prescribes how client.framework.id 
should be used in general, so covering this part could also make sense?



The KIP adds two tagged fields to v6.

Three, not two, right?



The maximum length of the fields is 249 characters

Should this be "each field" instead of "the fields" which reads as "the 
sum of all field should not exceed 249 chars"?





In the "Clients configuration" section, ie, the table listing the three 
new configs, importance level for client.framework.id is missing. Should 
be LOW.



if specified, must match the regular expression

What is the purpose to enforce such a pattern? My understanding was, 
that such an encoding is a recommendation only, not something the KIP 
dictates?



We should describe how Kafka Connect and Kafka Streams react, if one of 
the new configs is specified by the user. The KIP only say very high 
level that

If a framework provides a way for the application to set its own Kafka configs, 
the framework's own values for the client framework name, version and ID take 
precedence over the values the application sets for these configs; they are 
provided to be used by a framework.

But this statement is generic, and not limited to Connect and Streams, 
and instructions how it _should_ work for external frameworks.

For Connect and Streams we should either refuse to startup, ie, Connect 
worker throws ConfigException, deployment of a connector fails (not sure 
if applicable?), and `StreamsConfig` throws, or we ignore the config and 
only log a warning about it. Personally, I think throwing an exception 
might be better.




About Chris comment:

I wonder if there could be a sort of middle ground between "client" and
"framework" that would be a good fit for connector type/version--what do
you think about "application", as in "client.application.name" and
"client.application.version"? The Kafka Streams equivalents for these would
be the existing "application.id" property and a new "application.version"
property. This could also be independently useful for Kafka Streams
developers who want to track rollouts that update Streams configs and/or
topologies.

I see his point about Connect, but I am not sure right now, if/how it 
would apply to Streams and if it would indeed be helpful or not... 
Didn't form a full opinion on it yet, but will monitor the discussion 
and follow up as needed.




-Matthias


On 10/5/26 4:36 PM, Chris Egerton wrote:
Hi Andrew,

I'm not sure about using the instance ID to report connector type and
version. Concatenating the two and putting them in a single field removes
structure and forces extra work on users who want to, e.g., view aggregate
metrics for all versions of a connector. In addition, using it to
identify unique Kafka Streams processes makes sense, but (to me at least)
that means the intuitive use for it by Kafka Connect would be to report the
worker ID (i.e., the advertised URL) or the exact entity that the client is
used for (connector name for connectors, task ID for tasks). What do you
think about using the instance ID field to report the Connect worker ID
instead, since the connector name/task ID are already set in the default
client IDs used by Kafka Connect?

I wonder if there could be a sort of middle ground between "client" and
"framework" that would be a good fit for connector type/version--what do
you think about "application", as in "client.application.name" and
"client.application.version"? The Kafka Streams equivalents for these would
be the existing "application.id" property and a new "application.version"
property. This could also be independently useful for Kafka Streams
developers who want to track rollouts that update Streams configs and/or
topologies.

Also, I think that the Kafka Connect runtime should disallow setting any of
these properties, either in the worker config or in a connector config.

Cheers,

Chris

On Mon, Oct 5, 2026, 17:36 Andrew Schofield <schofie...@apache.org> wrote:

Hi Lianet,
Thanks for your reply. I have posted an updated version of the KIP which
adds a third config client.framework.id which a framework can use to set
an additional framework-specific ID. For Kafka Streams, it's the process
ID which can be used to group together connections which are part of the
same logical application. For Kafka Connect, I suggest that the
connector class and version are used. The validation rules for the
framework ID are slightly different because the ID is derived from
sources which the user does not directly control (such as the version
string from a Kafka Connect connector).

LM1: The ClientTelemetryContext interface is implemented by the broker
with the intent that the metrics plugin can use it to obtain the context
it needs to annotate the metrics. As such, there is no need to provide
default implementations of the new methods.

LM2: If an app is setting the configs intended for the framework to set
the framework's values win.

LM3: The new framework ID has a more lenient regex and the KIP describes
how Kafka Connect forms the framework ID from the class name and the
connector version.

I think this update should answer most of the questions, and probably
spark some more.

Thanks,
Andrew

On 2026/10/02 20:17:23 Lianet Magrans wrote:
Hi Andrew, thanks for the replies.

Agreed on the override policy as you describe it (the framework owns
the config, not the apps/connectors).

Following that ownership pov, I would like to bring back the Kafka
Streams case and the missing bit: identifying the instance of the
framework. For KS it would be valuable to know not only the framework
name but also the framework instance behind a client connection, so we
can reason about all the connections coming from one Streams instance
(embedded consumers/producer/admin). Streams already has a UUID for
this, the process id, and already sends it to the broker, but only on
individual paths (e.g, StreamsGroupHeartbeat). The process id is also
part of the default client.id, so the identifier is already there, but
this is where your ownership point matters: client.id is owned
by the user app, who can change it freely, vs. a framework id that
would be owned by the framework (just as the name and version). This
would be optional for the frameworks to set of course, just pointing
out that it's an extension to this intention of identifying the
framework, that seems relevant for KS. What do you think?

Thanks!
Lianet


On Tue, Sep 29, 2026 at 3:28 PM Andrew Schofield <schofie...@apache.org>
wrote:

Hi Lianet,
Thanks for your response.

LM3: I'm opposed to arbitrary concatenation because it makes it
impossible
to do sensible matching and the like. If we did the following, that
would be
fine:

* Kafka Connect uses
org.apache.kafka.connect+your.connector.class.name.here
for framework name, and x.y.z+a.b.c for framework version.
* Explicitly prevent overriding of these configs by the connectors.
Kafka
Connect owns the configs, not the connectors.

Then we get flexibility and sanity together. Maybe that's the best way
forward.

I'll see whether consensus emerges before updating the KIP.

Thanks,
Andrew

On 2026/09/29 19:06:48 Lianet Magrans wrote:
One more detail (again related to Chris' comment):

LM3:  if we want to leave the door open for concatenations, we relax
the validations to allow + (Chris used the connect example "client
framework of
"org.apache.kafka.connect+com.acme.FooConnector" and a version of
"4.4.0+1.2.3"), but something like that wouldn't be allowed with the
current regex.

Thanks,
Lianet

On Tue, Sep 29, 2026 at 2:50 PM Lianet Magrans <lian...@apache.org>
wrote:

Hi Andrew,

LM1: Just checking that it's intentional that the new methods
added to
the ClientTelemetryContext interface have no default
implementation?
It will break implementations but I guess we're relying on the fact
that implementations live in the broker itself?

LM2: related to Chris' comments and the overrides, if we have an
app
setting the config and the framwork setting it internally too,
seems
we're saying the user config wins?

Thanks!
Lianet

On Tue, Sep 29, 2026 at 2:26 PM Andrew Schofield <
schofie...@apache.org> wrote:

Hi Chris,
Thanks for your message.

1. I don't want to pander to people who want to concatenate
identifiers.
We could have ConnectorName and ConnectorVersion fields along
the same
lines as the SoftwareFrameworkName and SoftwareFrameworkVersion.
That
would be a cleaner way of doing it. Maybe we can come up with a
concrete
proposal easily, but the risk is that we end up adding a heap of
other
Connect concepts too and that's way beyond the original intent
of the KIP.
Let me know what you think.

2. There is nothing to prevent a connector config overriding
these
configs. Do you think this is something thet should be prevented
to
dissuade the kind of shenanigans that you illustrated with
concatenation?

Thanks,
Andrew

On 2026/09/28 21:57:43 Chris Egerton wrote:
Hi,

Forgive me if this is already addressed; there's a lot of
context here and
I haven't read the entire thread. I think the section on Kafka
Connect is a
little thin at the moment.

1. The type and version of the connector is important
information and it'd
be nice if there were a way to include that. This could be
done by
embedding the relevant information inside the strings for the
proposed
properties; e.g., a client framework of
"org.apache.kafka.connect+com.acme.FooConnector" and a version
of
"4.4.0+1.2.3" (where the Kafka Connect runtime version is
first and the
connector's version is second); however, that feels like a
hack (a
sentiment Andrew seems to share given this remark: "We could
support
concatenation of framework information, but that's just
pandering to these
people.").

2. Would users be allowed to override either the framework or
version by
explicitly setting values for them in a connector config,
e.g., "
producer.override.client.framework.name"?

Cheers,

Chris

On Mon, Sep 28, 2026 at 5:33 PM Lianet Magrans <
lian...@apache.org> wrote:

Ok, good that the concept was already considered. I will
bring it up for
discussion soon with KIP-1324 then.

Thanks!
Lianet

On Mon, Sep 28, 2026, 2:18 p.m. Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Lianet,
Thanks for your response.

I did discuss something similar with Matthias but decided
that we weren't
quite ready to commit to a specific behaviour in this KIP.
I suggest that
you include it in KIP-1324. There's a strong chance that
KIPs 1313, 1368
and 1324 all land in the same 4.5 release, and that will
mean the RPC
changes can be coalesced into the same version bump.

Thanks,
Andrew

On 2026/09/28 17:38:12 Lianet Magrans wrote:
Hi Andrew, just one question here, did we consider
identifying not just
the
framework type and version, but also the "framework
instance"? Similar
to
what KIP-1313 did for clients (UUID for
consumer/prodcuer/admin), but
closing the loop to have a way to identify instances of
this new
"logical
client" which is the framework.

That would be useful the moment we have several
instances of the
"framework" running together I expect, e.g., several
streams instances,
KIP-1313 would give us UUIDs for all the
consumers/producer/admins
running
underneath, and KIP-1368 tell us they are behind a
"logical
client"/framework" ("streams"), but no way to tell here
one of the
"streams" framwork instances from the other one. Streams
already has a
process ID that fills that gap, the point to consider is
if it's
something
we would want along with the framework type.

I'm thinking about this coming from KIP-1324 btw
(pushing configs from
the
client to the broker). For features like that, seems
useful to identify
not
only the framework type, but also the "framework
instance".  I'm happy
to
consider it as part of KIP-1324 (that I plan to resume
soon), but
raising
it for consideration, as it's closely related to the
framework type.

Thanks!
Lianet

On Thu, Sep 24, 2026 at 2:45 PM Jun Rao via dev <
dev@kafka.apache.org>
wrote:

Hi, Andrew,

Thanks for the updated KIP. LGTM

Jun

On Thu, Sep 24, 2026 at 10:58 AM Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Jun,

JR7: Done.

Thanks,
Andrew

On 2026/09/24 16:20:08 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the updated KIP.

JR7. Piggybacking on the version bump in 4.5
sounds good to me.
Could
we
document the version bump in the KIP?

Jun

On Thu, Sep 24, 2026 at 7:32 AM Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for your reply.

JR7: There is a tension here between bumping the
version when
adding
these
tagged fields because it is a new feature and
not doing so
because of
the
negotiation aspects of this. I believe that AK
4.5 will
introduce a
new
version of ApiVersions anyway because of
KIP-1313, so the best
path
is
probably to piggyback the tagged field addition
onto that new
version.

JR8.1: I've added this constraint in AK 5.0
since we need a
major
version
to enforce this. In practice, nobody will be hit
by this limit.
JR8.2: Yes, constraint added to the configs too.

Thanks,
Andrew

On 2026/09/24 00:08:16 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the updated KIP.

JR7. Currently, we always bump the version
when adding a
tagged
field.
The
intention of this KIP is to add the two tagged
fields
without a
version
bump. This choice seems reasonable since it
avoids an extra
round
of
ApiVersionRequest version negotiation when 4.5
clients talk
to
4.4
servers. It
would be useful to be explicit document there
is not version
bump
for
ApiVersionRequest.

JR8. "The maximum length of the fields is 249
characters"
JR8.1 Should the length constraint also apply
to the existing
fields
ClientSoftwareName and ClientSoftwareVersion?
JR8.2 Should we add a length constraint for
the two new
client
configurations?

Jun

On Wed, Sep 23, 2026 at 6:42 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for the reply.

JR6: Yes, that works. I've added the methods
to
ClientTelemetryContext.
Let me know what you think.

Thanks,
Andrew

On 2026/09/21 17:54:38 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the reply.

JR6. It seems weird that the
implementation of a KIP-714
plugin
requires
casting an authorizableRequestContext to
an internal
class
RequestContext.
Would it be better to expose all client
related resource
labels
through a
public API in ClientTelemetryContext?

Jun

On Mon, Sep 21, 2026 at 6:43 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for the reply.

JR4: Done. Producer, consumer and admin.

JR5: Yes. I've reworded slightly.

JR6: I think the plugin implementation
already needs to
do
something
similar to get to software name/version.

ClientTelemetryContext.authorizableRequestContext()
returns
an
instance of
AuthorizableRequestContext. The
o.a.k.common.requests.RequestContext
class
implements that interface, and it also
lets you get to
ClientInformation
and that's where you can find the
software name/version
and
framework
name/version. You can see this in

o.a.k.server.metrics.ClientMetricsInstanceMetadata. I
would
describe
this
as grubby :)

One way forward here would be to remove
the
broker-added
resource
labels
from KIP-1368, but leave the additions
to the match
criteria.
Then
it
would
still be possible to specify, for
example, that
particular
metrics
should
be captured for Kafka Streams clients
only, without
adding to
the
grubbiness.

Thanks,
Andrew

On 2026/09/15 20:44:06 Jun Rao via dev
wrote:
Hi, Andrew,

Thanks for the reply. A few more
comments.

JR4. "The configuration keys are
defined in

org.apache.kafka.clients.CommonClientConfigs"
Could you explicitly list the clients
(producer,
consumer,
AdminClient,
etc.) that define the two new configs?

JR5. Should AK frameworks such as
kstream and connect
set
client.framework.name and
client.framework.version
in
their
clients
automatically?

JR6. "Broker-added resources labels
for client
metrics"
Are the new labels added in
DefaultClientTelemetryContext.RequestContext?
Since
ClientTelemetryExporter.exportMetrics() only
takes
ClientTelemetryContext, does the
implementor need to
cast
it
to DefaultClientTelemetryContext to
retrieve the new
labels?

Jun

On Fri, Sep 11, 2026 at 12:48 PM
Andrew Schofield <
schofie...@apache.org

wrote:

Hi Jun,
JR1: I dug into the KIP-714 client
metrics stuff a
bit
more
and I
have
revised the KIP. I have added
client-framework-name
and
client-framework-version to the set
of keys in the
matching for
client
metrics, and also added these
broker-added metrics
tags.
This
is
how
these
pieces of information would have
been incorporated
into
KIP_714 had
they
been part of Kafka at that point.

Thanks,
Andrew

On 2026/08/04 20:29:19 Andrew
Schofield wrote:
Hi Jun,
Thanks for the response.

JR1: My intent here is only
request logging. It
would
be
possible to
add
client-framework-name and
client-framework-version
to the
match
criteria
for client metrics, which could
conceivably permit
scenarios
such
as
capturing metrics from specific
versions of the
Spring
framework.
This
is
beyond the scope I have in mind.

JR3: Thanks for clarifying the
tagged field
conventions.
Version
bump
reverted.

Thanks,
Andrew

On 2026/08/04 18:38:40 Jun Rao via
dev wrote:
Hi, Andrew,

Thanks for the reply.

JR1. ClientSoftwareName and
ClientSoftwareVersion are
used
in
metric
names
and match predicates for
configuring client
metrics.
Are
ClientFrameworkName
and ClientFrameworkVersion used
in those places
too
or
are
they
only
used
in request logging?

JR3. The general rule for adding
a new field in
RPC
is
that
(1) if
it's
truly optional, we add it as a
tagged field
with
no
version
bump;
(2)
otherwise, we add it as a
non-tagged field
with a
version
bump. One
exception is when adding a
non-optional field
in
the
request
header.
Because of the limitation in
existing
implementation,
we
need
to
add
it as
a tagged field with a version
bump.

Jun

On Tue, Aug 4, 2026 at 6:04 AM
Andrew
Schofield <
schofie...@apache.org

wrote:

Hi Jun,
Thanks for your response.

JR1: I've expanded the
motivation section in
the
KIP.
They
are
truly
optional, I feel. When
diagnosing an issue
using
the
client
logs,
having
this information can
illuminate why the
application
is
behaving
in a
particular way. If the
application is using a
framework,
behaviours
such as
retries will typically not be
in the
application
code
itself
because
they're implemented in the
framework.
There's a
difference
between
what the
application developer thinks
their code does
and
what we
see
in
the
logs.
That's the point.

JR2: Done. They're
Type.STRING, default
null. I
think
they
should be
importance LOW because that
affects the
prominence
of the
configs in
the
documentation, but I wonder
whether you
agree.

Thanks,
Andrew

On 2026/08/03 21:39:06 Jun Rao
via dev wrote:
Hi, Andrew,

Thanks for the KIP.

JR1. Could you describe the
use cases of
the
two
new
configs
and
are they
truly optional?

JR2. Could you add the type
and the default
value
for
the
new
configs?

Jun


On Fri, Jul 31, 2026 at
10:23 AM Matthias
J.
Sax
<
mj...@apache.org

wrote:

Thanks for the KIP Andrew.
I think it
will
be
very
useful if
we
can
identify frameworks,
especially our own
ones
(Connect and
Streams).

About Aditya's point: I am
wondering
where
to
draw
the
line.
Many
examples seems to be
metadata that
belongs
into
he
Kafka
record
`Headers` at the
application level,
rather
than
the
lower
level
request
headers?

There if of course no
strict logical
dividing
line
between
both.
And
yes, the broker does not
access
application
level
Kafka
record
`Headers`. But if it would
be useful to
let the
broker
tap
into
application level record
`Headers` we
should
tackle
it
independently?

Personally, I don't think
it would be the
right
design to
push
too many
thing into the lower level
request
headers. So
I
am
in
favor
of
only
added the two new propose
`ClientFrameworkName`
and
`ClientFrameworkVersion`
fields.


-Matthias

On 7/31/26 7:56 AM,
Federico Valeri
wrote:
Changes look good.
Thanks.

On Fri, Jul 31, 2026 at
11:17 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Fede,
Thanks for your
response.

FV1: They are set using
the regular
config
properties.
Yes,
it is
possible for an end user
to set arbitrary
values, but
then
that
would
also
be true of a builder or
internal
constructor.
I've
added
a
bit
more
information in the KIP and
beefed up the
config
descriptions
to
discourage
application use.

FV2: I've never
encountered such
nightmares
myself.
The
framework
which
sets the config last would
win. Maybe
this
is a
motivation
for
having a
programmatic way of
setting the
information. We
could
support
concatenation
of framework information,
but that's just
pandering
to
these
people.
Let me
know what you think.

I've also updated the
KIP with a
maximum
length
for
these
pieces of
information since all
identifiers should
have
defined
bounds.

Thanks,
Andrew

On 2026/07/31 08:50:11
Federico Valeri
wrote:
Hi Andrew, the
motivation looks
good. A
couple of
questions:

FV1: The KIP says
frameworks should
set
them,
but it
does not
specify
the mechanism. If
these are ordinary
user-facing
configs,
nothing
prevents an end user
from setting
arbitrary
values,
which
defeats
the
purpose. Should the
framework set
them
programmatically
(builder or
    internal
constructor) or is user
override
intentional?

FV2: What if we have
multiple layers
of
frameworks?
Let's
say a
custom
framework on top of
SpringBoot. I've
seen
similar
nightmares
in the
past.

Thanks
Fede

On Tue, Jul 28, 2026
at 8:27 AM
Aditya
Kousik <
adityakou...@icloud.com>
wrote:

Hi Andrew,

We’re definitely
agreed on the
increased
use of
frameworks
over
than
the client directly. I’ve
cited the
different
patterns
Spring,
Micronaut,
smallrye and
company-internal frameworks
have
APIs
built
on
top
of the
Kafka client, in a couple
of KIPs
already.

The project has
enough traction
that I
think of
it
as
its
own
network
client with distributed
log semantics for
which
frameworks
are
written,
much like gRPC over netty.
People want to
just
write
the
business
logic and
leave the plumbing and
threading to the
frameworks.

All of this to say,
I’m fine to ship
the
client
framework
as the
new
identifying parameter for
frameworks to
set. It
will
be
mighty
useful.

My addendum, rather
than a pushback
is
that:
guilty
as
charged,
I’m
driven by the otel/DD
telemetry/observability
of
using
Apache
Kafka in
applications. The
client.framework.version
config for
instance
can be
used
to detect regressions and
isolate root
causes.
But I
feel it
is
only
one of
many such facets. You
mentioned that you
would
use
client.id
and
clientInstanceId to
identify a client but
that
these
do
not
help
with
aggregate/fleet-wide
issues. As an
example, if
I
tag
an
app’s AZ
it can
help me write alerts on
spike in latency
in
us-west-2.
Or,
detect stuck
partitions across multiple
client.id/application.names
none
of
whom
share
the same
client.framework.name.

Other tags that come
to mind:
application.name,
az,
team,
env,
rack.
All fields users usually
hijack
client.id
for.

An otel JavaAgent can
capture the
client
metadata
registered and
attach it as tags with
each resource
span.
Users
who
use
frameworks but
rely on datadog/otel will
get visibility
into
the
client
metadata for
free.

The KIP as I read it,
serves as a
foundation for
future
use. So I
don’t want to shoehorn a
new behaviour if
it
explodes the
scope
too
much.

Best regards,
Aditya

On Jul 27, 2026, at
13:52, Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Aditya,
Thanks for your
response.

AK1: I chose
framework as the
blessed
abstraction
because
my
focus
was problem determination
for client
applications. We
often
find
that
users
with client problems have
not coded
directly to
the
Kafka
client
interface
because they are using a
framework. As a
result,
their
knowledge
of the
application code is one
level removed
from
the
Kafka
client.
Frameworks can
override configuration
defaults,
introduce
different
retry
behaviour
and so
on. Lots of companies have
their own
internal
frameworks, so
this KIP
can
be used by them too. I'm
trying to make
it
easier to
work out
when a
user
is making use of a
framework and knowing
what
it
is.

Sometimes,
particularly for
non-Java
clients,
people
have
overridden
the
ClientSoftwareName/Version
themselves,
which
makes
those
concepts
much
less useful than they
should be. By
providing
ClientFrameworkName/Version,
there is no longer any
need to do so.
That's
another
motivation
here,
even
though KIPs don't concern
themselves with
non-Java
clients as
such.

We could go for a
more flexible
key-value
approach,
but a
simple
name and version is
sufficient for what I
had
in
mind.
Feel
free
to
push
back with additional
justification and
examples.

AK2: Done.
o.a.k.clients.CommonClientConfigs.

AK3: To identify a
particular
client, I
would
use
client
ID and
client-instance ID. I
think these are
generally
more
useful
concepts
than
the framework name and
version which are
extra
information
for
the
person
trying to figure out why a
client is not
behaving as
expected.

AK4: I'm sure you
have more
experience of
OTel/DataDog
collectors.
You may well be correct
that they would
be
helpful
for
the
collectors.

Thanks,
Andrew

On 2026/07/26
07:48:20 Aditya
Kousik
wrote:
Hello Andrew,

I’m reminded of the
client.id
discussion
we
had
back
in
KIP-1313
re: client instance id.
After that
discussion,
I
have a
WIP
KIP
that
sets a
foundation for shipping
client metadata
tags to
be
sent
for
telemetry.
I
was hoping we could
discuss if part of
that
approach
could
fit
this
KIP.

AK1. I had a note
on the
motivation
of
selecting a
“framework”
as a
blessed abstraction. The
KIP mentions
it’s
for
the
client
metadata and
easier problem diagnosis.
This is akin to
an
“application id”
that
non-framework clients
usually tag with.

KIP-606 took an
approach like
metrics.context.<key>=<val>. If we
allow
client.metadata.<key>=<val>, then a
framework/application
name
and
version can sit in such a
metadata
context
and
be
sent
with
ApiVersion
RPC.
Of course, this is an open
box approach
rather
than
just
the
framework
name/version (just two
fields) we’re
adding to
the
protocol.
But
I’m
curious about the tier of
importance of
framework
alone.

AK2. Can you
clarify if this
config
goes
into
CommonClientConfigs.java,
referenced
across
producer/consumer/share
etc? I
know some share props have
the “share.”
prefix
going.

AK3. In another
thread you
mentioned
that
the
broker
may
add it
to
the request context. In
client logs,
clientId
is
a
very
useful
string
to
identify WARN logs when
things go
sideways
like
broker
disconnected,
rebalance in progress etc.
Can you cherry
pick
and
highlight
some
useful
log places that these
strings can go? I
suppose
adding
clientId/framework
to the MDC context might
be too
voluminous.

AK4. I foresee
collectors like
Datadog/OTel
might
find
these
tags
useful in each span
exported.

Looking forward to
your thoughts
on
this.

Regards,
Aditya

On Jul 17, 2026,
at 11:53,
Andrew
Schofield
<

andrew_schofield_j...@outlook.com>
wrote:

Hi,
I'd like to open
discussion on
KIP-1368:
Client
framework name
and
version.

Applications often
use
application
frameworks
such as
Spring to
connect to Kafka. To
assist with problem
diagnosis,
this
KIP
introduces a
way to provide the
framework name and
version


On Tue, Sep 29, 2026 at 3:28 PM Andrew Schofield <schofie...@apache.org>
wrote:

Hi Lianet,
Thanks for your response.

LM3: I'm opposed to arbitrary concatenation because it makes it
impossible
to do sensible matching and the like. If we did the following, that
would be
fine:

* Kafka Connect uses
org.apache.kafka.connect+your.connector.class.name.here
for framework name, and x.y.z+a.b.c for framework version.
* Explicitly prevent overriding of these configs by the connectors.
Kafka
Connect owns the configs, not the connectors.

Then we get flexibility and sanity together. Maybe that's the best way
forward.

I'll see whether consensus emerges before updating the KIP.

Thanks,
Andrew

On 2026/09/29 19:06:48 Lianet Magrans wrote:
One more detail (again related to Chris' comment):

LM3:  if we want to leave the door open for concatenations, we relax
the validations to allow + (Chris used the connect example "client
framework of
"org.apache.kafka.connect+com.acme.FooConnector" and a version of
"4.4.0+1.2.3"), but something like that wouldn't be allowed with the
current regex.

Thanks,
Lianet

On Tue, Sep 29, 2026 at 2:50 PM Lianet Magrans <lian...@apache.org>
wrote:

Hi Andrew,

LM1: Just checking that it's intentional that the new methods
added to
the ClientTelemetryContext interface have no default
implementation?
It will break implementations but I guess we're relying on the fact
that implementations live in the broker itself?

LM2: related to Chris' comments and the overrides, if we have an
app
setting the config and the framwork setting it internally too,
seems
we're saying the user config wins?

Thanks!
Lianet

On Tue, Sep 29, 2026 at 2:26 PM Andrew Schofield <
schofie...@apache.org> wrote:

Hi Chris,
Thanks for your message.

1. I don't want to pander to people who want to concatenate
identifiers.
We could have ConnectorName and ConnectorVersion fields along
the same
lines as the SoftwareFrameworkName and SoftwareFrameworkVersion.
That
would be a cleaner way of doing it. Maybe we can come up with a
concrete
proposal easily, but the risk is that we end up adding a heap of
other
Connect concepts too and that's way beyond the original intent
of the KIP.
Let me know what you think.

2. There is nothing to prevent a connector config overriding
these
configs. Do you think this is something thet should be prevented
to
dissuade the kind of shenanigans that you illustrated with
concatenation?

Thanks,
Andrew

On 2026/09/28 21:57:43 Chris Egerton wrote:
Hi,

Forgive me if this is already addressed; there's a lot of
context here and
I haven't read the entire thread. I think the section on Kafka
Connect is a
little thin at the moment.

1. The type and version of the connector is important
information and it'd
be nice if there were a way to include that. This could be
done by
embedding the relevant information inside the strings for the
proposed
properties; e.g., a client framework of
"org.apache.kafka.connect+com.acme.FooConnector" and a version
of
"4.4.0+1.2.3" (where the Kafka Connect runtime version is
first and the
connector's version is second); however, that feels like a
hack (a
sentiment Andrew seems to share given this remark: "We could
support
concatenation of framework information, but that's just
pandering to these
people.").

2. Would users be allowed to override either the framework or
version by
explicitly setting values for them in a connector config,
e.g., "
producer.override.client.framework.name"?

Cheers,

Chris

On Mon, Sep 28, 2026 at 5:33 PM Lianet Magrans <
lian...@apache.org> wrote:

Ok, good that the concept was already considered. I will
bring it up for
discussion soon with KIP-1324 then.

Thanks!
Lianet

On Mon, Sep 28, 2026, 2:18 p.m. Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Lianet,
Thanks for your response.

I did discuss something similar with Matthias but decided
that we weren't
quite ready to commit to a specific behaviour in this KIP.
I suggest that
you include it in KIP-1324. There's a strong chance that
KIPs 1313, 1368
and 1324 all land in the same 4.5 release, and that will
mean the RPC
changes can be coalesced into the same version bump.

Thanks,
Andrew

On 2026/09/28 17:38:12 Lianet Magrans wrote:
Hi Andrew, just one question here, did we consider
identifying not just
the
framework type and version, but also the "framework
instance"? Similar
to
what KIP-1313 did for clients (UUID for
consumer/prodcuer/admin), but
closing the loop to have a way to identify instances of
this new
"logical
client" which is the framework.

That would be useful the moment we have several
instances of the
"framework" running together I expect, e.g., several
streams instances,
KIP-1313 would give us UUIDs for all the
consumers/producer/admins
running
underneath, and KIP-1368 tell us they are behind a
"logical
client"/framework" ("streams"), but no way to tell here
one of the
"streams" framwork instances from the other one. Streams
already has a
process ID that fills that gap, the point to consider is
if it's
something
we would want along with the framework type.

I'm thinking about this coming from KIP-1324 btw
(pushing configs from
the
client to the broker). For features like that, seems
useful to identify
not
only the framework type, but also the "framework
instance".  I'm happy
to
consider it as part of KIP-1324 (that I plan to resume
soon), but
raising
it for consideration, as it's closely related to the
framework type.

Thanks!
Lianet

On Thu, Sep 24, 2026 at 2:45 PM Jun Rao via dev <
dev@kafka.apache.org>
wrote:

Hi, Andrew,

Thanks for the updated KIP. LGTM

Jun

On Thu, Sep 24, 2026 at 10:58 AM Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Jun,

JR7: Done.

Thanks,
Andrew

On 2026/09/24 16:20:08 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the updated KIP.

JR7. Piggybacking on the version bump in 4.5
sounds good to me.
Could
we
document the version bump in the KIP?

Jun

On Thu, Sep 24, 2026 at 7:32 AM Andrew Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for your reply.

JR7: There is a tension here between bumping the
version when
adding
these
tagged fields because it is a new feature and
not doing so
because of
the
negotiation aspects of this. I believe that AK
4.5 will
introduce a
new
version of ApiVersions anyway because of
KIP-1313, so the best
path
is
probably to piggyback the tagged field addition
onto that new
version.

JR8.1: I've added this constraint in AK 5.0
since we need a
major
version
to enforce this. In practice, nobody will be hit
by this limit.
JR8.2: Yes, constraint added to the configs too.

Thanks,
Andrew

On 2026/09/24 00:08:16 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the updated KIP.

JR7. Currently, we always bump the version
when adding a
tagged
field.
The
intention of this KIP is to add the two tagged
fields
without a
version
bump. This choice seems reasonable since it
avoids an extra
round
of
ApiVersionRequest version negotiation when 4.5
clients talk
to
4.4
servers. It
would be useful to be explicit document there
is not version
bump
for
ApiVersionRequest.

JR8. "The maximum length of the fields is 249
characters"
JR8.1 Should the length constraint also apply
to the existing
fields
ClientSoftwareName and ClientSoftwareVersion?
JR8.2 Should we add a length constraint for
the two new
client
configurations?

Jun

On Wed, Sep 23, 2026 at 6:42 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for the reply.

JR6: Yes, that works. I've added the methods
to
ClientTelemetryContext.
Let me know what you think.

Thanks,
Andrew

On 2026/09/21 17:54:38 Jun Rao via dev wrote:
Hi, Andrew,

Thanks for the reply.

JR6. It seems weird that the
implementation of a KIP-714
plugin
requires
casting an authorizableRequestContext to
an internal
class
RequestContext.
Would it be better to expose all client
related resource
labels
through a
public API in ClientTelemetryContext?

Jun

On Mon, Sep 21, 2026 at 6:43 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Jun,
Thanks for the reply.

JR4: Done. Producer, consumer and admin.

JR5: Yes. I've reworded slightly.

JR6: I think the plugin implementation
already needs to
do
something
similar to get to software name/version.

ClientTelemetryContext.authorizableRequestContext()
returns
an
instance of
AuthorizableRequestContext. The
o.a.k.common.requests.RequestContext
class
implements that interface, and it also
lets you get to
ClientInformation
and that's where you can find the
software name/version
and
framework
name/version. You can see this in

o.a.k.server.metrics.ClientMetricsInstanceMetadata. I
would
describe
this
as grubby :)

One way forward here would be to remove
the
broker-added
resource
labels
from KIP-1368, but leave the additions
to the match
criteria.
Then
it
would
still be possible to specify, for
example, that
particular
metrics
should
be captured for Kafka Streams clients
only, without
adding to
the
grubbiness.

Thanks,
Andrew

On 2026/09/15 20:44:06 Jun Rao via dev
wrote:
Hi, Andrew,

Thanks for the reply. A few more
comments.

JR4. "The configuration keys are
defined in

org.apache.kafka.clients.CommonClientConfigs"
Could you explicitly list the clients
(producer,
consumer,
AdminClient,
etc.) that define the two new configs?

JR5. Should AK frameworks such as
kstream and connect
set
client.framework.name and
client.framework.version
in
their
clients
automatically?

JR6. "Broker-added resources labels
for client
metrics"
Are the new labels added in
DefaultClientTelemetryContext.RequestContext?
Since
ClientTelemetryExporter.exportMetrics() only
takes
ClientTelemetryContext, does the
implementor need to
cast
it
to DefaultClientTelemetryContext to
retrieve the new
labels?

Jun

On Fri, Sep 11, 2026 at 12:48 PM
Andrew Schofield <
schofie...@apache.org

wrote:

Hi Jun,
JR1: I dug into the KIP-714 client
metrics stuff a
bit
more
and I
have
revised the KIP. I have added
client-framework-name
and
client-framework-version to the set
of keys in the
matching for
client
metrics, and also added these
broker-added metrics
tags.
This
is
how
these
pieces of information would have
been incorporated
into
KIP_714 had
they
been part of Kafka at that point.

Thanks,
Andrew

On 2026/08/04 20:29:19 Andrew
Schofield wrote:
Hi Jun,
Thanks for the response.

JR1: My intent here is only
request logging. It
would
be
possible to
add
client-framework-name and
client-framework-version
to the
match
criteria
for client metrics, which could
conceivably permit
scenarios
such
as
capturing metrics from specific
versions of the
Spring
framework.
This
is
beyond the scope I have in mind.

JR3: Thanks for clarifying the
tagged field
conventions.
Version
bump
reverted.

Thanks,
Andrew

On 2026/08/04 18:38:40 Jun Rao via
dev wrote:
Hi, Andrew,

Thanks for the reply.

JR1. ClientSoftwareName and
ClientSoftwareVersion are
used
in
metric
names
and match predicates for
configuring client
metrics.
Are
ClientFrameworkName
and ClientFrameworkVersion used
in those places
too
or
are
they
only
used
in request logging?

JR3. The general rule for adding
a new field in
RPC
is
that
(1) if
it's
truly optional, we add it as a
tagged field
with
no
version
bump;
(2)
otherwise, we add it as a
non-tagged field
with a
version
bump. One
exception is when adding a
non-optional field
in
the
request
header.
Because of the limitation in
existing
implementation,
we
need
to
add
it as
a tagged field with a version
bump.

Jun

On Tue, Aug 4, 2026 at 6:04 AM
Andrew
Schofield <
schofie...@apache.org

wrote:

Hi Jun,
Thanks for your response.

JR1: I've expanded the
motivation section in
the
KIP.
They
are
truly
optional, I feel. When
diagnosing an issue
using
the
client
logs,
having
this information can
illuminate why the
application
is
behaving
in a
particular way. If the
application is using a
framework,
behaviours
such as
retries will typically not be
in the
application
code
itself
because
they're implemented in the
framework.
There's a
difference
between
what the
application developer thinks
their code does
and
what we
see
in
the
logs.
That's the point.

JR2: Done. They're
Type.STRING, default
null. I
think
they
should be
importance LOW because that
affects the
prominence
of the
configs in
the
documentation, but I wonder
whether you
agree.

Thanks,
Andrew

On 2026/08/03 21:39:06 Jun Rao
via dev wrote:
Hi, Andrew,

Thanks for the KIP.

JR1. Could you describe the
use cases of
the
two
new
configs
and
are they
truly optional?

JR2. Could you add the type
and the default
value
for
the
new
configs?

Jun


On Fri, Jul 31, 2026 at
10:23 AM Matthias
J.
Sax
<
mj...@apache.org

wrote:

Thanks for the KIP Andrew.
I think it
will
be
very
useful if
we
can
identify frameworks,
especially our own
ones
(Connect and
Streams).

About Aditya's point: I am
wondering
where
to
draw
the
line.
Many
examples seems to be
metadata that
belongs
into
he
Kafka
record
`Headers` at the
application level,
rather
than
the
lower
level
request
headers?

There if of course no
strict logical
dividing
line
between
both.
And
yes, the broker does not
access
application
level
Kafka
record
`Headers`. But if it would
be useful to
let the
broker
tap
into
application level record
`Headers` we
should
tackle
it
independently?

Personally, I don't think
it would be the
right
design to
push
too many
thing into the lower level
request
headers. So
I
am
in
favor
of
only
added the two new propose
`ClientFrameworkName`
and
`ClientFrameworkVersion`
fields.


-Matthias

On 7/31/26 7:56 AM,
Federico Valeri
wrote:
Changes look good.
Thanks.

On Fri, Jul 31, 2026 at
11:17 AM Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Fede,
Thanks for your
response.

FV1: They are set using
the regular
config
properties.
Yes,
it is
possible for an end user
to set arbitrary
values, but
then
that
would
also
be true of a builder or
internal
constructor.
I've
added
a
bit
more
information in the KIP and
beefed up the
config
descriptions
to
discourage
application use.

FV2: I've never
encountered such
nightmares
myself.
The
framework
which
sets the config last would
win. Maybe
this
is a
motivation
for
having a
programmatic way of
setting the
information. We
could
support
concatenation
of framework information,
but that's just
pandering
to
these
people.
Let me
know what you think.

I've also updated the
KIP with a
maximum
length
for
these
pieces of
information since all
identifiers should
have
defined
bounds.

Thanks,
Andrew

On 2026/07/31 08:50:11
Federico Valeri
wrote:
Hi Andrew, the
motivation looks
good. A
couple of
questions:

FV1: The KIP says
frameworks should
set
them,
but it
does not
specify
the mechanism. If
these are ordinary
user-facing
configs,
nothing
prevents an end user
from setting
arbitrary
values,
which
defeats
the
purpose. Should the
framework set
them
programmatically
(builder or
    internal
constructor) or is user
override
intentional?

FV2: What if we have
multiple layers
of
frameworks?
Let's
say a
custom
framework on top of
SpringBoot. I've
seen
similar
nightmares
in the
past.

Thanks
Fede

On Tue, Jul 28, 2026
at 8:27 AM
Aditya
Kousik <
adityakou...@icloud.com>
wrote:

Hi Andrew,

We’re definitely
agreed on the
increased
use of
frameworks
over
than
the client directly. I’ve
cited the
different
patterns
Spring,
Micronaut,
smallrye and
company-internal frameworks
have
APIs
built
on
top
of the
Kafka client, in a couple
of KIPs
already.

The project has
enough traction
that I
think of
it
as
its
own
network
client with distributed
log semantics for
which
frameworks
are
written,
much like gRPC over netty.
People want to
just
write
the
business
logic and
leave the plumbing and
threading to the
frameworks.

All of this to say,
I’m fine to ship
the
client
framework
as the
new
identifying parameter for
frameworks to
set. It
will
be
mighty
useful.

My addendum, rather
than a pushback
is
that:
guilty
as
charged,
I’m
driven by the otel/DD
telemetry/observability
of
using
Apache
Kafka in
applications. The
client.framework.version
config for
instance
can be
used
to detect regressions and
isolate root
causes.
But I
feel it
is
only
one of
many such facets. You
mentioned that you
would
use
client.id
and
clientInstanceId to
identify a client but
that
these
do
not
help
with
aggregate/fleet-wide
issues. As an
example, if
I
tag
an
app’s AZ
it can
help me write alerts on
spike in latency
in
us-west-2.
Or,
detect stuck
partitions across multiple
client.id/application.names
none
of
whom
share
the same
client.framework.name.

Other tags that come
to mind:
application.name,
az,
team,
env,
rack.
All fields users usually
hijack
client.id
for.

An otel JavaAgent can
capture the
client
metadata
registered and
attach it as tags with
each resource
span.
Users
who
use
frameworks but
rely on datadog/otel will
get visibility
into
the
client
metadata for
free.

The KIP as I read it,
serves as a
foundation for
future
use. So I
don’t want to shoehorn a
new behaviour if
it
explodes the
scope
too
much.

Best regards,
Aditya

On Jul 27, 2026, at
13:52, Andrew
Schofield <
schofie...@apache.org>
wrote:

Hi Aditya,
Thanks for your
response.

AK1: I chose
framework as the
blessed
abstraction
because
my
focus
was problem determination
for client
applications. We
often
find
that
users
with client problems have
not coded
directly to
the
Kafka
client
interface
because they are using a
framework. As a
result,
their
knowledge
of the
application code is one
level removed
from
the
Kafka
client.
Frameworks can
override configuration
defaults,
introduce
different
retry
behaviour
and so
on. Lots of companies have
their own
internal
frameworks, so
this KIP
can
be used by them too. I'm
trying to make
it
easier to
work out
when a
user
is making use of a
framework and knowing
what
it
is.

Sometimes,
particularly for
non-Java
clients,
people
have
overridden
the
ClientSoftwareName/Version
themselves,
which
makes
those
concepts
much
less useful than they
should be. By
providing
ClientFrameworkName/Version,
there is no longer any
need to do so.
That's
another
motivation
here,
even
though KIPs don't concern
themselves with
non-Java
clients as
such.

We could go for a
more flexible
key-value
approach,
but a
simple
name and version is
sufficient for what I
had
in
mind.
Feel
free
to
push
back with additional
justification and
examples.

AK2: Done.
o.a.k.clients.CommonClientConfigs.

AK3: To identify a
particular
client, I
would
use
client
ID and
client-instance ID. I
think these are
generally
more
useful
concepts
than
the framework name and
version which are
extra
information
for
the
person
trying to figure out why a
client is not
behaving as
expected.

AK4: I'm sure you
have more
experience of
OTel/DataDog
collectors.
You may well be correct
that they would
be
helpful
for
the
collectors.

Thanks,
Andrew

On 2026/07/26
07:48:20 Aditya
Kousik
wrote:
Hello Andrew,

I’m reminded of the
client.id
discussion
we
had
back
in
KIP-1313
re: client instance id.
After that
discussion,
I
have a
WIP
KIP
that
sets a
foundation for shipping
client metadata
tags to
be
sent
for
telemetry.
I
was hoping we could
discuss if part of
that
approach
could
fit
this
KIP.

AK1. I had a note
on the
motivation
of
selecting a
“framework”
as a
blessed abstraction. The
KIP mentions
it’s
for
the
client
metadata and
easier problem diagnosis.
This is akin to
an
“application id”
that
non-framework clients
usually tag with.

KIP-606 took an
approach like
metrics.context.<key>=<val>. If we
allow
client.metadata.<key>=<val>, then a
framework/application
name
and
version can sit in such a
metadata
context
and
be
sent
with
ApiVersion
RPC.
Of course, this is an open
box approach
rather
than
just
the<