Hi Travis, Please, excuse me for my late reply.
02. Yeah, I agree that it is more convenient to rely on the api versions but that does not protect us from misuages. 03. Yeah, I was about to suggest the same. Adding the information to the DescribeCluster API would work and we also have the Admin.describeCluster API to access it. We could provide the software name and version of the broker that services the request. Adding it per broker is challenging because the broker doesn't know the version of the others. If you use the API directly, you can always send it to all brokers in the cluster to get all the versions. This would also mitigate 02. because clients won't use the DescribeCluster API to gate features. Best, David On Fri, Nov 11, 2022 at 5:50 PM Travis Bischel <travis.bisc...@gmail.com> wrote: > > Two quick mistakes to clarify: > > When I say ClusterMetadata, I mean request 60, DescribeCluster. > > Also, the email subject of this entire thread should be "[DISCUSS] KIP-885: > Expose Broker's Name and Version to Clients”. I must have either accidentally > pasted the “Skip to end of metadata”, or did not delete something. > > Cheers, > -Travis > > On 2022/11/11 16:45:12 Travis Bischel wrote: > > Thanks for the replies David and Magnus > > > > David: > > > > 02: From a client implementation perspective, it is easier to gate features > > based on the max version numbers returned per request, rather than any > > textual representation of a string. I’m not really envisioning a client > > implementation trying to match on an undefined string, especially if it’s > > documented as just metadata information. > > > > 03: Interesting, I may be one of the few that does query the version > > directly. Perhaps this can be some new information that is instead added to > > request 60, ClusterMetadata? The con with ClusterMetadata is that I’m > > interested in this information on a per-broker basis. We could add these > > fields per each broker in the Brokers field, though. > > > > Magnus: > > > > As far as I can see, only my franz-go client offers this ability to “guess” > > the version of a broker — and it’s historically done so through > > ApiVersions, which was the only way to do this. This feature was used in > > three projects by two people: my kcl project, and the formerly-known-as > > Kowl and Kminion projects. > > > > After reading through most of the discussion thread on KIP-35, it seems > > that the conversation about using a broker version string / cluster > > aggregate version was specifically related to having the client choose how > > to behave (i.e., choose what versions of requests to use). The discussion > > was not around having the broker version as a piece of information that a > > client can use in log lines or for end-user presentation purposes. > > > > It seems a bit of an misdirected worry that a client implementor may > > accidentally use an unstructured string field for versioning purposes, > > especially when another field (ApiKeys) exists for versioning purposes and > > is widely known. Implementing a Kafka client is quite complex and there are > > so many other areas an implementor can go wrong, I’m not sure that we > > should be worried about an unstructured and documented metadata field. > > > > "the existing ApiVersionsReq … this information is richer than a single > > version string" > > > > Agree, this true for clients. However, it’s completely useless visually for > > end users. > > > > The reason Kminion used the version guess was two fold: to emit log a log > > on startup that the process was talking to Kafka v#.#, and to emit a const > > gauge metric for Prometheus where people could monitor external to Kafka > > what version each broker was running. > > > > Kowl uses the version guess to display the Kafka version the process is > > talking to immediately when you go to the Brokers panel. I envision that > > this same UI display can be added to Conduktor, even Confluent, etc. > > > > kcl uses the version guess as an extremely quick debugging utility: > > software engineers (not cluster administrators) might not always know what > > version of Kafka they are talking to, but they are trying to use a client. > > I often receive questions about “why isn’t this xyz thing working”, I ask > > for their cluster version with kcl, and then we can jump into diagnosing > > the problem much quicker. > > > > I think, if we focus on the persona of a cluster administrator, it’s not > > obvious what the need for this KIP is. For me, focusing on the perspective > > of an end-user of a client makes the need a bit clearer. It is not the > > responsibility of an end-user to manage the cluster version, but it is the > > responsibility of an end-user to know which version of a cluster they are > > talking to so that they know which fields / requests / behaviors are > > supported in a client > > > > All that said: I think this information is worth it and unlikely to be > > misused. IMO, ApiVersions is the main place to include this information, > > but another alternative is ClusterMetadata. Adding these fields to > > ClusterMetadata might be a bit awkward and may return stale information > > sometimes during a rolling upgrade. > > > > Curious to know your thoughts, and again thank you for the consideration > > and replies, > > - Travis > > > > On 2022/11/11 13:07:37 Magnus Edenhill wrote: > > > Hi Travis and thanks for the KIP, two comments below: > > > > > > > > > Den fre 11 nov. 2022 kl 13:37 skrev David Jacot <da...@gmail.com>: > > > > > > > 02: I am a bit concerned by clients that could misuse these information. > > > > For instance, one may be tempted to rely on the version to decide > > > > whether a > > > > feature is enabled or not. The api versions should remain the source of > > > > truth but we cannot enforce it with the proposed approach. That may be > > > > acceptable though. > > > > > > > > > > We proposed including this in the original ApiVersionRequest KIP-35 (waaay > > > back), but it was rejected > > > for exactly this reason; that it may(will) be misused by clients. > > > > > > > > > > > > I would also like to question the actual end-user use of this information, > > > the existing ApiVersionsReq > > > already provides - on a detailed level - what features and functionality > > > the broker supports - > > > this information is richer than a single version string. > > > > > > The KIP says "End users can quickly check from a client if the cluster is > > > up to date" and > > > "Clients can also use the broker version in log lines so that end users > > > can > > > quickly see > > > if a version looks about right or if something is seriously broken.": > > > > > > In my mind that's not typically the end-users role or responsibility, but > > > that of the Kafka cluster operator, > > > who'll already know the version being deployed. > > > > > > > > > Regards, > > > Magnus > > > > > > > > > > > > > Le jeu. 10 nov. 2022 à 19:10, Travis Bischel <tr...@gmail.com> a > > > > écrit : > > > > > > > > > Hi all, > > > > > > > > > > I've written a KIP to expose the BrokerSoftwareName and > > > > > BrokerSoftwareVersion to clients: > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-885%3A+Expose+Broker%27s+Name+and+Version+to+Clients > > > > > > > > > > If we agree this is useful, it would be great to have this in by 3.4. > > > > > > > > > > Thank you, > > > > > - Travis > > > > > > > > > > > >
