I'd also add that there is some discussion of the mixed-broker issue on the KIP-35 wiki page as well:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature Of course, since 0.9 (and earlier) brokers do not support ApiVersion, you're still in a bit of a pickle. I think like most shiny new stuff, we'll probably have to wait and see how different clients approach the problem and see what works best over time. The 2 non-java clients that I'm aware of that are attempting to tackle broker backwards-compatibility are librdkafka (Magnus) and kafka-python (myself). Asish or others can correct me if I'm wrong here, but I believe the java client is not attempting to do this yet outside of a single request flow (SASL authentication). FWIW, the approach we take in kafka-python is to pick a single version-state for the cluster and use it for all brokers. At the present that version-state is chosen during bootstrap by checking the first contacted broker. That works great for most clusters, where all brokers are the same version. A more robust approach might be to check the versions for all brokers in the cluster, not rely on the first contacted, and pick the most conservative / lowest version-state found. That's not likely to be perfect, but would support clients that bootstrap into a mixed cluster. But in any event, providing a manual override for users to specifically configure the version-state and skip the dynamic broker version checks is always a good fallback for managing cluster upgrades. HTH, -Dana On Tue, May 17, 2016 at 6:21 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Oleksiy, > > Ashish has just created a PR to add some information about how to use > KIP-35: > > https://github.com/apache/kafka/pull/1395 > > Maybe you can have a look and leave a comment with areas that are not > covered by the first draft. > > Ismael > > On Tue, May 17, 2016 at 1:34 PM, Oleksiy Krivoshey <oleks...@gmail.com> > wrote: > >> Given lots of protocol changes in 0.10 compared to 0.9, with many API >> methods now having up to 3 possible versions, what is the suggested way for >> the client library to coordinate with Kafka brokers? I can see there is >> ApiVersions request which is great, but what happens if Kafka cluster >> consists of mixed 0.9 and 0.10 brokers (for example during upgrade), should >> the library keep and refresh the state of supported APIs for each broker >> the same way it does for topic metadata? >> Another example is the Metadata request (Api Key 3) which has two versions, >> how would the client library in the first place know which version of the >> request to send to each of the initial brokers? >> >> I can't see an easy way for the client library to support for example both >> 0.9 and 0.10 without complicated state switches and error handling. I'm not >> talking about official Java client, but about other language libraries. >> >> What would you suggest? Thanks! >>
