I have updated the config description in the KIP, made the example more clear
The proposed change allows setting different min versions for different APIs, and the ApiVersionRequest change is already in the KIP. On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote: > Hi Ismael, > I meant to say blocking clients based on their API version > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 > But If I understand what you are saying, since each client release can > support different versions for each of fetch, produce, offset commit etc.. > and it's harder to block just based on single min.api.version setting > across different clients. > The idea I had in my mind was to do this via ApiVersionRequest, when a > client makes api request to broker in response we return min and max > version supported for each Api. When min.api.version enabled on broker, it > returns the maxVersion it supports for each of the requests in that release > as min versions to the clients. > > Example: > Kafka 1.1.1 broker and min.api.verson set to > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79 > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for > example produce request > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112 > Instead of returning all of the supported versions it will return > PRODUCE_REQUEST_V5 as the only supported version. > > Irrespective of the above approach I understand your point still stands > which is sarama might not choose to implement all the higher version > protocols for Kafka 1.1 release and they might introduce higher version of > produce request in a subsequent minor release and it will be harder for > users to figure out which release of sarama client they can use. > > > Ying, if you have a different apporach which might address this issue > please add. > > > Thanks, > Harsha > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote: > > Hi Harsha, > > > > There is no such thing as 1.1 protocol. I encourage you to describe an > > example config that achieves what you are suggesting here. It's pretty > > complicated because the versions are per API and each client evolves > > independently. > > > > Ismael > > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote: > > > > > Hi, > > > > > > "Relying on min.version seems like a pretty clunky way to achieve the > above > > > > list. The challenge is that it's pretty difficult to do it in a way > that > > > > works for clients across languages. They each add support for new > > > protocol > > > > versions independently (it could even happen in a bug fix release). > So, > > > if > > > > you tried to block Sarama in #2, you may block Java clients too." > > > > > > That's the intended effect, right? if you as the admin/operator > > > configures the broker to have min.api.version to be 1.1 > > > it should block java , sarama clients etc.. which are below the 1.1 > > > protocol. As mentioned this is not just related to log.format upgrade > > > problem but in general a forcing cause to get the users to upgrade > their > > > client version in a multi-tenant environment. > > > > > > "> For #3, it seems simplest to have a config that requires clients to > > > support > > > > a given message format version (or higher). For #2, it seems like > you'd > > > > want clients to advertise their versions. That would be useful for > > > multiple > > > > reasons." > > > This kip offers the ability to block clients based on the protocol they > > > support. This should be independent of the message format upgrade. Not > all > > > of the features or bugs are dependent on a message format and having a > > > message format dependency to block clients means we have to upgrade to > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2 > message > > > format and now we want to block all 0.8.x clients. > > > > > > min.api.version helps at the cluster level to say that all users > required > > > to upgrade clients to the at minimum need to speak the min.api.version > and > > > not tie to message.format because not all cases one wants to upgrade > the > > > message format and block the old clients. > > > > > > > > > To Gwen's point, I think we should also return in the error message > that > > > the broker only supports min.api.version and above. So that users can > see a > > > clear message and upgrade to a newer version. > > > > > > > > > Thanks, > > > Harsha > > > > > > > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote: > > > > Hi Ying, > > > > > > > > The actual reasons are important so that people can evaluate the KIP > (and > > > > vote). :) Thanks for providing a few more: > > > > > > > > (1) force users to check pointing in Kafka instead of zookeeper > > > > (2) forbid an old go (sarama) client library which is known to have > some > > > > serious bugs > > > > (3) force kafka 1.x clients with the ability to roll back if there's > an > > > > issue (unlike a message format upgrade) > > > > > > > > Relying on min.version seems like a pretty clunky way to achieve the > > > above > > > > list. The challenge is that it's pretty difficult to do it in a way > that > > > > works for clients across languages. They each add support for new > > > protocol > > > > versions independently (it could even happen in a bug fix release). > So, > > > if > > > > you tried to block Sarama in #2, you may block Java clients too. > > > > > > > > For #3, it seems simplest to have a config that requires clients to > > > support > > > > a given message format version (or higher). For #2, it seems like > you'd > > > > want clients to advertise their versions. That would be useful for > > > multiple > > > > reasons. > > > > > > > > Ismael > > > > > > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid> > > > wrote: > > > > > > > > > Hi Ismael, > > > > > > > > > > Those are just examples. I think the administrators should be able > to > > > block > > > > > certain client libraries for whatever reason. Some other possible > > > reasons > > > > > include, force users to check pointing in Kafka instead of > zookeeper, > > > > > forbid an old go (sarama) client library which is known to have > some > > > > > serious bugs. > > > > > > > > > > message.downconversion.enable does not solve our problems. We are > now > > > > > planning to upgrade to message format V3, and force users to > upgrade to > > > > > Kafka 1.x clients. With the proposed min.api.version setting, in > case > > > of > > > > > there is anything wrong, we can roll back the setting. If we > upgrade > > > the > > > > > file format, there is no way to rollback (Kafka doesn't support > > > downgrading > > > > > message format). > > > > > > > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <ism...@juma.me.uk> > wrote: > > > > > > > > > > > Hi Ying, > > > > > > > > > > > > It looks to me that all the examples given in the KIP can be > handled > > > with > > > > > > the existing "message.downconversion.enable" config and by > > > configuring > > > > > the > > > > > > message format to be the latest: > > > > > > > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains > message > > > > > header > > > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers > > > > > RESOLVED ) > > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( > > > KAFKA-3160 - > > > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED > ) > > > > > > > 3. Performance penalty of converting message format from V3 to > V1 > > > or V2 > > > > > > > for the old consumers (KIP-31 - Move to relative offsets in > > > compressed > > > > > > > message sets) > > > > > > > > > > > > > > > > > > Am I missing something? Are there other examples that are not > > > related to > > > > > > message conversion? > > > > > > > > > > > > Ismael > > > > > > > > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng > <yi...@uber.com.invalid> > > > > > > wrote: > > > > > > > > > > > > > Hi here, > > > > > > > > > > > > > > Please vote for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > > > > > > > > > > > > > >
