Gwen, I do not think it will break clients. Just to be clear we have following options.
1. Versioned apiVersion request and response. KIP-35 aware clients will send apiRequest, version Vn, to get api versions from a broker. If a broker supports Vn of apiVersion request, it will respond. If not, client will face a connection closure, which can be due to many reasons. a. Broker has older version of Kafka and it does not support apiVersion request at all. b. Broker does not support version Vn of apiVersion request. c. Connection closure was caused by issues other than unknown apiVersion request. In case of connection closures, the KIP recommends that clients should use some other method of determining the apiRequest version to use, like, probing. For instance, client will send V0 version of apiVersion request and will try higher versions incrementally. In case b, client will eventually get apiVersion response and know what api versions it should use. For case a and c, client will eventually give up and propagate an error to application. Note that though the KIP suggests to keep apiVersion request versioned, there is no realistic reason, at this time, that warrants a version bump. However, it is keeping the option open. 2. apiVersion request and response with no versioning. KIP-35 aware clients will send apiRequest to get api versions from a broker. If a broker supports apiVersion request, it will respond. If not, client will face a connection closure, which can be due to following reasons. a. Broker has older version of Kafka and it does not support apiVersion request at all. b. Connection closure was caused by issues other than unknown apiVersion request. This approach does not require client to do any linear probing in case of connection closure, however there will be no way in future to change apiVersion req/resp format. Hopefully this gives more information to decide between the above mentioned options. What do you think now? On Fri, Apr 1, 2016 at 2:06 PM, Gwen Shapira <g...@confluent.io> wrote: > It isn't a guarantee, it is a limitation. > > It doesn't even matter what we do (have versions, no versions, deprecate or > not) - clients will never be able to use version other than v0 (how will > they know it is there?) and if we ever remove v0, we will immediately break > all clients. > > If we are ok making changes that immediately break all clients, we don't > need this KIP. I thought the entire point was to fix this. > > So, by having this KIP we agree to never ever remove VersionRequest v0 (as > it is in the KIP). I agree it could be hard, but no one promised that > implementing a server that is backward and forward compatible is easy. > Everyone who votes on this KIP must be aware that whatever we do here is > here to stay. > > Which is exactly why this discussion is so important. > > Gwen > > On Fri, Apr 1, 2016 at 1:52 PM, Ashish Singh <asi...@cloudera.com> wrote: > > > Never ever changing the version of VersionReq/resp will be ideal, but it > is > > hard to guarantee that we will never need a change. We should definitely > > try to avoid it and honestly I do not see a reason, as of now, why it > will > > change. I would say, we should keep it versioned and document any risk of > > changing it. Does that sound OK? > > > > On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > I am happy with: > > > 1. Adding only validation support in the client for now, deferring > > backward > > > compatibility for later. With the odd side-effect of having the C and > > > Python clients actually better than Java for a bit. > > > 2. Never ever changing the version of VersionRequest and > VersionResponse. > > > > > > I'd also like to see: > > > 1. At least one client implement backward compatibility (so we can use > it > > > in testing) > > > 2. Backward compatibility tests using that client (including an upgrade > > > test) > > > 3. Checklist for reviewers on when protocol bumps are required > > > > > > I am strongly against releasing an API that no one is using as > > intended... > > > so either we do it in our client or use a 3rd party client. > > > > > > Gwen > > > > > > On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson <ja...@confluent.io> > > > wrote: > > > > > > > I'm sympathetic to Jay's argument, but I feel the version API can be > > > > justified on its own without requiring a big change in the way we > > manage > > > > compatibility. For the official client compatibility story (i.e. what > > the > > > > Java client implements), we can use this API to verify compatibility > > with > > > > the broker, which is a significant gap at the moment (we can do this > > now > > > > before the release if we all agree without much delay). It also > happens > > > to > > > > help clients trying to maintain backwards compatibility, which is > > great. > > > > > > > > Maybe the question is whether this API will restrict later efforts to > > > > change the way we manage compatibility? I don't see much risk in > terms > > of > > > > maintenance of the API, but there is some risk that it could become > > > > superfluous at some point. For example, if we changed the response > > header > > > > to include an error code for an unknown request version, then maybe > the > > > > Java client would no longer need the initial call to the version API > > > > (though it might arguably still be a good idea to do so to detect > > > > incompatibility sooner). Maybe there is some risk also that it makes > > the > > > > compatibility story more confusing? I actually think it fits pretty > > well > > > > with the versioned API approach generally, though it's admittedly a > > > little > > > > odd to have the version API versioned itself. Practically speaking, I > > > think > > > > this means that we shouldn't increment the version. Otherwise the > > client > > > > will need another round trip (after always starting with v0), which > > might > > > > be doable even if it seems a little messy. > > > > > > > > Thoughts? > > > > > > > > -Jason > > > > > > > > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira <g...@confluent.io> > > wrote: > > > > > > > > > My understanding is that we are not adding the client portion to > the > > > KIP > > > > is > > > > > because we believe it will require a lot of discussion (read: will > be > > > > > really hard to get right). > > > > > > > > > > Maybe it is a hint that this protocol is too difficult for clients > to > > > > > implement? > > > > > I can't see why is it easy in C and Python and super difficult in > > Java. > > > > > > > > > > Even if it is too hard to implement just in Java for some reason > (god > > > > knows > > > > > the language has issues), isn't it a good reason to come up with > > > > something > > > > > we can implement in a reasonable way? > > > > > > > > > > Gwen > > > > > > > > > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com > > > > > > wrote: > > > > > > > > > > > That is a fair concern and I think eventually we might want to > have > > > > java > > > > > > clients backwards compatible. However, blocking KIP-35 on that > > might > > > > not > > > > > be > > > > > > the best idea. The reason I say so is due to following reasons. > > > > > > > > > > > > 1. Backwards compatibility in java clients is a larger > discussion, > > as > > > > we > > > > > > have already seen. Having a separate KIP focussing exactly on > that > > > will > > > > > > help in reducing moving pieces in one KIP. > > > > > > 2. It probably will fall out of 0.10 due to tight timeline, I am > > > > assuming > > > > > > we do not want to delay 0.10 a lot. > > > > > > 3. Even if we make java clients backward compatible in 0.10, I do > > not > > > > > think > > > > > > it will be able to work with older releases as older broker > > versions > > > > > still > > > > > > won't provide info on supported api versions. If we add api > > versions > > > > > > req/resp in 0.10, and add backwards compatibility in java clients > > in > > > > > later > > > > > > versions, they will probably work with 0.10 and we will be able > to > > > test > > > > > > that. > > > > > > > > > > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava < > > > > > e...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira < > g...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > > > > > I have two concerns with the proposal as it is: > > > > > > > > > > > > > > > > 1. Having an API that publishes protocol is useless for > clients > > > if > > > > > > > > developers don't bump the API when they should. > > > > > > > > I would like to see good documentation on when protocols are > > > bumped > > > > > > and a > > > > > > > > proposal on how this will be automatically validated to the > > > extent > > > > > > > possible > > > > > > > > (and to what extent are we still at risk of accidental > > breakage). > > > > > > > > Even if we don't implement all of this at first pass, I want > to > > > > know > > > > > > > which > > > > > > > > direction we are going in order to solve the client > > compatibility > > > > > > issue. > > > > > > > > > > > > > > > > 2. In addition to third-party clients, there are stream > > > processing > > > > > > > > frameworks who use the Java client we publish as their client > > and > > > > > would > > > > > > > > also like to enjoy the same compatibility benefits C and > Python > > > > > clients > > > > > > > > will enjoy. It will be very silly if Apache Kafka clients are > > the > > > > > worst > > > > > > > > clients out there from compatibility POV. The KIP can be > > > > implemented > > > > > in > > > > > > > > parts, but I really want to see an effort to build the Java > > > client > > > > > > > > compatibility into the KIP and if possible into the release > > too. > > > > > > > > > > > > > > > > > > > > > > I wouldn't conflate those two things. Changing the > compatibility > > > > > approach > > > > > > > of the Java clients could easily be another KIP (and probably a > > > large > > > > > > one, > > > > > > > too). There are already high quality, well maintained clients > > > trying > > > > to > > > > > > use > > > > > > > a different approach and this addresses their needs. Blocking > the > > > > > entire > > > > > > > ecosystem on the Java client seems problematic. > > > > > > > > > > > > > > That said, agreed that it would be bad for the Java client to > > have > > > > the > > > > > > > worst compatibility story. > > > > > > > > > > > > > > -Ewen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Gwen > > > > > > > > > > > > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson < > > > > ja...@confluent.io > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Bumping this thread. I talked with Ashish and Magnus about > > this > > > > KIP > > > > > > > > offline > > > > > > > > > and I'm gradually coming over. The new API actually stands > by > > > > > itself > > > > > > > > > outside of the discussion about whether the client should > > > support > > > > > > > > backwards > > > > > > > > > compatibility or not. For the Java client, we could > continue > > to > > > > > > support > > > > > > > > the > > > > > > > > > current compatibility approach in which the client supports > > > only > > > > > > > brokers > > > > > > > > > with the same version or greater. In that case, we would > use > > > this > > > > > API > > > > > > > > only > > > > > > > > > to assert that the current API versions are all supported, > > and > > > > > raise > > > > > > an > > > > > > > > > exception if they are not. This gives us the capability > going > > > > > forward > > > > > > > to > > > > > > > > > detect when the client is talking to an older broker, which > > we > > > > > don't > > > > > > > have > > > > > > > > > right now. This check should be straightforward, so we > could > > do > > > > it > > > > > > now, > > > > > > > > > which would resolve some of the uneasiness about having an > > > unused > > > > > > > feature > > > > > > > > > which we depended on other clients to test for us. Does > that > > > make > > > > > > sense > > > > > > > > or > > > > > > > > > not? > > > > > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh < > > > > asi...@cloudera.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > We have proposed and discussed majorly three approaches > so > > > far, > > > > > > there > > > > > > > > > were > > > > > > > > > > many minor versions with small variations. Comparing them > > > > really > > > > > > > > > requires a > > > > > > > > > > side by side proposal and their pros/cons, and I agree > with > > > > > others > > > > > > > that > > > > > > > > > > this has been lacking in the KIP. We just updated the KIP > > > with > > > > > > > > following > > > > > > > > > > details. > > > > > > > > > > > > > > > > > > > > 1. Provide proposed changes in all the three proposals we > > > have > > > > > > > > discussed > > > > > > > > > so > > > > > > > > > > far. Except the current proposal, these proposals are in > > > > rejected > > > > > > > > > > alternatives. > > > > > > > > > > 2. Provide reasoning on why the rejected proposals were > > > > rejected. > > > > > > > > > > 3. Add scenarios for all of these proposals from a client > > > > > developer > > > > > > > and > > > > > > > > > > core Kafka developer point of view. > > > > > > > > > > > > > > > > > > > > As we are really close to 0.10 deadline, a quick round of > > > > voting > > > > > > will > > > > > > > > > > really help. If you really do not like the idea, please > > feel > > > > free > > > > > > to > > > > > > > > say > > > > > > > > > > so. If the vote fails for the current proposal, it can at > > > lease > > > > > > > provide > > > > > > > > > > recommendations that we should consider for next version > of > > > > > > proposal > > > > > > > > and > > > > > > > > > > put it up for vote again for next release. However, as > > stated > > > > > > earlier > > > > > > > > by > > > > > > > > > > multiple people having this ASAP will be awesome. > > > > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < > > > > > > dana.pow...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira < > > > > > g...@confluent.io > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > "I think I would make this approach work by looking > at > > > the > > > > > > > released > > > > > > > > > > > server > > > > > > > > > > > > version documentation for each version that I am > trying > > > to > > > > > > > support > > > > > > > > > and > > > > > > > > > > > test > > > > > > > > > > > > against*, manually identify the expected "protocol > > > vectors" > > > > > > each > > > > > > > > > > > supports, > > > > > > > > > > > > store that as a map of vectors to "broker versions", > > > check > > > > > each > > > > > > > > > vector > > > > > > > > > > at > > > > > > > > > > > > runtime until I find a match, and write code > > > compatibility > > > > > > checks > > > > > > > > > from > > > > > > > > > > > > there." > > > > > > > > > > > > > > > > > > > > > > > > How is this better than a global version ID? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As a client developer, it seems roughly the same. I > think > > > it > > > > > > > probably > > > > > > > > > > > avoids the server development workflow issues, and > > possibly > > > > the > > > > > > > need > > > > > > > > to > > > > > > > > > > > agree on semantics of the global version ID? But others > > > > surely > > > > > > are > > > > > > > > more > > > > > > > > > > > qualified than me to comment on that part. > > > > > > > > > > > > > > > > > > > > > > -Dana > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Thanks, > > > > > > > Ewen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > Ashish > > > -- Regards, Ashish
