Also, just wanted to add that I get that people are impatient with this. It's super frustrating to kind of think on a plan for a chunk of time without checking in any code. But I do think we only just got to a workable proposal so most of the delay hasn't been process but actual genuine iteration on the proposal. And I do believe that even though it is a bit maddening to do so much up-front thinking there are a couple of areas where it is really the faster approach. I really think protocol design is one of these areas, and compatibility models are another, which I guess makes protocol support for compatibility the perfect storm of bike shedding. :-)
-Jay On Fri, Apr 1, 2016 at 11:22 AM, Jay Kreps <j...@confluent.io> wrote: > Hey Ewen, with protocol design I actually do think good is the enemy of > perfect (not sure if that makes sense but I think you get what I mean). > Your comment seemed to be along the lines of "this isn't very good, but > let's do something". Can you elaborate on what you think we should be > doing? > > The only two things I'd add to the comments so far: > 1. We will probably live with this scheme for five years if we make this > change. Let's for the love of god not do something half assed. It would be > far far better not to do anything yet and come at this when we have the > time to think it through, then to do something that doesn't make sense. > 2. As Gwen says unless/until someone actually sees this through--gets the > docs updated, implements the java client support, etc. we will be in a > worse situation than we currently are. Is anyone actually signing up for > that? I do think these things are coupled in actually getting us to a > better place though they are separate chunks of work. > > I get that people want to see forward motion here but I really think this > is an area where thoughtful beats fast. If we do something half-assed we > first do the work of converting the world to that, then the work of undoing > it and doing the thing we should have originally done so it really doesn't > speed things up. > > -Jay > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> wrote: > >> That is a fair concern and I think eventually we might want to have java >> clients backwards compatible. However, blocking KIP-35 on that might not >> be >> the best idea. The reason I say so is due to following reasons. >> >> 1. Backwards compatibility in java clients is a larger discussion, as we >> have already seen. Having a separate KIP focussing exactly on that will >> help in reducing moving pieces in one KIP. >> 2. It probably will fall out of 0.10 due to tight timeline, I am assuming >> we do not want to delay 0.10 a lot. >> 3. Even if we make java clients backward compatible in 0.10, I do not >> think >> it will be able to work with older releases as older broker versions still >> won't provide info on supported api versions. If we add api versions >> req/resp in 0.10, and add backwards compatibility in java clients in later >> versions, they will probably work with 0.10 and we will be able to test >> that. >> >> On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava <e...@confluent.io >> > >> wrote: >> >> > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io> >> wrote: >> > >> > > I have two concerns with the proposal as it is: >> > > >> > > 1. Having an API that publishes protocol is useless for clients if >> > > developers don't bump the API when they should. >> > > I would like to see good documentation on when protocols are bumped >> and a >> > > proposal on how this will be automatically validated to the extent >> > possible >> > > (and to what extent are we still at risk of accidental breakage). >> > > Even if we don't implement all of this at first pass, I want to know >> > which >> > > direction we are going in order to solve the client compatibility >> issue. >> > > >> > > 2. In addition to third-party clients, there are stream processing >> > > frameworks who use the Java client we publish as their client and >> would >> > > also like to enjoy the same compatibility benefits C and Python >> clients >> > > will enjoy. It will be very silly if Apache Kafka clients are the >> worst >> > > clients out there from compatibility POV. The KIP can be implemented >> in >> > > parts, but I really want to see an effort to build the Java client >> > > compatibility into the KIP and if possible into the release too. >> > > >> > >> > I wouldn't conflate those two things. Changing the compatibility >> approach >> > of the Java clients could easily be another KIP (and probably a large >> one, >> > too). There are already high quality, well maintained clients trying to >> use >> > a different approach and this addresses their needs. Blocking the entire >> > ecosystem on the Java client seems problematic. >> > >> > That said, agreed that it would be bad for the Java client to have the >> > worst compatibility story. >> > >> > -Ewen >> > >> > >> > > >> > > Gwen >> > > >> > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson <ja...@confluent.io> >> > > wrote: >> > > >> > > > Bumping this thread. I talked with Ashish and Magnus about this KIP >> > > offline >> > > > and I'm gradually coming over. The new API actually stands by itself >> > > > outside of the discussion about whether the client should support >> > > backwards >> > > > compatibility or not. For the Java client, we could continue to >> support >> > > the >> > > > current compatibility approach in which the client supports only >> > brokers >> > > > with the same version or greater. In that case, we would use this >> API >> > > only >> > > > to assert that the current API versions are all supported, and >> raise an >> > > > exception if they are not. This gives us the capability going >> forward >> > to >> > > > detect when the client is talking to an older broker, which we don't >> > have >> > > > right now. This check should be straightforward, so we could do it >> now, >> > > > which would resolve some of the uneasiness about having an unused >> > feature >> > > > which we depended on other clients to test for us. Does that make >> sense >> > > or >> > > > not? >> > > > >> > > > -Jason >> > > > >> > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh <asi...@cloudera.com> >> > > wrote: >> > > > >> > > > > We have proposed and discussed majorly three approaches so far, >> there >> > > > were >> > > > > many minor versions with small variations. Comparing them really >> > > > requires a >> > > > > side by side proposal and their pros/cons, and I agree with others >> > that >> > > > > this has been lacking in the KIP. We just updated the KIP with >> > > following >> > > > > details. >> > > > > >> > > > > 1. Provide proposed changes in all the three proposals we have >> > > discussed >> > > > so >> > > > > far. Except the current proposal, these proposals are in rejected >> > > > > alternatives. >> > > > > 2. Provide reasoning on why the rejected proposals were rejected. >> > > > > 3. Add scenarios for all of these proposals from a client >> developer >> > and >> > > > > core Kafka developer point of view. >> > > > > >> > > > > As we are really close to 0.10 deadline, a quick round of voting >> will >> > > > > really help. If you really do not like the idea, please feel free >> to >> > > say >> > > > > so. If the vote fails for the current proposal, it can at lease >> > provide >> > > > > recommendations that we should consider for next version of >> proposal >> > > and >> > > > > put it up for vote again for next release. However, as stated >> earlier >> > > by >> > > > > multiple people having this ASAP will be awesome. >> > > > > >> > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < >> dana.pow...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira < >> g...@confluent.io> >> > > > wrote: >> > > > > > >> > > > > > > "I think I would make this approach work by looking at the >> > released >> > > > > > server >> > > > > > > version documentation for each version that I am trying to >> > support >> > > > and >> > > > > > test >> > > > > > > against*, manually identify the expected "protocol vectors" >> each >> > > > > > supports, >> > > > > > > store that as a map of vectors to "broker versions", check >> each >> > > > vector >> > > > > at >> > > > > > > runtime until I find a match, and write code compatibility >> checks >> > > > from >> > > > > > > there." >> > > > > > > >> > > > > > > How is this better than a global version ID? >> > > > > > >> > > > > > >> > > > > > As a client developer, it seems roughly the same. I think it >> > probably >> > > > > > avoids the server development workflow issues, and possibly the >> > need >> > > to >> > > > > > agree on semantics of the global version ID? But others surely >> are >> > > more >> > > > > > qualified than me to comment on that part. >> > > > > > >> > > > > > -Dana >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > Regards, >> > > > > Ashish >> > > > > >> > > > >> > > >> > >> > >> > >> > -- >> > Thanks, >> > Ewen >> > >> >> >> >> -- >> >> Regards, >> Ashish >> > >
