Never ever changing the version of VersionReq/resp will be ideal, but it is
hard to guarantee that we will never need a change. We should definitely
try to avoid it and honestly I do not see a reason, as of now, why it will
change. I would say, we should keep it versioned and document any risk of
changing it. Does that sound OK?

On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira <g...@confluent.io> wrote:

> I am happy with:
> 1. Adding only validation support in the client for now, deferring backward
> compatibility for later. With the odd side-effect of having the C and
> Python clients actually better than Java for a bit.
> 2. Never ever changing the version of VersionRequest and VersionResponse.
>
> I'd also like to see:
> 1. At least one client implement backward compatibility (so we can use it
> in testing)
> 2. Backward compatibility tests using that client (including an upgrade
> test)
> 3. Checklist for reviewers on when protocol bumps are required
>
> I am strongly against releasing an API that no one is using as intended...
> so either we do it in our client or use a 3rd party client.
>
> Gwen
>
> On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > I'm sympathetic to Jay's argument, but I feel the version API can be
> > justified on its own without requiring a big change in the way we manage
> > compatibility. For the official client compatibility story (i.e. what the
> > Java client implements), we can use this API to verify compatibility with
> > the broker, which is a significant gap at the moment (we can do this now
> > before the release if we all agree without much delay). It also happens
> to
> > help clients trying to maintain backwards compatibility, which is great.
> >
> > Maybe the question is whether this API will restrict later efforts to
> > change the way we manage compatibility? I don't see much risk in terms of
> > maintenance of the API, but there is some risk that it could become
> > superfluous at some point. For example, if we changed the response header
> > to include an error code for an unknown request version, then maybe the
> > Java client would no longer need the initial call to the version API
> > (though it might arguably still be a good idea to do so to detect
> > incompatibility sooner). Maybe there is some risk also that it makes the
> > compatibility story more confusing? I actually think it fits pretty well
> > with the versioned API approach generally, though it's admittedly a
> little
> > odd to have the version API versioned itself. Practically speaking, I
> think
> > this means that we shouldn't increment the version. Otherwise the client
> > will need another round trip (after always starting with v0), which might
> > be doable even if it seems a little messy.
> >
> > Thoughts?
> >
> > -Jason
> >
> > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira <g...@confluent.io> wrote:
> >
> > > My understanding is that we are not adding the client portion to the
> KIP
> > is
> > > because we believe it will require a lot of discussion (read: will be
> > > really hard to get right).
> > >
> > > Maybe it is a hint that this protocol is too difficult for clients to
> > > implement?
> > > I can't see why is it easy in C and Python and super difficult in Java.
> > >
> > > Even if it is too hard to implement just in Java for some reason (god
> > knows
> > > the language has issues), isn't it a good reason to come up with
> > something
> > > we can implement in a reasonable way?
> > >
> > > Gwen
> > >
> > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com>
> > wrote:
> > >
> > > > That is a fair concern and I think eventually we might want to have
> > java
> > > > clients backwards compatible. However, blocking KIP-35 on that might
> > not
> > > be
> > > > the best idea. The reason I say so is due to following reasons.
> > > >
> > > > 1. Backwards compatibility in java clients is a larger discussion, as
> > we
> > > > have already seen. Having a separate KIP focussing exactly on that
> will
> > > > help in reducing moving pieces in one KIP.
> > > > 2. It probably will fall out of 0.10 due to tight timeline, I am
> > assuming
> > > > we do not want to delay 0.10 a lot.
> > > > 3. Even if we make java clients backward compatible in 0.10, I do not
> > > think
> > > > it will be able to work with older releases as older broker versions
> > > still
> > > > won't provide info on supported api versions. If we add api versions
> > > > req/resp in 0.10, and add backwards compatibility in java clients in
> > > later
> > > > versions, they will probably work with 0.10 and we will be able to
> test
> > > > that.
> > > >
> > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > > >
> > > > > > I have two concerns with the proposal as it is:
> > > > > >
> > > > > > 1. Having an API that publishes protocol is useless for clients
> if
> > > > > > developers don't bump the API when they should.
> > > > > > I would like to see good documentation on when protocols are
> bumped
> > > > and a
> > > > > > proposal on how this will be automatically validated to the
> extent
> > > > > possible
> > > > > > (and to what extent are we still at risk of accidental breakage).
> > > > > > Even if we don't implement all of this at first pass, I want to
> > know
> > > > > which
> > > > > > direction we are going in order to solve the client compatibility
> > > > issue.
> > > > > >
> > > > > > 2. In addition to third-party clients, there are stream
> processing
> > > > > > frameworks who use the Java client we publish as their client and
> > > would
> > > > > > also like to enjoy the same compatibility benefits C and Python
> > > clients
> > > > > > will enjoy. It will be very silly if Apache Kafka clients are the
> > > worst
> > > > > > clients out there from compatibility POV. The KIP can be
> > implemented
> > > in
> > > > > > parts, but I really want to see an effort to build the Java
> client
> > > > > > compatibility into the KIP and if possible into the release too.
> > > > > >
> > > > >
> > > > > I wouldn't conflate those two things. Changing the compatibility
> > > approach
> > > > > of the Java clients could easily be another KIP (and probably a
> large
> > > > one,
> > > > > too). There are already high quality, well maintained clients
> trying
> > to
> > > > use
> > > > > a different approach and this addresses their needs. Blocking the
> > > entire
> > > > > ecosystem on the Java client seems problematic.
> > > > >
> > > > > That said, agreed that it would be bad for the Java client to have
> > the
> > > > > worst compatibility story.
> > > > >
> > > > > -Ewen
> > > > >
> > > > >
> > > > > >
> > > > > > Gwen
> > > > > >
> > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson <
> > ja...@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Bumping this thread. I talked with Ashish and Magnus about this
> > KIP
> > > > > > offline
> > > > > > > and I'm gradually coming over. The new API actually stands by
> > > itself
> > > > > > > outside of the discussion about whether the client should
> support
> > > > > > backwards
> > > > > > > compatibility or not. For the Java client, we could continue to
> > > > support
> > > > > > the
> > > > > > > current compatibility approach in which the client supports
> only
> > > > > brokers
> > > > > > > with the same version or greater. In that case, we would use
> this
> > > API
> > > > > > only
> > > > > > > to assert that the current API versions are all supported, and
> > > raise
> > > > an
> > > > > > > exception if they are not. This gives us the capability going
> > > forward
> > > > > to
> > > > > > > detect when the client is talking to an older broker, which we
> > > don't
> > > > > have
> > > > > > > right now. This check should be straightforward, so we could do
> > it
> > > > now,
> > > > > > > which would resolve some of the uneasiness about having an
> unused
> > > > > feature
> > > > > > > which we depended on other clients to test for us. Does that
> make
> > > > sense
> > > > > > or
> > > > > > > not?
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh <
> > asi...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > We have proposed and discussed majorly three approaches so
> far,
> > > > there
> > > > > > > were
> > > > > > > > many minor versions with small variations. Comparing them
> > really
> > > > > > > requires a
> > > > > > > > side by side proposal and their pros/cons, and I agree with
> > > others
> > > > > that
> > > > > > > > this has been lacking in the KIP. We just updated the KIP
> with
> > > > > > following
> > > > > > > > details.
> > > > > > > >
> > > > > > > > 1. Provide proposed changes in all the three proposals we
> have
> > > > > > discussed
> > > > > > > so
> > > > > > > > far. Except the current proposal, these proposals are in
> > rejected
> > > > > > > > alternatives.
> > > > > > > > 2. Provide reasoning on why the rejected proposals were
> > rejected.
> > > > > > > > 3. Add scenarios for all of these proposals from a client
> > > developer
> > > > > and
> > > > > > > > core Kafka developer point of view.
> > > > > > > >
> > > > > > > > As we are really close to 0.10 deadline, a quick round of
> > voting
> > > > will
> > > > > > > > really help. If you really do not like the idea, please feel
> > free
> > > > to
> > > > > > say
> > > > > > > > so. If the vote fails for the current proposal, it can at
> lease
> > > > > provide
> > > > > > > > recommendations that we should consider for next version of
> > > > proposal
> > > > > > and
> > > > > > > > put it up for vote again for next release. However, as stated
> > > > earlier
> > > > > > by
> > > > > > > > multiple people having this ASAP will be awesome.
> > > > > > > >
> > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers <
> > > > dana.pow...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira <
> > > g...@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > "I think I would make this approach work by looking at
> the
> > > > > released
> > > > > > > > > server
> > > > > > > > > > version documentation for each version that I am trying
> to
> > > > > support
> > > > > > > and
> > > > > > > > > test
> > > > > > > > > > against*, manually identify the expected "protocol
> vectors"
> > > > each
> > > > > > > > > supports,
> > > > > > > > > > store that as a map of vectors to "broker versions",
> check
> > > each
> > > > > > > vector
> > > > > > > > at
> > > > > > > > > > runtime until I find a match, and write code
> compatibility
> > > > checks
> > > > > > > from
> > > > > > > > > > there."
> > > > > > > > > >
> > > > > > > > > > How is this better than a global version ID?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As a client developer, it seems roughly the same. I think
> it
> > > > > probably
> > > > > > > > > avoids the server development workflow issues, and possibly
> > the
> > > > > need
> > > > > > to
> > > > > > > > > agree on semantics of the global version ID? But others
> > surely
> > > > are
> > > > > > more
> > > > > > > > > qualified than me to comment on that part.
> > > > > > > > >
> > > > > > > > > -Dana
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Ashish
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ewen
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
>



-- 

Regards,
Ashish

Reply via email to