Good summary!
Some comments inline:
2016-03-11 22:59 GMT+01:00 Ashish Singh <asi...@cloudera.com>:
> Sounds like we are mostly in agreement. Following are the key points.
>
> 1. Every time a protocol version changes, for any request/response,
> broker version, ApiVersion, will be bumped up.
>
Maybe clarify that the protocol version (=broker version) bump is implicit
by a release (formal/final or interim).
> 2. Protocol documentation will be versioned with broker version. Every
> time there is a broker version change, protocol documentation version
> needs
> to be updated and linked to main documentation page.
> 3. Deprecation of protocol version will be done via marking the version
> as deprecated on the protocol documentation.
> 4. On getting unknown protocol version, broker will send an empty
> response, instead of simply closing client connection.
> 5. Metadata response will be enhanced to also contain broker version,
> VersionInt and VersionString. VersionString will contain internal
> version information.
>
Clarification: only needs to include version of the responding broker, not
for all returned brokers.
> 6. Metadata request with single null topic and size of -1 can be used to
> fetch metadata response with only broker version information and no
> topic/broker info.
7. On receiving a metadata request with single null topic with size of
> -1, broker will respond with only broker version.
>
Clarification: It should just skip the topic enumeration, broker
enumeration should still be included
since this is used to seed the broker list in the client.
>
> Please correct/add, if I missed out something. If the aforementioned
> changes sound good, I can update the KIP-35 wiki, WIP PR and start a Vote
> thread.
>
> On Fri, Mar 11, 2016 at 12:48 PM, Magnus Edenhill <mag...@edenhill.se>
> wrote:
>
> > I'm not sure supporting specific interim versions between releases are
> > really that big of a concern,
> > for a start the protocol changes may be in flux and not settle until the
> > formal release, secondly
> > the 3rd party clients typically lag behind at least until the formal
> > release before they implement support (for the first stated reason..).
> > But this is still a good point and if we could use the version fields to
> > specify a point between
> > two formal releases then that would be useful to ease client development
> > during that period.
> > Grabbing 0.10.0 from versionInt and "IV<N>" from versionString is an
> > acceptable solution as long
> > as there is some way for a client to distinguish the formal release.
> >
> >
> > /Magnus
> >
> >
> >
> >
> > 2016-03-11 20:27 GMT+01:00 Gwen Shapira <g...@confluent.io>:
> >
> > > Yeah, I'm not sure that 0.10.0-IV1 and 0.10.0-IV2 is what Magnus had
> > > in mind when he was advocating for release versions in the protocol.
> > >
> > > But - if we serialize both the string and the integer Id of ApiVersion
> > > into the Metadata object, I think both Magnus and Jay will be happy :)
> > >
> > > Gwen
> > >
> > > On Fri, Mar 11, 2016 at 11:22 AM, Ismael Juma <ism...@juma.me.uk>
> wrote:
> > > > We introduced a way to bump the API version in between releases as
> part
> > > of
> > > > the KIP-31/KIP-32 by the way. Extending that could maybe work. Take a
> > > look
> > > > at the ApiVersion class and its documentation.
> > > >
> > > > Ismael
> > > > On 11 Mar 2016 19:06, "Gwen Shapira" <g...@confluent.io> wrote:
> > > >
> > > >> Magnus,
> > > >>
> > > >> If we go with release version as protocol version (which I agree is
> > > >> much more user-friendly) - what will be the release version on
> trunk?
> > > >> 0.10.0-SNAPSHOT?
> > > >> How will clients handle the fact that some 0.10.0-SNAPSHOT will have
> > > >> different protocol than others (because we modify the protocol
> > > >> multiple times between releases)?
> > > >>
> > > >> Gwen
> > > >>
> > > >> On Thu, Mar 10, 2016 at 1:52 PM, Magnus Edenhill <
> mag...@edenhill.se>
> > > >> wrote:
> > > >> > Hi all,
> > > >> >
> > > >> > sorry for joining late in the game, the carribean got in the way.
> > > >> >
> > > >> > My thoughts:
> > > >> >
> > > >> > There is no way around the chicken&egg problem, so the sooner we
> can
> > > >> > add protocol versioning functionality the better and we'll add
> > > heuristics
> > > >> > in clients to
> > > >> > handle the migration period (e.g, what Dana has done in
> > kafka-python).
> > > >> > The focus at this point should be to mitigate the core issue
> (allow
> > > >> clients
> > > >> > to know what is supported)
> > > >> > in the least intrusive way. Hopefully we can redesign the protocol
> > in
> > > the
> > > >> > future to add proper
> > > >> > response headers, etc.
> > > >> >
> > > >> > I'm with Data that reusing the broker version as a protocol
> version
> > > will
> > > >> > work just fine and
> > > >> > saves us from administrating another version.
> > > >> > From a client's perspective an explicit protocol version doesn't
> > > really
> > > >> add
> > > >> > any value.
> > > >> > I'd rather maintain a mapping of actual broker versions to
> supported
> > > >> > protocol requests rather than
> > > >> > some independent protocol version that still needs to be
> translated
> > > to a
> > > >> > broker version for
> > > >> > proper code maintainability / error messages / etc.
> > > >> >
> > > >> >
> > > >> > Thus my suggestion is in line with some of the previous speakers,
> > > that is
> > > >> > is to keep things
> > > >> > simple and bump the MetadataRequest version to 1 by adding a
> > > >> VersionString
> > > >> > ("0.9.1.0")
> > > >> > and VersionInt (0x00090100) field to the response.
> > > >> > These fields return version information for the current
> connection's
> > > >> broker
> > > >> > only, not for other broker's
> > > >> > in the cluster:
> > > >> > Providing version information for other brokers doesn't really
> serve
> > > any
> > > >> > purpose:
> > > >> > a) the information is cached by the responding broker so it might
> > be
> > > >> > outdated ( = cant be trusted)
> > > >> > b) by the time the client connects to a given broker it might
> have
> > > >> upgraded
> > > >> >
> > > >> > This means that a client (that is interested in protocol
> versioning)
> > > will
> > > >> > need to query each
> > > >> > connection's version any way. Since MetadataRequets are typically
> > > already
> > > >> > sent on connection set up
> > > >> > this seems to be the proper place to put it.
> > > >> >
> > > >> > The MetadataRequest semantics should also be extended to allow
> > asking
> > > >> only
> > > >> > for cluster and version information,
> > > >> > but not the topic list since this might have negative performance
> > > impact
> > > >> on
> > > >> > large clusters with many topics.
> > > >> > One way to achieve this would be to provide one single Null topic
> in
> > > the
> > > >> > request (length=-1).
> > > >> >
> > > >> > Sending a new Metadata V1 request to an old broker will cause the
> > > >> > connection to be closed and
> > > >> > the client will need to use this as a heuristic to downgrade its
> > > protocol
> > > >> > ambitions to an older version
> > > >> > (either by some default value or by user configuration).
> > > >> >
> > > >> >
> > > >> > /Magnus
> > > >> >
> > > >> >
> > > >> > 2016-03-10 20:04 GMT+01:00 Ashish Singh <asi...@cloudera.com>:
> > > >> >
> > > >> >> @Magnus,
> > > >> >>
> > > >> >> Does the latest suggestion sound OK to you. I am planning to
> update
> > > PR
> > > >> >> based on latest suggestion.
> > > >> >>
> > > >> >> On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh <
> asi...@cloudera.com
> > >
> > > >> wrote:
> > > >> >>
> > > >> >> >
> > > >> >> >
> > > >> >> > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps <j...@confluent.io>
> > > wrote:
> > > >> >> >
> > > >> >> >> Hey Ashish,
> > > >> >> >>
> > > >> >> >> Both good points.
> > > >> >> >>
> > > >> >> >> I think the issue with the general metadata request is the
> same
> > as
> > > >> the
> > > >> >> >> issue with a version-specific metadata request from the other
> > > >> >> >> proposal--basically it's a chicken and egg problem, to find
> out
> > > >> anything
> > > >> >> >> about the cluster you have to be able to communicate something
> > in
> > > a
> > > >> >> format
> > > >> >> >> the server can understand without knowing a priori what
> version
> > > it's
> > > >> >> on. I
> > > >> >> >> guess the question is how can you continue to evolve the
> > metadata
> > > >> >> request
> > > >> >> >> (whether it is the existing metadata or a protocol-version
> > > specific
> > > >> >> >> metadata request) given that you need this information to
> > > bootstrap
> > > >> you
> > > >> >> >> have to be more careful in how that request evolves.
> > > >> >> >>
> > > >> >> > You are correct. It's just that protocol version request would
> be
> > > very
> > > >> >> > specific to retrieve the protocol versions. Changes to protocol
> > > >> version
> > > >> >> > request itself should be very rare, if at all. However, the
> > general
> > > >> >> > metadata request carries a lot more information and its format
> is
> > > more
> > > >> >> > probable to evolve. This boils down to higher probability of
> > change
> > > >> vs a
> > > >> >> > definite network round-trip for each re/connect. It does sound
> > > like,
> > > >> it
> > > >> >> is
> > > >> >> > better to avoid a definite penalty than to avoid a probable
> rare
> > > >> issue.
> > > >> >> >
> > > >> >> >>
> > > >> >> >> I think deprecation/removal may be okay. Ultimately clients
> will
> > > >> always
> > > >> >> >> use
> > > >> >> >> the highest possible version of the protocol the server
> supports
> > > so
> > > >> if
> > > >> >> >> we've already deprecated and removed your highest version then
> > you
> > > >> are
> > > >> >> >> screwed and you're going to get an error no matter what,
> right?
> > > >> >> Basically
> > > >> >> >> there is nothing dynamic you can do in that case.
> > > >> >> >>
> > > >> >> > Sure, this should be expected. Just wanted to make sure
> > > deprecation is
> > > >> >> > still on the table.
> > > >> >> >
> > > >> >> >>
> > > >> >> >> -Jay
> > > >> >> >>
> > > >> >> >> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh <
> > asi...@cloudera.com
> > > >
> > > >> >> wrote:
> > > >> >> >>
> > > >> >> >> > Hello Jay,
> > > >> >> >> >
> > > >> >> >> > The overall approach sounds good. I do realize that this
> > > discussion
> > > >> >> has
> > > >> >> >> > gotten too lengthy and is starting to shoot tangents. Maybe
> a
> > > KIP
> > > >> call
> > > >> >> >> will
> > > >> >> >> > help us getting to a decision faster. I do have a few
> > questions
> > > >> >> though.
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps <j...@confluent.io
> >
> > > >> wrote:
> > > >> >> >> >
> > > >> >> >> > > Yeah here is my summary of my take:
> > > >> >> >> > >
> > > >> >> >> > > 1. Negotiating a per-connection protocol actually does
> add a
> > > lot
> > > >> of
> > > >> >> >> > > complexity to clients (many more failure states to get
> > right).
> > > >> >> >> > >
> > > >> >> >> > > 2. Having the client configure the protocol version
> manually
> > > is
> > > >> >> doable
> > > >> >> >> > now
> > > >> >> >> > > but probably a worse state. I suspect this will lead to
> more
> > > not
> > > >> >> less
> > > >> >> >> > > confusion.
> > > >> >> >> > >
> > > >> >> >> > > 3. I don't think the current state is actually that bad.
> > > >> Integrators
> > > >> >> >> > pick a
> > > >> >> >> > > conservative version and build against that. There is a
> > > tradeoff
> > > >> >> >> between
> > > >> >> >> > > getting the new features and being compatible with old
> Kafka
> > > >> >> versions.
> > > >> >> >> > But
> > > >> >> >> > > a large part of this tradeoff is essential since new
> > features
> > > >> aren't
> > > >> >> >> > going
> > > >> >> >> > > to magically appear on old servers, so even if you upgrade
> > > your
> > > >> >> client
> > > >> >> >> > you
> > > >> >> >> > > likely aren't going to get the new stuff (since we will
> end
> > up
> > > >> >> >> > dynamically
> > > >> >> >> > > turning it off). Having client features that are there but
> > > don't
> > > >> >> work
> > > >> >> >> > > because you're on an old cluster may actually be a worse
> > > >> experience
> > > >> >> if
> > > >> >> >> > not
> > > >> >> >> > > handled very carefully..
> > > >> >> >> > >
> > > >> >> >> > > 4. The problems Dana brought up are totally orthogonal to
> > the
> > > >> >> problem
> > > >> >> >> of
> > > >> >> >> > > having per-api versions or overall versions. The problem
> was
> > > >> that we
> > > >> >> >> > > changed behavior subtly without changing the version. This
> > > will
> > > >> be
> > > >> >> an
> > > >> >> >> > issue
> > > >> >> >> > > regardless of whether the version is global or not.
> > > >> >> >> > >
> > > >> >> >> > > 5. Using the broker release as the version is strictly
> worse
> > > than
> > > >> >> >> using a
> > > >> >> >> > > global protocol version (0, 1, 2, ...) that increments any
> > > time
> > > >> any
> > > >> >> >> api
> > > >> >> >> > > changes but doesn't increment just because non-protocol
> code
> > > is
> > > >> >> >> changed.
> > > >> >> >> > > The problem with using the broker release version is we
> want
> > > to
> > > >> be
> > > >> >> >> able
> > > >> >> >> > to
> > > >> >> >> > > keep Kafka releasable from any commit which means there
> > isn't
> > > as
> > > >> >> >> clear a
> > > >> >> >> > > sequencing of releases as you would think.
> > > >> >> >> > >
> > > >> >> >> > > 6. We need to consider the case of mixed version clusters
> > > during
> > > >> the
> > > >> >> >> time
> > > >> >> >> > > period when you are upgrading Kafka.
> > > >> >> >> > >
> > > >> >> >> > > So overall I think this is not a critical thing to do
> right
> > > now,
> > > >> but
> > > >> >> >> if
> > > >> >> >> > we
> > > >> >> >> > > are going to do it we should do it in a way that actually
> > > >> improves
> > > >> >> >> > things.
> > > >> >> >> > >
> > > >> >> >> > > Here would be one proposal for that:
> > > >> >> >> > > a. Add a global protocol version that increments with any
> > api
> > > >> >> version
> > > >> >> >> > > update. Move the documentation so that the docs are by
> > > version.
> > > >> This
> > > >> >> >> is
> > > >> >> >> > > basically just a short-hand for a complete set of
> supported
> > > api
> > > >> >> >> versions.
> > > >> >> >> > > b. Include a field in the metadata response for each
> broker
> > > that
> > > >> >> adds
> > > >> >> >> the
> > > >> >> >> > > protocol version.
> > > >> >> >> > >
> > > >> >> >> > There might be an issue here where the metadata request
> > version
> > > >> sent
> > > >> >> by
> > > >> >> >> > client is not supported by broker, an older broker. However,
> > if
> > > we
> > > >> are
> > > >> >> >> > clearly stating that a client is not guaranteed to work with
> > an
> > > >> older
> > > >> >> >> > broker then this becomes expected. This will potentially
> limit
> > > us
> > > >> in
> > > >> >> >> terms
> > > >> >> >> > of supporting downgrades though, if we ever want to.
> > > >> >> >> >
> > > >> >> >> > > c. To maintain the protocol version this information will
> > > have to
> > > >> >> get
> > > >> >> >> > > propagated with the rest of the broker metadata like host,
> > > port,
> > > >> id,
> > > >> >> >> etc.
> > > >> >> >> > >
> > > >> >> >> > > The instructions to clients would be:
> > > >> >> >> > > - By default you build against a single conservative Kafka
> > > >> protocol
> > > >> >> >> > version
> > > >> >> >> > > and we carry that support forward, as today
> > > >> >> >> > >
> > > >> >> >> > If I am getting this correct, this will mean we will never
> > > >> >> >> deprecate/remove
> > > >> >> >> > any protocol version in future. Having some way to
> > > deprecate/remove
> > > >> >> >> older
> > > >> >> >> > protocol versions will probably be a good idea. It is
> possible
> > > with
> > > >> >> the
> > > >> >> >> > global protocol version approach, it could be as simple as
> > > marking
> > > >> a
> > > >> >> >> > protocol deprecated in protocol doc before removing it. Just
> > > want
> > > >> to
> > > >> >> >> make
> > > >> >> >> > sure deprecation is still on the table.
> > > >> >> >> >
> > > >> >> >> > > - If you want to get fancy you can use the protocol
> version
> > > >> field in
> > > >> >> >> the
> > > >> >> >> > > metadata request to more dynamically chose what features
> are
> > > >> >> available
> > > >> >> >> > and
> > > >> >> >> > > select api versions appropriately. This is purely
> optional.
> > > >> >> >> > >
> > > >> >> >> > > -Jay
> > > >> >> >> > >
> > > >> >> >> > > On Thu, Mar 3, 2016 at 9:38 PM, Jason Gustafson <
> > > >> ja...@confluent.io
> > > >> >> >
> > > >> >> >> > > wrote:
> > > >> >> >> > >
> > > >> >> >> > > > I talked with Jay about this KIP briefly this morning,
> so
> > > let
> > > >> me
> > > >> >> >> try to
> > > >> >> >> > > > summarize the discussion (I'm sure he'll jump in if I
> get
> > > >> anything
> > > >> >> >> > > wrong).
> > > >> >> >> > > > Apologies in advance for the length.
> > > >> >> >> > > >
> > > >> >> >> > > > I think we both share some skepticism that a request
> with
> > > all
> > > >> the
> > > >> >> >> > > supported
> > > >> >> >> > > > versions of all the request APIs is going to be a useful
> > > >> primitive
> > > >> >> >> to
> > > >> >> >> > try
> > > >> >> >> > > > and build client compatibility around. In practice I
> think
> > > >> people
> > > >> >> >> would
> > > >> >> >> > > end
> > > >> >> >> > > > up checking for particular request versions in order to
> > > >> determine
> > > >> >> if
> > > >> >> >> > the
> > > >> >> >> > > > broker is 0.8 or 0.9 or whatever, and then change
> behavior
> > > >> >> >> accordingly.
> > > >> >> >> > > I'm
> > > >> >> >> > > > wondering if there's a reasonable way to handle the
> > version
> > > >> >> >> responses
> > > >> >> >> > > that
> > > >> >> >> > > > doesn't amount to that. Maybe you could try to capture
> > > feature
> > > >> >> >> > > > compatibility by checking the versions for a subset of
> > > request
> > > >> >> >> types?
> > > >> >> >> > For
> > > >> >> >> > > > example, to ensure that you can use the new consumer
> API,
> > > you
> > > >> >> check
> > > >> >> >> > that
> > > >> >> >> > > > the group coordinator request is present, the offset
> > commit
> > > >> >> request
> > > >> >> >> > > version
> > > >> >> >> > > > is greater than 2, the offset fetch request is greater
> > than
> > > 1,
> > > >> and
> > > >> >> >> the
> > > >> >> >> > > join
> > > >> >> >> > > > group request is present. And to ensure compatibility
> with
> > > >> KIP-32,
> > > >> >> >> > maybe
> > > >> >> >> > > > you only need to check the appropriate versions of the
> > fetch
> > > >> and
> > > >> >> >> > produce
> > > >> >> >> > > > requests. That sounds kind of complicated to keep track
> of
> > > and
> > > >> you
> > > >> >> >> > > probably
> > > >> >> >> > > > end up trying to handle combinations which aren't even
> > > >> possible in
> > > >> >> >> > > > practice.
> > > >> >> >> > > >
> > > >> >> >> > > > The alternative is to use a single API version. It could
> > be
> > > the
> > > >> >> >> Kafka
> > > >> >> >> > > > release version, but then you need to figure out how to
> > > handle
> > > >> >> users
> > > >> >> >> > who
> > > >> >> >> > > > are running off of trunk since multiple API versions
> will
> > > >> >> typically
> > > >> >> >> > > change
> > > >> >> >> > > > between releases. Perhaps it makes more sense to keep a
> > > >> separate
> > > >> >> API
> > > >> >> >> > > > version number which is incremented every time any one
> of
> > > the
> > > >> API
> > > >> >> >> > > versions
> > > >> >> >> > > > increases? This also decouples the protocol from the
> Kafka
> > > >> >> >> > distribution.
> > > >> >> >> > > >
> > > >> >> >> > > > As far as whether there should be a separate request or
> > > not, I
> > > >> get
> > > >> >> >> > > Becket's
> > > >> >> >> > > > point that you would only need to do the version check
> > once
> > > >> when a
> > > >> >> >> > > > connection is established, but another round trip still
> > > >> >> complicates
> > > >> >> >> the
> > > >> >> >> > > > picture quite a bit. Before you just need to send a
> > metadata
> > > >> >> >> request to
> > > >> >> >> > > > bootstrap yourself to the cluster, but now you need to
> do
> > > >> version
> > > >> >> >> > > > negotiation before you can even do that, and then you
> need
> > > to
> > > >> try
> > > >> >> >> adapt
> > > >> >> >> > > to
> > > >> >> >> > > > the versions reported. Jay brought up the point that you
> > > >> probably
> > > >> >> >> > > wouldn't
> > > >> >> >> > > > design a protocol from scratch to work this way. Using
> the
> > > >> >> metadata
> > > >> >> >> > > request
> > > >> >> >> > > > would be better if it's possible, but you need a way to
> > > handle
> > > >> the
> > > >> >> >> fact
> > > >> >> >> > > > that a broker's version might be stale by the time you
> > > connect
> > > >> to
> > > >> >> >> it.
> > > >> >> >> > And
> > > >> >> >> > > > even then you're going to have to deal internally with
> the
> > > >> >> >> complexity
> > > >> >> >> > > > involved in trying to upgrade/downgrade dynamically,
> which
> > > >> sounds
> > > >> >> >> to me
> > > >> >> >> > > > like it would have a ton of edge cases.
> > > >> >> >> > > >
> > > >> >> >> > > > Taking a bit of a step back, any solution is probably
> > going
> > > to
> > > >> be
> > > >> >> >> > painful
> > > >> >> >> > > > since the Kafka protocol was not designed for this use
> > case.
> > > >> >> >> Currently
> > > >> >> >> > > what
> > > >> >> >> > > > that means for clients that /want/ to support
> > compatibility
> > > >> across
> > > >> >> >> > broker
> > > >> >> >> > > > versions is that they need to have the user tell them
> the
> > > >> broker
> > > >> >> >> > version
> > > >> >> >> > > > through configuration (e.g. librdkafka has a
> > > "protocol.version"
> > > >> >> >> field
> > > >> >> >> > for
> > > >> >> >> > > > this purpose). The only real problem with this in my
> mind
> > is
> > > >> that
> > > >> >> we
> > > >> >> >> > > don't
> > > >> >> >> > > > have a graceful way to detect request incompatibility,
> > > which is
> > > >> >> why
> > > >> >> >> > there
> > > >> >> >> > > > are so many questions on the user list which basically
> > > amount
> > > >> to
> > > >> >> the
> > > >> >> >> > > client
> > > >> >> >> > > > hanging because the broker refuses to respond to a
> request
> > > it
> > > >> >> >> doesn't
> > > >> >> >> > > > understand. If you solve this problem, then depending on
> > > >> >> >> configuration
> > > >> >> >> > > > seems totally reasonable and we can skip trying to
> > implement
> > > >> >> request
> > > >> >> >> > > > version negotiation. Magnus's solution in this KIP may
> > seem
> > > a
> > > >> >> little
> > > >> >> >> > > hacky,
> > > >> >> >> > > > but it also seems like the only way to do it without
> > > changing
> > > >> the
> > > >> >> >> > header.
> > > >> >> >> > > >
> > > >> >> >> > > > The Spark problem mentioned above is interesting and I
> > agree
> > > >> that
> > > >> >> it
> > > >> >> >> > > sucks
> > > >> >> >> > > > for frameworks that need to ship the kafka client
> library
> > > since
> > > >> >> they
> > > >> >> >> > have
> > > >> >> >> > > > to figure out how to bundle multiple versions.
> Ultimately
> > > if we
> > > >> >> >> want to
> > > >> >> >> > > > solve this problem, then it sounds like we need to
> commit
> > to
> > > >> >> >> > maintaining
> > > >> >> >> > > > compatibility with older versions of Kafka in the client
> > > going
> > > >> >> >> forward.
> > > >> >> >> > > > That's a lot bigger decision and it matters less whether
> > the
> > > >> >> broker
> > > >> >> >> > > version
> > > >> >> >> > > > is found through configuration, topic metadata, or a new
> > > >> request
> > > >> >> >> type.
> > > >> >> >> > > >
> > > >> >> >> > > > -Jason
> > > >> >> >> > > >
> > > >> >> >> > > >
> > > >> >> >> > > > On Thu, Mar 3, 2016 at 3:59 PM, Becket Qin <
> > > >> becket....@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> > > >
> > > >> >> >> > > > > Hi Ashish,
> > > >> >> >> > > > >
> > > >> >> >> > > > > In approach (1), the clients will still be able to
> > talked
> > > to
> > > >> >> >> multiple
> > > >> >> >> > > > > versions of Kafka brokers as long as the clients
> version
> > > is
> > > >> not
> > > >> >> >> > higher
> > > >> >> >> > > > than
> > > >> >> >> > > > > the broker version, right?
> > > >> >> >> > > > >
> > > >> >> >> > > > > From Spark's point of view, it seems the difference is
> > > >> whether
> > > >> >> >> Spark
> > > >> >> >> > > can
> > > >> >> >> > > > > independently update their Kafka clients dependency or
> > > not.
> > > >> More
> > > >> >> >> > > > > specifically, consider the following three scenarios:
> > > >> >> >> > > > > A. Spark has some new features that do not rely on
> > > clients or
> > > >> >> >> brokers
> > > >> >> >> > > in
> > > >> >> >> > > > a
> > > >> >> >> > > > > new Kafka release.
> > > >> >> >> > > > > B. Spark has some new features that only rely on the
> > > clients
> > > >> in
> > > >> >> a
> > > >> >> >> new
> > > >> >> >> > > > Kafka
> > > >> >> >> > > > > release, but not rely on the brokers in a new Kafka
> > > release.
> > > >> >> e.g.
> > > >> >> >> New
> > > >> >> >> > > > > client provides a listTopic() method.
> > > >> >> >> > > > > C. Spark has some new features that rely on both the
> > > clients
> > > >> and
> > > >> >> >> > > brokers
> > > >> >> >> > > > in
> > > >> >> >> > > > > a new Kafka release. e.g timestamp field.
> > > >> >> >> > > > >
> > > >> >> >> > > > > For A, Spark does not need to update the Kafka
> > dependency
> > > >> >> because
> > > >> >> >> > there
> > > >> >> >> > > > is
> > > >> >> >> > > > > no need and the old clients can talk to both new and
> old
> > > >> Kafka
> > > >> >> >> > brokers.
> > > >> >> >> > > > > For C, Spark has to wait for broker upgrade anyways.
> > > >> >> >> > > > > So in the above two scenarios, there is not much
> > > difference
> > > >> >> >> between
> > > >> >> >> > > > > approach (1) and (2).
> > > >> >> >> > > > >
> > > >> >> >> > > > > B is a tricky scenario. Because it is possible that we
> > > >> introduce
> > > >> >> >> both
> > > >> >> >> > > > > listTopic() and the timestamp field in the same Kafka
> > > >> release,
> > > >> >> >> and we
> > > >> >> >> > > > don't
> > > >> >> >> > > > > know if Spark needs both or only uses listTopic().
> > > >> >> >> > > > > This indicates the client should work fine if a method
> > is
> > > >> >> >> supported
> > > >> >> >> > and
> > > >> >> >> > > > > should throw exception when a method is not
> supported. I
> > > >> think
> > > >> >> we
> > > >> >> >> can
> > > >> >> >> > > do
> > > >> >> >> > > > > the following:
> > > >> >> >> > > > > 0. Clients always use its highest request version. The
> > > >> clients
> > > >> >> >> keeps
> > > >> >> >> > a
> > > >> >> >> > > > > static final map recording the minimum required
> > ApiVersion
> > > >> for
> > > >> >> >> each
> > > >> >> >> > > > > request.
> > > >> >> >> > > > > 1. When connect to a broker, the clients always send
> an
> > > >> >> >> > > ApiVersionRequest
> > > >> >> >> > > > > to the broker.
> > > >> >> >> > > > > 2. The broker replies with the its highest supported
> > > >> ApiVersion.
> > > >> >> >> > > > > 3. Before sending a request, the clients checks the
> > > minimum
> > > >> >> >> required
> > > >> >> >> > > > > ApiVersion for that request. If the broker returned
> > > >> ApiVersion
> > > >> >> is
> > > >> >> >> > > higher
> > > >> >> >> > > > > than this minimum required ApiVersion, then we can
> > > proceed.
> > > >> >> >> Otherwise
> > > >> >> >> > > we
> > > >> >> >> > > > > throw something like NotSupportedOperationException.
> > > >> >> >> > > > >
> > > >> >> >> > > > > With this approach, scenario B will also work unless
> > Spark
> > > >> calls
> > > >> >> >> some
> > > >> >> >> > > > > function that is not supported by the Kafka broker,
> > which
> > > >> makes
> > > >> >> it
> > > >> >> >> > > become
> > > >> >> >> > > > > scenario C.
> > > >> >> >> > > > >
> > > >> >> >> > > > > Thoughts?
> > > >> >> >> > > > >
> > > >> >> >> > > > > Thanks,
> > > >> >> >> > > > >
> > > >> >> >> > > > > Jiangjie (Becket) Qin
> > > >> >> >> > > > >
> > > >> >> >> > > > > On Thu, Mar 3, 2016 at 11:24 AM, Ashish Singh <
> > > >> >> >> asi...@cloudera.com>
> > > >> >> >> > > > wrote:
> > > >> >> >> > > > >
> > > >> >> >> > > > > > On Wed, Mar 2, 2016 at 8:29 PM, Becket Qin <
> > > >> >> >> becket....@gmail.com>
> > > >> >> >> > > > wrote:
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > > Hi Jason,
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > I was thinking that every time when we connect to
> a
> > > >> broker,
> > > >> >> we
> > > >> >> >> > > first
> > > >> >> >> > > > > send
> > > >> >> >> > > > > > > the version check request. (The version check
> > request
> > > >> itself
> > > >> >> >> > should
> > > >> >> >> > > > be
> > > >> >> >> > > > > > very
> > > >> >> >> > > > > > > simple and never changes across all server
> > releases.)
> > > >> This
> > > >> >> >> does
> > > >> >> >> > add
> > > >> >> >> > > > an
> > > >> >> >> > > > > > > additional round trip, but given reconnect is
> rare,
> > > it is
> > > >> >> >> > probably
> > > >> >> >> > > > > fine.
> > > >> >> >> > > > > > On
> > > >> >> >> > > > > > > the client side, the client will always send
> request
> > > >> using
> > > >> >> the
> > > >> >> >> > > lowest
> > > >> >> >> > > > > > > supported version across all brokers. That means
> if
> > a
> > > >> Kafka
> > > >> >> >> > cluster
> > > >> >> >> > > > is
> > > >> >> >> > > > > > > downgrading, we will use the downgraded protocol
> as
> > > soon
> > > >> as
> > > >> >> >> the
> > > >> >> >> > > > client
> > > >> >> >> > > > > > > connected to an older broker.
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > This sounds interesting and very similar to current
> > > >> >> suggestion.
> > > >> >> >> > > > However,
> > > >> >> >> > > > > > just to make sure I am getting it right, you are
> > > suggesting
> > > >> >> >> send a
> > > >> >> >> > > > > separate
> > > >> >> >> > > > > > request only for release version?
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > @Ashish,
> > > >> >> >> > > > > > > Can you help me understand the pain points from
> > other
> > > >> open
> > > >> >> >> source
> > > >> >> >> > > > > > projects
> > > >> >> >> > > > > > > that you mentioned a little more? There are two
> > > different
> > > >> >> >> levels
> > > >> >> >> > of
> > > >> >> >> > > > > > > requirements:
> > > >> >> >> > > > > > > 1. User wants to know if the client is compatible
> > with
> > > >> the
> > > >> >> >> broker
> > > >> >> >> > > or
> > > >> >> >> > > > > not.
> > > >> >> >> > > > > > > 2. User wants the client and the broker to
> negotiate
> > > the
> > > >> >> >> protocol
> > > >> >> >> > > on
> > > >> >> >> > > > > > their
> > > >> >> >> > > > > > > own.
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > Not sure which category it falls in, but below is
> the
> > > >> excerpt
> > > >> >> >> from
> > > >> >> >> > > > Mark,
> > > >> >> >> > > > > a
> > > >> >> >> > > > > > spark dev, who has been trying to upgrade spark
> kafka
> > > >> >> >> integration
> > > >> >> >> > to
> > > >> >> >> > > > use
> > > >> >> >> > > > > > 0.9 clients.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > Based on what I understand, users of Kafka need to
> > > upgrade
> > > >> >> their
> > > >> >> >> > > > brokers
> > > >> >> >> > > > > to
> > > >> >> >> > > > > > Kafka 0.9.x first, before they upgrade their clients
> > to
> > > >> Kafka
> > > >> >> >> > 0.9.x.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > > However, that presents a problem to other projects
> > > that
> > > >> >> >> integrate
> > > >> >> >> > > > with
> > > >> >> >> > > > > > Kafka (Spark, Flume, Storm, etc.). From here on, I
> > will
> > > >> speak
> > > >> >> >> for
> > > >> >> >> > > > Spark +
> > > >> >> >> > > > > > Kafka, since that's the one I am most familiar with.
> > > >> >> >> > > > > > In the light of compatibility (or the lack thereof)
> > > between
> > > >> >> >> 0.8.x
> > > >> >> >> > and
> > > >> >> >> > > > > > 0.9.x, Spark is faced with a problem of what
> > version(s)
> > > of
> > > >> >> >> Kafka to
> > > >> >> >> > > be
> > > >> >> >> > > > > > compatible with, and has 2 options (discussed in
> this
> > PR
> > > >> >> >> > > > > > <https://github.com/apache/spark/pull/11143>):
> > > >> >> >> > > > > > 1. We either upgrade to Kafka 0.9, dropping support
> > for
> > > >> 0.8.
> > > >> >> >> Storm
> > > >> >> >> > > and
> > > >> >> >> > > > > > Flume are already on this path.
> > > >> >> >> > > > > > 2. We introduce complexity in our code to support
> both
> > > 0.8
> > > >> and
> > > >> >> >> 0.9
> > > >> >> >> > > for
> > > >> >> >> > > > > the
> > > >> >> >> > > > > > entire duration of our next major release (Apache
> > Spark
> > > >> 2.x).
> > > >> >> >> > > > > > I'd love to hear your thoughts on which option, you
> > > >> recommend.
> > > >> >> >> > > > > > Long term, I'd really appreciate if Kafka could do
> > > >> something
> > > >> >> >> that
> > > >> >> >> > > > doesn't
> > > >> >> >> > > > > > make Spark having to support two, or even more
> > versions
> > > of
> > > >> >> >> Kafka.
> > > >> >> >> > > And,
> > > >> >> >> > > > if
> > > >> >> >> > > > > > there is something that I, personally, and Spark
> > project
> > > >> can
> > > >> >> do
> > > >> >> >> in
> > > >> >> >> > > your
> > > >> >> >> > > > > > next release candidate phase to make things easier,
> > > please
> > > >> do
> > > >> >> >> let
> > > >> >> >> > us
> > > >> >> >> > > > > know.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > This issue has made other projects worry about how
> > they
> > > are
> > > >> >> >> going
> > > >> >> >> > to
> > > >> >> >> > > > keep
> > > >> >> >> > > > > > up with Kafka releases. Last I heard, take this
> with a
> > > >> pinch
> > > >> >> of
> > > >> >> >> > salt,
> > > >> >> >> > > > > Spark
> > > >> >> >> > > > > > folks are discussing about using Maven profiles to
> > build
> > > >> >> against
> > > >> >> >> > > > multiple
> > > >> >> >> > > > > > Kafka versions at compile time, etc. Also, there are
> > > >> clients
> > > >> >> who
> > > >> >> >> > are
> > > >> >> >> > > > > > relying on class-loading tricks with custom
> > > implementation
> > > >> of
> > > >> >> >> OSGi
> > > >> >> >> > to
> > > >> >> >> > > > > solve
> > > >> >> >> > > > > > such issues. Don't quote me on the stuff I just
> > > mentioned,
> > > >> as
> > > >> >> >> this
> > > >> >> >> > is
> > > >> >> >> > > > > what
> > > >> >> >> > > > > > I have heard during casual discussions. The point I
> am
> > > >> trying
> > > >> >> to
> > > >> >> >> > make
> > > >> >> >> > > > is
> > > >> >> >> > > > > > that Kafka clients are worried about being able to
> > > support
> > > >> >> >> multiple
> > > >> >> >> > > > Kafka
> > > >> >> >> > > > > > broker versions. I am sure we all agree on that.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > I think the second requirement makes more sense
> from a
> > > >> client
> > > >> >> >> > > > > perspective.
> > > >> >> >> > > > > > First req will just tell them that there is a
> problem,
> > > but
> > > >> no
> > > >> >> >> way
> > > >> >> >> > to
> > > >> >> >> > > > work
> > > >> >> >> > > > > > around it.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > Currently in Kafka the principle we are following
> is
> > > to
> > > >> let
> > > >> >> >> > clients
> > > >> >> >> > > > > stick
> > > >> >> >> > > > > > > to a certain version and server will adapt to the
> > > clients
> > > >> >> >> > > > accordingly.
> > > >> >> >> > > > > > > If this KIP doesn't want to break this rule, it
> > seems
> > > we
> > > >> >> >> should
> > > >> >> >> > > > simply
> > > >> >> >> > > > > > let
> > > >> >> >> > > > > > > the clients send the ApiVersion it is using to the
> > > >> brokers
> > > >> >> and
> > > >> >> >> > the
> > > >> >> >> > > > > > brokers
> > > >> >> >> > > > > > > will decide whether to accept or reject the
> clients.
> > > This
> > > >> >> >> means
> > > >> >> >> > > user
> > > >> >> >> > > > > have
> > > >> >> >> > > > > > > to upgrade broker before they upgrade clients.
> This
> > > >> >> satisfies
> > > >> >> >> (1)
> > > >> >> >> > > so
> > > >> >> >> > > > > > that a
> > > >> >> >> > > > > > > newer client will know it does not compatible with
> > an
> > > >> older
> > > >> >> >> > server
> > > >> >> >> > > > > > > immediately.
> > > >> >> >> > > > > > > If this KIP will change that to let the newer
> > clients
> > > >> adapt
> > > >> >> to
> > > >> >> >> > the
> > > >> >> >> > > > > older
> > > >> >> >> > > > > > > brokers, compatibility wise it is a good thing to
> > > have.
> > > >> >> With
> > > >> >> >> > this
> > > >> >> >> > > > now
> > > >> >> >> > > > > > > users are able to upgrade clients before they
> > upgrade
> > > >> Kafka
> > > >> >> >> > > brokers.
> > > >> >> >> > > > > This
> > > >> >> >> > > > > > > means user can upgrade clients even before upgrade
> > > >> servers.
> > > >> >> >> This
> > > >> >> >> > > > > > satisfies
> > > >> >> >> > > > > > > (2) as the newer clients can also talk to the
> older
> > > >> servers.
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > More importantly, this will allow a client to talk
> to
> > > >> multiple
> > > >> >> >> > > versions
> > > >> >> >> > > > > of
> > > >> >> >> > > > > > Kafka.
> > > >> >> >> > > > >
> > > >> >> >> > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > If we decide to go with (2). The benefit is that a
> > > newer
> > > >> >> >> client
> > > >> >> >> > > won't
> > > >> >> >> > > > > > break
> > > >> >> >> > > > > > > when talking to an older broker. But functionality
> > > wise,
> > > >> it
> > > >> >> >> might
> > > >> >> >> > > be
> > > >> >> >> > > > > the
> > > >> >> >> > > > > > > same as an older clients.
> > > >> >> >> > > > > > > In the downgrading case, we probably still have to
> > > notify
> > > >> >> all
> > > >> >> >> the
> > > >> >> >> > > > > users.
> > > >> >> >> > > > > > > For example, if application is sending messages
> with
> > > >> >> timestamp
> > > >> >> >> > and
> > > >> >> >> > > > the
> > > >> >> >> > > > > > > broker got downgraded to an older version that
> does
> > > not
> > > >> >> >> support
> > > >> >> >> > > > > > timestamp.
> > > >> >> >> > > > > > > The clients will suddenly start to throw away
> > > timestamps.
> > > >> >> This
> > > >> >> >> > > might
> > > >> >> >> > > > > > affect
> > > >> >> >> > > > > > > the application logic. In this case even if we
> have
> > > >> clients
> > > >> >> >> > > > > automatically
> > > >> >> >> > > > > > > adapted to a lower version broker, the
> applications
> > > might
> > > >> >> >> still
> > > >> >> >> > > > break.
> > > >> >> >> > > > > > > Hence we still need to notify the users about the
> > case
> > > >> when
> > > >> >> >> the
> > > >> >> >> > > > clients
> > > >> >> >> > > > > > is
> > > >> >> >> > > > > > > newer than the brokers. This is the same for both
> > (1)
> > > and
> > > >> >> (2).
> > > >> >> >> > > > > > > Supporting (2) will introduce more complication on
> > the
> > > >> >> client
> > > >> >> >> > side.
> > > >> >> >> > > > And
> > > >> >> >> > > > > > we
> > > >> >> >> > > > > > > may also have to communicate with users about what
> > > >> function
> > > >> >> is
> > > >> >> >> > > > > supported
> > > >> >> >> > > > > > in
> > > >> >> >> > > > > > > the new clients and what is not supported after
> the
> > > >> protocol
> > > >> >> >> > > > > negotiation
> > > >> >> >> > > > > > > finishes.
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > Totally agreed, however only if clients want to
> > support
> > > >> >> multiple
> > > >> >> >> > > broker
> > > >> >> >> > > > > > versions. If they want to, then I am sure they are
> > > willing
> > > >> to
> > > >> >> >> add
> > > >> >> >> > > some
> > > >> >> >> > > > > > logic on their end.
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > Thanks,
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > Jiangjie (Becket) Qin
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > On Wed, Mar 2, 2016 at 5:58 PM, Dana Powers <
> > > >> >> >> > dana.pow...@gmail.com
> > > >> >> >> > > >
> > > >> >> >> > > > > > wrote:
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > > > > In kafka-python we've been doing something like:
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > > > if version >= (0, 9):
> > > >> >> >> > > > > > > > Do cool new stuff
> > > >> >> >> > > > > > > > elif version >= (0, 8, 2):
> > > >> >> >> > > > > > > > Do some older stuff
> > > >> >> >> > > > > > > > ....
> > > >> >> >> > > > > > > > else:
> > > >> >> >> > > > > > > > raise UnsupportedVersionError
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > > > This will break if / when the new 0.9 apis are
> > > >> completely
> > > >> >> >> > removed
> > > >> >> >> > > > > from
> > > >> >> >> > > > > > > some
> > > >> >> >> > > > > > > > future release, but should handle intermediate
> > > broker
> > > >> >> >> upgrades.
> > > >> >> >> > > > > Because
> > > >> >> >> > > > > > > we
> > > >> >> >> > > > > > > > can't add support for future apis a priori, I
> > think
> > > the
> > > >> >> >> best we
> > > >> >> >> > > > could
> > > >> >> >> > > > > > do
> > > >> >> >> > > > > > > > here is throw an error that request protocol
> > > version X
> > > >> is
> > > >> >> >> not
> > > >> >> >> > > > > > supported.
> > > >> >> >> > > > > > > > For now that comes through as a broken socket
> > > >> connection,
> > > >> >> so
> > > >> >> >> > > there
> > > >> >> >> > > > is
> > > >> >> >> > > > > > an
> > > >> >> >> > > > > > > > error - just not a super helpful one.
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > > > For that reason I'm also in favor of a generic
> > error
> > > >> >> >> response
> > > >> >> >> > > when
> > > >> >> >> > > > a
> > > >> >> >> > > > > > > > protocol req is not recognized.
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > > > -Dana
> > > >> >> >> > > > > > > > On Mar 2, 2016 5:38 PM, "Jay Kreps" <
> > > j...@confluent.io>
> > > >> >> >> wrote:
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > > > > But won't it be the case that what clients end
> > up
> > > >> doing
> > > >> >> >> would
> > > >> >> >> > > be
> > > >> >> >> > > > > > > > something
> > > >> >> >> > > > > > > > > like
> > > >> >> >> > > > > > > > > if(version != 0.8.1)
> > > >> >> >> > > > > > > > > throw new UnsupportedVersionException()
> > > >> >> >> > > > > > > > > which then means the client is broken as soon
> as
> > > we
> > > >> >> >> release a
> > > >> >> >> > > new
> > > >> >> >> > > > > > > server
> > > >> >> >> > > > > > > > > version even though the protocol didn't
> change.
> > > I'm
> > > >> >> >> actually
> > > >> >> >> > > not
> > > >> >> >> > > > > sure
> > > >> >> >> > > > > > > how
> > > >> >> >> > > > > > > > > you could use that information in a forward
> > > >> compatible
> > > >> >> way
> > > >> >> >> > > since
> > > >> >> >> > > > > you
> > > >> >> >> > > > > > > > can't
> > > >> >> >> > > > > > > > > know a priori if you will work with the next
> > > release
> > > >> >> until
> > > >> >> >> > you
> > > >> >> >> > > > know
> > > >> >> >> > > > > > if
> > > >> >> >> > > > > > > > the
> > > >> >> >> > > > > > > > > protocol changed.
> > > >> >> >> > > > > > > > >
> > > >> >> >> > > > > > > > > -Jay
> > > >> >> >> > > > > > > > >
> > > >> >> >> > > > > > > > > On Wed, Mar 2, 2016 at 5:28 PM, Jason
> Gustafson
> > <
> > > >> >> >> > > > > ja...@confluent.io>
> > > >> >> >> > > > > > > > > wrote:
> > > >> >> >> > > > > > > > >
> > > >> >> >> > > > > > > > > > Hey Jay,
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > > > Yeah, I wasn't suggesting that we eliminate
> > > request
> > > >> >> API
> > > >> >> >> > > > versions.
> > > >> >> >> > > > > > > > They're
> > > >> >> >> > > > > > > > > > definitely needed on the broker to support
> > > >> >> >> compatibility. I
> > > >> >> >> > > was
> > > >> >> >> > > > > > just
> > > >> >> >> > > > > > > > > saying
> > > >> >> >> > > > > > > > > > that if a client wants to support multiple
> > > broker
> > > >> >> >> versions
> > > >> >> >> > > > (e.g.
> > > >> >> >> > > > > > 0.8
> > > >> >> >> > > > > > > > and
> > > >> >> >> > > > > > > > > > 0.9), then it makes more sense to me to make
> > the
> > > >> kafka
> > > >> >> >> > > release
> > > >> >> >> > > > > > > version
> > > >> >> >> > > > > > > > > > available in order to determine which
> version
> > of
> > > >> the
> > > >> >> >> > request
> > > >> >> >> > > > API
> > > >> >> >> > > > > > > should
> > > >> >> >> > > > > > > > > be
> > > >> >> >> > > > > > > > > > used rather than adding a new request type
> > which
> > > >> >> exposes
> > > >> >> >> > all
> > > >> >> >> > > of
> > > >> >> >> > > > > the
> > > >> >> >> > > > > > > > > > different supported versions for all of the
> > > request
> > > >> >> >> types.
> > > >> >> >> > > > > Request
> > > >> >> >> > > > > > > API
> > > >> >> >> > > > > > > > > > versions all change in lockstep with Kafka
> > > releases
> > > >> >> >> anyway.
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > > > -Jason
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > > > On Wed, Mar 2, 2016 at 5:15 PM, Becket Qin <
> > > >> >> >> > > > becket....@gmail.com
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > > > wrote:
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > > > > I think using Kafka release version makes
> > > sense.
> > > >> >> More
> > > >> >> >> > > > > > particularly,
> > > >> >> >> > > > > > > > we
> > > >> >> >> > > > > > > > > > can
> > > >> >> >> > > > > > > > > > > use the ApiVersion and this will cover all
> > the
> > > >> >> >> interval
> > > >> >> >> > > > version
> > > >> >> >> > > > > > as
> > > >> >> >> > > > > > > > > well.
> > > >> >> >> > > > > > > > > > In
> > > >> >> >> > > > > > > > > > > KAFKA-3025, we added the ApiVersion to
> > message
> > > >> >> format
> > > >> >> >> > > version
> > > >> >> >> > > > > > > > mapping,
> > > >> >> >> > > > > > > > > We
> > > >> >> >> > > > > > > > > > > can add the ApiKey to version mapping to
> > > >> ApiVersion
> > > >> >> as
> > > >> >> >> > > well.
> > > >> >> >> > > > We
> > > >> >> >> > > > > > can
> > > >> >> >> > > > > > > > > move
> > > >> >> >> > > > > > > > > > > ApiVersion class to o.a.k.c package and
> use
> > it
> > > >> for
> > > >> >> >> both
> > > >> >> >> > > > server
> > > >> >> >> > > > > > and
> > > >> >> >> > > > > > > > > > clients.
> > > >> >> >> > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > @Jason, if we cache the release info in
> > > metadata
> > > >> and
> > > >> >> >> not
> > > >> >> >> > > > > > > re-validate
> > > >> >> >> > > > > > > > > the
> > > >> >> >> > > > > > > > > > > release on reconnect, would it still work
> if
> > > we
> > > >> do a
> > > >> >> >> > > rolling
> > > >> >> >> > > > > > > > downgrade?
> > > >> >> >> > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > On Wed, Mar 2, 2016 at 3:16 PM, Jason
> > > Gustafson <
> > > >> >> >> > > > > > > ja...@confluent.io>
> > > >> >> >> > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > I think Dana's suggestion to include the
> > > Kafka
> > > >> >> >> release
> > > >> >> >> > > > > version
> > > >> >> >> > > > > > > > makes
> > > >> >> >> > > > > > > > > a
> > > >> >> >> > > > > > > > > > > lot
> > > >> >> >> > > > > > > > > > > > of sense. I'm actually wondering why you
> > > would
> > > >> >> need
> > > >> >> >> the
> > > >> >> >> > > > > > > individual
> > > >> >> >> > > > > > > > > API
> > > >> >> >> > > > > > > > > > > > versions if you have that? It sounds
> like
> > > >> keeping
> > > >> >> >> track
> > > >> >> >> > > of
> > > >> >> >> > > > > all
> > > >> >> >> > > > > > > the
> > > >> >> >> > > > > > > > > api
> > > >> >> >> > > > > > > > > > > > version information would add a lot of
> > > >> complexity
> > > >> >> to
> > > >> >> >> > > > clients
> > > >> >> >> > > > > > > since
> > > >> >> >> > > > > > > > > > > they'll
> > > >> >> >> > > > > > > > > > > > have to try to handle different version
> > > >> >> permutations
> > > >> >> >> > > which
> > > >> >> >> > > > > are
> > > >> >> >> > > > > > > not
> > > >> >> >> > > > > > > > > > > actually
> > > >> >> >> > > > > > > > > > > > possible in practice. Wouldn't it be
> > > simpler to
> > > >> >> know
> > > >> >> >> > that
> > > >> >> >> > > > > > you're
> > > >> >> >> > > > > > > > > > talking
> > > >> >> >> > > > > > > > > > > to
> > > >> >> >> > > > > > > > > > > > an 0.9 broker than that you're talking
> to
> > a
> > > >> broker
> > > >> >> >> > which
> > > >> >> >> > > > > > supports
> > > >> >> >> > > > > > > > > > > version 2
> > > >> >> >> > > > > > > > > > > > of the group coordinator request,
> version
> > 1
> > > of
> > > >> >> fetch
> > > >> >> >> > > > request,
> > > >> >> >> > > > > > > etc?
> > > >> >> >> > > > > > > > > > Also,
> > > >> >> >> > > > > > > > > > > > the release version could be included in
> > the
> > > >> >> broker
> > > >> >> >> > > > > information
> > > >> >> >> > > > > > > in
> > > >> >> >> > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > topic metadata request which would save
> > the
> > > >> need
> > > >> >> for
> > > >> >> >> > the
> > > >> >> >> > > > > > > additional
> > > >> >> >> > > > > > > > > > round
> > > >> >> >> > > > > > > > > > > > trip on every reconnect.
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > -Jason
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > On Tue, Mar 1, 2016 at 7:59 PM, Ashish
> > > Singh <
> > > >> >> >> > > > > > > asi...@cloudera.com>
> > > >> >> >> > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > On Tue, Mar 1, 2016 at 6:30 PM, Gwen
> > > Shapira
> > > >> <
> > > >> >> >> > > > > > > g...@confluent.io>
> > > >> >> >> > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > One more thing, the KIP actually
> had 3
> > > >> parts:
> > > >> >> >> > > > > > > > > > > > > > 1. The version protocol
> > > >> >> >> > > > > > > > > > > > > > 2. New response on messages of wrong
> > API
> > > >> key
> > > >> >> or
> > > >> >> >> > wrong
> > > >> >> >> > > > > > version
> > > >> >> >> > > > > > > > > > > > > > 3. Protocol documentation
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > There is a WIP patch for adding
> protocol
> > > >> docs,
> > > >> >> >> > > > > > > > > > > > >
> > https://github.com/apache/kafka/pull/970
> > > .
> > > >> By
> > > >> >> >> > protocol
> > > >> >> >> > > > > > > > > > documentation,
> > > >> >> >> > > > > > > > > > > > you
> > > >> >> >> > > > > > > > > > > > > mean updating this, right?
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > I understand that you are offering
> to
> > > only
> > > >> >> >> > implement
> > > >> >> >> > > > part
> > > >> >> >> > > > > > 1?
> > > >> >> >> > > > > > > > > > > > > > But the KIP discussion and vote
> should
> > > >> still
> > > >> >> >> cover
> > > >> >> >> > > all
> > > >> >> >> > > > > > three
> > > >> >> >> > > > > > > > > parts,
> > > >> >> >> > > > > > > > > > > > > > they will just be implemented in
> > > separate
> > > >> >> JIRA?
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > The patch for KAFKA-3307,
> > > >> >> >> > > > > > > > https://github.com/apache/kafka/pull/986
> > > >> >> >> > > > > > > > > ,
> > > >> >> >> > > > > > > > > > > > covers
> > > >> >> >> > > > > > > > > > > > > 1 and 2. KAFKA-3309 tracks
> documentation
> > > >> part.
> > > >> >> >> Yes,
> > > >> >> >> > we
> > > >> >> >> > > > > should
> > > >> >> >> > > > > > > > > include
> > > >> >> >> > > > > > > > > > > all
> > > >> >> >> > > > > > > > > > > > > the three points you mentioned while
> > > >> discussing
> > > >> >> or
> > > >> >> >> > > voting
> > > >> >> >> > > > > for
> > > >> >> >> > > > > > > > > KIP-35.
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > On Tue, Mar 1, 2016 at 5:25 PM,
> Ashish
> > > >> Singh <
> > > >> >> >> > > > > > > > > asi...@cloudera.com>
> > > >> >> >> > > > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > > > > > On Tue, Mar 1, 2016 at 5:11 PM,
> Gwen
> > > >> >> Shapira <
> > > >> >> >> > > > > > > > > g...@confluent.io>
> > > >> >> >> > > > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >> I don't see a use for the name -
> > > clients
> > > >> >> >> should
> > > >> >> >> > be
> > > >> >> >> > > > > able
> > > >> >> >> > > > > > to
> > > >> >> >> > > > > > > > > > > translate
> > > >> >> >> > > > > > > > > > > > > > >> ApiKey to name for any API they
> > > support,
> > > >> >> and
> > > >> >> >> I'm
> > > >> >> >> > > not
> > > >> >> >> > > > > > sure
> > > >> >> >> > > > > > > > why
> > > >> >> >> > > > > > > > > > > would
> > > >> >> >> > > > > > > > > > > > a
> > > >> >> >> > > > > > > > > > > > > > >> client need to log anything about
> > > APIs
> > > >> it
> > > >> >> >> does
> > > >> >> >> > not
> > > >> >> >> > > > > > > support.
> > > >> >> >> > > > > > > > > Am I
> > > >> >> >> > > > > > > > > > > > > > >> missing something?
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > > Yea, it is a fair assumption that
> > > client
> > > >> >> would
> > > >> >> >> > know
> > > >> >> >> > > > > about
> > > >> >> >> > > > > > > > APIs
> > > >> >> >> > > > > > > > > it
> > > >> >> >> > > > > > > > > > > > > > supports.
> > > >> >> >> > > > > > > > > > > > > > > It could have been helpful for
> > client
> > > >> users
> > > >> >> to
> > > >> >> >> > see
> > > >> >> >> > > > new
> > > >> >> >> > > > > > APIs
> > > >> >> >> > > > > > > > > > though,
> > > >> >> >> > > > > > > > > > > > > > however
> > > >> >> >> > > > > > > > > > > > > > > users can always refer to protocol
> > > doc of
> > > >> >> new
> > > >> >> >> > > version
> > > >> >> >> > > > > to
> > > >> >> >> > > > > > > find
> > > >> >> >> > > > > > > > > > > > > > corresponding
> > > >> >> >> > > > > > > > > > > > > > > names of the new APIs.
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > >> On a related note, Magnus is
> > > currently
> > > >> on
> > > >> >> >> > > vacation,
> > > >> >> >> > > > > but
> > > >> >> >> > > > > > he
> > > >> >> >> > > > > > > > > > should
> > > >> >> >> > > > > > > > > > > be
> > > >> >> >> > > > > > > > > > > > > > >> back at the end of next week. I'd
> > > like
> > > >> to
> > > >> >> >> hold
> > > >> >> >> > off
> > > >> >> >> > > > on
> > > >> >> >> > > > > > the
> > > >> >> >> > > > > > > > vote
> > > >> >> >> > > > > > > > > > > until
> > > >> >> >> > > > > > > > > > > > > > >> he gets back since his experience
> > in
> > > >> >> >> > implementing
> > > >> >> >> > > > > > clients
> > > >> >> >> > > > > > > > and
> > > >> >> >> > > > > > > > > > his
> > > >> >> >> > > > > > > > > > > > > > >> opinions will be very valuable
> for
> > > this
> > > >> >> >> > > discussion.
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > > That is great. It will be valuable
> > to
> > > >> have
> > > >> >> his
> > > >> >> >> > > > > feedback.
> > > >> >> >> > > > > > I
> > > >> >> >> > > > > > > > will
> > > >> >> >> > > > > > > > > > > hold
> > > >> >> >> > > > > > > > > > > > > off
> > > >> >> >> > > > > > > > > > > > > > on
> > > >> >> >> > > > > > > > > > > > > > > removing "api_name" and
> > > >> >> >> "api_deprecated_versions"
> > > >> >> >> > > or
> > > >> >> >> > > > > > adding
> > > >> >> >> > > > > > > > > > release
> > > >> >> >> > > > > > > > > > > > > > version.
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > >> Gwen
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > >> On Tue, Mar 1, 2016 at 4:24 PM,
> > > Ashish
> > > >> >> Singh
> > > >> >> >> <
> > > >> >> >> > > > > > > > > > asi...@cloudera.com
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > > > > >> > Works with me. I will update PR
> > to
> > > >> remove
> > > >> >> >> > this.
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> > Also, "api_name" have been
> > pointed
> > > out
> > > >> >> as a
> > > >> >> >> > > > concern.
> > > >> >> >> > > > > > > > > However,
> > > >> >> >> > > > > > > > > > it
> > > >> >> >> > > > > > > > > > > > can
> > > >> >> >> > > > > > > > > > > > > > be
> > > >> >> >> > > > > > > > > > > > > > >> > handy for logging and similar
> > > >> purposes.
> > > >> >> Any
> > > >> >> >> > take
> > > >> >> >> > > > on
> > > >> >> >> > > > > > > that?
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> > On Tue, Mar 1, 2016 at 3:46 PM,
> > > Gwen
> > > >> >> >> Shapira <
> > > >> >> >> > > > > > > > > > g...@confluent.io
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> Jay also mentioned:
> > > >> >> >> > > > > > > > > > > > > > >> >> "Or, alternately, since
> > > deprecation
> > > >> has
> > > >> >> no
> > > >> >> >> > > > > functional
> > > >> >> >> > > > > > > > > impact
> > > >> >> >> > > > > > > > > > > and
> > > >> >> >> > > > > > > > > > > > is
> > > >> >> >> > > > > > > > > > > > > > >> >> just a message
> > > >> >> >> > > > > > > > > > > > > > >> >> to developers, we could just
> > > leave it
> > > >> >> out
> > > >> >> >> of
> > > >> >> >> > > the
> > > >> >> >> > > > > > > protocol
> > > >> >> >> > > > > > > > > and
> > > >> >> >> > > > > > > > > > > > just
> > > >> >> >> > > > > > > > > > > > > > have
> > > >> >> >> > > > > > > > > > > > > > >> it
> > > >> >> >> > > > > > > > > > > > > > >> >> in release notes etc."
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >> >> I'm in favor of leaving it out
> > of
> > > the
> > > >> >> >> > > protocol. I
> > > >> >> >> > > > > > can't
> > > >> >> >> > > > > > > > > > really
> > > >> >> >> > > > > > > > > > > > see
> > > >> >> >> > > > > > > > > > > > > a
> > > >> >> >> > > > > > > > > > > > > > >> >> use-case.
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >> >> Gwen
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >> >> On Mon, Feb 29, 2016 at 3:55
> PM,
> > > >> Ashish
> > > >> >> >> > Singh <
> > > >> >> >> > > > > > > > > > > > asi...@cloudera.com
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >> wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > I hope it is OK for me to
> make
> > > some
> > > >> >> >> > progress
> > > >> >> >> > > > > here.
> > > >> >> >> > > > > > I
> > > >> >> >> > > > > > > > have
> > > >> >> >> > > > > > > > > > > made
> > > >> >> >> > > > > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > > >> >> > following changes.
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > 1. Updated KIP-35, to adopt
> > > Jay's
> > > >> >> >> > suggestion
> > > >> >> >> > > on
> > > >> >> >> > > > > > > > > maintaining
> > > >> >> >> > > > > > > > > > > > > > separate
> > > >> >> >> > > > > > > > > > > > > > >> list
> > > >> >> >> > > > > > > > > > > > > > >> >> > of deprecated versions,
> > instead
> > > of
> > > >> >> >> using a
> > > >> >> >> > > > > version
> > > >> >> >> > > > > > of
> > > >> >> >> > > > > > > > -1.
> > > >> >> >> > > > > > > > > > > > > > >> >> > 2. Added information on
> > required
> > > >> >> >> > permissions,
> > > >> >> >> > > > > > > Describe
> > > >> >> >> > > > > > > > > > action
> > > >> >> >> > > > > > > > > > > > on
> > > >> >> >> > > > > > > > > > > > > > >> Cluster
> > > >> >> >> > > > > > > > > > > > > > >> >> > resource, to be able to
> > retrieve
> > > >> >> >> protocol
> > > >> >> >> > > > > versions
> > > >> >> >> > > > > > > > from a
> > > >> >> >> > > > > > > > > > > auth
> > > >> >> >> > > > > > > > > > > > > > enabled
> > > >> >> >> > > > > > > > > > > > > > >> >> > Kafka cluster.
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > Created
> > > >> >> >> > > > > > > >
> https://issues.apache.org/jira/browse/KAFKA-3304
> > > >> >> >> > > > > > > > > .
> > > >> >> >> > > > > > > > > > > > > Primary
> > > >> >> >> > > > > > > > > > > > > > >> patch
> > > >> >> >> > > > > > > > > > > > > > >> >> is
> > > >> >> >> > > > > > > > > > > > > > >> >> > available to review,
> > > >> >> >> > > > > > > > > > > https://github.com/apache/kafka/pull/986
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > On Thu, Feb 25, 2016 at 1:27
> > PM,
> > > >> >> Ashish
> > > >> >> >> > > Singh <
> > > >> >> >> > > > > > > > > > > > > asi...@cloudera.com
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > > Kafka clients in Hadoop
> > > >> ecosystem,
> > > >> >> >> Flume,
> > > >> >> >> > > > > Spark,
> > > >> >> >> > > > > > > etc,
> > > >> >> >> > > > > > > > > > have
> > > >> >> >> > > > > > > > > > > > > found
> > > >> >> >> > > > > > > > > > > > > > it
> > > >> >> >> > > > > > > > > > > > > > >> >> > really
> > > >> >> >> > > > > > > > > > > > > > >> >> > > difficult to cope up with
> > > Kafka
> > > >> >> >> releases
> > > >> >> >> > as
> > > >> >> >> > > > > they
> > > >> >> >> > > > > > > want
> > > >> >> >> > > > > > > > > to
> > > >> >> >> > > > > > > > > > > > > support
> > > >> >> >> > > > > > > > > > > > > > >> users
> > > >> >> >> > > > > > > > > > > > > > >> >> on
> > > >> >> >> > > > > > > > > > > > > > >> >> > > different Kafka versions.
> > > >> Capability
> > > >> >> >> to
> > > >> >> >> > > > > retrieve
> > > >> >> >> > > > > > > > > protocol
> > > >> >> >> > > > > > > > > > > > > version
> > > >> >> >> > > > > > > > > > > > > > >> will
> > > >> >> >> > > > > > > > > > > > > > >> >> > go a
> > > >> >> >> > > > > > > > > > > > > > >> >> > > long way to ease out those
> > > pain
> > > >> >> >> points. I
> > > >> >> >> > > > will
> > > >> >> >> > > > > be
> > > >> >> >> > > > > > > > happy
> > > >> >> >> > > > > > > > > > to
> > > >> >> >> > > > > > > > > > > > help
> > > >> >> >> > > > > > > > > > > > > > out
> > > >> >> >> > > > > > > > > > > > > > >> >> with
> > > >> >> >> > > > > > > > > > > > > > >> >> > > the work on this KIP.
> > @Magnus,
> > > >> >> thanks
> > > >> >> >> for
> > > >> >> >> > > > > driving
> > > >> >> >> > > > > > > > this,
> > > >> >> >> > > > > > > > > > is
> > > >> >> >> > > > > > > > > > > it
> > > >> >> >> > > > > > > > > > > > > OK
> > > >> >> >> > > > > > > > > > > > > > if
> > > >> >> >> > > > > > > > > > > > > > >> I
> > > >> >> >> > > > > > > > > > > > > > >> >> > carry
> > > >> >> >> > > > > > > > > > > > > > >> >> > > forward the work from
> here.
> > It
> > > >> will
> > > >> >> be
> > > >> >> >> > > ideal
> > > >> >> >> > > > to
> > > >> >> >> > > > > > > have
> > > >> >> >> > > > > > > > > this
> > > >> >> >> > > > > > > > > > > in
> > > >> >> >> > > > > > > > > > > > > > >> 0.10.0.0.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > > On Mon, Oct 12, 2015 at
> 9:29
> > > PM,
> > > >> Jay
> > > >> >> >> > Kreps
> > > >> >> >> > > <
> > > >> >> >> > > > > > > > > > > j...@confluent.io
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >> wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> I wonder if we need to
> > solve
> > > the
> > > >> >> >> error
> > > >> >> >> > > > > problem?
> > > >> >> >> > > > > > I
> > > >> >> >> > > > > > > > > think
> > > >> >> >> > > > > > > > > > > this
> > > >> >> >> > > > > > > > > > > > > KIP
> > > >> >> >> > > > > > > > > > > > > > >> >> gives a
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> descent work around.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> Probably we should have
> > > >> included an
> > > >> >> >> > error
> > > >> >> >> > > in
> > > >> >> >> > > > > the
> > > >> >> >> > > > > > > > > > response
> > > >> >> >> > > > > > > > > > > > > > header,
> > > >> >> >> > > > > > > > > > > > > > >> but
> > > >> >> >> > > > > > > > > > > > > > >> >> we
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> debated it at the time
> > > decided
> > > >> not
> > > >> >> to
> > > >> >> >> > and
> > > >> >> >> > > > now
> > > >> >> >> > > > > it
> > > >> >> >> > > > > > > is
> > > >> >> >> > > > > > > > > > pretty
> > > >> >> >> > > > > > > > > > > > > hard
> > > >> >> >> > > > > > > > > > > > > > to
> > > >> >> >> > > > > > > > > > > > > > >> add
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> because the headers
> aren't
> > > >> >> versioned
> > > >> >> >> > > (d'oh).
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> It seems like any other
> > > >> solution is
> > > >> >> >> > going
> > > >> >> >> > > to
> > > >> >> >> > > > > be
> > > >> >> >> > > > > > > kind
> > > >> >> >> > > > > > > > > of
> > > >> >> >> > > > > > > > > > a
> > > >> >> >> > > > > > > > > > > > > hack,
> > > >> >> >> > > > > > > > > > > > > > >> right?
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> Sending malformed
> responses
> > > back
> > > >> >> >> seems
> > > >> >> >> > > like
> > > >> >> >> > > > > not
> > > >> >> >> > > > > > a
> > > >> >> >> > > > > > > > > clean
> > > >> >> >> > > > > > > > > > > > > > solution...
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> (Not sure if I was pro-
> > > having a
> > > >> >> >> > top-level
> > > >> >> >> > > > > error
> > > >> >> >> > > > > > > or
> > > >> >> >> > > > > > > > > not,
> > > >> >> >> > > > > > > > > > > but
> > > >> >> >> > > > > > > > > > > > > in
> > > >> >> >> > > > > > > > > > > > > > any
> > > >> >> >> > > > > > > > > > > > > > >> >> case
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> the rationale for the
> > > decision
> > > >> was
> > > >> >> >> that
> > > >> >> >> > so
> > > >> >> >> > > > > many
> > > >> >> >> > > > > > of
> > > >> >> >> > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > requests
> > > >> >> >> > > > > > > > > > > > > > >> were
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> per-partition or
> per-topic
> > or
> > > >> >> >> whatever
> > > >> >> >> > and
> > > >> >> >> > > > > hence
> > > >> >> >> > > > > > > > fail
> > > >> >> >> > > > > > > > > or
> > > >> >> >> > > > > > > > > > > > > > succeed at
> > > >> >> >> > > > > > > > > > > > > > >> >> that
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> level and this makes it
> > hard
> > > to
> > > >> >> know
> > > >> >> >> > what
> > > >> >> >> > > > the
> > > >> >> >> > > > > > > right
> > > >> >> >> > > > > > > > > > > > top-level
> > > >> >> >> > > > > > > > > > > > > > error
> > > >> >> >> > > > > > > > > > > > > > >> >> code
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> is
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> and hard for the client
> to
> > > >> figure
> > > >> >> out
> > > >> >> >> > what
> > > >> >> >> > > > to
> > > >> >> >> > > > > do
> > > >> >> >> > > > > > > > with
> > > >> >> >> > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > top
> > > >> >> >> > > > > > > > > > > > > > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > error
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> if some of the partitions
> > > >> succeed
> > > >> >> but
> > > >> >> >> > > there
> > > >> >> >> > > > > is a
> > > >> >> >> > > > > > > > > > top-level
> > > >> >> >> > > > > > > > > > > > > > error).
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> I think actually this new
> > API
> > > >> >> >> actually
> > > >> >> >> > > > gives a
> > > >> >> >> > > > > > way
> > > >> >> >> > > > > > > > to
> > > >> >> >> > > > > > > > > > > handle
> > > >> >> >> > > > > > > > > > > > > > this
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> gracefully on the client
> > > side by
> > > >> >> just
> > > >> >> >> > > having
> > > >> >> >> > > > > > > clients
> > > >> >> >> > > > > > > > > > that
> > > >> >> >> > > > > > > > > > > > want
> > > >> >> >> > > > > > > > > > > > > > to
> > > >> >> >> > > > > > > > > > > > > > >> be
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> graceful check for
> support
> > > for
> > > >> >> their
> > > >> >> >> > > > version.
> > > >> >> >> > > > > > > > Clients
> > > >> >> >> > > > > > > > > > that
> > > >> >> >> > > > > > > > > > > > do
> > > >> >> >> > > > > > > > > > > > > > that
> > > >> >> >> > > > > > > > > > > > > > >> >> will
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> have a graceful message.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> At some point if we're
> ever
> > > >> >> reworking
> > > >> >> >> > the
> > > >> >> >> > > > > > headers
> > > >> >> >> > > > > > > we
> > > >> >> >> > > > > > > > > > > should
> > > >> >> >> > > > > > > > > > > > > > really
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> consider
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> (a) versioning them and
> (b)
> > > >> adding
> > > >> >> a
> > > >> >> >> > > > top-level
> > > >> >> >> > > > > > > error
> > > >> >> >> > > > > > > > > > code
> > > >> >> >> > > > > > > > > > > in
> > > >> >> >> > > > > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > > >> >> > response.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> But given this would be a
> > big
> > > >> >> >> breaking
> > > >> >> >> > > > change
> > > >> >> >> > > > > > and
> > > >> >> >> > > > > > > > this
> > > >> >> >> > > > > > > > > > is
> > > >> >> >> > > > > > > > > > > > > really
> > > >> >> >> > > > > > > > > > > > > > >> just
> > > >> >> >> > > > > > > > > > > > > > >> >> to
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> give a nicer error
> message
> > > seems
> > > >> >> >> like it
> > > >> >> >> > > > > > probably
> > > >> >> >> > > > > > > > > isn't
> > > >> >> >> > > > > > > > > > > > worth
> > > >> >> >> > > > > > > > > > > > > > it to
> > > >> >> >> > > > > > > > > > > > > > >> >> try
> > > >> >> >> > > > > > > > > > > > > > >> >> > to
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> do something now.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> -Jay
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> On Mon, Oct 12, 2015 at
> > 8:11
> > > PM,
> > > >> >> >> > Jiangjie
> > > >> >> >> > > > Qin
> > > >> >> >> > > > > > > > > > > > > > >> >> <j...@linkedin.com.invalid
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > I am thinking instead
> of
> > > >> >> returning
> > > >> >> >> an
> > > >> >> >> > > > empty
> > > >> >> >> > > > > > > > > response,
> > > >> >> >> > > > > > > > > > it
> > > >> >> >> > > > > > > > > > > > > > would be
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> better to
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > return an explicit
> > > >> >> >> > > > > UnsupportedVersionException
> > > >> >> >> > > > > > > > code.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Today KafkaApis handles
> > the
> > > >> error
> > > >> >> >> in
> > > >> >> >> > the
> > > >> >> >> > > > > > > following
> > > >> >> >> > > > > > > > > > way:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 1. For
> requests/responses
> > > >> using
> > > >> >> old
> > > >> >> >> > > Scala
> > > >> >> >> > > > > > > classes,
> > > >> >> >> > > > > > > > > > > > KafkaApis
> > > >> >> >> > > > > > > > > > > > > > uses
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> RequestOrResponse.handleError()
> > > >> >> to
> > > >> >> >> > > return
> > > >> >> >> > > > an
> > > >> >> >> > > > > > > error
> > > >> >> >> > > > > > > > > > > > response.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 2. For
> requests/response
> > > using
> > > >> >> Java
> > > >> >> >> > > > classes
> > > >> >> >> > > > > > > (only
> > > >> >> >> > > > > > > > > > > > > > >> JoinGroupRequest
> > > >> >> >> > > > > > > > > > > > > > >> >> and
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Heartbeat now),
> KafkaApis
> > > >> calls
> > > >> >> >> > > > > > > > > > > > > > >>
> AbstractRequest.getErrorResponse()
> > > >> >> >> > > > > > > > > > > > > > >> >> to
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > return an error
> response.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > In KAFKA-2512, I am
> > > returning
> > > >> an
> > > >> >> >> > > > > > > > > > > > UnsupportedVersionException
> > > >> >> >> > > > > > > > > > > > > > for
> > > >> >> >> > > > > > > > > > > > > > >> >> case
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> [1]
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > when see an unsupported
> > > >> version.
> > > >> >> >> This
> > > >> >> >> > > will
> > > >> >> >> > > > > put
> > > >> >> >> > > > > > > the
> > > >> >> >> > > > > > > > > > error
> > > >> >> >> > > > > > > > > > > > > code
> > > >> >> >> > > > > > > > > > > > > > per
> > > >> >> >> > > > > > > > > > > > > > >> >> > topic
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> or
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > partition for most of
> the
> > > >> >> requests,
> > > >> >> >> > but
> > > >> >> >> > > > > might
> > > >> >> >> > > > > > > not
> > > >> >> >> > > > > > > > > work
> > > >> >> >> > > > > > > > > > > all
> > > >> >> >> > > > > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > > >> time.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> e.g.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > TopicMetadataRequest
> with
> > > an
> > > >> >> empty
> > > >> >> >> > topic
> > > >> >> >> > > > > set.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Case [2] does not quite
> > > work
> > > >> for
> > > >> >> >> > > > unsupported
> > > >> >> >> > > > > > > > > version,
> > > >> >> >> > > > > > > > > > > > > because
> > > >> >> >> > > > > > > > > > > > > > we
> > > >> >> >> > > > > > > > > > > > > > >> >> will
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > thrown an uncaught
> > > exception
> > > >> when
> > > >> >> >> > > version
> > > >> >> >> > > > is
> > > >> >> >> > > > > > not
> > > >> >> >> > > > > > > > > > > > recognized
> > > >> >> >> > > > > > > > > > > > > > (BTW
> > > >> >> >> > > > > > > > > > > > > > >> >> this
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> is a
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > bug). Part of the
> reason
> > is
> > > >> that
> > > >> >> >> for
> > > >> >> >> > > some
> > > >> >> >> > > > > > > response
> > > >> >> >> > > > > > > > > > > types,
> > > >> >> >> > > > > > > > > > > > > > error
> > > >> >> >> > > > > > > > > > > > > > >> code
> > > >> >> >> > > > > > > > > > > > > > >> >> > is
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> not
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > part of the response
> > level
> > > >> field.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Maybe it worth checking
> > how
> > > >> each
> > > >> >> >> > > response
> > > >> >> >> > > > is
> > > >> >> >> > > > > > > > dealing
> > > >> >> >> > > > > > > > > > > with
> > > >> >> >> > > > > > > > > > > > > > error
> > > >> >> >> > > > > > > > > > > > > > >> code
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> today.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > A scan of the response
> > > formats
> > > >> >> >> gives
> > > >> >> >> > the
> > > >> >> >> > > > > > > following
> > > >> >> >> > > > > > > > > > > result:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 1.
> TopicMetadataResponse
> > -
> > > per
> > > >> >> >> topic
> > > >> >> >> > > error
> > > >> >> >> > > > > > code,
> > > >> >> >> > > > > > > > > does
> > > >> >> >> > > > > > > > > > > not
> > > >> >> >> > > > > > > > > > > > > work
> > > >> >> >> > > > > > > > > > > > > > >> when
> > > >> >> >> > > > > > > > > > > > > > >> >> > the
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > topic set is empty in
> the
> > > >> >> request.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 2. ProduceResonse - per
> > > >> partition
> > > >> >> >> > error
> > > >> >> >> > > > > code.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 3.
> OffsetCommitResponse -
> > > per
> > > >> >> >> > partition.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 4. OffsetFetchResponse
> -
> > > per
> > > >> >> >> > partition.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 5. OffsetResponse - per
> > > >> >> partition.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 6. FetchResponse - per
> > > >> partition
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 7.
> > > ConsumerMetadataResponse -
> > > >> >> >> response
> > > >> >> >> > > > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 8.
> > > ControlledShutdownResponse
> > > >> -
> > > >> >> >> > response
> > > >> >> >> > > > > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 9. JoinGroupResponse -
> > > >> response
> > > >> >> >> level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 10. HearbeatResponse -
> > > >> response
> > > >> >> >> level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 11.
> LeaderAndIsrResponse
> > -
> > > >> >> response
> > > >> >> >> > > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 12.
> StopReplicaResponse -
> > > >> >> response
> > > >> >> >> > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > 13.
> > UpdateMetadataResponse
> > > -
> > > >> >> >> response
> > > >> >> >> > > > level
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > So from the list above
> it
> > > >> looks
> > > >> >> for
> > > >> >> >> > each
> > > >> >> >> > > > > > > response
> > > >> >> >> > > > > > > > we
> > > >> >> >> > > > > > > > > > are
> > > >> >> >> > > > > > > > > > > > > > actually
> > > >> >> >> > > > > > > > > > > > > > >> >> able
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> to
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > return an error code,
> as
> > > long
> > > >> as
> > > >> >> we
> > > >> >> >> > make
> > > >> >> >> > > > > sure
> > > >> >> >> > > > > > > the
> > > >> >> >> > > > > > > > > > topic
> > > >> >> >> > > > > > > > > > > or
> > > >> >> >> > > > > > > > > > > > > > >> partition
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> won't
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > be empty when the error
> > > code
> > > >> is
> > > >> >> at
> > > >> >> >> > topic
> > > >> >> >> > > > or
> > > >> >> >> > > > > > > > > partition
> > > >> >> >> > > > > > > > > > > > level.
> > > >> >> >> > > > > > > > > > > > > > >> Luckily
> > > >> >> >> > > > > > > > > > > > > > >> >> > in
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> the
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > above list we only need
> > to
> > > >> worry
> > > >> >> >> about
> > > >> >> >> > > > > > > > > > > > > TopicMetadataResponse.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Maybe error handling is
> > > out of
> > > >> >> the
> > > >> >> >> > scope
> > > >> >> >> > > > of
> > > >> >> >> > > > > > this
> > > >> >> >> > > > > > > > > KIP,
> > > >> >> >> > > > > > > > > > > but
> > > >> >> >> > > > > > > > > > > > I
> > > >> >> >> > > > > > > > > > > > > > >> prefer
> > > >> >> >> > > > > > > > > > > > > > >> >> we
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> think
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > through how to deal
> with
> > > error
> > > >> >> code
> > > >> >> >> > for
> > > >> >> >> > > > the
> > > >> >> >> > > > > > > > > requests,
> > > >> >> >> > > > > > > > > > > > > because
> > > >> >> >> > > > > > > > > > > > > > >> there
> > > >> >> >> > > > > > > > > > > > > > >> >> > are
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > more request types to
> be
> > > >> added in
> > > >> >> >> > > > KAFKA-2464
> > > >> >> >> > > > > > and
> > > >> >> >> > > > > > > > > > future
> > > >> >> >> > > > > > > > > > > > > > patches.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Thanks,
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > Jiangjie (Becket) Qin
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > On Mon, Oct 12, 2015 at
> > > 6:04
> > > >> PM,
> > > >> >> >> Jay
> > > >> >> >> > > > Kreps <
> > > >> >> >> > > > > > > > > > > > > j...@confluent.io>
> > > >> >> >> > > > > > > > > > > > > > >> >> wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > Two quick pieces of
> > > >> feedback:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > 1. The use of a
> version
> > > of
> > > >> -1
> > > >> >> as
> > > >> >> >> > > magical
> > > >> >> >> > > > > > entry
> > > >> >> >> > > > > > > > > > > dividing
> > > >> >> >> > > > > > > > > > > > > > between
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > deprecated
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > versions is a bit
> > hacky.
> > > >> What
> > > >> >> >> about
> > > >> >> >> > > > > instead
> > > >> >> >> > > > > > > > having
> > > >> >> >> > > > > > > > > > an
> > > >> >> >> > > > > > > > > > > > > array
> > > >> >> >> > > > > > > > > > > > > > of
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> supported
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > versions and a
> separate
> > > >> array
> > > >> >> of
> > > >> >> >> > > > > deprecated
> > > >> >> >> > > > > > > > > > versions.
> > > >> >> >> > > > > > > > > > > > The
> > > >> >> >> > > > > > > > > > > > > > >> >> deprecated
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > versions would always
> > be
> > > a
> > > >> >> >> subset of
> > > >> >> >> > > the
> > > >> >> >> > > > > > > > supported
> > > >> >> >> > > > > > > > > > > > > versions.
> > > >> >> >> > > > > > > > > > > > > > >> Or,
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > alternately, since
> > > >> deprecation
> > > >> >> >> has
> > > >> >> >> > no
> > > >> >> >> > > > > > > functional
> > > >> >> >> > > > > > > > > > > impact
> > > >> >> >> > > > > > > > > > > > > and
> > > >> >> >> > > > > > > > > > > > > > is
> > > >> >> >> > > > > > > > > > > > > > >> >> just
> > > >> >> >> > > > > > > > > > > > > > >> >> > a
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > message to
> developers,
> > we
> > > >> could
> > > >> >> >> just
> > > >> >> >> > > > leave
> > > >> >> >> > > > > > it
> > > >> >> >> > > > > > > > out
> > > >> >> >> > > > > > > > > of
> > > >> >> >> > > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > > >> protocol
> > > >> >> >> > > > > > > > > > > > > > >> >> > and
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > just
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > have it in release
> > notes
> > > >> etc.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > 2. I think including
> > the
> > > api
> > > >> >> name
> > > >> >> >> > may
> > > >> >> >> > > > > cause
> > > >> >> >> > > > > > > some
> > > >> >> >> > > > > > > > > > > > problems.
> > > >> >> >> > > > > > > > > > > > > > >> >> Currently
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> the
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > api key is the
> primary
> > > key
> > > >> that
> > > >> >> >> we
> > > >> >> >> > > keep
> > > >> >> >> > > > > > > > consistent
> > > >> >> >> > > > > > > > > > but
> > > >> >> >> > > > > > > > > > > > we
> > > >> >> >> > > > > > > > > > > > > > have
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> actually
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > evolved the english
> > > >> description
> > > >> >> >> of
> > > >> >> >> > the
> > > >> >> >> > > > > apis
> > > >> >> >> > > > > > as
> > > >> >> >> > > > > > > > > they
> > > >> >> >> > > > > > > > > > > have
> > > >> >> >> > > > > > > > > > > > > > >> changed.
> > > >> >> >> > > > > > > > > > > > > > >> >> > The
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > only
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > use I can think of
> for
> > > the
> > > >> name
> > > >> >> >> > would
> > > >> >> >> > > be
> > > >> >> >> > > > > if
> > > >> >> >> > > > > > > > people
> > > >> >> >> > > > > > > > > > > used
> > > >> >> >> > > > > > > > > > > > > the
> > > >> >> >> > > > > > > > > > > > > > >> >> logical
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> name
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > and tried to resolve
> > the
> > > api
> > > >> >> key,
> > > >> >> >> > but
> > > >> >> >> > > > that
> > > >> >> >> > > > > > > would
> > > >> >> >> > > > > > > > > be
> > > >> >> >> > > > > > > > > > > > wrong.
> > > >> >> >> > > > > > > > > > > > > > Not
> > > >> >> >> > > > > > > > > > > > > > >> >> sure
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> if we
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > actually need the
> > english
> > > >> name,
> > > >> >> >> if
> > > >> >> >> > > there
> > > >> >> >> > > > > is
> > > >> >> >> > > > > > a
> > > >> >> >> > > > > > > > use
> > > >> >> >> > > > > > > > > > > case I
> > > >> >> >> > > > > > > > > > > > > > guess
> > > >> >> >> > > > > > > > > > > > > > >> >> we'll
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> just
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > have to be very clear
> > > that
> > > >> the
> > > >> >> >> name
> > > >> >> >> > is
> > > >> >> >> > > > > just
> > > >> >> >> > > > > > > > > > > > documentation
> > > >> >> >> > > > > > > > > > > > > > and
> > > >> >> >> > > > > > > > > > > > > > >> can
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> change
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > any time.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > -Jay
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > On Fri, Sep 25, 2015
> at
> > > 2:53
> > > >> >> PM,
> > > >> >> >> > > Magnus
> > > >> >> >> > > > > > > > Edenhill <
> > > >> >> >> > > > > > > > > > > > > > >> >> > mag...@edenhill.se>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > wrote:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > Good evening,
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > KIP-35 was created
> to
> > > >> address
> > > >> >> >> > > current
> > > >> >> >> > > > > and
> > > >> >> >> > > > > > > > future
> > > >> >> >> > > > > > > > > > > > > > >> broker-client
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > compatibility.
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > >
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > >
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > >
> > > >> >> >> > > > >
> > > >> >> >> > > >
> > > >> >> >> > >
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > Summary:
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > * allow clients to
> > > >> retrieve
> > > >> >> >> the
> > > >> >> >> > > > > broker's
> > > >> >> >> > > > > > > > > protocol
> > > >> >> >> > > > > > > > > > > > > version
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > * make broker
> handle
> > > >> unknown
> > > >> >> >> > > protocol
> > > >> >> >> > > > > > > > requests
> > > >> >> >> > > > > > > > > > > > > gracefully
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > Feedback and
> comments
> > > >> >> welcome!
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > Regards,
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > > Magnus
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >>
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > > --
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> > > Regards,
> > > >> >> >> > > > > > > > > > > > > > >> >> > > Ashish
> > > >> >> >> > > > > > > > > > > > > > >> >> > >
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > --
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >> > Regards,
> > > >> >> >> > > > > > > > > > > > > > >> >> > Ashish
> > > >> >> >> > > > > > > > > > > > > > >> >> >
> > > >> >> >> > > > > > > > > > > > > > >> >>
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> > --
> > > >> >> >> > > > > > > > > > > > > > >> >
> > > >> >> >> > > > > > > > > > > > > > >> > Regards,
> > > >> >> >> > > > > > > > > > > > > > >> > Ashish
> > > >> >> >> > > > > > > > > > > > > > >>
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > > --
> > > >> >> >> > > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > > > Regards,
> > > >> >> >> > > > > > > > > > > > > > > Ashish
> > > >> >> >> > > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > --
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > > > Regards,
> > > >> >> >> > > > > > > > > > > > > Ashish
> > > >> >> >> > > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > > >
> > > >> >> >> > > > > > > > > > >
> > > >> >> >> > > > > > > > > >
> > > >> >> >> > > > > > > > >
> > > >> >> >> > > > > > > >
> > > >> >> >> > > > > > >
> > > >> >> >> > > > > >
> > > >> >> >> > > > > >
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > --
> > > >> >> >> > > > > >
> > > >> >> >> > > > > > Regards,
> > > >> >> >> > > > > > Ashish
> > > >> >> >> > > > > >
> > > >> >> >> > > > >
> > > >> >> >> > > >
> > > >> >> >> > >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > --
> > > >> >> >> >
> > > >> >> >> > Regards,
> > > >> >> >> > Ashish
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> > Ashish
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >>
> > > >> >> Regards,
> > > >> >> Ashish
> > > >> >>
> > > >>
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>
