Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Ashish, Both good points. I think the issue with the general metadata request is the same as the issue with a version-specific metadata request from the other proposal--basically it's a chicken and egg problem, to find out anything about the cluster you have to be able to communicate

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hello Jay, The overall approach sounds good. I do realize that this discussion has gotten too lengthy and is starting to shoot tangents. Maybe a KIP call will help us getting to a decision faster. I do have a few questions though. On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah here is my summary of my take: 1. Negotiating a per-connection protocol actually does add a lot of complexity to clients (many more failure states to get right). 2. Having the client configure the protocol version manually is doable now but probably a worse state. I suspect this will lead

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I talked with Jay about this KIP briefly this morning, so let me try to summarize the discussion (I'm sure he'll jump in if I get anything wrong). Apologies in advance for the length. I think we both share some skepticism that a request with all the supported versions of all the request APIs is

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> I think using Kafka release version makes sense. More particularly, we can use the ApiVersion and this will cover all the interval version as well. In KAFKA-3025, we added the ApiVersion to message format version mapping, We can add the ApiKey to version mapping to ApiVersion as well. We can

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> Also, the release version could be included in the broker information in the topic metadata request which would save the need for the additional round trip on every reconnect. Wouldn't this cause a catch-22 situation, where you need to know protocol version of metadata request to get supported

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Jason, I was thinking that every time when we connect to a broker, we first send the version check request. (The version check request itself should be very simple and never changes across all server releases.) This does add an additional round trip, but given reconnect is rare, it is probably

Re: [DISCUSS] KIP-35 - Retrieve protocol version

In kafka-python we've been doing something like: if version >= (0, 9): Do cool new stuff elif version >= (0, 8, 2): Do some older stuff else: raise UnsupportedVersionError This will break if / when the new 0.9 apis are completely removed from some future release, but should handle

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Becket, I was thinking about that too. I guess there are two scenarios: upgrades and downgrades. For upgrades, the client may continue using the old API version until the next metadata refresh, which seems ok. For downgrades, we still need some way to signal the client that an API version is

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I'm assuming that the broker would continue to support older versions of the requests, so the check would probably be more like this: if (version > 0.9) // send 0.9 requests else if (version > 0.8) // send 0.8 requests Does that not work? This version can also can be used by clients to tell

Re: [DISCUSS] KIP-35 - Retrieve protocol version

But won't it be the case that what clients end up doing would be something like if(version != 0.8.1) throw new UnsupportedVersionException() which then means the client is broken as soon as we release a new server version even though the protocol didn't change. I'm actually not sure how

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Jay, Yeah, I wasn't suggesting that we eliminate request API versions. They're definitely needed on the broker to support compatibility. I was just saying that if a client wants to support multiple broker versions (e.g. 0.8 and 0.9), then it makes more sense to me to make the kafka release

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I think using Kafka release version makes sense. More particularly, we can use the ApiVersion and this will cover all the interval version as well. In KAFKA-3025, we added the ApiVersion to message format version mapping, We can add the ApiKey to version mapping to ApiVersion as well. We can move

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey guys, I think we want the clients to depend not on the broker version but rather on the protocol version. If you build against protocol version x it is valid with any Kafka version that supports x even as Kafka releases progress. We discussed originally whether to version the protocol as a

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I think Dana's suggestion to include the Kafka release version makes a lot of sense. I'm actually wondering why you would need the individual API versions if you have that? It sounds like keeping track of all the api version information would add a lot of complexity to clients since they'll have

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Tue, Mar 1, 2016 at 6:30 PM, Gwen Shapira wrote: > One more thing, the KIP actually had 3 parts: > 1. The version protocol > 2. New response on messages of wrong API key or wrong version > 3. Protocol documentation > There is a WIP patch for adding protocol docs,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

One more thing, the KIP actually had 3 parts: 1. The version protocol 2. New response on messages of wrong API key or wrong version 3. Protocol documentation I understand that you are offering to only implement part 1? But the KIP discussion and vote should still cover all three parts, they will

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Tue, Mar 1, 2016 at 5:11 PM, Gwen Shapira wrote: > I don't see a use for the name - clients should be able to translate > ApiKey to name for any API they support, and I'm not sure why would a > client need to log anything about APIs it does not support. Am I > missing

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I don't see a use for the name - clients should be able to translate ApiKey to name for any API they support, and I'm not sure why would a client need to log anything about APIs it does not support. Am I missing something? On a related note, Magnus is currently on vacation, but he should be back

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Works with me. I will update PR to remove this. Also, "api_name" have been pointed out as a concern. However, it can be handy for logging and similar purposes. Any take on that? On Tue, Mar 1, 2016 at 3:46 PM, Gwen Shapira wrote: > Jay also mentioned: > "Or, alternately,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Jay also mentioned: "Or, alternately, since deprecation has no functional impact and is just a message to developers, we could just leave it out of the protocol and just have it in release notes etc." I'm in favor of leaving it out of the protocol. I can't really see a use-case. Gwen On Mon,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hello Dana, On Mon, Feb 29, 2016 at 4:11 PM, Dana Powers wrote: > This is a fantastic and much-needed KIP. All third-party clients have had > to deal with this issue. In my experience most clients are either declaring > they only support version broker version X, or are

Re: [DISCUSS] KIP-35 - Retrieve protocol version

This is a fantastic and much-needed KIP. All third-party clients have had to deal with this issue. In my experience most clients are either declaring they only support version broker version X, or are spending a lot of time hacking around the issue. I think the community of non-java drivers would

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I hope it is OK for me to make some progress here. I have made the following changes. 1. Updated KIP-35, to adopt Jay's suggestion on maintaining separate list of deprecated versions, instead of using a version of -1. 2. Added information on required permissions, Describe action on Cluster

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Kafka clients in Hadoop ecosystem, Flume, Spark, etc, have found it really difficult to cope up with Kafka releases as they want to support users on different Kafka versions. Capability to retrieve protocol version will go a long way to ease out those pain points. I will be happy to help out with

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Two quick pieces of feedback: 1. The use of a version of -1 as magical entry dividing between deprecated versions is a bit hacky. What about instead having an array of supported versions and a separate array of deprecated versions. The deprecated versions would always be a subset of the supported

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I am thinking instead of returning an empty response, it would be better to return an explicit UnsupportedVersionException code. Today KafkaApis handles the error in the following way: 1. For requests/responses using old Scala classes, KafkaApis uses RequestOrResponse.handleError() to return an

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I wonder if we need to solve the error problem? I think this KIP gives a descent work around. Probably we should have included an error in the response header, but we debated it at the time decided not to and now it is pretty hard to add because the headers aren't versioned (d'oh). It seems like

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I agree with Jason we should aim to have clients to properly detect and handle unknown-protocol error. The hack of empty response should be shorten solution. After reading the earlier discussion and wiki, I have two questions: - In the future, do we expect all clients to send

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-10-07 9:49 GMT+02:00 Dong Lin : > I agree with Jason we should aim to have clients to properly detect and > handle unknown-protocol error. The hack of empty response should be shorten > solution. > Yes, in the long run we want to modify the common protocol response

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Thanks for the write-up and discussion. This would have been super useful in our last round of deployments at LinkedIn for which we ended up having to hack around a number of incompatibilities. I could list all of the compatibility issues that have hit us, but some are irrelevant to this specific

Re: [DISCUSS] KIP-35 - Retrieve protocol version

After today's KIP call we decided on the following regarding KIP-35: * limit scope to just propagate supported API versions (no key-value tags, broker info, etc) * let the new API return the full list of broker's supported ApiKeys and ApiVersions, rather than an aggregated global version *

Re: [DISCUSS] KIP-35 - Retrieve protocol version

It would be nice to get the unknown api workaround into 0.9, even if we can't have ProtocolVersionRequest. It should be a small change and it allows clients going forward to detect when they have connected to an old broker, which lets them surface a helpful error to the user. This is much better

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-29 3:36 GMT+02:00 Jiangjie Qin : > Thanks for the writeup. I also think having a specific protocol for > client-broker version negotiation is better. > > I'm wondering is it better to let the broker to decide the version to use? > It might have some value If

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Very good points, Todd, totally agree. 2015-09-30 1:04 GMT+02:00 Todd Palino : > We should also consider what else should be negotiated between the broker > and the client as this comes together. The version is definitely first, but > there are other things, such as the max

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-26 6:34 GMT+02:00 Ewen Cheslack-Postava : > The basic functionality is definitely useful here. I'm generally in favor > of exposing some info about broker versions to clients. > > I'd prefer to keep the key/values explicit. Making them extensible > string/string pairs

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-29 17:25 GMT+02:00 Grant Henke : > If we create a protocol version negotiation api for clients, can we use it > to replace or improve the ease of upgrades that break inter-broker > messaging? > > Currently upgrades that break the wire protocol take 2 rolling restarts.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Everyone, thanks for your comments and input this far, here follows an update of the proposal based on the discussions. BrokerMetadataRequest => [NodeId] NodeId => int32 // Request Metadata for these brokers only. // Empty array: retrieve for all brokers.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-28 19:47 GMT+02:00 Jason Gustafson : > Having the version API can make clients more robust, so I'm in favor. One > note on the addition of the "rack" field. Since this is a broker-specific > setting, the client would have to query BrokerMetadata for every new broker >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-28 20:59 GMT+02:00 Jun Rao : > I agree with Ewen that having the keys explicitly specified in the response > is better. > > In addition to the supported protocol version, there are other interesting > metadata at the broker level that could be of interest to things like

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2015-09-29 5:26 GMT+02:00 Mayuresh Gharat : > Nice write-up. > > Just had a question, instead of returning an empty response back to the > client, would it be better for the broker to return a response that gives > some more info to the client regarding the min version

Re: [DISCUSS] KIP-35 - Retrieve protocol version

If we create a protocol version negotiation api for clients, can we use it to replace or improve the ease of upgrades that break inter-broker messaging? Currently upgrades that break the wire protocol take 2 rolling restarts. The first restart we set inter.broker.protocol.version telling all

Re: [DISCUSS] KIP-35 - Retrieve protocol version

We should also consider what else should be negotiated between the broker and the client as this comes together. The version is definitely first, but there are other things, such as the max message size, that should not need to be replicated on both the broker and the client. Granted, max message

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Right. But there should be a max old version that the broker should support to avoid these incompatibility issues. For example, if the broker is at version X, it should be able to support the versions (clients and interbroker) till X-2. In case we have brokers and clients older than that it can

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Nice write-up. Just had a question, instead of returning an empty response back to the client, would it be better for the broker to return a response that gives some more info to the client regarding the min version they need to upgrade to in order to communicate with the broker. Thanks,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I agree with Ewen that having the keys explicitly specified in the response is better. In addition to the supported protocol version, there are other interesting metadata at the broker level that could be of interest to things like admin tools (e.g., used disk space, remaining disk space, etc). I

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Having the version API can make clients more robust, so I'm in favor. One note on the addition of the "rack" field. Since this is a broker-specific setting, the client would have to query BrokerMetadata for every new broker it connects to (unless we also expose rack in TopicMetadata). This is also

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Thanks for the writeup. I also think having a specific protocol for client-broker version negotiation is better. I'm wondering is it better to let the broker to decide the version to use? It might have some value If brokers have preference for a particular version. Using a global version is a

Re: [DISCUSS] KIP-35 - Retrieve protocol version

The basic functionality is definitely useful here. I'm generally in favor of exposing some info about broker versions to clients. I'd prefer to keep the key/values explicit. Making them extensible string/string pairs is good for avoiding unnecessary version changes in the protocol, but I think we

<    1   2