[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012529#comment-15012529
 ] 

ASF GitHub Bot commented on KAFKA-2845:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/537


> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.1.0
>
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012528#comment-15012528
 ] 

Geoff Anderson commented on KAFKA-2845:
---

[~guozhang] An example of the behavior we see:

0.9 java producer issues V1 produce request
0.8.2.2 broker returns V0 produce request response
0.9 java producer tries to parse the response it receives (V0) as V1 since that 
is what it expects. These differ => parse error on producer


> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012465#comment-15012465
 ] 

Geoff Anderson commented on KAFKA-2845:
---

[~gwenshap] That is the behavior we expected... but our discovery when adding 
these tests was that the 0.8.X brokers *don't* close the connection for at 
least some requests, and the error occurs on the client side when it tries to 
parse the response.

I looked at NetworkClient.handleCompletedRecieves in 0.8.2.2, and maybe I'm 
missing something, but I don't see what appears to be a version check there? It 
seems like ProtoUtils.currentResponseSchema just finds the latest version for 
the given apiKey?

{code:title=NetworkClient.java}
private void handleCompletedReceives(List responses, long 
now) {
for (NetworkReceive receive : this.selector.completedReceives()) {
int source = receive.source();
ClientRequest req = inFlightRequests.completeNext(source);
ResponseHeader header = ResponseHeader.parse(receive.payload());
short apiKey = req.request().header().apiKey();
Struct body = (Struct) 
ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
correlate(req.request().header(), header);
if (apiKey == ApiKeys.METADATA.id) {
handleMetadataResponse(req.request().header(), body, now);
} else {
// need to add body/header to response here
responses.add(new ClientResponse(req, now, false, body));
}
}
}
{code}

{code:title=ProtoUtils.java}
public static Schema currentResponseSchema(int apiKey) {
return schemaFor(Protocol.RESPONSES, apiKey, latestVersion(apiKey));
}
{code}


> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012516#comment-15012516
 ] 

Guozhang Wang commented on KAFKA-2845:
--

[~geoffra] In 0.8.2.2 we do not use the matching version id of the request but 
always the latest version; in 0.9.0.0 we have changed this behavior, on the 
other hand in 0.8.2.2 there is no version changes against any older versions of 
0.8.2.2.

> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012438#comment-15012438
 ] 

Gwen Shapira commented on KAFKA-2845:
-

[~granders] The broker responds not with the latest version, but with the 
version that matches what it got from the client.  You can look up the code in 
NetworkClient.handleCompletedRecieves, to see how the response schema is 
generated with the request version. 

In an older broker: if the client sends V1, the broker should see it is V1, 
figure out its not a version it can handle (since its an old broker), write an 
error and close the connection.

Thats while new brokers work fine with old clients but not vice-versa.

> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007529#comment-15007529
 ] 

ASF GitHub Bot commented on KAFKA-2845:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/537

KAFKA-2845: new client old broker compatibility



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
KAFKA-2845-new-client-old-broker-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #537


commit f8cb7f9ba8f4c6f08d15af4c1df8d439739e15a9
Author: Geoff Anderson 
Date:   2015-10-29T21:43:40Z

Sketch of compatibility test

commit 2c3b65ef7c26fef035b83e44d6ed1f301cc80a34
Author: Geoff Anderson 
Date:   2015-11-16T22:15:46Z

Add test for 0.9 consumer against 0.8 brokers

commit 924a281eeace248f852551d30bf55e8c5d79a7b8
Author: Geoff Anderson 
Date:   2015-11-16T22:40:03Z

Added compatibility test




> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007575#comment-15007575
 ] 

Grant Henke commented on KAFKA-2845:


This is a great test to have. I recently tested this manually and found the 
same result. 

A few questions based on the expected result and the perspective of Kafka 
clients not being backward compatible with old brokers:

* I understand our current perspective is that clients do not need to be 
backward compatible with brokers in major releases. However, that doesn't mean 
that we _have_ to break unnecessarily. The buffer underflow is due to an extra 
field added for quotas. Is there a way we could handle things more gracefully 
instead of breaking? If so I am happy to brainstorm and help.
* If we can't handle things more gracefully, I think we should handle the issue 
more gracefully and improve the error messaging. I can open a Jira/pr for that 
as well.

> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007673#comment-15007673
 ] 

Geoff Anderson commented on KAFKA-2845:
---

[~granthenke] credit to [~nehanarkhede] for pushing for these tests

There seem to be a couple issues at play here:
- If you poke around RequestKeys.scala, KafkaApis.scala, and 
RequestChannel.scala, it seems that neither 0.8.X nor 0.9.X really seems to do 
any version validation on the broker side, and instead sends back the latest 
request response version it knows. 
In the case of the producer test, the producer sends a v1 produce request, the 
broker returns a v0 response, and then the producer tries to parse the response 
as v1, and fails with BufferUnderflowException. Similarly, if the consumer 
issues a v1 fetch request it receives a v0 response from the old broker, 
resulting in a parse error on the client side.

- Protocol.java at least tries to handle this by closing the connection on the 
broker side, but not all requests have been ported to Protocol.java.

- Note that there is a proposal on the table to improve how both brokers and 
clients handle unsupported requests and request versions:
[KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version]
 - See "Improved handling of unsupported requests on broker"


> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)