[jira] [Commented] (KAFKA-1888) Add a "rolling upgrade" system test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977270#comment-14977270
 ] 

ASF GitHub Bot commented on KAFKA-1888:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/229


> Add a "rolling upgrade" system test
> ---
>
> Key: KAFKA-1888
> URL: https://issues.apache.org/jira/browse/KAFKA-1888
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Reporter: Gwen Shapira
>Assignee: Geoff Anderson
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-1888_2015-03-23_11:54:25.patch
>
>
> To help test upgrades and compatibility between versions, it will be cool to 
> add a rolling-upgrade test to system tests:
> Given two versions (just a path to the jars?), check that you can do a
> rolling upgrade of the brokers from one version to another (using clients 
> from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a "rolling upgrade" system test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901420#comment-14901420
 ] 

ASF GitHub Bot commented on KAFKA-1888:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/229

KAFKA-1888 [WIP]: rolling upgrade test

@ewencp @gwenshap 
This needs some refactoring to avoid the duplicated code between 
replication test and upgrade test, but in shape for initial feedback.

I'm interested in feedback on the added `KafkaConfig` class and 
`kafka_props` file. This addition makes it:
- easier to attach different configs to different nodes (e.g. during broker 
upgrade process)
- easier to reason about the configuration of a particular node

Notes:
- in the default values in the KafkaConfig class, I removed many properties 
which were in kafka.properties before. This is because most of those properties 
were set to what is already the default value.
- when running non-trunk VerifiableProducer, I append the trunk tools jar 
to the classpath, and run it with the non-trunk kafka-run-class.sh script

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka KAFKA-1888-upgrade-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/229.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #229


commit 22eae20df7ced03ca49f2f97033c21809c19c2e9
Author: Geoff Anderson 
Date:   2015-09-02T23:24:01Z

Added simple mechanism for setting version on a Kafka node. Also updated 
provisioning script to bring in kafka 0.8.2.1, and included simple sanity tests.

commit 1fe461d1e6a9aa28fa23ce61a85ad5722691e003
Author: Geoff Anderson 
Date:   2015-09-15T01:04:44Z

Added sanity checks on VerifiableProducer - check that we can successfully 
start and produce on trunk and on 0.8.2.1

commit 06dcb337ced72876bc03d0242e88971ea4bf57a6
Author: Geoff Anderson 
Date:   2015-09-15T01:13:55Z

Added versioning on console consumer service

commit 80cbe0a0b03411e43439cfedab4bb2b63b26f869
Author: Geoff Anderson 
Date:   2015-09-15T01:18:56Z

Added versioning for verifiable producer, as well as recoverable log4j 
logging

commit 4b8cfc72cb00474f2072cc7bc3136151688277f3
Author: Geoff Anderson 
Date:   2015-09-15T01:23:32Z

Updated Kafka to include versioning and use KafkaConfig class instead of 
properties template file, and added upgrade test




> Add a "rolling upgrade" system test
> ---
>
> Key: KAFKA-1888
> URL: https://issues.apache.org/jira/browse/KAFKA-1888
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Reporter: Gwen Shapira
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-1888_2015-03-23_11:54:25.patch
>
>
> To help test upgrades and compatibility between versions, it will be cool to 
> add a rolling-upgrade test to system tests:
> Given two versions (just a path to the jars?), check that you can do a
> rolling upgrade of the brokers from one version to another (using clients 
> from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Abhishek Nigam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547566#comment-14547566
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Geoffrey,
Thanks for the heads up. I saw a related article that you are planning to work 
on the API compatibility testing as well.
I am taking myself off of this ticket as it looks like this ticket will be 
subsumed by your work.

-Abhishek

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Abhishek Nigam
 Fix For: 0.9.0

 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Geoffrey Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541118#comment-14541118
 ] 

Geoffrey Anderson commented on KAFKA-1888:
--

Just making sure you're aware of work we're doing at Confluent on system tests. 
I'll be posting a KIP for this soon, but here's some info:

The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements

This is the core library/test framework (WIP) which aids in writing and running 
the tests
https://github.com/confluentinc/ducktape/

This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Abhishek Nigam
 Fix For: 0.9.0

 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Abhishek Nigam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393806#comment-14393806
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Hi Gwen/Ashish,
I need to finish up something else and I will only be able to come back to this 
ticket in 2-3 weeks. 

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Abhishek Nigam
 Fix For: 0.9.0

 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Abhishek Nigam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376399#comment-14376399
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Updated reviewboard https://reviews.apache.org/r/30809/diff/
 against branch origin/trunk

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Abhishek Nigam
 Fix For: 0.9.0

 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Abhishek Nigam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355921#comment-14355921
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Hi Gwen,
I am not sure why the link is not showing up. Here you go:
https://reviews.apache.org/r/30809/

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Abhishek Nigam
 Fix For: 0.9.0


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291482#comment-14291482
 ] 

Ashish Kumar Singh commented on KAFKA-1888:
---

[~gwenshap] this is indeed an important step towards easing out upgrade pains. 
[~joestein] gauntlet looks really nice. However, I agree with [~nehanarkhede] 
that we should try to improve upon existing framework and save ourselves from 
YAFS and maintenance cost. Just my 2cents. However, if you guys are not working 
on this, then I can take this one.

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Gwen Shapira
 Fix For: 0.9.0


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291340#comment-14291340
 ] 

Neha Narkhede commented on KAFKA-1888:
--

This is a very useful test. However, I'd recommend at least looking into the 
system test framework before deciding on adding yet another way to do write a 
Kafka test. It will be helpful to see a summary of why this cannot be done 
using the existing system test framework. Arguably, there may be another way to 
design the system test framework which is easier to use, but given that we have 
the current one, we should know what it doesn't support so we can address that 
in the next redesign of the test framework :)

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Gwen Shapira
 Fix For: 0.9.0


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288694#comment-14288694
 ] 

Joe Stein commented on KAFKA-1888:
--

I agree this is something that we need. I don't know if the current system 
tests are the right vehicle for the effort. The system tests haven't been able 
to help compatibility with clients or tooling. We started on some spark jobs 
https://github.com/stealthly/gauntlet which I think we can make work for this 
type of test too. If this is an approach that folks might be interested in the 
core project I could write up a KIP.

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Gwen Shapira
 Fix For: 0.9.0


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test

[Abhishek Nigam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288206#comment-14288206
 ] 

Abhishek Nigam commented on KAFKA-1888:
---

Gwen,
If you are not actively working on it can I pick it up.

 Add a rolling upgrade system test
 ---

 Key: KAFKA-1888
 URL: https://issues.apache.org/jira/browse/KAFKA-1888
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Gwen Shapira
Assignee: Gwen Shapira
 Fix For: 0.9.0


 To help test upgrades and compatibility between versions, it will be cool to 
 add a rolling-upgrade test to system tests:
 Given two versions (just a path to the jars?), check that you can do a
 rolling upgrade of the brokers from one version to another (using clients 
 from the old version) without losing data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)