Review Request 31633: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31633/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31633/diff/ Testing --- Thanks, Geoffrey Anderson
Review Request 31715: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31715/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Added config for backoff jitter; added randomization to a few more client parameters; updated randomization function. Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 06fcfe62cc1fe76f58540221698ef076fe150e96 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 5fb21001abd77cac839bd724afa04e377a3e82aa clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 122375c473bf73caf05299b9f5174c6b226ca863 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31715/diff/ Testing --- Thanks, Geoffrey Anderson
adding replicas to existing topic partitions - KAFKA-1313
Hi dev list, I have a few questions regarding KAFKA-1313 ( https://issues.apache.org/jira/browse/KAFKA-1313) It is currently possible albeit in a kindof hacky way to increase replication factor on a topic by using the partition reassignment tool with a hand-crafted reassignment plan. The preferred approach to making this better is to add support for replication-factor in the alter-topic command. For example, bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --alter --replication-factor 2 Another approach is use the partition reassignment tool to auto-generate a reassignment plan which results in increased replication. The advantage of the alter topics approach is that it seems logical to group this functionality with the other alter topic stuff. However, a couple questions: - Should partitions automatically be reassigned to brokers and shuffled when the replication factor is changed? - If so, it would be best to be able to update the replication factor of multiple topics at once so that the automatic reassignment would take place only once for the whole batch of changes. One advantage of the partition reassignment approach is that it's easier to generate a reassignment plan which updates multiple topics at once, and you have a chance to review the reassignment plan before putting it into effect with the --execute flag. Thanks, Geoff
kafka system tests
Hi dev list, I've been discussing the current state of system tests with Jun and others, and have summarized goals moving forward at: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements Feedback is welcome! Thanks, Geoff
Re: kafka system tests
Hi Gwen, Sorry about that, the ducttape repository was not yet public, but now it is. Cheers, Geoff On Wed, Mar 25, 2015 at 12:08 PM, Gwen Shapira wrote: > Thanks for summarizing! I think we are all feeling the pain here and want > to make life easier moving forward. > > The framework discussion is particularly interesting - unfortunately, the > link to ducttape is broken at the moment. > > Gwen > > On Wed, Mar 25, 2015 at 11:46 AM, Geoffrey Anderson > wrote: > > > Hi dev list, > > > > I've been discussing the current state of system tests with Jun and > others, > > and have summarized goals moving forward at: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements > > > > Feedback is welcome! > > > > Thanks, > > Geoff > > >
[DISCUSS] KIP-25 System test improvements
Hi, Just kicking off the discussion thread on KIP-25 https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements Thanks, Geoff
Re: [DISCUSS] KIP-25 System test improvements
Great, I'll work on putting together a more detailed map of this replacement process. On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira wrote: > Love this idea :) > > I took a look at Ducktape API and it looks like a good fit - clean API, > extensible, easy to use and powerful enough for our use-case. > > Something I'd like to see as part of the KIP is a map of what system-test > currently tests, which ones we want to replace and a JIRA for replacing > (possibly one for each group of tests). > Basically, I know we all want to use the new system for new test cases > (upgrades, etc), but I really want to make sure we don't get stuck with > both systems forever. > > Gwen > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson > wrote: > > > Hi, > > > > Just kicking off the discussion thread on KIP-25 > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements > > > > Thanks, > > Geoff > > >
Re: [DISCUSS] KIP-25 System test improvements
Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test "service" logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava wrote: > Ashish, > > 1. That was the plan. We put some effort into cleanly separating the > framework so it would be reusable across many projects. > 2. I think you're seeing a test in progress where the final report hasn't > been created yet. If you visit one of the older ones you'll see it has a > landing page with links: > http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently > we need to adjust when we update the 'latest' symlink. The logs that are > collected for tests are configurable, and service implementations include > sane defaults (so, e.g., you will always get the normal log file for Kafka, > but only get the data files if the test asks for them). > 3. No code coverage support. Haven't looked into it, so I couldn't comment > on how hard it would be to add. > > -Ewen > > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh wrote: > > > Geoffrey, > > > > This looks great! > > > > A few questions. > > 1. Will ducktape be maintained separately as a github repo? > > 2. How easy is viewing the test results and logs. The link in KIP, > > http://testing.confluent.io/confluent_platform/latest/, lists a bunch of > > files and dirs. Could you add to KIP how the result and logs for the > tests > > will be organized. > > 3. Does it support code coverage? If not, how easy/ difficult would it > be? > > > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson > > wrote: > > > > > Great, I'll work on putting together a more detailed map of this > > > replacement process. > > > > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira > > > wrote: > > > > > > > Love this idea :) > > > > > > > > I took a look at Ducktape API and it looks like a good fit - clean > API, > > > > extensible, easy to use and powerful enough for our use-case. > > > > > > > > Something I'd like to see as part of the KIP is a map of what > > system-test > > > > currently tests, which ones we want to replace and a JIRA for > replacing > > > > (possibly one for each group of tests). > > > > Basically, I know we all want to use the new system for new test > cases > > > > (upgrades, etc), but I really want to make sure we don't get stuck > with > > > > both systems forever. > > > > > > > > Gwen > > > > > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson < > ge...@confluent.io > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Just kicking off the discussion thread on KIP-25 > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements > > > > > > > > > > Thanks, > > > > > Geoff > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > Ashish > > > > > > -- > Thanks, > Ewen >
Re: [DISCUSS] KIP-25 System test improvements
Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira wrote: > Not completely random places :) > People may use Cloudera / HWX distributions which include Kafka, but want > to verify that these bits match a specific upstream release. > > I think having the tests separately will be useful for this. In this case, > finding the tests are not a big issue - we'll add a download link :) > > On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin > wrote: > > > Hey Gwen, > > > > Currently the test and code are downloaded at the same time. Supposedly > > the tests in the same repository should cover match the code. > > Are you saying people downloaded a release from some random place and > want > > to verify it? If that is the case, does that mean people still need to > > find the correct place to download the right test artifact? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > On 6/4/15, 4:29 PM, "Gwen Shapira" wrote: > > > > >Hi, > > > > > >Reviving the discussion a bit :) > > > > > >I think it will be nice if each Kafka version that we release will also > > >have a separate "tests" artifact that users can download, untar and > easily > > >run against a Kafka cluster of the same version. > > > > > >The idea is that if someone downloads packages that claim to contain > > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), > > >users > > >can easily download the tests and verify that it indeed passes the tests > > >for this version and therefore behaves the way this version is expected > to > > >behave. > > > > > >Does it make sense? > > > > > >Gwen > > > > > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson > > >wrote: > > > > > >> Hi Ashish, > > >> > > >> Looks like Ewen already hit the main points, but a few additions: > > >> > > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape > > >> ducktape itself will be pip installable in the near future, and Kafka > > >> system tests will be able to depend on a particular version of > ducktape. > > >> > > >> 2. The reporting is nothing fancy. We're definitely open to feedback, > > >>but > > >> it consists of: > > >> - top level summary of the test run (simple PASS/FAIL for each test) > > >> - top level info and debug logs > > >> - per-test info and debug logs > > >> - per-test "service" logs gathered from each service used in the test. > > >>For > > >> example, if your test pulls up a Kafka cluster with 5 brokers, the end > > >> result will have the Kafka logs from each of those 5 machines. > > >> > > >> Cheers, > > >> Geoff > > >> > > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava > > >> > > >> wrote: > > >> > > >> > Ashish, > > >> > > > >> > 1. That was the plan. We put some effort into cleanly separating the > > >> > framework so it would be reusable across many projects. > > >> > 2. I think you're seeing a test in progress where the final report > > >>hasn't > > >> > been created yet. If you visit one of the older ones you'll see it > > >>has a > > >> > landing page with links: > > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/ > > >> Apparently > > >> > we need to adjust when we update the 'latest' symlink. The logs that > > >>are > > >> > collected for tests are configurable, and service implementations > > >>include > > >> > sane defaults (so, e.g., you will always get the normal log file for > > >> Kafka, > > >> > but only get the data files if the test asks for them). > > >> > 3. No code coverage support. Haven't looked into it, so I couldn't > > >> comment > > >> > on how hard it would be to add. > > >> > > > >> > -Ewen > > >> > > > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh > > >> wrote: > > >> > > > >> > > Geoffrey, > >
Re: [DISCUSS] KIP-25 System test improvements
Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate "tests" artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson wrote: > Hi Gwen, > > I don't see any problem with this as long as we're convinced there's a > good use case, which seems to be true. > > Cheers, > Geoff > > On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira > wrote: > >> Not completely random places :) >> People may use Cloudera / HWX distributions which include Kafka, but want >> to verify that these bits match a specific upstream release. >> >> I think having the tests separately will be useful for this. In this case, >> finding the tests are not a big issue - we'll add a download link :) >> >> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin >> wrote: >> >> > Hey Gwen, >> > >> > Currently the test and code are downloaded at the same time. Supposedly >> > the tests in the same repository should cover match the code. >> > Are you saying people downloaded a release from some random place and >> want >> > to verify it? If that is the case, does that mean people still need to >> > find the correct place to download the right test artifact? >> > >> > Thanks, >> > >> > Jiangjie (Becket) Qin >> > >> > >> > >> > On 6/4/15, 4:29 PM, "Gwen Shapira" wrote: >> > >> > >Hi, >> > > >> > >Reviving the discussion a bit :) >> > > >> > >I think it will be nice if each Kafka version that we release will also >> > >have a separate "tests" artifact that users can download, untar and >> easily >> > >run against a Kafka cluster of the same version. >> > > >> > >The idea is that if someone downloads packages that claim to contain >> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), >> > >users >> > >can easily download the tests and verify that it indeed passes the >> tests >> > >for this version and therefore behaves the way this version is >> expected to >> > >behave. >> > > >> > >Does it make sense? >> > > >> > >Gwen >> > > >> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson > > >> > >wrote: >> > > >> > >> Hi Ashish, >> > >> >> > >> Looks like Ewen already hit the main points, but a few additions: >> > >> >> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape >> > >> ducktape itself will be pip installable in the near future, and Kafka >> > >> system tests will be able to depend on a particular version of >> ducktape. >> > >> >> > >> 2. The reporting is nothing fancy. We're definitely open to >> feedback, >> > >>but >> > >> it consists of: >> > >> - top level summary of the test run (simple PASS/FAIL for each test) >> > >> - top level info and debug logs >> > >> - per-test info and debug logs >> > >> - per-test "service" logs gathered from each service used in the >> test. >> > >>For >> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the &g
[VOTE] KIP-25 System test improvements
Hi Kafka, After a few rounds of discussion on KIP-25, there doesn't seem to be opposition, so I'd like to propose a vote. Thanks, Geoff On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson wrote: > Hi KIP-25 thread, > > I consolidated some of the questions from this thread and elsewhere. > > Q: Can we see a map of what system-test currently tests, which ones we > want to replace and JIRAs for replacing? > A: Initial draft here: > https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests > > Q: Will ducktape be maintained separately as a github repo? > A: Yes https://github.com/confluentinc/ducktape > > Q: How easy is viewing the test results and logs, how will test output be > structured? > A: Hierarchical structure as outlined here: > https://github.com/confluentinc/ducktape/wiki/Design-overview#output > > Q: Does it support code coverage? If not, how easy/ difficult would it be > to support? > A: It does not, and we have no immediate plans to support this. Difficulty > unclear. > > Q: It would be nice if each Kafka version that we release will also > have a separate "tests" artifact that users can download, untar and easily > run against a Kafka cluster of the same version. > A: This seems reasonable and not too much extra work. Definitely open to > discussion on this. > > Q: Why not share running services across multiple tests? > A: Prefer to optimize for simplicity and correctness over what might be a > questionable improvement in run-time. > > Q: Are regressions - in the road map? > A: yes > > Q: Are Jepsen style tests involving network failures in the road map? > A: yes > > Thanks much, > Geoff > > >
Re: [DISCUSS] KIP-25 System test improvements
Hi Aditya, (1) There are a currently a few different ways to target a specific test or subset of tests. If for example tests were organized like the current system tests, where suites are grouped by directory, you could run cd ducktape replication_testsuite/ You can also target tests in a particular file (ducktape ), all tests in a test class (ducktape ::), or a particular test method in a class (ducktape ::.) (2) ducktape runs on a vagrant cluster by parsing the information returned by the "vagrant ssh-config" command into JSON configuration used by the JsonCluster class, so in that sense we are already using the JSON flavor. I see a few potential challenges, but nothing too bad. - There may be some work involved in getting ssh configs right - A couple assumptions about where files are deployed on remote machines are baked into some of the Service classes, so some minor refactoring may be needed to make this a little more general. This would be a good thing. In any case, we're happy to help out/take pull requests on ducktape etc. Best, Geoff On Thu, Jun 11, 2015 at 10:13 AM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Hi Geoffrey, > > Thanks for the writeup. Couple of questions: > - Is it possible to configure suites using ducktape? For example: assume > all the tests in system_tests have been migrated to ducktape. Can I run a > subset of all tests grouped by functional areas i.e. replication, broker > failure etc. > > - Ducktape allows us to run tests on a vagrant cluster or on a static > cluster configured via JSON. Once ported to ducktape, can we very easily > run the existing system tests in both flavors? > > Thanks, > Aditya > > > From: Geoffrey Anderson [ge...@confluent.io] > Sent: Monday, June 08, 2015 10:56 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-25 System test improvements > > Hi KIP-25 thread, > > I consolidated some of the questions from this thread and elsewhere. > > Q: Can we see a map of what system-test currently tests, which ones we want > to replace and JIRAs for replacing? > A: Initial draft here: > > https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests > > Q: Will ducktape be maintained separately as a github repo? > A: Yes https://github.com/confluentinc/ducktape > > Q: How easy is viewing the test results and logs, how will test output be > structured? > A: Hierarchical structure as outlined here: > https://github.com/confluentinc/ducktape/wiki/Design-overview#output > > Q: Does it support code coverage? If not, how easy/ difficult would it be > to support? > A: It does not, and we have no immediate plans to support this. Difficulty > unclear. > > Q: It would be nice if each Kafka version that we release will also > have a separate "tests" artifact that users can download, untar and easily > run against a Kafka cluster of the same version. > A: This seems reasonable and not too much extra work. Definitely open to > discussion on this. > > Q: Why not share running services across multiple tests? > A: Prefer to optimize for simplicity and correctness over what might be a > questionable improvement in run-time. > > Q: Are regressions - in the road map? > A: yes > > Q: Are Jepsen style tests involving network failures in the road map? > A: yes > > Thanks much, > Geoff > > > > On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson > wrote: > > > Hi Gwen, > > > > I don't see any problem with this as long as we're convinced there's a > > good use case, which seems to be true. > > > > Cheers, > > Geoff > > > > On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira > > wrote: > > > >> Not completely random places :) > >> People may use Cloudera / HWX distributions which include Kafka, but > want > >> to verify that these bits match a specific upstream release. > >> > >> I think having the tests separately will be useful for this. In this > case, > >> finding the tests are not a big issue - we'll add a download link :) > >> > >> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin > > >> wrote: > >> > >> > Hey Gwen, > >> > > >> > Currently the test and code are downloaded at the same time. > Supposedly > >> > the tests in the same repository should cover match the code. > >> > Are you saying people downloaded a release from some random place and > >> want > >> > to verify it? If that is the case, does that mean people still need to > >> > find the correct place to download the ri
Re: [VOTE] KIP-25 System test improvements
Hi Gwen, Yes, it looks like it passed with a lazy majority :) I should be submitting a patch with a few initial tests today or tomorrow. Cheers, Geoff On Mon, Jun 15, 2015 at 2:10 PM, Gwen Shapira wrote: > I counted 3 binding votes (Jun, Neha, Joe) and two non-binding (Ewen and > I). > So I assume the KIP passed :) > > What's the next step? Are we expecting a patch with few initial tests soon? > > On Thu, Jun 11, 2015 at 5:41 PM, Jun Rao wrote: > > +1 > > > > Thanks, > > > > Jun > > > > On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson > > wrote: > > > >> Hi Kafka, > >> > >> After a few rounds of discussion on KIP-25, there doesn't seem to be > >> opposition, so I'd like to propose a vote. > >> > >> Thanks, > >> Geoff > >> > >> On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson > >> wrote: > >> > >> > Hi KIP-25 thread, > >> > > >> > I consolidated some of the questions from this thread and elsewhere. > >> > > >> > Q: Can we see a map of what system-test currently tests, which ones we > >> > want to replace and JIRAs for replacing? > >> > A: Initial draft here: > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests > >> > > >> > Q: Will ducktape be maintained separately as a github repo? > >> > A: Yes https://github.com/confluentinc/ducktape > >> > > >> > Q: How easy is viewing the test results and logs, how will test > output be > >> > structured? > >> > A: Hierarchical structure as outlined here: > >> > https://github.com/confluentinc/ducktape/wiki/Design-overview#output > >> > > >> > Q: Does it support code coverage? If not, how easy/ difficult would > it be > >> > to support? > >> > A: It does not, and we have no immediate plans to support this. > >> Difficulty > >> > unclear. > >> > > >> > Q: It would be nice if each Kafka version that we release will also > >> > have a separate "tests" artifact that users can download, untar and > >> easily > >> > run against a Kafka cluster of the same version. > >> > A: This seems reasonable and not too much extra work. Definitely open > to > >> > discussion on this. > >> > > >> > Q: Why not share running services across multiple tests? > >> > A: Prefer to optimize for simplicity and correctness over what might > be a > >> > questionable improvement in run-time. > >> > > >> > Q: Are regressions - in the road map? > >> > A: yes > >> > > >> > Q: Are Jepsen style tests involving network failures in the road map? > >> > A: yes > >> > > >> > Thanks much, > >> > Geoff > >> > > >> > > >> > > >> >
Re: [GitHub] kafka pull request: Kafka 2276
Hi, I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) again since it sounds like at least one person I spoke with did not see the initial pull request. Pull request: https://github.com/apache/kafka/pull/70/ JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 Thanks! Geoff On Tue, Jun 16, 2015 at 2:50 PM, granders wrote: > GitHub user granders opened a pull request: > > https://github.com/apache/kafka/pull/70 > > Kafka 2276 > > Initial patch for KIP-25 > > Note that to install ducktape, do *not* use pip to install ducktape. > Instead: > > ``` > $ git clone g...@github.com:confluentinc/ducktape.git > $ cd ducktape > $ python setup.py install > ``` > > > You can merge this pull request into a Git repository by running: > > $ git pull https://github.com/confluentinc/kafka KAFKA-2276 > > Alternatively you can review and apply these changes as the patch at: > > https://github.com/apache/kafka/pull/70.patch > > To close this pull request, make a commit to your master/trunk branch > with (at least) the following in the commit message: > > This closes #70 > > > commit 81e41562f3836e95e89e12f215c82b1b2d505381 > Author: Liquan Pei > Date: 2015-04-24T01:32:54Z > > Bootstrap Kafka system tests > > commit f1914c3ba9b52d0f8db3989c8b031127b42ac59e > Author: Liquan Pei > Date: 2015-04-24T01:33:44Z > > Merge pull request #2 from confluentinc/system_tests > > Bootstrap Kafka system tests > > commit a2789885806f98dcd1fd58edc9a10a30e4bd314c > Author: Geoff Anderson > Date: 2015-05-26T22:21:23Z > > fixed typos > > commit 07cd1c66a952ee29fc3c8e85464acb43a6981b8a > Author: Geoff Anderson > Date: 2015-05-26T22:22:14Z > > Added simple producer which prints status of produced messages to > stdout. > > commit da94b8cbe79e6634cc32fbe8f6deb25388923029 > Author: Geoff Anderson > Date: 2015-05-27T21:07:20Z > > Added number of messages option. > > commit 212b39a2d75027299fbb1b1008d463a82aab > Author: Geoff Anderson > Date: 2015-05-27T22:35:06Z > > Added some metadata to producer output. > > commit 8b4b1f2aa9681632ef65aa92dfd3066cd7d62851 > Author: Geoff Anderson > Date: 2015-05-29T23:38:32Z > > Minor updates to VerboseProducer > > commit c0526fe44cea739519a0889ebe9ead01b406b365 > Author: Geoff Anderson > Date: 2015-06-01T02:27:15Z > > Updates per review comments. > > commit bc009f218e00241cbdd23931d01b52c442eef6b7 > Author: Geoff Anderson > Date: 2015-06-01T02:28:28Z > > Got rid of VerboseProducer in core (moved to clients) > > commit 475423bb642ac8f816e8080f891867a6362c17fa > Author: Geoff Anderson > Date: 2015-06-01T04:05:09Z > > Convert class to string before adding to json object. > > commit 0a5de8e0590e3a8dce1a91769ad41497b5e07d17 > Author: Geoff Anderson > Date: 2015-06-02T22:46:52Z > > Fixed checkstyle errors. Changed name to VerifiableProducer. Added > synchronization for thread safety on println statements. > > commit 9100417ce0717a71c822c5a279fe7858bfe7a7ee > Author: Geoff Anderson > Date: 2015-06-03T19:50:11Z > > Updated command-line options for VerifiableProducer. Extracted > throughput logic to make it reusable. > > commit 1228eefc4e52b58c214b3ad45feab36a475d5a66 > Author: Geoff Anderson > Date: 2015-06-04T01:09:14Z > > Renamed throttler > > commit 6842ed1ffad62a84df67a0f0b6a651a6df085d12 > Author: Geoff Anderson > Date: 2015-06-04T01:12:11Z > > left out a file from last commit > > commit d586fb0eb63409807c02f280fae786cec55fb348 > Author: Geoff Anderson > Date: 2015-06-04T01:22:34Z > > Updated comments to reflect that throttler is not message-specific > > commit a80a4282ba9a288edba7cdf409d31f01ebf3d458 > Author: Geoff Anderson > Date: 2015-06-04T20:47:21Z > > Added shell program for VerifiableProducer. > > commit 51a94fd6ece926bcdd864af353efcf4c4d1b8ad8 > Author: Geoff Anderson > Date: 2015-06-04T20:55:02Z > > Use argparse4j instead of joptsimple. ThroughputThrottler now has more > intuitive behavior when targetThroughput is 0. > > commit 632be12d2384bfd1ed3b057913dfd363cab71726 > Author: Geoff > Date: 2015-06-04T22:22:44Z > > Merge pull request #3 from confluentinc/verbose-client > > Verbose client > > commit fc7c81c1f6cce497c19da34f7c452ee44800ab6d > Author: Geoff Anderson > Date: 2015-06-11T01:01:39Z > > added setup.py > > commit 884b20e3a7ce7a94f22594782322e4366b51f7eb > Author: Geoff Anderson > Date: 2015-06-11T01:02:11Z > > Moved a bunch of files to kafkatest directory > > commit 25a413d6ae938e9773eb2b20509760bab464 > Author: Geoff > Date: 2015-06-11T20:29:21Z > > Update aws-example-Vagrantfile.local > > commit 96533c3718a9285d78393fb453b951592c72a490 > Author: Geoff > Date: 2015-06-11T20:36:33Z > > Update aws-access-keys-commands > > commit e5edf031aeb99b9176a6ae8375963f2aedaaa6d7 > Author: Geoff Anderson > Date: 2015-06-12T00:27:49Z > > Updated example aws Vagrant
Re: [GitHub] kafka pull request: Kafka 2276
Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx62k1vbv+ycknuxq...@mail.gmail.com%3E Ismael's script is here: https://issues.apache.org/jira/browse/KAFKA-2187 Thanks, Geoff On Mon, Jun 22, 2015 at 9:59 PM, Gwen Shapira wrote: > Thanks, I indeed missed the original :) > > Is the plan to squash the commits and merge a pull request with single > commit that matches the JIRA #? > This will be more in line with how commits were organized until now > and will make life much easier when cherry-picking. > > Gwen > > On Mon, Jun 22, 2015 at 1:58 PM, Geoffrey Anderson > wrote: > > Hi, > > > > I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) > again > > since it sounds like at least one person I spoke with did not see the > > initial pull request. > > > > Pull request: https://github.com/apache/kafka/pull/70/ > > JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 > > > > Thanks! > > Geoff > > > > > > On Tue, Jun 16, 2015 at 2:50 PM, granders wrote: > > > >> GitHub user granders opened a pull request: > >> > >> https://github.com/apache/kafka/pull/70 > >> > >> Kafka 2276 > >> > >> Initial patch for KIP-25 > >> > >> Note that to install ducktape, do *not* use pip to install ducktape. > >> Instead: > >> > >> ``` > >> $ git clone g...@github.com:confluentinc/ducktape.git > >> $ cd ducktape > >> $ python setup.py install > >> ``` > >> > >> > >> You can merge this pull request into a Git repository by running: > >> > >> $ git pull https://github.com/confluentinc/kafka KAFKA-2276 > >> > >> Alternatively you can review and apply these changes as the patch at: > >> > >> https://github.com/apache/kafka/pull/70.patch > >> > >> To close this pull request, make a commit to your master/trunk branch > >> with (at least) the following in the commit message: > >> > >> This closes #70 > >> > >> > >> commit 81e41562f3836e95e89e12f215c82b1b2d505381 > >> Author: Liquan Pei > >> Date: 2015-04-24T01:32:54Z > >> > >> Bootstrap Kafka system tests > >> > >> commit f1914c3ba9b52d0f8db3989c8b031127b42ac59e > >> Author: Liquan Pei > >> Date: 2015-04-24T01:33:44Z > >> > >> Merge pull request #2 from confluentinc/system_tests > >> > >> Bootstrap Kafka system tests > >> > >> commit a2789885806f98dcd1fd58edc9a10a30e4bd314c > >> Author: Geoff Anderson > >> Date: 2015-05-26T22:21:23Z > >> > >> fixed typos > >> > >> commit 07cd1c66a952ee29fc3c8e85464acb43a6981b8a > >> Author: Geoff Anderson > >> Date: 2015-05-26T22:22:14Z > >> > >> Added simple producer which prints status of produced messages to > >> stdout. > >> > >> commit da94b8cbe79e6634cc32fbe8f6deb25388923029 > >> Author: Geoff Anderson > >> Date: 2015-05-27T21:07:20Z > >> > >> Added number of messages option. > >> > >> commit 212b39a2d75027299fbb1b1008d463a82aab > >> Author: Geoff Anderson > >> Date: 2015-05-27T22:35:06Z > >> > >> Added some metadata to producer output. > >> > >> commit 8b4b1f2aa9681632ef65aa92dfd3066cd7d62851 > >> Author: Geoff Anderson > >> Date: 2015-05-29T23:38:32Z > >> > >> Minor updates to VerboseProducer > >> > >> commit c0526fe44cea739519a0889ebe9ead01b406b365 > >> Author: Geoff Anderson > >> Date: 2015-06-01T02:27:15Z > >> > >> Updates per review comments. > >> > >> commit bc009f218e00241cbdd23931d01b52c442eef6b7 > >> Author: Geoff Anderson > >> Date: 2015-06-01T02:28:28Z > >> > >> Got rid of VerboseProducer in core (moved to clients) > >> > >> commit 475423bb642ac8f816e8080f891867a6362c17fa > >> Author: Geoff Anderson > >> Date: 2015-06-01T04:05:09Z > >> > >> Convert class to string before adding to json object. > >> > >> commit 0a5de8e0590e3a8dce1a91
Re: [GitHub] kafka pull request: Kafka 2276
Hi Gwen, A couple other thoughts: services/templates These are jinja2 templates which can be populated with values as desired. Although currently many individual configs in various templates have preset values, we can easily make these more parametrizable (presumably with default values). There is no uniform config file strategy or convention. The templates will be reused, but where actual values used to populate the templates come from is totally flexible. My experience with the existing tests suggests it's not that useful or clear to mandate per-test config files for every service used, but there's nothing here preventing some configs from living in files. multi-version When I prototyped some of this, it was fairly reasonable to have individual services support multiple versions, and this makes it possible to have clusters with *mixed* versions (i.e. some nodes on some versions and some nodes on other versions). kafkatest/tests It's not totally clear whether we'll need to break this into subdirectories. My inclination is to start with modules and break into subdirectories if a particular module becomes complex/big enough. With ducktape, running all tests in a directory is no harder than running all tests in a file, so subdirectories don't change anything from a user perspective. Thanks, Geoff On Tue, Jun 23, 2015 at 7:56 PM, Gwen Shapira wrote: > I'm unclear on the directory structure and few high level things (and > I can't figure out how to comment on more than a single line in > github): > > tests/kafkatest - isn't it redundant? do we expect any non-kafka tests? > > services/templates - those are basically all configuration files to be > used by the services, right? Do we expect a single set for the entire > system? or different templates for different tests? I'm checking > because "systemtests" had config files per test. > > any thoughts on how multi-version tests will work? will we have > service per version? > > kafkatest/tests - do we expect every test to be a single script? or > should we have subdirectories here, to start things in the right > direction? Maybe subdirectories for planned groups of tests? > > kafka/vagrant - is this intentionally not under tests? Is this at all > stand-alone? If so, maybe a separate Jira? This has a bunch of stuff > that I'm not sure we want in the project at all... specific IPs, > Vagrant install from debian packages, etc. > > Gwen > > > > > > > > > > On Tue, Jun 23, 2015 at 10:45 AM, Gwen Shapira > wrote: > > Awesome, thanks :) > > > > On Tue, Jun 23, 2015 at 10:32 AM, Geoffrey Anderson > wrote: > >> Hi Gwen, > >> > >> That is indeed the plan, and my understanding is that the merge script > >> Ismael has been working on helps committers with this step. > >> > >> I'm trying out the Github flow roughly as outlined here: > >> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx62k1vbv+ycknuxq...@mail.gmail.com%3E > >> > >> Ismael's script is here: > https://issues.apache.org/jira/browse/KAFKA-2187 > >> > >> > >> Thanks, > >> > >> Geoff > >> > >> On Mon, Jun 22, 2015 at 9:59 PM, Gwen Shapira > wrote: > >> > >>> Thanks, I indeed missed the original :) > >>> > >>> Is the plan to squash the commits and merge a pull request with single > >>> commit that matches the JIRA #? > >>> This will be more in line with how commits were organized until now > >>> and will make life much easier when cherry-picking. > >>> > >>> Gwen > >>> > >>> On Mon, Jun 22, 2015 at 1:58 PM, Geoffrey Anderson > > >>> wrote: > >>> > Hi, > >>> > > >>> > I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) > >>> again > >>> > since it sounds like at least one person I spoke with did not see the > >>> > initial pull request. > >>> > > >>> > Pull request: https://github.com/apache/kafka/pull/70/ > >>> > JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 > >>> > > >>> > Thanks! > >>> > Geoff > >>> > > >>> > > >>> > On Tue, Jun 16, 2015 at 2:50 PM, granders > wrote: > >>> > > >>> >> GitHub user granders opened a pull request: > >>> >> > >>> >> https://github.com/apache/kafka/pull/70 > >>> >> > >>> >>
Re: Newbie ducktape questions :)
Hi Gwen,
Congrats on becoming a committer!
I submitted a (ducktape) pull request for the cluster size issue here:
https://github.com/confluentinc/ducktape/pull/67 which hopefully makes the
error less confusing.
Maybe we can punt on the slightly funky Vagrantfile.local setup in the
Kafka repository.
Anything else to add? I'm thinking it would be nice to move forward with
the merge and iterate on the workflow/user experience when more people get
a chance to try it.
Thanks,
Geoff
On Thu, Jun 25, 2015 at 12:17 PM, Ewen Cheslack-Postava
wrote:
> To add some reasoning to Geoff's explanation, there are a couple of
> reasons I think separating node configuration from test execution is better:
>
> 1. Allows for (slightly) more sane configuration across different
> platforms -- if we wanted an RPM-based distro or something, that would just
> be a separate script that needs to setup the same stuff. The
> 2. Makes some config options, like JDK version, easily testable without
> requiring test support.
> 3. You don't want to have to handle node configuration as part of a normal
> test run. For jenkins-style tests that might make sense, but often you want
> that step isolated.
> 4. It's obviously generalizable and makes the config as easy as the method
> you use to script it. Docker would be a good example here. You could do it
> all through SSH, but that might actually suck if the DockerCluster
> implementation for ducktape allocates new containers for each test run
> (which would require configuring from scratch every time, if you already
> had an image that would imply you already ran that configuration as a
> pre-process).
>
> It has some drawbacks:
>
> 1. More things to look at/be aware of. Setup process is more complex, and
> I think this is part of what you encountered yesterday.
> 2. Configuration scripts for different cluster types can get disconnected.
> Ideally automated testing across all types of configs would make this
> better, but if people add even just a couple of platforms, that's going to
> be a *lot* of overhead to get setup.
>
> Maybe we should formalize that configuration step somehow in ducktape?
> i.e. add some support to the cluster interface? Currently we assume the
> cluster is already there + configured, and I'm not sure how this would work
> out for "dynamic" clusters (like localhost or docker where you can add
> nodes arbitrarily). But it doesn't sound like a bad idea to be able to boil
> the commands down to:
>
> ducktape cluster configure
> ducktape tests/
>
> -Ewen
>
> P.S. We don't have a ducktape mailing list or anything right now (and it's
> be pretty lonely if we did), but we might also want to move some of this
> discussion back to public lists/JIRAs. There are lots of good ideas flowing
> here, don't want to lose any!
>
> On Thu, Jun 25, 2015 at 12:15 PM, Geoffrey Anderson
> wrote:
>
>> Hi Gwen,
>>
>> Provisioning and installation is *not* baked into ducktape
>> (intentionally), so it's expected that the user has a mechanism to
>> provision whatever machines they're using. In our kafkatest case, the
>> Vagrant provisioning scripts do the work of installing zk, kafka etc on the
>> slave machines.
>>
>> Within ducktape there currently exist three Cluster classes:
>> LocalHostCluster, VagrantCluster, JsonCluster. The --cluster option
>> specifies which cluster class to use.
>>
>> The most generic of these is JsonCluster, which searches for a
>> "cluster.json" file or takes JSON in its constructor, and this JSON data
>> specifies enough information so that ssh commands can be run on individual
>> nodes.
>>
>> VagrantCluster is actually just a JsonCluster but it parses vagrant
>> ssh-config to create its json data. For example:
>>
>> {'nodes': [{'ssh_hostname': '127.0.0.1', 'hostname': 'worker1', 'user':
>> 'vagrant', 'ssh_args': "-o 'HostName 127.0.0.1' -o 'Port ' -o
>> 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o
>> 'PasswordAuthentication no' -o 'IdentityFile
>> /Users/geoffreyanderson/Google_Drive/Confluent_code/cp_system_tests/muckrake/.vagrant/machines/worker1/virtualbox/private_key'
>> -o 'IdentitiesOnly yes' -o 'LogLevel FATAL' "}, {'ssh_hostname':
>> '127.0.0.1', 'hostname': 'worker2', 'user': 'vagrant', 'ssh_args': "-o
>> 'HostName 127.0.0.1' -o 'Port 2200' -o 'UserKnownHostsFile /dev/null&#
Re: [DISCUSS] Json libraries for Kafka
Hi all, my pull request here: https://github.com/apache/kafka/pull/70/files#diff-59f3fe36571d1eee9f923df927a643eb would introduce a client-side dependency on the json-simple package. It is only used in a tool (VerifiableProducer.java), but it sounds like I should probably use Jackson instead? Feel free to make comments on the pull request itself if this is too tangential to the discussion. Thanks, Geoff On Tue, Jul 14, 2015 at 5:23 PM, Jay Kreps wrote: > Ah, makes sense. Yes that addresses my concerns. > > -Jay > > On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma wrote: > > > Hi Jay, > > > > Comments inline. > > > > On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps wrote: > > > > > Is this going to become a dependency for core and then transitively for > > the > > > old clients? > > > > > > That's right. > > > > > > > The current json library is definitely not great, but it does > > > parse json and it's not used in any context where performance is a > > concern. > > > > > > > The issue seemed to indicate that the blocking and slow performance were > > causing issues: > > > > https://issues.apache.org/jira/browse/KAFKA-1595 > > > > > > > > > > Because the older clients aren't well modularized, adding core > > dependencies > > > sucks these up into every user of the clients. This particularly > becomes > > a > > > problem with common libraries since it will turn out we require > version X > > > but other code in the same app requires version Y. > > > > > > > Yes, I understand. Still, if we use Jackson and only use methods that > > existed in 2.0, then it's unlikely to cause issues. I checked it with > > Jackson's author: https://twitter.com/ijuma/status/621106341503991808. > > > > The reasoning is as follows: > > > >- Jackson 1 and Jackson 2 use different packages, so they co-exist > >peacefully. > >- Jackson 2.x releases are backwards compatible so clients are free to > >choose whichever version they want. If they don't pick a version, then > > the > >highest version among the dependencies will be chosen. > >- It is possible that bugs in an untested release would cause issues, > >but that could affect the existing JSON parser too (clients may use > >different Scala versions). > > > > Limiting ourselves to methods from Jackson 2.0 is not as hard as it > sounds > > because the code only interacts with our thin wrapper of Jackson, all the > > code that deals directly with Jackson is isolated. > > > > The new clients fix this issue but not everyone is using them yet. > > > > > > If there is a pressing need maybe we should just do it and people who > > have > > > problems can just hack their build to exclude the dependency (since the > > > client code won't need it). If not it might be better just to leave it > > for > > > a bit until we have at least get a couple releases with both the new > > > producer and the new consumer. > > > > > > > Hacking the builds to exclude the transitive dependency would be a last > > resort, but not needed in most (hopefully all) cases. > > > > Does this make it less problematic in your view? > > > > If people are concerned about this, we can delay it, of course. A bit of > a > > shame though, as this change improves performance, makes the code more > > readable and makes it safer. > > > > Ismael > > >
Review Request 26575: Fix for KAFKA-1700
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26575/ --- Review request for kafka. Bugs: KAFKA-1700 https://issues.apache.org/jira/browse/KAFKA-1700 Repository: kafka Description --- Tweaked README, updated shell scripts so they call kafka-run-class.sh instead of manually adding to the CLASSPATH. Diffs - examples/README 61de2868de29e7c04811bfe12ccabc50b45d148e examples/bin/java-producer-consumer-demo.sh 29e01c2dcf82365c28fef0836e1771282cb49bc1 examples/bin/java-simple-consumer-demo.sh 4716a098c7d404477e0e7254e65e0f509e9df92e Diff: https://reviews.apache.org/r/26575/diff/ Testing --- Thanks, Geoffrey Anderson
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343607#comment-14343607 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31633/diff/ against branch origin/trunk > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede > Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Status: Patch Available (was: Open) > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede > Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede > Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346083#comment-14346083 ] Geoffrey Anderson commented on KAFKA-902: - Thanks for the feedback. Is there any reason to keep 0 -> 0 behavior? In other words, is there any harm to making 0 less magical and perturbing it just like any other value? Also, I like the idea of scaling the upper bound on jitter with backoff_ms, though it seems like we still want some amount of jitter even if backoff_ms is small (in your example, if backoff_ms < 5, then jitter is always 0) In that case, we might want to make sure that the jitter can be non-zero with something like max(3, min(20, 0.2 * backoff_ms)) But then we end up with a couple more semi-arbitrary parameters - a scaling parameter and a hard minimum. > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede >Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede > Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch, KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346149#comment-14346149 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31715/diff/ against branch origin/trunk > Randomize backoff on the clients for metadata requests > -- > > Key: KAFKA-902 > URL: https://issues.apache.org/jira/browse/KAFKA-902 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.0 >Reporter: Neha Narkhede > Assignee: Geoffrey Anderson >Priority: Critical > Labels: newbie > Attachments: KAFKA-902.patch, KAFKA-902.patch > > > If a Kafka broker dies and there are a large number of clients talking to the > Kafka cluster, each of the clients can end up shooting metadata requests at > around the same time. It is better to randomize the backoff on the clients so > the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1313) Support adding replicas to existing topic partitions
[
https://issues.apache.org/jira/browse/KAFKA-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347795#comment-14347795
]
Geoffrey Anderson commented on KAFKA-1313:
--
Quoting the Kafka docs:
The first step is to hand craft the custom reassignment plan in a json file-
> cat increase-replication-factor.json
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
Then, use the json file with the --execute option to start the reassignment
process-
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
> --reassignment-json-file increase-replication-factor.json --execute
---
As I understand it, the pain point is "hand crafting" the reassignment plan.
From there on out, updating replication factor is functionally identical to any
other reassignment. So my proposal is to add another file type
(--replication-factor-json-file) to kafka-reassign-partitions.sh --generate
(and/or --rebalance per KIP 6) which allows users to specify desired
replication_factor per topic/partition.
For example:
>cat update-replication-factor.json
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replication_factor":3}]}
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
--replication-factor-json-file update-replication-factor.json --generate
The user would then feed generated output into kafka-reassign-partitions.sh
--execute as before.
[~nehanarkhede], let me know if this seems reasonable/is in the ballpark of
what you had in mind.
> Support adding replicas to existing topic partitions
>
>
> Key: KAFKA-1313
> URL: https://issues.apache.org/jira/browse/KAFKA-1313
> Project: Kafka
> Issue Type: New Feature
> Components: tools
>Affects Versions: 0.8.0
>Reporter: Marc Labbe
>Assignee: Geoffrey Anderson
>Priority: Critical
> Labels: newbie++
> Fix For: 0.9.0
>
>
> There is currently no easy way to add replicas to an existing topic
> partitions.
> For example, topic create-test has been created with ReplicationFactor=1:
> Topic:create-test PartitionCount:3ReplicationFactor:1 Configs:
> Topic: create-test Partition: 0Leader: 1 Replicas: 1 Isr: 1
> Topic: create-test Partition: 1Leader: 2 Replicas: 2 Isr: 2
> Topic: create-test Partition: 2Leader: 3 Replicas: 3 Isr: 3
> I would like to increase the ReplicationFactor=2 (or more) so it shows up
> like this instead.
> Topic:create-test PartitionCount:3ReplicationFactor:2 Configs:
> Topic: create-test Partition: 0Leader: 1 Replicas: 1,2 Isr: 1,2
> Topic: create-test Partition: 1Leader: 2 Replicas: 2,3 Isr: 2,3
> Topic: create-test Partition: 2Leader: 3 Replicas: 3,1 Isr: 3,1
> Use cases for this:
> - adding brokers and thus increase fault tolerance
> - fixing human errors for topics created with wrong values
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541116#comment-14541116 ] Geoffrey Anderson commented on KAFKA-2171: -- Just pinging this thread to make sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > System Test for Quotas > -- > > Key: KAFKA-2171 > URL: https://issues.apache.org/jira/browse/KAFKA-2171 > Project: Kafka > Issue Type: Sub-task >Reporter: Dong Lin >Assignee: Dong Lin > Original Estimate: 336h > Remaining Estimate: 336h > > Motivation and goal: > We want to make sure that following features are working properly for both > consumer and producer: default quota, client-specific quota, and quota > sharing per clientId. > The tests and configuration described aims to cover most of the scenarios. > More test cases with varying configurations (e.g. ackNum) can be added if > there is good reason to do so. > Initial setup and configuration: > In all scenarios, we first create kafka brokers and topic as follows: > - create two kafka broker processes (by default local) > - create a topic with replication factor = 2 and ackNum = -1 > - let max_read = max_write = 5MB. The test machine is expected to provide > read (write) throughput at least max_read (max_write). > - we consider two rates are approximately the same if they differ by at most > 5%. > Scenario 1: Validate that max_read and max_write are provided by the test > machine(s) using 1 producer and 1 consumer > 1) produce data to the topic without rate limit for 30 seconds > 2) record the rate of producer > 3) then, consume data from the topic without rate limit until he finishes > 4) record the rate of consumer > 5) verify that the data consumed is identical to the data produced > 6) verify that producer rate >= max_write and consumer rate >= max_read > Scenario 2: validate the effectiveness of default write and read quota using > 1 producer and 1 consumer > 1) configure brokers to use max_write/2 as default write quota and max_read/2 > as default read quota > 2) produce data to the topic for 30 seconds > 3) record the rate of producer > 4) then, consume data from the topic until he finishes > 5) record the rate of consumer > 6) verify that the data consumed is identical to the data produced > 7) verify that recorded write (read) rate is within 5% of max_write/2 > (max_read/2). > Scenario 3: validate the effectiveness of client-specific write and read > quota using 2 producers and 2 consumers > 1) configure brokers to use max_write/2 as default write quota and max_read/2 > as default read quota; configure brokers to use max_write/4 for producer_2 > and max_read/4 for consumer_2 > 2) both producers produce data to the topic for 30 seconds. They use > different clientId. > 3) record the rate of producer > 4) both consumers consume data from the topic until they finish. They use > different clientId and groupId. > 5) record the rate of consumer > 6) verify that the data consumed is identical to the data produced > 7) verify that producer_1 and producer_2 rates are approximately max_write/2 > and max_write/4; verify that consumer_1 and consumer_2 rates are > approximately max_read/2 and max_read/4. > Scenario 4: validate the effectiveness of write and read quota sharing among > clients of same clientId using 2 producers and 2 consumers. > 1) configure brokers to use max_write/2 as default write quota and max_read/2 > as default read quota > 2) both producers produce data to the topic for 30 seconds. They use same > clientId. > 3) record the rate of producer > 4) both consumers consume data from the topic until they finish. They use > same clientId but different groupId. > 5) record the rate of consumer > 6) verify that the data consumed is identical to the data produced > 7) verify that total rate of producer_1 and producer_2 is approximately > max_write/2; verify that total rate of consumer_1 and consumer_2 is > approximately max_read/2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1888) Add a "rolling upgrade" system test
[ https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541118#comment-14541118 ] Geoffrey Anderson commented on KAFKA-1888: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > Add a "rolling upgrade" system test > --- > > Key: KAFKA-1888 > URL: https://issues.apache.org/jira/browse/KAFKA-1888 > Project: Kafka > Issue Type: Improvement > Components: system tests >Reporter: Gwen Shapira >Assignee: Abhishek Nigam > Fix For: 0.9.0 > > Attachments: KAFKA-1888_2015-03-23_11:54:25.patch > > > To help test upgrades and compatibility between versions, it will be cool to > add a rolling-upgrade test to system tests: > Given two versions (just a path to the jars?), check that you can do a > rolling upgrade of the brokers from one version to another (using clients > from the old version) without losing data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1904) run sanity failed test
[ https://issues.apache.org/jira/browse/KAFKA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541120#comment-14541120 ] Geoffrey Anderson commented on KAFKA-1904: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > run sanity failed test > -- > > Key: KAFKA-1904 > URL: https://issues.apache.org/jira/browse/KAFKA-1904 > Project: Kafka > Issue Type: Bug > Components: system tests >Reporter: Joe Stein >Priority: Blocker > Fix For: 0.8.3 > > Attachments: run_sanity.log.gz > > > _test_case_name : testcase_1 > _test_class_name : ReplicaBasicTest > arg : bounce_broker : true > arg : broker_type : leader > arg : message_producing_free_time_sec : 15 > arg : num_iteration : 2 > arg : num_messages_to_produce_per_producer_call : 50 > arg : num_partition : 2 > arg : replica_factor : 3 > arg : sleep_seconds_between_producer_calls : 1 > validation_status : > Test completed : FAILED -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1898) compatibility testing framework
[
https://issues.apache.org/jira/browse/KAFKA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541122#comment-14541122
]
Geoffrey Anderson commented on KAFKA-1898:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
> compatibility testing framework
>
>
> Key: KAFKA-1898
> URL: https://issues.apache.org/jira/browse/KAFKA-1898
> Project: Kafka
> Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8.3
>
> Attachments: cctk.png
>
>
> There are a few different scenarios where you want/need to know the
> status/state of a client library that works with Kafka. Client library
> development is not just about supporting the wire protocol but also the
> implementations around specific interactions of the API. The API has
> blossomed into a robust set of producer, consumer, broker and administrative
> calls all of which have layers of logic above them. A Client Library may
> choose to deviate from the path the project sets out and that is ok. The goal
> of this ticket is to have a system for Kafka that can help to explain what
> the library is or isn't doing (regardless of what it claims).
> The idea behind this stems in being able to quickly/easily/succinctly analyze
> the topic message data. Once you can analyze the topic(s) message you can
> gather lots of information about what the client library is doing, is not
> doing and such. There are a few components to this.
> 1) dataset-generator
> Test Kafka dataset generation tool. Generates a random text file with given
> params:
> --filename, -f - output file name.
> --filesize, -s - desired size of output file. The actual size will always be
> a bit larger (with a maximum size of $filesize + $max.length - 1)
> --min.length, -l - minimum generated entry length.
> --max.length, -h - maximum generated entry length.
> Usage:
> ./gradlew build
> java -jar dataset-generator/build/libs/dataset-generator-*.jar -s 10 -l 2
> -h 20
> 2) dataset-producer
> Test Kafka dataset producer tool. Able to produce the given dataset to Kafka
> or Syslog server. The idea here is you already have lots of data sets that
> you want to test different things for. You might have different sized
> messages, formats, etc and want a repeatable benchmark to run and re-run the
> testing on. You could just have a days worth of data and just choose to
> replay it. The CCTK idea is that you are always starting from CONSUME in
> your state of library. If your library is only producing then you will fail a
> bunch of tests and that might be ok for people.
> Accepts following params:
> {code}
> --filename, -f - input file name.
> --kafka, -k - Kafka broker address in host:port format. If this parameter is
> set, --producer.config and --topic must be set too (otherwise they're
> ignored).
> --producer.config, -p - Kafka producer properties file location.
> --topic, -t - Kafka topic to produce to.
> --syslog, -s - Syslog server address. Format: protocol://host:port
> (tcp://0.0.0.0:5140 or udp://0.0.0.0:5141 for example)
> --loop, -l - flag to loop through file until shut off manually. False by
> default.
> Usage:
> ./gradlew build
> java -jar dataset-producer/build/libs/dataset-producer-*.jar --filename
> dataset --syslog tcp://0.0.0.0:5140 --loop true
> {code}
> 3) extract
> This step is good so you can save data and compare tests. It could also be
> removed if folks are just looking for a real live test (and we could support
> that too). Here we are taking data out of Kafka and putting it into
> Cassandra (but other data stores can be used too and we should come up with a
> way to abstract this out completely so folks could implement whatever they
> wanted.
> {code}
> package ly.stealth.shaihulud.reader
> import java.util.UUID
> import com.datastax.spark.connector._
> import com.datastax.spark.connector.cql.CassandraConnector
> import consumer.kafka.MessageAndMetadata
> import consumer.kafka.client.KafkaReceiver
> import org.apache.spark._
> import org.apache.spark.storage.StorageLevel
> import org.apache.spark.streaming._
> import
[jira] [Commented] (KAFKA-1604) System Test for Transaction Management
[ https://issues.apache.org/jira/browse/KAFKA-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541123#comment-14541123 ] Geoffrey Anderson commented on KAFKA-1604: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > System Test for Transaction Management > -- > > Key: KAFKA-1604 > URL: https://issues.apache.org/jira/browse/KAFKA-1604 > Project: Kafka > Issue Type: Test >Reporter: Dong Lin >Assignee: Dong Lin > Labels: transactions > Attachments: KAFKA-1604_2014-08-19_17:31:16.patch, > KAFKA-1604_2014-08-19_21:07:35.patch > > > Perform end-to-end transaction management test in the following steps: > 1) Start Zookeeper. > 2) Start multiple brokers. > 3) Create topic. > 4) Start transaction-aware ProducerPerformance to generate transactional > messages to topic. > 5) Start transaction-aware ConsoleConsumer to read messages from topic. > 6) Bounce brokers (optional). > 7) Verify that same number of messages are sent and received. > This patch depends on KAFKA-1524, KAFKA-1526 and KAFKA-1601. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2003) Add upgrade tests
[ https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541121#comment-14541121 ] Geoffrey Anderson commented on KAFKA-2003: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > Add upgrade tests > - > > Key: KAFKA-2003 > URL: https://issues.apache.org/jira/browse/KAFKA-2003 > Project: Kafka > Issue Type: Improvement > Components: system tests >Reporter: Gwen Shapira >Assignee: Ashish K Singh > > To test protocol changes, compatibility and upgrade process, we need a good > way to test different versions of the product together and to test end-to-end > upgrade process. > For example, for 0.8.2 to 0.8.3 test we want to check: > * Can we start a cluster with a mix of 0.8.2 and 0.8.3 brokers? > * Can a cluster of 0.8.3 brokers bump the protocol level one broker at a time? > * Can 0.8.2 clients run against a cluster of 0.8.3 brokers? > There are probably more questions. But an automated framework that can test > those and report results will be a good start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[ https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541124#comment-14541124 ] Geoffrey Anderson commented on KAFKA-976: - Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > Order-Preserving Mirror Maker Testcase > -- > > Key: KAFKA-976 > URL: https://issues.apache.org/jira/browse/KAFKA-976 > Project: Kafka > Issue Type: Test >Reporter: Guozhang Wang >Assignee: John Fung > Attachments: kafka-976-v1.patch > > > A new testcase (5007) for mirror_maker_testsuite is needed for the > key-dependent order-preserving mirror maker, this is related to KAFKA-957. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1918) System test for ZooKeeper quorum failure scenarios
[
https://issues.apache.org/jira/browse/KAFKA-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541127#comment-14541127
]
Geoffrey Anderson commented on KAFKA-1918:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
> System test for ZooKeeper quorum failure scenarios
> --
>
> Key: KAFKA-1918
> URL: https://issues.apache.org/jira/browse/KAFKA-1918
> Project: Kafka
> Issue Type: Test
>Reporter: Omid Aladini
>
> Following up on the [conversation on the mailing
> list|http://mail-archives.apache.org/mod_mbox/kafka-users/201502.mbox/%3CCAHwHRrX3SAWDUGF5LjU4rrMUsqv%3DtJcyjX7OENeL5C_V5o3tCw%40mail.gmail.com%3E],
> the FAQ writes:
> {quote}
> Once the Zookeeper quorum is down, brokers could result in a bad state and
> could not normally serve client requests, etc. Although when Zookeeper quorum
> recovers, the Kafka brokers should be able to resume to normal state
> automatically, _there are still a few +corner cases+ the they cannot and a
> hard kill-and-recovery is required to bring it back to normal_. Hence it is
> recommended to closely monitor your zookeeper cluster and provision it so
> that it is performant.
> {quote}
> As ZK quorum failures are inevitable (due to rolling upgrades of ZK, leader
> hardware failure, etc), it would be great to identify the corner cases (if
> they still exist) and fix them if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1347) Create a system test for network partitions
[ https://issues.apache.org/jira/browse/KAFKA-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541125#comment-14541125 ] Geoffrey Anderson commented on KAFKA-1347: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake > Create a system test for network partitions > --- > > Key: KAFKA-1347 > URL: https://issues.apache.org/jira/browse/KAFKA-1347 > Project: Kafka > Issue Type: Test >Reporter: Jay Kreps > > We got some free and rather public QA here: > http://aphyr.com/posts/293-call-me-maybe-kafka > We have since added a configuration to disable unclean leader election which > allows you to prefer consistency over availability when all brokers fail. > This has some unit tests, but ultimately there is no reason to believe this > works unless we have a fairly aggressive system test case for it. > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests > It would be good to add support for network partitions. I don't think we > actually need to try to use the jepsen stuff directly, we can just us the > underlying tools it uses--iptables and tc. These are linux specific, but that > is prolly okay. You can see these at work here: > https://github.com/aphyr/jepsen/blob/master/src/jepsen/control/net.clj > Having this would help provide better evidence that this works now, and would > keep it working in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2256) Port system tests
Geoffrey Anderson created KAFKA-2256: Summary: Port system tests Key: KAFKA-2256 URL: https://issues.apache.org/jira/browse/KAFKA-2256 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 This is a tracking issue for the system test suites to be ported per KIP-25 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2257) port replication_testsuite
Geoffrey Anderson created KAFKA-2257: Summary: port replication_testsuite Key: KAFKA-2257 URL: https://issues.apache.org/jira/browse/KAFKA-2257 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port subset of replication_testsuite to run on ducktape. Details to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2258) Port mirrormaker_testsuite
Geoffrey Anderson created KAFKA-2258: Summary: Port mirrormaker_testsuite Key: KAFKA-2258 URL: https://issues.apache.org/jira/browse/KAFKA-2258 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Port mirrormaker_testsuite to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2259) port offset_management_testsuite
Geoffrey Anderson created KAFKA-2259: Summary: port offset_management_testsuite Key: KAFKA-2259 URL: https://issues.apache.org/jira/browse/KAFKA-2259 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2276) Initial patch for KIP-25
Geoffrey Anderson created KAFKA-2276: Summary: Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson commented on KAFKA-2276: -- Pull request is here: https://github.com/apache/kafka/pull/70 > Initial patch for KIP-25 > > > Key: KAFKA-2276 > URL: https://issues.apache.org/jira/browse/KAFKA-2276 > Project: Kafka > Issue Type: Bug > Reporter: Geoffrey Anderson >Assignee: Geoffrey Anderson > > Submit initial patch for KIP-25 > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) > This patch should contain a few Service classes and a few tests which can > serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson edited comment on KAFKA-2276 at 6/22/15 9:00 PM: --- Pull request is here: https://github.com/apache/kafka/pull/70 More info: https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests was (Author: granders): Pull request is here: https://github.com/apache/kafka/pull/70 > Initial patch for KIP-25 > > > Key: KAFKA-2276 > URL: https://issues.apache.org/jira/browse/KAFKA-2276 > Project: Kafka > Issue Type: Bug > Reporter: Geoffrey Anderson >Assignee: Geoffrey Anderson > > Submit initial patch for KIP-25 > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) > This patch should contain a few Service classes and a few tests which can > serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
Geoffrey Anderson created KAFKA-2327:
Summary: broker doesn't start if config defines advertised.host
but not advertised.port
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse/KAFKA-2327
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
Priority: Minor
To reproduce locally, in server.properties, define "advertised.host" and
"port", but not "advertised.port"
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
getInt(KafkaConfig.AdvertisedPortProp))
->
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
advertisedHostName + ":" + advertisedPort
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define "advertised.host" and
"port", but not "advertised.port"
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
getInt(KafkaConfig.AdvertisedPortProp))
->
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
advertisedHostName + ":" + advertisedPort
was:
To reproduce locally, in server.properties, define "advertised.host" and
"port", but not "advertised.port"
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
getInt(KafkaConfig.AdvertisedPortProp))
->
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
advertisedHostName + ":" + advertisedPort
> broker doesn't start if config defines advertised.host but not adverti
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define "advertised.host" and
"port", but not "advertised.port"
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
{code}
...
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
getInt(KafkaConfig.AdvertisedPortProp))
...
{code}
->
{code}
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
advertisedHostName + ":" + advertisedPort
{code}
was:
To reproduce locally, in server.properties, define "advertised.host" and
"port", but not "advertised.port"
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
getInt(KafkaConfig.AdvertisedPortProp))
->
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
advertisedHostName + ":" + advertisedPort
> broker doesn't start if config defines ad
[jira] [Commented] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621091#comment-14621091
]
Geoffrey Anderson commented on KAFKA-2327:
--
Oh quick reply! Yup I just opened a PR for the patch :)
> broker doesn't start if config defines advertised.host but not advertised.port
> --
>
> Key: KAFKA-2327
> URL: https://issues.apache.org/jira/browse/KAFKA-2327
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8.3
> Reporter: Geoffrey Anderson
>Assignee: Geoffrey Anderson
>Priority: Minor
>
> To reproduce locally, in server.properties, define "advertised.host" and
> "port", but not "advertised.port"
> port=9092
> advertised.host.name=localhost
> Then start zookeeper and try to start kafka. The result is an error like so:
> [2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
> kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
> broker endpoint
> at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
> at
> kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
> at
> kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
> at
> kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
> at kafka.server.KafkaConfig.(KafkaConfig.scala:668)
> at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
> at kafka.Kafka$.main(Kafka.scala:58)
> at kafka.Kafka.main(Kafka.scala)
> Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
> the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
> method, and I believe the fix is (starting at line 727)
> {code}
> ...
> } else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
> getInt(KafkaConfig.AdvertisedPortProp) != null) {
> CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
> getString(KafkaConfig.AdvertisedHostNameProp) + ":" +
> getInt(KafkaConfig.AdvertisedPortProp))
> ...
> {code}
> ->
> {code}
> } else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
> getInt(KafkaConfig.AdvertisedPortProp) != null) {
> CoreUtils.listenerListToEndPoints("PLAINTEXT://" +
> advertisedHostName + ":" + advertisedPort
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628928#comment-14628928 ] Geoffrey Anderson commented on KAFKA-2330: -- +1 from me I confirmed on my mac that changes in the kafka directory are immediately visible within the vm's /vagrant directory, and that aws requires rsync. Note that the rsync excludes don't work, but this may actually be a problem with vagrant-aws: https://github.com/mitchellh/vagrant-aws/issues/304 At first I thought this might be an issue with the variable name 'vagrant_excludes': the current Vagrant docs show 'vagrant__exclude' (double underscore, singular - http://docs.vagrantup.com/v2/synced-folders/rsync.html), however you can find both usages in a quick google search, so it's not clear to me which if any is incorrect. I did a quick trial of both variable names, and neither seems to work. > Vagrantfile sets global configs instead of per-provider override configs > > > Key: KAFKA-2330 > URL: https://issues.apache.org/jira/browse/KAFKA-2330 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2.1 >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava >Priority: Minor > Fix For: 0.8.3 > > Attachments: KAFKA-2330.patch > > > There's a couple of minor incorrect usages of the global configuration object > in the Vagrantfile inside provider-specific override blocks where we should > be using the override config object. Two end up being harmless since they > have no affect on other providers (but should still be corrected). The third > results in using rsync when using Virtualbox, which is unnecessary, slower, > and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629060#comment-14629060 ] Geoffrey Anderson commented on KAFKA-2330: -- Sorry, for reference, the aws machine I tested this on had: $ vagrant --version Vagrant 1.7.2 $ vagrant plugin list vagrant-aws (0.6.0) vagrant-hostmanager (1.5.0) vagrant-share (1.1.3, system) > Vagrantfile sets global configs instead of per-provider override configs > > > Key: KAFKA-2330 > URL: https://issues.apache.org/jira/browse/KAFKA-2330 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.2.1 >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava >Priority: Minor > Fix For: 0.8.3 > > Attachments: KAFKA-2330.patch > > > There's a couple of minor incorrect usages of the global configuration object > in the Vagrantfile inside provider-specific override blocks where we should > be using the override config object. Two end up being harmless since they > have no affect on other providers (but should still be corrected). The third > results in using rsync when using Virtualbox, which is unnecessary, slower, > and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
Geoffrey Anderson created KAFKA-2408: Summary: (new) system tests: ConsoleConsumerService occasionally fails to register consumed message Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
[ https://issues.apache.org/jira/browse/KAFKA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2408: - Reviewer: Ewen Cheslack-Postava Description: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. was: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. > (new) system tests: ConsoleConsumerService occasionally fails to register > consumed message > -- > > Key: KAFKA-2408 > URL: https://issues.apache.org/jira/browse/KAFKA-2408 > Project: Kafka > Issue Type: Bug > Reporter: Geoffrey Anderson > Assignee: Geoffrey Anderson > Fix For: 0.8.3 > > > There have been a few spurious failures in ReplicationTest.test_hard_bounce, > where it was reported that a few of the acked messages were not consumed. > Checking the logs, however, it is clear that they were consumed, but > ConsoleConsumerService failed to parse. > Lines causing parsing failure looks something like: > 779725[2015-08-03 07:25:47,757] ERROR > [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], > Error for partition [test_topic,0] to broker 1:class > kafka.common.NotLeaderForPartitionException > (kafka.consumer.ConsumerFetcherThread) > (i.e. the consumed message, and a log message appear on the same line) > ConsoleConsumerService simply tries to strip each line of whitespace and > parse as an integer, which will clearly fail in this case. > Solution should either redirect stderr elsewhere or update parsing to handle > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
Geoffrey Anderson created KAFKA-2439: Summary: Add MirrorMakerService to ducktape system tests Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
[ https://issues.apache.org/jira/browse/KAFKA-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2439: - Reviewer: Ewen Cheslack-Postava > Add MirrorMakerService to ducktape system tests > --- > > Key: KAFKA-2439 > URL: https://issues.apache.org/jira/browse/KAFKA-2439 > Project: Kafka > Issue Type: Sub-task > Components: system tests > Reporter: Geoffrey Anderson > Assignee: Geoffrey Anderson > Fix For: 0.8.3 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1700) examples directory - README and shell scripts are out of date
Geoffrey Anderson created KAFKA-1700: Summary: examples directory - README and shell scripts are out of date Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Fix For: 0.8.2 sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-1700: - Attachment: KAFKA-1700.patch > examples directory - README and shell scripts are out of date > - > > Key: KAFKA-1700 > URL: https://issues.apache.org/jira/browse/KAFKA-1700 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.1 > Reporter: Geoffrey Anderson >Priority: Minor > Labels: newbie > Fix For: 0.8.2 > > Attachments: KAFKA-1700.patch > > > sbt build files were removed during resolution of KAFKA-1254, so the README > under the examples directory should no longer make reference to sbt. > Also, the paths added to CLASSPATH variable in the example shell script are > no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167501#comment-14167501 ] Geoffrey Anderson commented on KAFKA-1700: -- Created reviewboard https://reviews.apache.org/r/26575/diff/ against branch origin/trunk > examples directory - README and shell scripts are out of date > - > > Key: KAFKA-1700 > URL: https://issues.apache.org/jira/browse/KAFKA-1700 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Geoffrey Anderson >Priority: Minor > Labels: newbie > Fix For: 0.8.2 > > Attachments: KAFKA-1700.patch > > > sbt build files were removed during resolution of KAFKA-1254, so the README > under the examples directory should no longer make reference to sbt. > Also, the paths added to CLASSPATH variable in the example shell script are > no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
