[jira] [Updated] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
[ https://issues.apache.org/jira/browse/KAFKA-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2439: - Reviewer: Ewen Cheslack-Postava Add MirrorMakerService to ducktape system tests --- Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2439) Add MirrorMakerService to ducktape system tests
Geoffrey Anderson created KAFKA-2439: Summary: Add MirrorMakerService to ducktape system tests Key: KAFKA-2439 URL: https://issues.apache.org/jira/browse/KAFKA-2439 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
[ https://issues.apache.org/jira/browse/KAFKA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-2408: - Reviewer: Ewen Cheslack-Postava Description: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. was: There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. (new) system tests: ConsoleConsumerService occasionally fails to register consumed message -- Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message
Geoffrey Anderson created KAFKA-2408: Summary: (new) system tests: ConsoleConsumerService occasionally fails to register consumed message Key: KAFKA-2408 URL: https://issues.apache.org/jira/browse/KAFKA-2408 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 There have been a few spurious failures in ReplicationTest.test_hard_bounce, where it was reported that a few of the acked messages were not consumed. Checking the logs, however, it is clear that they were consumed, but ConsoleConsumerService failed to parse. Lines causing parsing failure looks something like: 779725[2015-08-03 07:25:47,757] ERROR [ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1], Error for partition [test_topic,0] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) (i.e. the consumed message, and a log message appear on the same line) ConsoleConsumerService simply tries to strip each line of whitespace and parse as an integer, which will clearly fail in this case. Solution should either redirect stderr elsewhere or update parsing to handle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629060#comment-14629060 ] Geoffrey Anderson commented on KAFKA-2330: -- Sorry, for reference, the aws machine I tested this on had: $ vagrant --version Vagrant 1.7.2 $ vagrant plugin list vagrant-aws (0.6.0) vagrant-hostmanager (1.5.0) vagrant-share (1.1.3, system) Vagrantfile sets global configs instead of per-provider override configs Key: KAFKA-2330 URL: https://issues.apache.org/jira/browse/KAFKA-2330 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-2330.patch There's a couple of minor incorrect usages of the global configuration object in the Vagrantfile inside provider-specific override blocks where we should be using the override config object. Two end up being harmless since they have no affect on other providers (but should still be corrected). The third results in using rsync when using Virtualbox, which is unnecessary, slower, and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2330) Vagrantfile sets global configs instead of per-provider override configs
[ https://issues.apache.org/jira/browse/KAFKA-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628928#comment-14628928 ] Geoffrey Anderson commented on KAFKA-2330: -- +1 from me I confirmed on my mac that changes in the kafka directory are immediately visible within the vm's /vagrant directory, and that aws requires rsync. Note that the rsync excludes don't work, but this may actually be a problem with vagrant-aws: https://github.com/mitchellh/vagrant-aws/issues/304 At first I thought this might be an issue with the variable name 'vagrant_excludes': the current Vagrant docs show 'vagrant__exclude' (double underscore, singular - http://docs.vagrantup.com/v2/synced-folders/rsync.html), however you can find both usages in a quick google search, so it's not clear to me which if any is incorrect. I did a quick trial of both variable names, and neither seems to work. Vagrantfile sets global configs instead of per-provider override configs Key: KAFKA-2330 URL: https://issues.apache.org/jira/browse/KAFKA-2330 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-2330.patch There's a couple of minor incorrect usages of the global configuration object in the Vagrantfile inside provider-specific override blocks where we should be using the override config object. Two end up being harmless since they have no affect on other providers (but should still be corrected). The third results in using rsync when using Virtualbox, which is unnecessary, slower, and requires copying the entire kafka directory to every VM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Json libraries for Kafka
Hi all, my pull request here: https://github.com/apache/kafka/pull/70/files#diff-59f3fe36571d1eee9f923df927a643eb would introduce a client-side dependency on the json-simple package. It is only used in a tool (VerifiableProducer.java), but it sounds like I should probably use Jackson instead? Feel free to make comments on the pull request itself if this is too tangential to the discussion. Thanks, Geoff On Tue, Jul 14, 2015 at 5:23 PM, Jay Kreps j...@confluent.io wrote: Ah, makes sense. Yes that addresses my concerns. -Jay On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma ism...@juma.me.uk wrote: Hi Jay, Comments inline. On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps j...@confluent.io wrote: Is this going to become a dependency for core and then transitively for the old clients? That's right. The current json library is definitely not great, but it does parse json and it's not used in any context where performance is a concern. The issue seemed to indicate that the blocking and slow performance were causing issues: https://issues.apache.org/jira/browse/KAFKA-1595 Because the older clients aren't well modularized, adding core dependencies sucks these up into every user of the clients. This particularly becomes a problem with common libraries since it will turn out we require version X but other code in the same app requires version Y. Yes, I understand. Still, if we use Jackson and only use methods that existed in 2.0, then it's unlikely to cause issues. I checked it with Jackson's author: https://twitter.com/ijuma/status/621106341503991808. The reasoning is as follows: - Jackson 1 and Jackson 2 use different packages, so they co-exist peacefully. - Jackson 2.x releases are backwards compatible so clients are free to choose whichever version they want. If they don't pick a version, then the highest version among the dependencies will be chosen. - It is possible that bugs in an untested release would cause issues, but that could affect the existing JSON parser too (clients may use different Scala versions). Limiting ourselves to methods from Jackson 2.0 is not as hard as it sounds because the code only interacts with our thin wrapper of Jackson, all the code that deals directly with Jackson is isolated. The new clients fix this issue but not everyone is using them yet. If there is a pressing need maybe we should just do it and people who have problems can just hack their build to exclude the dependency (since the client code won't need it). If not it might be better just to leave it for a bit until we have at least get a couple releases with both the new producer and the new consumer. Hacking the builds to exclude the transitive dependency would be a last resort, but not needed in most (hopefully all) cases. Does this make it less problematic in your view? If people are concerned about this, we can delay it, of course. A bit of a shame though, as this change improves performance, makes the code more readable and makes it safer. Ismael
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
was:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse
[jira] [Created] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
Geoffrey Anderson created KAFKA-2327:
Summary: broker doesn't start if config defines advertised.host
but not advertised.port
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse/KAFKA-2327
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
Priority: Minor
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Anderson updated KAFKA-2327:
-
Description:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
{code}
...
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
...
{code}
-
{code}
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
{code}
was:
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
-
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL: https
[jira] [Commented] (KAFKA-2327) broker doesn't start if config defines advertised.host but not advertised.port
[
https://issues.apache.org/jira/browse/KAFKA-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621091#comment-14621091
]
Geoffrey Anderson commented on KAFKA-2327:
--
Oh quick reply! Yup I just opened a PR for the patch :)
broker doesn't start if config defines advertised.host but not advertised.port
--
Key: KAFKA-2327
URL: https://issues.apache.org/jira/browse/KAFKA-2327
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.3
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
Priority: Minor
To reproduce locally, in server.properties, define advertised.host and
port, but not advertised.port
port=9092
advertised.host.name=localhost
Then start zookeeper and try to start kafka. The result is an error like so:
[2015-07-09 11:29:20,760] FATAL (kafka.Kafka$)
kafka.common.KafkaException: Unable to parse PLAINTEXT://localhost:null to a
broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:49)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
kafka.utils.CoreUtils$$anonfun$listenerListToEndPoints$1.apply(CoreUtils.scala:309)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)
at
kafka.server.KafkaConfig.getAdvertisedListeners(KafkaConfig.scala:728)
at kafka.server.KafkaConfig.init(KafkaConfig.scala:668)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:541)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Looks like this was changed in 5c9040745466945a04ea0315de583ccdab0614ac
the cause seems to be in KafkaConfig.scala in the getAdvertisedListeners
method, and I believe the fix is (starting at line 727)
{code}
...
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
getString(KafkaConfig.AdvertisedHostNameProp) + : +
getInt(KafkaConfig.AdvertisedPortProp))
...
{code}
-
{code}
} else if (getString(KafkaConfig.AdvertisedHostNameProp) != null ||
getInt(KafkaConfig.AdvertisedPortProp) != null) {
CoreUtils.listenerListToEndPoints(PLAINTEXT:// +
advertisedHostName + : + advertisedPort
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Newbie ducktape questions :)
Hi Gwen,
Congrats on becoming a committer!
I submitted a (ducktape) pull request for the cluster size issue here:
https://github.com/confluentinc/ducktape/pull/67 which hopefully makes the
error less confusing.
Maybe we can punt on the slightly funky Vagrantfile.local setup in the
Kafka repository.
Anything else to add? I'm thinking it would be nice to move forward with
the merge and iterate on the workflow/user experience when more people get
a chance to try it.
Thanks,
Geoff
On Thu, Jun 25, 2015 at 12:17 PM, Ewen Cheslack-Postava e...@confluent.io
wrote:
To add some reasoning to Geoff's explanation, there are a couple of
reasons I think separating node configuration from test execution is better:
1. Allows for (slightly) more sane configuration across different
platforms -- if we wanted an RPM-based distro or something, that would just
be a separate script that needs to setup the same stuff. The
2. Makes some config options, like JDK version, easily testable without
requiring test support.
3. You don't want to have to handle node configuration as part of a normal
test run. For jenkins-style tests that might make sense, but often you want
that step isolated.
4. It's obviously generalizable and makes the config as easy as the method
you use to script it. Docker would be a good example here. You could do it
all through SSH, but that might actually suck if the DockerCluster
implementation for ducktape allocates new containers for each test run
(which would require configuring from scratch every time, if you already
had an image that would imply you already ran that configuration as a
pre-process).
It has some drawbacks:
1. More things to look at/be aware of. Setup process is more complex, and
I think this is part of what you encountered yesterday.
2. Configuration scripts for different cluster types can get disconnected.
Ideally automated testing across all types of configs would make this
better, but if people add even just a couple of platforms, that's going to
be a *lot* of overhead to get setup.
Maybe we should formalize that configuration step somehow in ducktape?
i.e. add some support to the cluster interface? Currently we assume the
cluster is already there + configured, and I'm not sure how this would work
out for dynamic clusters (like localhost or docker where you can add
nodes arbitrarily). But it doesn't sound like a bad idea to be able to boil
the commands down to:
ducktape cluster configure
ducktape tests/
-Ewen
P.S. We don't have a ducktape mailing list or anything right now (and it's
be pretty lonely if we did), but we might also want to move some of this
discussion back to public lists/JIRAs. There are lots of good ideas flowing
here, don't want to lose any!
On Thu, Jun 25, 2015 at 12:15 PM, Geoffrey Anderson ge...@confluent.io
wrote:
Hi Gwen,
Provisioning and installation is *not* baked into ducktape
(intentionally), so it's expected that the user has a mechanism to
provision whatever machines they're using. In our kafkatest case, the
Vagrant provisioning scripts do the work of installing zk, kafka etc on the
slave machines.
Within ducktape there currently exist three Cluster classes:
LocalHostCluster, VagrantCluster, JsonCluster. The --cluster option
specifies which cluster class to use.
The most generic of these is JsonCluster, which searches for a
cluster.json file or takes JSON in its constructor, and this JSON data
specifies enough information so that ssh commands can be run on individual
nodes.
VagrantCluster is actually just a JsonCluster but it parses vagrant
ssh-config to create its json data. For example:
{'nodes': [{'ssh_hostname': '127.0.0.1', 'hostname': 'worker1', 'user':
'vagrant', 'ssh_args': -o 'HostName 127.0.0.1' -o 'Port ' -o
'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' -o
'PasswordAuthentication no' -o 'IdentityFile
/Users/geoffreyanderson/Google_Drive/Confluent_code/cp_system_tests/muckrake/.vagrant/machines/worker1/virtualbox/private_key'
-o 'IdentitiesOnly yes' -o 'LogLevel FATAL' }, {'ssh_hostname':
'127.0.0.1', 'hostname': 'worker2', 'user': 'vagrant', 'ssh_args': -o
'HostName 127.0.0.1' -o 'Port 2200' -o 'UserKnownHostsFile /dev/null' -o
'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
/Users/geoffreyanderson/Google_Drive/Confluent_code/cp_system_tests/muckrake/.vagrant/machines/worker2/virtualbox/private_key'
-o 'IdentitiesOnly yes' -o 'LogLevel FATAL' }, {'ssh_hostname':
'127.0.0.1', 'hostname': 'worker3', 'user': 'vagrant', 'ssh_args': -o
'HostName 127.0.0.1' -o 'Port 2201' -o 'UserKnownHostsFile /dev/null' -o
'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
/Users/geoffreyanderson/Google_Drive/Confluent_code/cp_system_tests/muckrake/.vagrant/machines/worker3/virtualbox/private_key'
-o 'IdentitiesOnly yes' -o 'LogLevel FATAL' }]}
Does that make sense?
Cheers,
Geoff
On Thu, Jun 25, 2015
Re: [GitHub] kafka pull request: Kafka 2276
Hi Gwen, A couple other thoughts: services/templates These are jinja2 templates which can be populated with values as desired. Although currently many individual configs in various templates have preset values, we can easily make these more parametrizable (presumably with default values). There is no uniform config file strategy or convention. The templates will be reused, but where actual values used to populate the templates come from is totally flexible. My experience with the existing tests suggests it's not that useful or clear to mandate per-test config files for every service used, but there's nothing here preventing some configs from living in files. multi-version When I prototyped some of this, it was fairly reasonable to have individual services support multiple versions, and this makes it possible to have clusters with *mixed* versions (i.e. some nodes on some versions and some nodes on other versions). kafkatest/tests It's not totally clear whether we'll need to break this into subdirectories. My inclination is to start with modules and break into subdirectories if a particular module becomes complex/big enough. With ducktape, running all tests in a directory is no harder than running all tests in a file, so subdirectories don't change anything from a user perspective. Thanks, Geoff On Tue, Jun 23, 2015 at 7:56 PM, Gwen Shapira gshap...@cloudera.com wrote: I'm unclear on the directory structure and few high level things (and I can't figure out how to comment on more than a single line in github): tests/kafkatest - isn't it redundant? do we expect any non-kafka tests? services/templates - those are basically all configuration files to be used by the services, right? Do we expect a single set for the entire system? or different templates for different tests? I'm checking because systemtests had config files per test. any thoughts on how multi-version tests will work? will we have service per version? kafkatest/tests - do we expect every test to be a single script? or should we have subdirectories here, to start things in the right direction? Maybe subdirectories for planned groups of tests? kafka/vagrant - is this intentionally not under tests? Is this at all stand-alone? If so, maybe a separate Jira? This has a bunch of stuff that I'm not sure we want in the project at all... specific IPs, Vagrant install from debian packages, etc. Gwen On Tue, Jun 23, 2015 at 10:45 AM, Gwen Shapira gshap...@cloudera.com wrote: Awesome, thanks :) On Tue, Jun 23, 2015 at 10:32 AM, Geoffrey Anderson ge...@confluent.io wrote: Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx62k1vbv+ycknuxq...@mail.gmail.com%3E Ismael's script is here: https://issues.apache.org/jira/browse/KAFKA-2187 Thanks, Geoff On Mon, Jun 22, 2015 at 9:59 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks, I indeed missed the original :) Is the plan to squash the commits and merge a pull request with single commit that matches the JIRA #? This will be more in line with how commits were organized until now and will make life much easier when cherry-picking. Gwen On Mon, Jun 22, 2015 at 1:58 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi, I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) again since it sounds like at least one person I spoke with did not see the initial pull request. Pull request: https://github.com/apache/kafka/pull/70/ JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 Thanks! Geoff On Tue, Jun 16, 2015 at 2:50 PM, granders g...@git.apache.org wrote: GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/70 Kafka 2276 Initial patch for KIP-25 Note that to install ducktape, do *not* use pip to install ducktape. Instead: ``` $ git clone g...@github.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2276 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/70.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #70 commit 81e41562f3836e95e89e12f215c82b1b2d505381 Author: Liquan Pei liquan...@gmail.com Date: 2015-04-24T01:32:54Z Bootstrap Kafka system tests commit f1914c3ba9b52d0f8db3989c8b031127b42ac59e Author: Liquan
Re: [GitHub] kafka pull request: Kafka 2276
Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201504.mbox/%3ccad5tkzab-hkey-zcr8x4wtxawybxpojx62k1vbv+ycknuxq...@mail.gmail.com%3E Ismael's script is here: https://issues.apache.org/jira/browse/KAFKA-2187 Thanks, Geoff On Mon, Jun 22, 2015 at 9:59 PM, Gwen Shapira gshap...@cloudera.com wrote: Thanks, I indeed missed the original :) Is the plan to squash the commits and merge a pull request with single commit that matches the JIRA #? This will be more in line with how commits were organized until now and will make life much easier when cherry-picking. Gwen On Mon, Jun 22, 2015 at 1:58 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi, I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) again since it sounds like at least one person I spoke with did not see the initial pull request. Pull request: https://github.com/apache/kafka/pull/70/ JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 Thanks! Geoff On Tue, Jun 16, 2015 at 2:50 PM, granders g...@git.apache.org wrote: GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/70 Kafka 2276 Initial patch for KIP-25 Note that to install ducktape, do *not* use pip to install ducktape. Instead: ``` $ git clone g...@github.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2276 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/70.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #70 commit 81e41562f3836e95e89e12f215c82b1b2d505381 Author: Liquan Pei liquan...@gmail.com Date: 2015-04-24T01:32:54Z Bootstrap Kafka system tests commit f1914c3ba9b52d0f8db3989c8b031127b42ac59e Author: Liquan Pei liquan...@gmail.com Date: 2015-04-24T01:33:44Z Merge pull request #2 from confluentinc/system_tests Bootstrap Kafka system tests commit a2789885806f98dcd1fd58edc9a10a30e4bd314c Author: Geoff Anderson ge...@confluent.io Date: 2015-05-26T22:21:23Z fixed typos commit 07cd1c66a952ee29fc3c8e85464acb43a6981b8a Author: Geoff Anderson ge...@confluent.io Date: 2015-05-26T22:22:14Z Added simple producer which prints status of produced messages to stdout. commit da94b8cbe79e6634cc32fbe8f6deb25388923029 Author: Geoff Anderson ge...@confluent.io Date: 2015-05-27T21:07:20Z Added number of messages option. commit 212b39a2d75027299fbb1b1008d463a82aab Author: Geoff Anderson ge...@confluent.io Date: 2015-05-27T22:35:06Z Added some metadata to producer output. commit 8b4b1f2aa9681632ef65aa92dfd3066cd7d62851 Author: Geoff Anderson ge...@confluent.io Date: 2015-05-29T23:38:32Z Minor updates to VerboseProducer commit c0526fe44cea739519a0889ebe9ead01b406b365 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T02:27:15Z Updates per review comments. commit bc009f218e00241cbdd23931d01b52c442eef6b7 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T02:28:28Z Got rid of VerboseProducer in core (moved to clients) commit 475423bb642ac8f816e8080f891867a6362c17fa Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T04:05:09Z Convert class to string before adding to json object. commit 0a5de8e0590e3a8dce1a91769ad41497b5e07d17 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-02T22:46:52Z Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. commit 9100417ce0717a71c822c5a279fe7858bfe7a7ee Author: Geoff Anderson ge...@confluent.io Date: 2015-06-03T19:50:11Z Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. commit 1228eefc4e52b58c214b3ad45feab36a475d5a66 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:09:14Z Renamed throttler commit 6842ed1ffad62a84df67a0f0b6a651a6df085d12 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:12:11Z left out a file from last commit commit d586fb0eb63409807c02f280fae786cec55fb348 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:22:34Z Updated comments to reflect that throttler is not message-specific commit a80a4282ba9a288edba7cdf409d31f01ebf3d458 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T20:47:21Z
[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson commented on KAFKA-2276: -- Pull request is here: https://github.com/apache/kafka/pull/70 Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [GitHub] kafka pull request: Kafka 2276
Hi, I'm pinging the dev list regarding KAFKA-2276 (KIP-25 initial patch) again since it sounds like at least one person I spoke with did not see the initial pull request. Pull request: https://github.com/apache/kafka/pull/70/ JIRA: https://issues.apache.org/jira/browse/KAFKA-2276 Thanks! Geoff On Tue, Jun 16, 2015 at 2:50 PM, granders g...@git.apache.org wrote: GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/70 Kafka 2276 Initial patch for KIP-25 Note that to install ducktape, do *not* use pip to install ducktape. Instead: ``` $ git clone g...@github.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2276 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/70.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #70 commit 81e41562f3836e95e89e12f215c82b1b2d505381 Author: Liquan Pei liquan...@gmail.com Date: 2015-04-24T01:32:54Z Bootstrap Kafka system tests commit f1914c3ba9b52d0f8db3989c8b031127b42ac59e Author: Liquan Pei liquan...@gmail.com Date: 2015-04-24T01:33:44Z Merge pull request #2 from confluentinc/system_tests Bootstrap Kafka system tests commit a2789885806f98dcd1fd58edc9a10a30e4bd314c Author: Geoff Anderson ge...@confluent.io Date: 2015-05-26T22:21:23Z fixed typos commit 07cd1c66a952ee29fc3c8e85464acb43a6981b8a Author: Geoff Anderson ge...@confluent.io Date: 2015-05-26T22:22:14Z Added simple producer which prints status of produced messages to stdout. commit da94b8cbe79e6634cc32fbe8f6deb25388923029 Author: Geoff Anderson ge...@confluent.io Date: 2015-05-27T21:07:20Z Added number of messages option. commit 212b39a2d75027299fbb1b1008d463a82aab Author: Geoff Anderson ge...@confluent.io Date: 2015-05-27T22:35:06Z Added some metadata to producer output. commit 8b4b1f2aa9681632ef65aa92dfd3066cd7d62851 Author: Geoff Anderson ge...@confluent.io Date: 2015-05-29T23:38:32Z Minor updates to VerboseProducer commit c0526fe44cea739519a0889ebe9ead01b406b365 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T02:27:15Z Updates per review comments. commit bc009f218e00241cbdd23931d01b52c442eef6b7 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T02:28:28Z Got rid of VerboseProducer in core (moved to clients) commit 475423bb642ac8f816e8080f891867a6362c17fa Author: Geoff Anderson ge...@confluent.io Date: 2015-06-01T04:05:09Z Convert class to string before adding to json object. commit 0a5de8e0590e3a8dce1a91769ad41497b5e07d17 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-02T22:46:52Z Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. commit 9100417ce0717a71c822c5a279fe7858bfe7a7ee Author: Geoff Anderson ge...@confluent.io Date: 2015-06-03T19:50:11Z Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. commit 1228eefc4e52b58c214b3ad45feab36a475d5a66 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:09:14Z Renamed throttler commit 6842ed1ffad62a84df67a0f0b6a651a6df085d12 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:12:11Z left out a file from last commit commit d586fb0eb63409807c02f280fae786cec55fb348 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T01:22:34Z Updated comments to reflect that throttler is not message-specific commit a80a4282ba9a288edba7cdf409d31f01ebf3d458 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T20:47:21Z Added shell program for VerifiableProducer. commit 51a94fd6ece926bcdd864af353efcf4c4d1b8ad8 Author: Geoff Anderson ge...@confluent.io Date: 2015-06-04T20:55:02Z Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. commit 632be12d2384bfd1ed3b057913dfd363cab71726 Author: Geoff grand...@gmail.com Date: 2015-06-04T22:22:44Z Merge pull request #3 from confluentinc/verbose-client Verbose client commit fc7c81c1f6cce497c19da34f7c452ee44800ab6d Author: Geoff Anderson ge...@confluent.io Date: 2015-06-11T01:01:39Z added setup.py commit 884b20e3a7ce7a94f22594782322e4366b51f7eb Author: Geoff Anderson ge...@confluent.io Date: 2015-06-11T01:02:11Z Moved a bunch of files to kafkatest directory commit 25a413d6ae938e9773eb2b20509760bab464 Author: Geoff grand...@gmail.com Date: 2015-06-11T20:29:21Z Update aws-example-Vagrantfile.local commit 96533c3718a9285d78393fb453b951592c72a490
[jira] [Comment Edited] (KAFKA-2276) Initial patch for KIP-25
[ https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596624#comment-14596624 ] Geoffrey Anderson edited comment on KAFKA-2276 at 6/22/15 9:00 PM: --- Pull request is here: https://github.com/apache/kafka/pull/70 More info: https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests was (Author: granders): Pull request is here: https://github.com/apache/kafka/pull/70 Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2276) Initial patch for KIP-25
Geoffrey Anderson created KAFKA-2276: Summary: Initial patch for KIP-25 Key: KAFKA-2276 URL: https://issues.apache.org/jira/browse/KAFKA-2276 Project: Kafka Issue Type: Bug Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Submit initial patch for KIP-25 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements) This patch should contain a few Service classes and a few tests which can serve as examples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] KIP-25 System test improvements
Hi Gwen, Yes, it looks like it passed with a lazy majority :) I should be submitting a patch with a few initial tests today or tomorrow. Cheers, Geoff On Mon, Jun 15, 2015 at 2:10 PM, Gwen Shapira gshap...@cloudera.com wrote: I counted 3 binding votes (Jun, Neha, Joe) and two non-binding (Ewen and I). So I assume the KIP passed :) What's the next step? Are we expecting a patch with few initial tests soon? On Thu, Jun 11, 2015 at 5:41 PM, Jun Rao j...@confluent.io wrote: +1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Kafka, After a few rounds of discussion on KIP-25, there doesn't seem to be opposition, so I'd like to propose a vote. Thanks, Geoff On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff
Re: [DISCUSS] KIP-25 System test improvements
Hi Aditya, (1) There are a currently a few different ways to target a specific test or subset of tests. If for example tests were organized like the current system tests, where suites are grouped by directory, you could run cd system_test_dir ducktape replication_testsuite/ You can also target tests in a particular file (ducktape path_to_file), all tests in a test class (ducktape path_to_file::test_class), or a particular test method in a class (ducktape path_to_file::test_class.test_method) (2) ducktape runs on a vagrant cluster by parsing the information returned by the vagrant ssh-config command into JSON configuration used by the JsonCluster class, so in that sense we are already using the JSON flavor. I see a few potential challenges, but nothing too bad. - There may be some work involved in getting ssh configs right - A couple assumptions about where files are deployed on remote machines are baked into some of the Service classes, so some minor refactoring may be needed to make this a little more general. This would be a good thing. In any case, we're happy to help out/take pull requests on ducktape etc. Best, Geoff On Thu, Jun 11, 2015 at 10:13 AM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: Hi Geoffrey, Thanks for the writeup. Couple of questions: - Is it possible to configure suites using ducktape? For example: assume all the tests in system_tests have been migrated to ducktape. Can I run a subset of all tests grouped by functional areas i.e. replication, broker failure etc. - Ducktape allows us to run tests on a vagrant cluster or on a static cluster configured via JSON. Once ported to ducktape, can we very easily run the existing system tests in both flavors? Thanks, Aditya From: Geoffrey Anderson [ge...@confluent.io] Sent: Monday, June 08, 2015 10:56 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-25 System test improvements Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira gshap...@cloudera.com wrote: Not completely random places :) People may use Cloudera / HWX distributions which include Kafka, but want to verify that these bits match a specific upstream release. I think having the tests separately will be useful for this. In this case, finding the tests are not a big issue - we'll add a download link :) On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact? Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Reviving the discussion a bit :) I think it will be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. The idea is that if someone downloads packages that claim to contain something of a specific Kafka version (i.e. Kafka 0.8.2.0
[VOTE] KIP-25 System test improvements
Hi Kafka, After a few rounds of discussion on KIP-25, there doesn't seem to be opposition, so I'd like to propose a vote. Thanks, Geoff On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff
[jira] [Created] (KAFKA-2259) port offset_management_testsuite
Geoffrey Anderson created KAFKA-2259: Summary: port offset_management_testsuite Key: KAFKA-2259 URL: https://issues.apache.org/jira/browse/KAFKA-2259 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2258) Port mirrormaker_testsuite
Geoffrey Anderson created KAFKA-2258: Summary: Port mirrormaker_testsuite Key: KAFKA-2258 URL: https://issues.apache.org/jira/browse/KAFKA-2258 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Port mirrormaker_testsuite to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2257) port replication_testsuite
Geoffrey Anderson created KAFKA-2257: Summary: port replication_testsuite Key: KAFKA-2257 URL: https://issues.apache.org/jira/browse/KAFKA-2257 Project: Kafka Issue Type: Sub-task Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 Port subset of replication_testsuite to run on ducktape. Details to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2256) Port system tests
Geoffrey Anderson created KAFKA-2256: Summary: Port system tests Key: KAFKA-2256 URL: https://issues.apache.org/jira/browse/KAFKA-2256 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Geoffrey Anderson Assignee: Geoffrey Anderson Fix For: 0.8.3 This is a tracking issue for the system test suites to be ported per KIP-25 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-25 System test improvements
Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira gshap...@cloudera.com wrote: Not completely random places :) People may use Cloudera / HWX distributions which include Kafka, but want to verify that these bits match a specific upstream release. I think having the tests separately will be useful for this. In this case, finding the tests are not a big issue - we'll add a download link :) On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact? Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Reviving the discussion a bit :) I think it will be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. The idea is that if someone downloads packages that claim to contain something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), users can easily download the tests and verify that it indeed passes the tests for this version and therefore behaves the way this version is expected to behave. Does it make sense? Gwen On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test service logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Ashish, 1. That was the plan. We put some effort into cleanly separating the framework so it would be reusable across many projects. 2. I think you're seeing a test in progress where the final report hasn't been created yet. If you visit one of the older ones you'll see it has a landing page with links: http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently we need to adjust when we update the 'latest' symlink. The logs that are collected for tests are configurable, and service implementations include sane defaults (so, e.g., you will always get the normal log file for Kafka, but only get the data files if the test asks for them). 3. No code coverage support. Haven't looked into it, so I couldn't comment on how hard it would be to add. -Ewen On Thu, May 21, 2015 at 2:38 PM, Ashish Singh asi...@cloudera.com wrote: Geoffrey, This looks great! A few questions. 1. Will ducktape be maintained separately as a github repo? 2. How easy is viewing the test results and logs. The link in KIP, http://testing.confluent.io/confluent_platform/latest/, lists a bunch of files and dirs. Could you add to KIP how the result and logs for the tests will be organized. 3. Does it support code coverage? If not, how easy/ difficult would it be? On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson ge...@confluent.io wrote: Great, I'll work on putting together a more detailed map of this replacement process. On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira gshap...@cloudera.com wrote: Love this idea :) I took a look at Ducktape API and it looks like a good fit - clean API, extensible, easy to use and powerful enough for our use-case. Something I'd like to see as part of the KIP is a map of what system-test currently tests, which ones we want to replace and a JIRA for replacing (possibly one for each group of tests). Basically, I know we all want to use the new system for new test cases (upgrades, etc), but I really want to make sure we don't get stuck with both systems forever
Re: [DISCUSS] KIP-25 System test improvements
Hi KIP-25 thread, I consolidated some of the questions from this thread and elsewhere. Q: Can we see a map of what system-test currently tests, which ones we want to replace and JIRAs for replacing? A: Initial draft here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests Q: Will ducktape be maintained separately as a github repo? A: Yes https://github.com/confluentinc/ducktape Q: How easy is viewing the test results and logs, how will test output be structured? A: Hierarchical structure as outlined here: https://github.com/confluentinc/ducktape/wiki/Design-overview#output Q: Does it support code coverage? If not, how easy/ difficult would it be to support? A: It does not, and we have no immediate plans to support this. Difficulty unclear. Q: It would be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. A: This seems reasonable and not too much extra work. Definitely open to discussion on this. Q: Why not share running services across multiple tests? A: Prefer to optimize for simplicity and correctness over what might be a questionable improvement in run-time. Q: Are regressions - in the road map? A: yes Q: Are Jepsen style tests involving network failures in the road map? A: yes Thanks much, Geoff On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Gwen, I don't see any problem with this as long as we're convinced there's a good use case, which seems to be true. Cheers, Geoff On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira gshap...@cloudera.com wrote: Not completely random places :) People may use Cloudera / HWX distributions which include Kafka, but want to verify that these bits match a specific upstream release. I think having the tests separately will be useful for this. In this case, finding the tests are not a big issue - we'll add a download link :) On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the correct place to download the right test artifact? Thanks, Jiangjie (Becket) Qin On 6/4/15, 4:29 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Reviving the discussion a bit :) I think it will be nice if each Kafka version that we release will also have a separate tests artifact that users can download, untar and easily run against a Kafka cluster of the same version. The idea is that if someone downloads packages that claim to contain something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), users can easily download the tests and verify that it indeed passes the tests for this version and therefore behaves the way this version is expected to behave. Does it make sense? Gwen On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test service logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Ashish, 1. That was the plan. We put some effort into cleanly separating the framework so it would be reusable across many projects. 2. I think you're seeing a test in progress where the final report hasn't been created yet. If you visit one of the older ones you'll see it has a landing page with links: http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently we need to adjust when we update the 'latest' symlink. The logs that are collected for tests are configurable, and service implementations include sane defaults (so, e.g., you will always get the normal log file for Kafka, but only get the data files if the test asks for them). 3. No code coverage support. Haven't looked into it, so I couldn't comment on how hard it would be to add. -Ewen
Re: [DISCUSS] KIP-25 System test improvements
Great, I'll work on putting together a more detailed map of this replacement process. On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira gshap...@cloudera.com wrote: Love this idea :) I took a look at Ducktape API and it looks like a good fit - clean API, extensible, easy to use and powerful enough for our use-case. Something I'd like to see as part of the KIP is a map of what system-test currently tests, which ones we want to replace and a JIRA for replacing (possibly one for each group of tests). Basically, I know we all want to use the new system for new test cases (upgrades, etc), but I really want to make sure we don't get stuck with both systems forever. Gwen On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi, Just kicking off the discussion thread on KIP-25 https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements Thanks, Geoff
Re: [DISCUSS] KIP-25 System test improvements
Hi Ashish, Looks like Ewen already hit the main points, but a few additions: 1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test service logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Ashish, 1. That was the plan. We put some effort into cleanly separating the framework so it would be reusable across many projects. 2. I think you're seeing a test in progress where the final report hasn't been created yet. If you visit one of the older ones you'll see it has a landing page with links: http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently we need to adjust when we update the 'latest' symlink. The logs that are collected for tests are configurable, and service implementations include sane defaults (so, e.g., you will always get the normal log file for Kafka, but only get the data files if the test asks for them). 3. No code coverage support. Haven't looked into it, so I couldn't comment on how hard it would be to add. -Ewen On Thu, May 21, 2015 at 2:38 PM, Ashish Singh asi...@cloudera.com wrote: Geoffrey, This looks great! A few questions. 1. Will ducktape be maintained separately as a github repo? 2. How easy is viewing the test results and logs. The link in KIP, http://testing.confluent.io/confluent_platform/latest/, lists a bunch of files and dirs. Could you add to KIP how the result and logs for the tests will be organized. 3. Does it support code coverage? If not, how easy/ difficult would it be? On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson ge...@confluent.io wrote: Great, I'll work on putting together a more detailed map of this replacement process. On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira gshap...@cloudera.com wrote: Love this idea :) I took a look at Ducktape API and it looks like a good fit - clean API, extensible, easy to use and powerful enough for our use-case. Something I'd like to see as part of the KIP is a map of what system-test currently tests, which ones we want to replace and a JIRA for replacing (possibly one for each group of tests). Basically, I know we all want to use the new system for new test cases (upgrades, etc), but I really want to make sure we don't get stuck with both systems forever. Gwen On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi, Just kicking off the discussion thread on KIP-25 https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements Thanks, Geoff -- Regards, Ashish -- Thanks, Ewen
[DISCUSS] KIP-25 System test improvements
Hi, Just kicking off the discussion thread on KIP-25 https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements Thanks, Geoff
[jira] [Commented] (KAFKA-1347) Create a system test for network partitions
[ https://issues.apache.org/jira/browse/KAFKA-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541125#comment-14541125 ] Geoffrey Anderson commented on KAFKA-1347: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Create a system test for network partitions --- Key: KAFKA-1347 URL: https://issues.apache.org/jira/browse/KAFKA-1347 Project: Kafka Issue Type: Test Reporter: Jay Kreps We got some free and rather public QA here: http://aphyr.com/posts/293-call-me-maybe-kafka We have since added a configuration to disable unclean leader election which allows you to prefer consistency over availability when all brokers fail. This has some unit tests, but ultimately there is no reason to believe this works unless we have a fairly aggressive system test case for it. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+System+Tests It would be good to add support for network partitions. I don't think we actually need to try to use the jepsen stuff directly, we can just us the underlying tools it uses--iptables and tc. These are linux specific, but that is prolly okay. You can see these at work here: https://github.com/aphyr/jepsen/blob/master/src/jepsen/control/net.clj Having this would help provide better evidence that this works now, and would keep it working in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1604) System Test for Transaction Management
[ https://issues.apache.org/jira/browse/KAFKA-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541123#comment-14541123 ] Geoffrey Anderson commented on KAFKA-1604: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake System Test for Transaction Management -- Key: KAFKA-1604 URL: https://issues.apache.org/jira/browse/KAFKA-1604 Project: Kafka Issue Type: Test Reporter: Dong Lin Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1604_2014-08-19_17:31:16.patch, KAFKA-1604_2014-08-19_21:07:35.patch Perform end-to-end transaction management test in the following steps: 1) Start Zookeeper. 2) Start multiple brokers. 3) Create topic. 4) Start transaction-aware ProducerPerformance to generate transactional messages to topic. 5) Start transaction-aware ConsoleConsumer to read messages from topic. 6) Bounce brokers (optional). 7) Verify that same number of messages are sent and received. This patch depends on KAFKA-1524, KAFKA-1526 and KAFKA-1601. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2003) Add upgrade tests
[ https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541121#comment-14541121 ] Geoffrey Anderson commented on KAFKA-2003: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Add upgrade tests - Key: KAFKA-2003 URL: https://issues.apache.org/jira/browse/KAFKA-2003 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Gwen Shapira Assignee: Ashish K Singh To test protocol changes, compatibility and upgrade process, we need a good way to test different versions of the product together and to test end-to-end upgrade process. For example, for 0.8.2 to 0.8.3 test we want to check: * Can we start a cluster with a mix of 0.8.2 and 0.8.3 brokers? * Can a cluster of 0.8.3 brokers bump the protocol level one broker at a time? * Can 0.8.2 clients run against a cluster of 0.8.3 brokers? There are probably more questions. But an automated framework that can test those and report results will be a good start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1918) System test for ZooKeeper quorum failure scenarios
[
https://issues.apache.org/jira/browse/KAFKA-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541127#comment-14541127
]
Geoffrey Anderson commented on KAFKA-1918:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
System test for ZooKeeper quorum failure scenarios
--
Key: KAFKA-1918
URL: https://issues.apache.org/jira/browse/KAFKA-1918
Project: Kafka
Issue Type: Test
Reporter: Omid Aladini
Following up on the [conversation on the mailing
list|http://mail-archives.apache.org/mod_mbox/kafka-users/201502.mbox/%3CCAHwHRrX3SAWDUGF5LjU4rrMUsqv%3DtJcyjX7OENeL5C_V5o3tCw%40mail.gmail.com%3E],
the FAQ writes:
{quote}
Once the Zookeeper quorum is down, brokers could result in a bad state and
could not normally serve client requests, etc. Although when Zookeeper quorum
recovers, the Kafka brokers should be able to resume to normal state
automatically, _there are still a few +corner cases+ the they cannot and a
hard kill-and-recovery is required to bring it back to normal_. Hence it is
recommended to closely monitor your zookeeper cluster and provision it so
that it is performant.
{quote}
As ZK quorum failures are inevitable (due to rolling upgrades of ZK, leader
hardware failure, etc), it would be great to identify the corner cases (if
they still exist) and fix them if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[ https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541124#comment-14541124 ] Geoffrey Anderson commented on KAFKA-976: - Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Order-Preserving Mirror Maker Testcase -- Key: KAFKA-976 URL: https://issues.apache.org/jira/browse/KAFKA-976 Project: Kafka Issue Type: Test Reporter: Guozhang Wang Assignee: John Fung Attachments: kafka-976-v1.patch A new testcase (5007) for mirror_maker_testsuite is needed for the key-dependent order-preserving mirror maker, this is related to KAFKA-957. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1904) run sanity failed test
[ https://issues.apache.org/jira/browse/KAFKA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541120#comment-14541120 ] Geoffrey Anderson commented on KAFKA-1904: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake run sanity failed test -- Key: KAFKA-1904 URL: https://issues.apache.org/jira/browse/KAFKA-1904 Project: Kafka Issue Type: Bug Components: system tests Reporter: Joe Stein Priority: Blocker Fix For: 0.8.3 Attachments: run_sanity.log.gz _test_case_name : testcase_1 _test_class_name : ReplicaBasicTest arg : bounce_broker : true arg : broker_type : leader arg : message_producing_free_time_sec : 15 arg : num_iteration : 2 arg : num_messages_to_produce_per_producer_call : 50 arg : num_partition : 2 arg : replica_factor : 3 arg : sleep_seconds_between_producer_calls : 1 validation_status : Test completed : FAILED -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1898) compatibility testing framework
[
https://issues.apache.org/jira/browse/KAFKA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541122#comment-14541122
]
Geoffrey Anderson commented on KAFKA-1898:
--
Just making sure you're aware of work we're doing at Confluent on system tests.
I'll be posting a KIP for this soon, but here's some info:
The original plan is sketched here:
https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements
This is the core library/test framework (WIP) which aids in writing and running
the tests
https://github.com/confluentinc/ducktape/
This has system tests we've written to date for the Confluent Platform
https://github.com/confluentinc/muckrake
compatibility testing framework
Key: KAFKA-1898
URL: https://issues.apache.org/jira/browse/KAFKA-1898
Project: Kafka
Issue Type: Bug
Reporter: Joe Stein
Fix For: 0.8.3
Attachments: cctk.png
There are a few different scenarios where you want/need to know the
status/state of a client library that works with Kafka. Client library
development is not just about supporting the wire protocol but also the
implementations around specific interactions of the API. The API has
blossomed into a robust set of producer, consumer, broker and administrative
calls all of which have layers of logic above them. A Client Library may
choose to deviate from the path the project sets out and that is ok. The goal
of this ticket is to have a system for Kafka that can help to explain what
the library is or isn't doing (regardless of what it claims).
The idea behind this stems in being able to quickly/easily/succinctly analyze
the topic message data. Once you can analyze the topic(s) message you can
gather lots of information about what the client library is doing, is not
doing and such. There are a few components to this.
1) dataset-generator
Test Kafka dataset generation tool. Generates a random text file with given
params:
--filename, -f - output file name.
--filesize, -s - desired size of output file. The actual size will always be
a bit larger (with a maximum size of $filesize + $max.length - 1)
--min.length, -l - minimum generated entry length.
--max.length, -h - maximum generated entry length.
Usage:
./gradlew build
java -jar dataset-generator/build/libs/dataset-generator-*.jar -s 10 -l 2
-h 20
2) dataset-producer
Test Kafka dataset producer tool. Able to produce the given dataset to Kafka
or Syslog server. The idea here is you already have lots of data sets that
you want to test different things for. You might have different sized
messages, formats, etc and want a repeatable benchmark to run and re-run the
testing on. You could just have a days worth of data and just choose to
replay it. The CCTK idea is that you are always starting from CONSUME in
your state of library. If your library is only producing then you will fail a
bunch of tests and that might be ok for people.
Accepts following params:
{code}
--filename, -f - input file name.
--kafka, -k - Kafka broker address in host:port format. If this parameter is
set, --producer.config and --topic must be set too (otherwise they're
ignored).
--producer.config, -p - Kafka producer properties file location.
--topic, -t - Kafka topic to produce to.
--syslog, -s - Syslog server address. Format: protocol://host:port
(tcp://0.0.0.0:5140 or udp://0.0.0.0:5141 for example)
--loop, -l - flag to loop through file until shut off manually. False by
default.
Usage:
./gradlew build
java -jar dataset-producer/build/libs/dataset-producer-*.jar --filename
dataset --syslog tcp://0.0.0.0:5140 --loop true
{code}
3) extract
This step is good so you can save data and compare tests. It could also be
removed if folks are just looking for a real live test (and we could support
that too). Here we are taking data out of Kafka and putting it into
Cassandra (but other data stores can be used too and we should come up with a
way to abstract this out completely so folks could implement whatever they
wanted.
{code}
package ly.stealth.shaihulud.reader
import java.util.UUID
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql.CassandraConnector
import consumer.kafka.MessageAndMetadata
import consumer.kafka.client.KafkaReceiver
import org.apache.spark._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream.DStream
object Main extends App with Logging {
val parser = new scopt.OptionParser[ReaderConfiguration](spark-reader) {
head(Spark Reader for Kafka client applications, 1.0)
opt[String](testId) unbounded() optional() action { (x, c) =
c.copy(testId = x)
} text (Source topic with initial set of data
[jira] [Commented] (KAFKA-1888) Add a rolling upgrade system test
[ https://issues.apache.org/jira/browse/KAFKA-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541118#comment-14541118 ] Geoffrey Anderson commented on KAFKA-1888: -- Just making sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake Add a rolling upgrade system test --- Key: KAFKA-1888 URL: https://issues.apache.org/jira/browse/KAFKA-1888 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Gwen Shapira Assignee: Abhishek Nigam Fix For: 0.9.0 Attachments: KAFKA-1888_2015-03-23_11:54:25.patch To help test upgrades and compatibility between versions, it will be cool to add a rolling-upgrade test to system tests: Given two versions (just a path to the jars?), check that you can do a rolling upgrade of the brokers from one version to another (using clients from the old version) without losing data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2171) System Test for Quotas
[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541116#comment-14541116 ] Geoffrey Anderson commented on KAFKA-2171: -- Just pinging this thread to make sure you're aware of work we're doing at Confluent on system tests. I'll be posting a KIP for this soon, but here's some info: The original plan is sketched here: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements This is the core library/test framework (WIP) which aids in writing and running the tests https://github.com/confluentinc/ducktape/ This has system tests we've written to date for the Confluent Platform https://github.com/confluentinc/muckrake System Test for Quotas -- Key: KAFKA-2171 URL: https://issues.apache.org/jira/browse/KAFKA-2171 Project: Kafka Issue Type: Sub-task Reporter: Dong Lin Assignee: Dong Lin Original Estimate: 336h Remaining Estimate: 336h Motivation and goal: We want to make sure that following features are working properly for both consumer and producer: default quota, client-specific quota, and quota sharing per clientId. The tests and configuration described aims to cover most of the scenarios. More test cases with varying configurations (e.g. ackNum) can be added if there is good reason to do so. Initial setup and configuration: In all scenarios, we first create kafka brokers and topic as follows: - create two kafka broker processes (by default local) - create a topic with replication factor = 2 and ackNum = -1 - let max_read = max_write = 5MB. The test machine is expected to provide read (write) throughput at least max_read (max_write). - we consider two rates are approximately the same if they differ by at most 5%. Scenario 1: Validate that max_read and max_write are provided by the test machine(s) using 1 producer and 1 consumer 1) produce data to the topic without rate limit for 30 seconds 2) record the rate of producer 3) then, consume data from the topic without rate limit until he finishes 4) record the rate of consumer 5) verify that the data consumed is identical to the data produced 6) verify that producer rate = max_write and consumer rate = max_read Scenario 2: validate the effectiveness of default write and read quota using 1 producer and 1 consumer 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) produce data to the topic for 30 seconds 3) record the rate of producer 4) then, consume data from the topic until he finishes 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that recorded write (read) rate is within 5% of max_write/2 (max_read/2). Scenario 3: validate the effectiveness of client-specific write and read quota using 2 producers and 2 consumers 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota; configure brokers to use max_write/4 for producer_2 and max_read/4 for consumer_2 2) both producers produce data to the topic for 30 seconds. They use different clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use different clientId and groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that producer_1 and producer_2 rates are approximately max_write/2 and max_write/4; verify that consumer_1 and consumer_2 rates are approximately max_read/2 and max_read/4. Scenario 4: validate the effectiveness of write and read quota sharing among clients of same clientId using 2 producers and 2 consumers. 1) configure brokers to use max_write/2 as default write quota and max_read/2 as default read quota 2) both producers produce data to the topic for 30 seconds. They use same clientId. 3) record the rate of producer 4) both consumers consume data from the topic until they finish. They use same clientId but different groupId. 5) record the rate of consumer 6) verify that the data consumed is identical to the data produced 7) verify that total rate of producer_1 and producer_2 is approximately max_write/2; verify that total rate of consumer_1 and consumer_2 is approximately max_read/2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
kafka system tests
Hi dev list, I've been discussing the current state of system tests with Jun and others, and have summarized goals moving forward at: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements Feedback is welcome! Thanks, Geoff
adding replicas to existing topic partitions - KAFKA-1313
Hi dev list, I have a few questions regarding KAFKA-1313 ( https://issues.apache.org/jira/browse/KAFKA-1313) It is currently possible albeit in a kindof hacky way to increase replication factor on a topic by using the partition reassignment tool with a hand-crafted reassignment plan. The preferred approach to making this better is to add support for replication-factor in the alter-topic command. For example, bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --alter --replication-factor 2 Another approach is use the partition reassignment tool to auto-generate a reassignment plan which results in increased replication. The advantage of the alter topics approach is that it seems logical to group this functionality with the other alter topic stuff. However, a couple questions: - Should partitions automatically be reassigned to brokers and shuffled when the replication factor is changed? - If so, it would be best to be able to update the replication factor of multiple topics at once so that the automatic reassignment would take place only once for the whole batch of changes. One advantage of the partition reassignment approach is that it's easier to generate a reassignment plan which updates multiple topics at once, and you have a chance to review the reassignment plan before putting it into effect with the --execute flag. Thanks, Geoff
[jira] [Commented] (KAFKA-1313) Support adding replicas to existing topic partitions
[
https://issues.apache.org/jira/browse/KAFKA-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347795#comment-14347795
]
Geoffrey Anderson commented on KAFKA-1313:
--
Quoting the Kafka docs:
The first step is to hand craft the custom reassignment plan in a json file-
cat increase-replication-factor.json
{version:1,
partitions:[{topic:foo,partition:0,replicas:[5,6,7]}]}
Then, use the json file with the --execute option to start the reassignment
process-
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
--reassignment-json-file increase-replication-factor.json --execute
---
As I understand it, the pain point is hand crafting the reassignment plan.
From there on out, updating replication factor is functionally identical to any
other reassignment. So my proposal is to add another file type
(--replication-factor-json-file) to kafka-reassign-partitions.sh --generate
(and/or --rebalance per KIP 6) which allows users to specify desired
replication_factor per topic/partition.
For example:
cat update-replication-factor.json
{version:1,
partitions:[{topic:foo,partition:0,replication_factor:3}]}
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
--replication-factor-json-file update-replication-factor.json --generate
The user would then feed generated output into kafka-reassign-partitions.sh
--execute as before.
[~nehanarkhede], let me know if this seems reasonable/is in the ballpark of
what you had in mind.
Support adding replicas to existing topic partitions
Key: KAFKA-1313
URL: https://issues.apache.org/jira/browse/KAFKA-1313
Project: Kafka
Issue Type: New Feature
Components: tools
Affects Versions: 0.8.0
Reporter: Marc Labbe
Assignee: Geoffrey Anderson
Priority: Critical
Labels: newbie++
Fix For: 0.9.0
There is currently no easy way to add replicas to an existing topic
partitions.
For example, topic create-test has been created with ReplicationFactor=1:
Topic:create-test PartitionCount:3ReplicationFactor:1 Configs:
Topic: create-test Partition: 0Leader: 1 Replicas: 1 Isr: 1
Topic: create-test Partition: 1Leader: 2 Replicas: 2 Isr: 2
Topic: create-test Partition: 2Leader: 3 Replicas: 3 Isr: 3
I would like to increase the ReplicationFactor=2 (or more) so it shows up
like this instead.
Topic:create-test PartitionCount:3ReplicationFactor:2 Configs:
Topic: create-test Partition: 0Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: create-test Partition: 1Leader: 2 Replicas: 2,3 Isr: 2,3
Topic: create-test Partition: 2Leader: 3 Replicas: 3,1 Isr: 3,1
Use cases for this:
- adding brokers and thus increase fault tolerance
- fixing human errors for topics created with wrong values
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346083#comment-14346083 ] Geoffrey Anderson commented on KAFKA-902: - Thanks for the feedback. Is there any reason to keep 0 - 0 behavior? In other words, is there any harm to making 0 less magical and perturbing it just like any other value? Also, I like the idea of scaling the upper bound on jitter with backoff_ms, though it seems like we still want some amount of jitter even if backoff_ms is small (in your example, if backoff_ms 5, then jitter is always 0) In that case, we might want to make sure that the jitter can be non-zero with something like max(3, min(20, 0.2 * backoff_ms)) But then we end up with a couple more semi-arbitrary parameters - a scaling parameter and a hard minimum. Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31715: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31715/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Added config for backoff jitter; added randomization to a few more client parameters; updated randomization function. Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 06fcfe62cc1fe76f58540221698ef076fe150e96 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 5fb21001abd77cac839bd724afa04e377a3e82aa clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 122375c473bf73caf05299b9f5174c6b226ca863 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31715/diff/ Testing --- Thanks, Geoffrey Anderson
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346149#comment-14346149 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31715/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch, KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343607#comment-14343607 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31633/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Status: Patch Available (was: Open) Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31633: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31633/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31633/diff/ Testing --- Thanks, Geoffrey Anderson
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1700) examples directory - README and shell scripts are out of date
Geoffrey Anderson created KAFKA-1700: Summary: examples directory - README and shell scripts are out of date Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Fix For: 0.8.2 sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 26575: Fix for KAFKA-1700
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26575/ --- Review request for kafka. Bugs: KAFKA-1700 https://issues.apache.org/jira/browse/KAFKA-1700 Repository: kafka Description --- Tweaked README, updated shell scripts so they call kafka-run-class.sh instead of manually adding to the CLASSPATH. Diffs - examples/README 61de2868de29e7c04811bfe12ccabc50b45d148e examples/bin/java-producer-consumer-demo.sh 29e01c2dcf82365c28fef0836e1771282cb49bc1 examples/bin/java-simple-consumer-demo.sh 4716a098c7d404477e0e7254e65e0f509e9df92e Diff: https://reviews.apache.org/r/26575/diff/ Testing --- Thanks, Geoffrey Anderson
[jira] [Updated] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-1700: - Attachment: KAFKA-1700.patch examples directory - README and shell scripts are out of date - Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1700.patch sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1700) examples directory - README and shell scripts are out of date
[ https://issues.apache.org/jira/browse/KAFKA-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167501#comment-14167501 ] Geoffrey Anderson commented on KAFKA-1700: -- Created reviewboard https://reviews.apache.org/r/26575/diff/ against branch origin/trunk examples directory - README and shell scripts are out of date - Key: KAFKA-1700 URL: https://issues.apache.org/jira/browse/KAFKA-1700 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Geoffrey Anderson Priority: Minor Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1700.patch sbt build files were removed during resolution of KAFKA-1254, so the README under the examples directory should no longer make reference to sbt. Also, the paths added to CLASSPATH variable in the example shell script are no longer correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
