[jira] [Commented] (KAFKA-930) Integrate preferred replica election logic into kafka
[ https://issues.apache.org/jira/browse/KAFKA-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829130#comment-13829130 ] Sriram Subramanian commented on KAFKA-930: -- Updated reviewboard https://reviews.apache.org/r/15711/ against branch origin/trunk Integrate preferred replica election logic into kafka - Key: KAFKA-930 URL: https://issues.apache.org/jira/browse/KAFKA-930 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.9 Attachments: KAFKA-930.patch, KAFKA-930_2013-11-19_17:37:29.patch, KAFKA-930_2013-11-19_17:38:49.patch, KAFKA-930_2013-11-21_09:42:11.patch It seems useful to integrate the preferred replica election logic into kafka controller. A simple way to implement this would be to have a background thread that periodically finds the topic partitions that are not assigned to the preferred broker and initiate the move. We could come up with some heuristics to initiate the move only if the imbalance over a specific threshold in order to avoid rebalancing too aggressively. Making the software do this reduces operational cost. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1139) Topic data change handling callback should not call syncedRebalance directly
Guozhang Wang created KAFKA-1139: Summary: Topic data change handling callback should not call syncedRebalance directly Key: KAFKA-1139 URL: https://issues.apache.org/jira/browse/KAFKA-1139 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Guozhang Wang Assignee: Guozhang Wang .. but should just set the flag as consumer change handling callback, so that a sequence of data changes on topics will not trigger unnecessarily many rebalances. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829244#comment-13829244 ] Guozhang Wang commented on KAFKA-1103: -- Updated reviewboard https://reviews.apache.org/r/15659/ against branch origin/trunk Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch, KAFKA-1103_2013-11-21_11:22:04.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829078#comment-13829078 ] Jun Rao commented on KAFKA-1103: The latest patch reverted some of the old commits. Could you rebase your branch and submit a new patch? Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-404) When using chroot path, create chroot on startup if it doesn't exist
[ https://issues.apache.org/jira/browse/KAFKA-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829263#comment-13829263 ] Guozhang Wang commented on KAFKA-404: - Thanks for the patch, a few comments: 1. val parts = chroot.split(/).drop(1) // Do you need to drop(1) since you are spliting from chroot, not zkConnect string? 2. Could you try to reuse ZkUtil.createPersistentPath ? 3. Could you rename xxTmp to xxForChrootCreation? 4. info(Created zookeeper path +path) // Might be better just having one info logging at the end of the loop? Guozhang When using chroot path, create chroot on startup if it doesn't exist Key: KAFKA-404 URL: https://issues.apache.org/jira/browse/KAFKA-404 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.1 Environment: CentOS 5.5, Linux 2.6.18-194.32.1.el5 x86_64 GNU/Linux Reporter: Jonathan Creasy Labels: newbie, patch Fix For: 0.8.1 Attachments: KAFKA-404-0.7.1.patch, KAFKA-404-0.8.patch, KAFKA-404-auto-create-zookeeper-chroot-on-start-up-i.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15711: Patch for KAFKA-930
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 21, 2013, 5:42 p.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description (updated) --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs (updated) - core/src/main/scala/kafka/controller/KafkaController.scala 4c319aba97655e7c4ec97fac2e34de4e28c9f5d3 core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
[jira] [Resolved] (KAFKA-1137) Problem of FetchRequest API
[ https://issues.apache.org/jira/browse/KAFKA-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-1137. --- Resolution: Invalid Closing - this behavior is documented here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Network Problem of FetchRequest API --- Key: KAFKA-1137 URL: https://issues.apache.org/jira/browse/KAFKA-1137 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8 Reporter: aio Assignee: Neha Narkhede We meet this problem when use Fetch API [0] in Node.js client [1]. The detail of the problem as follow: Send 2 FetchRequest in sequence through one long polling socket connection, each request hold one different topic (topic1 and topic2), topic2 is sent after topic1,MaxWaitTime set 30s, MinBytes set 1 byte. As the protocol doc said, when topic has new produced message, Kafka server would write back response to the socket, but the fact is Kafka server write response only when the last topic (topic2) has new message, otherwise we have to wait 30s to get the fetchResponse of topic1. I think Kafka server only approve the the MaxWaitTime in last FetchRequest of one socket connection, but why respond the new message in all topic when the last request have message. [0] : https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchRequest [1] : https://github.com/SOHU-Co/kafka-node -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15659: Incorporate Joel/Jun's comments, MM system test passed, rebased
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/ --- (Updated Nov. 21, 2013, 7:22 p.m.) Review request for kafka. Summary (updated) - Incorporate Joel/Jun's comments, MM system test passed, rebased Bugs: KAFKA-1103 https://issues.apache.org/jira/browse/KAFKA-1103 Repository: kafka Description --- KAFKA-1103.v2 Dummy KAFKA-1103.v1 Diffs (updated) - core/src/main/scala/kafka/consumer/TopicFilter.scala cf3853b223095e1fe0921175c407a906828b8113 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 6d0cfa665e90a168a70501a81f10fa4d3c7a7f22 core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala a67c193df9f7cbfc52f75dc1b71dc017de1b5fe2 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 40a2bf7a9277eb5f94bc07b40d7726d81860cefc system_test/migration_tool_testsuite/0.7/config/test-log4j.properties a3ae33f20e4b7cff87d8cf8368d0639b8bea73a6 Diff: https://reviews.apache.org/r/15659/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 15711: Patch for KAFKA-930
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29242 --- core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56380 One other way is that we can let the watcher handler call back function to not explicitly execute the election procedure but enqueue the request into this scheduler so that all replica election procedure will be done by this thread, and hence we can to batch election? Also this can help make the handling function very light so the chance of missing an event can be reduced. - Guozhang Wang On Nov. 21, 2013, 5:42 p.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 21, 2013, 5:42 p.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 4c319aba97655e7c4ec97fac2e34de4e28c9f5d3 core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
[jira] [Commented] (KAFKA-1138) Remote producer uses the hostname defined in broker
[ https://issues.apache.org/jira/browse/KAFKA-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829077#comment-13829077 ] Jun Rao commented on KAFKA-1138: Do you think https://issues.apache.org/jira/browse/KAFKA-1092 addresses this issue? Remote producer uses the hostname defined in broker --- Key: KAFKA-1138 URL: https://issues.apache.org/jira/browse/KAFKA-1138 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8 Reporter: Hyun-Gul Roh Assignee: Jun Rao When the producer API in the node which is not the broker sends message to a broker, only TopicMetadataRequest is sent, but ProducerRequest is not by observing the log of kafka-request.log According to my analysis, when the producer api sends ProducerRequest, it seems to use the hostname defined in the broker. So, if the hostname is not the one registered in DNS, the producer cannot send the ProducerRequest. I am attaching the log: [2013-11-21 15:28:49,464] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(id:0,host:111.111.111.111,port:9092)] failed (kafka.producer.async.DefaultEventHandler) [2013-11-21 15:28:49,465] INFO Back off for 100 ms before retrying send. Remaining retries = 1 (kafka.producer.async.DefaultEventHandler) [2013-11-21 15:28:49,566] INFO Fetching metadata from broker id:0,host:111.111.111.111,port:9092 with correlation id 6 for 1 topic(s) Set(test) (kafka.client.ClientUtils$) [2013-11-21 15:28:49,819] ERROR Producer connection to 111.111.111.111:9092 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: 연결이 거부됨 at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:146) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) at kafka.producer.SyncProducer.send(SyncProducer.scala:112) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53) at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79) at kafka.utils.Utils$.swallow(Utils.scala:186) at kafka.utils.Logging$class.swallowError(Logging.scala:105) at kafka.utils.Utils$.swallowError(Utils.scala:45) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) at scala.collection.immutable.Stream.foreach(Stream.scala:254) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) [2013-11-21 15:28:49,821] WARN Fetching topic metadata with correlation id 6 for topics [Set(test)] from broker [id:0,host:111.111.111.111,port:9092] failed (kafka.client.ClientUtils$) java.net.ConnectException: 연결이 거부됨 at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:146) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) at kafka.producer.SyncProducer.send(SyncProducer.scala:112) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53) at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79) at kafka.utils.Utils$.swallow(Utils.scala:186) at kafka.utils.Logging$class.swallowError(Logging.scala:105) at kafka.utils.Utils$.swallowError(Utils.scala:45) at
Re: Review Request 15711: Patch for KAFKA-930
On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/controller/KafkaController.scala, lines 745-746 https://reviews.apache.org/r/15711/diff/3/?file=388714#file388714line745 Could we rename updateZk to sth like isTriggeredByCommandLine? Sriram Subramanian wrote: I dont like the external usage to leak into the code. I see your intent to make the usage of this flag more explicit. How about isTriggeredByAutoRebalance and not update zk if it is set? Jun Rao wrote: This is fine. My only concern is that updateZK is a bit misleading. We do update the ISR path in ZK. We just don't update the leader balancing path. Changed it to isTriggeredByAutoRebalance On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaConfig.scala, lines 236-239 https://reviews.apache.org/r/15711/diff/3/?file=388715#file388715line236 I am wondering if this config is really necessary. Wouldn't it be simpler to always do the balancing on all partitions that are not already on the preferred replica? Sriram Subramanian wrote: I do think there is value in this. To ensure rebalance happens always we can set it to 0. There are cases where few topic partition movements does not enforce a rebalance and hence cause unavailability. It is useful to have this to operationalize this feature and understand its behavior in production. Jun Rao wrote: I still don't see a strong reason why someone would want to leave some partitions' leader unbalanced. Even one unbalanced leader can cause significant extra load on the broker if the amount of data on that partition is large. The logic in KafkaController is also simplified if we always try to balance all leaders. Our balancing of topics across topics is very primitive. We do not balance based on load or storage. What you suggest will happen even if the number of topics are balanced across the brokers. It is valuable to avoid unnecessary churn till we understand this feature better in production. I do see a benefit in not balancing when only a few topic partitions are imbalanced. Additionally the logic to do this is very minimum. - Sriram --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29177 --- On Nov. 21, 2013, 5:42 p.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 21, 2013, 5:42 p.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 4c319aba97655e7c4ec97fac2e34de4e28c9f5d3 core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
Re: Review Request 15711: Patch for KAFKA-930
On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaConfig.scala, lines 236-239 https://reviews.apache.org/r/15711/diff/3/?file=388715#file388715line236 I am wondering if this config is really necessary. Wouldn't it be simpler to always do the balancing on all partitions that are not already on the preferred replica? Sriram Subramanian wrote: I do think there is value in this. To ensure rebalance happens always we can set it to 0. There are cases where few topic partition movements does not enforce a rebalance and hence cause unavailability. It is useful to have this to operationalize this feature and understand its behavior in production. I still don't see a strong reason why someone would want to leave some partitions' leader unbalanced. Even one unbalanced leader can cause significant extra load on the broker if the amount of data on that partition is large. The logic in KafkaController is also simplified if we always try to balance all leaders. On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/controller/KafkaController.scala, lines 745-746 https://reviews.apache.org/r/15711/diff/3/?file=388714#file388714line745 Could we rename updateZk to sth like isTriggeredByCommandLine? Sriram Subramanian wrote: I dont like the external usage to leak into the code. I see your intent to make the usage of this flag more explicit. How about isTriggeredByAutoRebalance and not update zk if it is set? This is fine. My only concern is that updateZK is a bit misleading. We do update the ISR path in ZK. We just don't update the leader balancing path. - Jun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29177 --- On Nov. 20, 2013, 1:38 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 20, 2013, 1:38 a.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
Re: Review Request 15711: Patch for KAFKA-930
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15711/#review29240
---
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56377
Does this need to be at info level?
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56373
Could we rewrite it as the following to make it clear?
groupBy { case(topicAndParttion, assignedReplicas) : = .. }
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56374
Can we do the following so that we can use named variables?
foreach { case(leaderBroker, (topicAndPartition, assignedReplicas)) = ..
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56375
Same here. Could we give item._1 a name so that it's clear what the
referenced value is?
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56376
Should this be info level logging?
core/src/main/scala/kafka/controller/KafkaController.scala
https://reviews.apache.org/r/15711/#comment56378
Same here. It would be better to give a referenced name for
topicPartition._1.
- Jun Rao
On Nov. 21, 2013, 5:42 p.m., Sriram Subramanian wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15711/
---
(Updated Nov. 21, 2013, 5:42 p.m.)
Review request for kafka.
Bugs: KAFKA-930
https://issues.apache.org/jira/browse/KAFKA-930
Repository: kafka
Description
---
Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into
trunk
commit missing code
some more changes
fix merge conflicts
Add auto leader rebalance support
Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into
trunk
Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into
trunk
Conflicts:
core/src/main/scala/kafka/admin/AdminUtils.scala
core/src/main/scala/kafka/admin/TopicCommand.scala
change comments
commit the remaining changes
Move AddPartitions into TopicCommand
Diffs
-
core/src/main/scala/kafka/controller/KafkaController.scala
4c319aba97655e7c4ec97fac2e34de4e28c9f5d3
core/src/main/scala/kafka/server/KafkaConfig.scala
b324344d0a383398db8bfe2cbeec2c1378fe13c9
Diff: https://reviews.apache.org/r/15711/diff/
Testing
---
Thanks,
Sriram Subramanian
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (75 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1136 Add subAppend in Log4jAppender for generic usage https://issues.apache.org/jira/browse/KAFKA-1136 KAFKA-1133 LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven https://issues.apache.org/jira/browse/KAFKA-1133 KAFKA-1130 log.dirs is a confusing property name https://issues.apache.org/jira/browse/KAFKA-1130 KAFKA-1116 Need to upgrade sbt-assembly to compile on scala 2.10.2 https://issues.apache.org/jira/browse/KAFKA-1116 KAFKA-1110 Unable to produce messages with snappy/gzip compression https://issues.apache.org/jira/browse/KAFKA-1110 KAFKA-1109 Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS https://issues.apache.org/jira/browse/KAFKA-1109 KAFKA-1106 HighwaterMarkCheckpoint failure puting broker into a bad state https://issues.apache.org/jira/browse/KAFKA-1106 KAFKA-1103 Consumer uses two zkclients https://issues.apache.org/jira/browse/KAFKA-1103 KAFKA-1093 Log.getOffsetsBefore(t, …) does not return the last confirmed offset before t https://issues.apache.org/jira/browse/KAFKA-1093 KAFKA-1086 Improve GetOffsetShell to find metadata automatically https://issues.apache.org/jira/browse/KAFKA-1086 KAFKA-1082 zkclient dies after UnknownHostException in zk reconnect https://issues.apache.org/jira/browse/KAFKA-1082 KAFKA-1079 Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually https://issues.apache.org/jira/browse/KAFKA-1079 KAFKA-1074 Reassign partitions should delete the old replicas from disk https://issues.apache.org/jira/browse/KAFKA-1074 KAFKA-1049 Encoder implementations are required to provide an undocumented constructor. https://issues.apache.org/jira/browse/KAFKA-1049 KAFKA-1032 Messages sent to the old leader will be lost on broker GC resulted failure https://issues.apache.org/jira/browse/KAFKA-1032 KAFKA-1020 Remove getAllReplicasOnBroker from KafkaController https://issues.apache.org/jira/browse/KAFKA-1020 KAFKA-1012 Implement an Offset Manager and hook offset requests to it https://issues.apache.org/jira/browse/KAFKA-1012 KAFKA-1011 Decompression and re-compression on MirrorMaker could result in messages being dropped in the pipeline https://issues.apache.org/jira/browse/KAFKA-1011 KAFKA-1005 kafka.perf.ConsumerPerformance not shutting down consumer https://issues.apache.org/jira/browse/KAFKA-1005 KAFKA-1004 Handle topic event for trivial whitelist topic filters https://issues.apache.org/jira/browse/KAFKA-1004 KAFKA-998 Producer should not retry on non-recoverable error codes https://issues.apache.org/jira/browse/KAFKA-998 KAFKA-997 Provide a strict verification mode when reading configuration properties https://issues.apache.org/jira/browse/KAFKA-997 KAFKA-996 Capitalize first letter for log entries https://issues.apache.org/jira/browse/KAFKA-996 KAFKA-984 Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same https://issues.apache.org/jira/browse/KAFKA-984 KAFKA-976 Order-Preserving Mirror Maker Testcase https://issues.apache.org/jira/browse/KAFKA-976 KAFKA-967 Use key range in ProducerPerformance https://issues.apache.org/jira/browse/KAFKA-967 KAFKA-917 Expose zk.session.timeout.ms in console consumer https://issues.apache.org/jira/browse/KAFKA-917 KAFKA-885 sbt package builds two kafka jars https://issues.apache.org/jira/browse/KAFKA-885 KAFKA-881 Kafka broker not respecting log.roll.hours https://issues.apache.org/jira/browse/KAFKA-881 KAFKA-873 Consider replacing zkclient with curator (with zkclient-bridge) https://issues.apache.org/jira/browse/KAFKA-873 KAFKA-868 System Test - add test case for rolling controlled shutdown https://issues.apache.org/jira/browse/KAFKA-868 KAFKA-863 System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases https://issues.apache.org/jira/browse/KAFKA-863 KAFKA-859 support basic auth protection of mx4j console https://issues.apache.org/jira/browse/KAFKA-859 KAFKA-855 Ant+Ivy build for Kafka https://issues.apache.org/jira/browse/KAFKA-855 KAFKA-854 Upgrade dependencies for 0.8 https://issues.apache.org/jira/browse/KAFKA-854 KAFKA-815 Improve SimpleConsumerShell to take in a max messages config option https://issues.apache.org/jira/browse/KAFKA-815 KAFKA-745 Remove getShutdownReceive() and other kafka
[jira] [Updated] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1103: - Attachment: KAFKA-1103_2013-11-21_11:22:04.patch Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch, KAFKA-1103_2013-11-21_11:22:04.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1140) Move the decoding logic from ConsumerIterator.makeNext to next
Guozhang Wang created KAFKA-1140:
Summary: Move the decoding logic from ConsumerIterator.makeNext to
next
Key: KAFKA-1140
URL: https://issues.apache.org/jira/browse/KAFKA-1140
Project: Kafka
Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
Usually people will write code around consumer like
while(iter.hasNext()) {
try {
msg = iter.next()
// do something
}
catch{
}
}
However, the iter.hasNext() call itself can throw exceptions due to decoding
failures. It would be better to move the decoding to the next function call.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
Re: Review Request 15711: Patch for KAFKA-930
On Nov. 21, 2013, 3:41 a.m., Swapnil Ghike wrote: core/src/main/scala/kafka/controller/KafkaController.scala, line 926 https://reviews.apache.org/r/15711/diff/3/?file=388714#file388714line926 rename to topicPartitionsNotLedByPreferredReplica? PreferredReplica cannot lead multiple topic partitions. - Sriram --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29207 --- On Nov. 21, 2013, 5:42 p.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 21, 2013, 5:42 p.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 4c319aba97655e7c4ec97fac2e34de4e28c9f5d3 core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
[jira] [Updated] (KAFKA-1140) Move the decoding logic from ConsumerIterator.makeNext to next
[
https://issues.apache.org/jira/browse/KAFKA-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Koshy updated KAFKA-1140:
--
Fix Version/s: 0.8.1
Move the decoding logic from ConsumerIterator.makeNext to next
--
Key: KAFKA-1140
URL: https://issues.apache.org/jira/browse/KAFKA-1140
Project: Kafka
Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
Fix For: 0.8.1
Usually people will write code around consumer like
while(iter.hasNext()) {
try {
msg = iter.next()
// do something
}
catch{
}
}
However, the iter.hasNext() call itself can throw exceptions due to decoding
failures. It would be better to move the decoding to the next function call.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (KAFKA-1136) Add subAppend in Log4jAppender for generic usage
[ https://issues.apache.org/jira/browse/KAFKA-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1136: -- Fix Version/s: 0.8.1 Add subAppend in Log4jAppender for generic usage Key: KAFKA-1136 URL: https://issues.apache.org/jira/browse/KAFKA-1136 Project: Kafka Issue Type: Improvement Components: producer Reporter: Jie Huang Assignee: Jun Rao Priority: Trivial Fix For: 0.8.1 Attachments: KAFKA-1136.diff KafkaLog4jAppender is quite useful for us to send our log4j logs to the Kafka system with ease. However, according to our experience, it is not so that convenient to customize the message content before emitting it out. Sometimes, we need to decorate the message like adding more system level information before passing it to the producer. I wonder if it is possible to add one subAppend() function, like org.apache.log4j.WriterAppender does. Thus, the end user can customize their message by overwriting the subAppend() only in their own hierarchy class, and re-use all the rest part. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-404) When using chroot path, create chroot on startup if it doesn't exist
[ https://issues.apache.org/jira/browse/KAFKA-404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-404: Affects Version/s: (was: 0.7.1) 0.8.1 When using chroot path, create chroot on startup if it doesn't exist Key: KAFKA-404 URL: https://issues.apache.org/jira/browse/KAFKA-404 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.1 Environment: CentOS 5.5, Linux 2.6.18-194.32.1.el5 x86_64 GNU/Linux Reporter: Jonathan Creasy Labels: newbie, patch Fix For: 0.8.1 Attachments: KAFKA-404-0.7.1.patch, KAFKA-404-0.8.patch, KAFKA-404-auto-create-zookeeper-chroot-on-start-up-i.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1136) Add subAppend in Log4jAppender for generic usage
[ https://issues.apache.org/jira/browse/KAFKA-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829610#comment-13829610 ] Jie Huang commented on KAFKA-1136: -- Thanks. Add subAppend in Log4jAppender for generic usage Key: KAFKA-1136 URL: https://issues.apache.org/jira/browse/KAFKA-1136 Project: Kafka Issue Type: Improvement Components: producer Reporter: Jie Huang Assignee: Jun Rao Priority: Trivial Fix For: 0.8.1 Attachments: KAFKA-1136.diff KafkaLog4jAppender is quite useful for us to send our log4j logs to the Kafka system with ease. However, according to our experience, it is not so that convenient to customize the message content before emitting it out. Sometimes, we need to decorate the message like adding more system level information before passing it to the producer. I wonder if it is possible to add one subAppend() function, like org.apache.log4j.WriterAppender does. Thus, the end user can customize their message by overwriting the subAppend() only in their own hierarchy class, and re-use all the rest part. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-933) Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception
[ https://issues.apache.org/jira/browse/KAFKA-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829627#comment-13829627 ] drunkedcat commented on KAFKA-933: -- as 0.8.0-bete1, the following change will work: byte[] bytes = timestamp.toString().getBytes(UTF8); - Message message = new Message(bytes); - list.add(new KeyedMessageInteger, Message(_topic, null, message)); + + list.add(new KeyedMessageInteger, byte[](_topic, null, bytes)); } Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception Key: KAFKA-933 URL: https://issues.apache.org/jira/browse/KAFKA-933 Project: Kafka Issue Type: Bug Components: contrib Affects Versions: 0.8 Environment: [amilkowski@localhost ~]$ uname -a Linux localhost.localdomain 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [amilkowski@localhost ~]$ Reporter: Andrew Milkowski Priority: Minor Labels: hadoop Fix For: 0.8.1 Working of git master codebase and following instructions at https://github.com/apache/kafka/blob/trunk/contrib/hadoop-consumer/README https://github.com/apache/kafka when running ./run-class.sh kafka.etl.impl.DataGenerator test/test.properties an exception is thrown Exception in thread main java.lang.ClassCastException: kafka.message.Message cannot be cast to [B at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:129) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:124) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at scala.collection.JavaConversions$JListWrapper.map(JavaConversions.scala:615) at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:124) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:54) at kafka.producer.Producer.send(Producer.scala:74) at kafka.javaapi.producer.Producer.send(Producer.scala:41) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (KAFKA-933) Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception
[ https://issues.apache.org/jira/browse/KAFKA-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829627#comment-13829627 ] drunkedcat edited comment on KAFKA-933 at 11/22/13 3:44 AM: as 0.8.0-bete1, the following change will work: byte[] bytes = timestamp.toString().getBytes(UTF8); -- Message message = new Message(bytes); -- list.add(new KeyedMessageInteger, Message(_topic, null, message)); ++ ++ list.add(new KeyedMessageInteger, byte[](_topic, null, bytes)); } was (Author: drunkedcat): as 0.8.0-bete1, the following change will work: byte[] bytes = timestamp.toString().getBytes(UTF8); - Message message = new Message(bytes); - list.add(new KeyedMessageInteger, Message(_topic, null, message)); + + list.add(new KeyedMessageInteger, byte[](_topic, null, bytes)); } Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception Key: KAFKA-933 URL: https://issues.apache.org/jira/browse/KAFKA-933 Project: Kafka Issue Type: Bug Components: contrib Affects Versions: 0.8 Environment: [amilkowski@localhost ~]$ uname -a Linux localhost.localdomain 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [amilkowski@localhost ~]$ Reporter: Andrew Milkowski Priority: Minor Labels: hadoop Fix For: 0.8.1 Working of git master codebase and following instructions at https://github.com/apache/kafka/blob/trunk/contrib/hadoop-consumer/README https://github.com/apache/kafka when running ./run-class.sh kafka.etl.impl.DataGenerator test/test.properties an exception is thrown Exception in thread main java.lang.ClassCastException: kafka.message.Message cannot be cast to [B at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:129) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:124) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at scala.collection.JavaConversions$JListWrapper.map(JavaConversions.scala:615) at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:124) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:54) at kafka.producer.Producer.send(Producer.scala:74) at kafka.javaapi.producer.Producer.send(Producer.scala:41) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (KAFKA-933) Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception
[ https://issues.apache.org/jira/browse/KAFKA-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829627#comment-13829627 ] drunkedcat edited comment on KAFKA-933 at 11/22/13 3:54 AM: as 0.8.0-beta1, the following change will work: byte[] bytes = timestamp.toString().getBytes(UTF8); -- Message message = new Message(bytes); -- list.add(new KeyedMessageInteger, Message(_topic, null, message)); ++ ++ list.add(new KeyedMessageInteger, byte[](_topic, null, bytes)); } was (Author: drunkedcat): as 0.8.0-bete1, the following change will work: byte[] bytes = timestamp.toString().getBytes(UTF8); -- Message message = new Message(bytes); -- list.add(new KeyedMessageInteger, Message(_topic, null, message)); ++ ++ list.add(new KeyedMessageInteger, byte[](_topic, null, bytes)); } Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception Key: KAFKA-933 URL: https://issues.apache.org/jira/browse/KAFKA-933 Project: Kafka Issue Type: Bug Components: contrib Affects Versions: 0.8 Environment: [amilkowski@localhost ~]$ uname -a Linux localhost.localdomain 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [amilkowski@localhost ~]$ Reporter: Andrew Milkowski Priority: Minor Labels: hadoop Fix For: 0.8.1 Working of git master codebase and following instructions at https://github.com/apache/kafka/blob/trunk/contrib/hadoop-consumer/README https://github.com/apache/kafka when running ./run-class.sh kafka.etl.impl.DataGenerator test/test.properties an exception is thrown Exception in thread main java.lang.ClassCastException: kafka.message.Message cannot be cast to [B at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:129) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:124) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at scala.collection.JavaConversions$JListWrapper.map(JavaConversions.scala:615) at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:124) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:54) at kafka.producer.Producer.send(Producer.scala:74) at kafka.javaapi.producer.Producer.send(Producer.scala:41) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1141) make changes to downloads for the archive old releases to new old_releases folder
Joe Stein created KAFKA-1141: Summary: make changes to downloads for the archive old releases to new old_releases folder Key: KAFKA-1141 URL: https://issues.apache.org/jira/browse/KAFKA-1141 Project: Kafka Issue Type: Bug Reporter: Joe Stein Priority: Blocker Fix For: 0.8 I copied the files from dist incubator to where they belong now in dist, gotta modifiy the downloads page too need to-do this before RC4 http://archive.apache.org/dist/kafka/ waiting on the artifacts to mirror -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (KAFKA-1141) make changes to downloads for the archive old releases to new old_releases folder
[ https://issues.apache.org/jira/browse/KAFKA-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein reassigned KAFKA-1141: Assignee: Joe Stein make changes to downloads for the archive old releases to new old_releases folder - Key: KAFKA-1141 URL: https://issues.apache.org/jira/browse/KAFKA-1141 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Joe Stein Priority: Blocker Fix For: 0.8 I copied the files from dist incubator to where they belong now in dist, gotta modifiy the downloads page too need to-do this before RC4 http://archive.apache.org/dist/kafka/ waiting on the artifacts to mirror -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13829671#comment-13829671 ] Guozhang Wang commented on KAFKA-1103: -- Wired. I already did a rebase and git diff did not show that.. Let me re-try. Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch, KAFKA-1103_2013-11-21_11:22:04.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (KAFKA-933) Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception
[ https://issues.apache.org/jira/browse/KAFKA-933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-933. --- Resolution: Fixed Thanks for the patch. Committed to trunk. Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception Key: KAFKA-933 URL: https://issues.apache.org/jira/browse/KAFKA-933 Project: Kafka Issue Type: Bug Components: contrib Affects Versions: 0.8 Environment: [amilkowski@localhost ~]$ uname -a Linux localhost.localdomain 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [amilkowski@localhost ~]$ Reporter: Andrew Milkowski Priority: Minor Labels: hadoop Fix For: 0.8.1 Working of git master codebase and following instructions at https://github.com/apache/kafka/blob/trunk/contrib/hadoop-consumer/README https://github.com/apache/kafka when running ./run-class.sh kafka.etl.impl.DataGenerator test/test.properties an exception is thrown Exception in thread main java.lang.ClassCastException: kafka.message.Message cannot be cast to [B at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:34) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:129) at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:124) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at scala.collection.JavaConversions$JListWrapper.map(JavaConversions.scala:615) at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:124) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:54) at kafka.producer.Producer.send(Producer.scala:74) at kafka.javaapi.producer.Producer.send(Producer.scala:41) -- This message was sent by Atlassian JIRA (v6.1#6144)
Changed ordering guarantee with multiple in-flight messages
Hi, I noticed Joel Koshy's update to the protocol guide wiki at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol This sentence was added: The broker allows only a single in-flight request per connection in order to guarantee this ordering Adding such a constraint for the number of in-flight requests is a performance killer, and it seems odd to do so at this point in time given the number of third-party clients implemented - at least some of them hopefully relying on multiple requests in flight being properly supported. It is also in contrast with the previous sentence: The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well. So, can you elaborate on this new constraint? In which cases with multiple in-flight requests may reordering reoccur? Regards, Magnus
Re: Changed ordering guarantee with multiple in-flight messages
I can elaborate further on the wiki tomorrow. The term in-flight in my edit is a bit incomplete. It refers to what's in-flight on the broker-side for actual handling - that is what provides the ordering guarantee. The client can continue to write requests to the socket even while the broker is handling a preceding request since those requests will sit in the socket buffer. So there is still significant benefit in using non-blocking IO/request pipelining on the client-side to achieve high throughput. Thanks, Joel On Thu, Nov 21, 2013 at 11:33 PM, Magnus Edenhill mag...@edenhill.se wrote: Hi, I noticed Joel Koshy's update to the protocol guide wiki at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol This sentence was added: The broker allows only a single in-flight request per connection in order to guarantee this ordering Adding such a constraint for the number of in-flight requests is a performance killer, and it seems odd to do so at this point in time given the number of third-party clients implemented - at least some of them hopefully relying on multiple requests in flight being properly supported. It is also in contrast with the previous sentence: The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well. So, can you elaborate on this new constraint? In which cases with multiple in-flight requests may reordering reoccur? Regards, Magnus
