[jira] [Updated] (KAFKA-1124) Sending to a new topic (with auto.create.topics.enable) returns ERROR
[ https://issues.apache.org/jira/browse/KAFKA-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1124: - Affects Version/s: 0.8.1 Sending to a new topic (with auto.create.topics.enable) returns ERROR - Key: KAFKA-1124 URL: https://issues.apache.org/jira/browse/KAFKA-1124 Project: Kafka Issue Type: Bug Affects Versions: 0.8, 0.8.1 Reporter: Jason Rosenberg I had thought this was reported issue, but can't seem to find a previous report for it. If auto.create.topics.enable is true, a producer still gets an ERROR logged on the first attempt to send a message to a new topic, e.g.: 2013-11-06 03:00:08,638 ERROR [Thread-1] async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: mynewtopic 2013-11-06 03:00:08,638 INFO [Thread-1] async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 3 This usually clears itself up immediately on retry (after 100 ms), as handled by the the kafka.producer.async.DefaultEventHandler (with retries enabled). However, this is logged to the client as an ERROR, and looks scary, when in fact it should have been a normal operation (since we have auto.create.topics.enable=true). There should be a better interaction here between the producer client and the server. Perhaps the server can create the topic in flight before returning the metadata request. Or, if it needs to be asynchronous, it could return a code which indicates something like: The topic doesn't exist yet, it is being created, try again shortly.and have the client automatically retry (even if retries not enabled, since it's not an ERROR condition, really). The ERROR log level is a problem since apps often have alert systems set up to notify when any ERROR happens, etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (KAFKA-1117) tool for checking the consistency among replicas
[ https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1117. Resolution: Fixed Thanks for the reviews. Committed to trunk. tool for checking the consistency among replicas Key: KAFKA-1117 URL: https://issues.apache.org/jira/browse/KAFKA-1117 Project: Kafka Issue Type: New Feature Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.1 Attachments: KAFKA-1117.patch, KAFKA-1117_2013-11-11_08:44:25.patch, KAFKA-1117_2013-11-12_08:34:53.patch, KAFKA-1117_2013-11-14_08:24:41.patch, KAFKA-1117_2013-11-18_09:58:23.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15665: Patch for KAFKA-1135
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15665/#review29176 --- Ship it! Ship It! - Jun Rao On Nov. 19, 2013, 3:21 a.m., Swapnil Ghike wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15665/ --- (Updated Nov. 19, 2013, 3:21 a.m.) Review request for kafka. Bugs: KAFKA-1135 https://issues.apache.org/jira/browse/KAFKA-1135 Repository: kafka Description --- iteration 2 json.encode Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8ff4bd5a5f6ea1a51df926c31155251bcc109238 core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 26beb9698422ceb6cc682b86913b4f9d2d4f1307 core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala 981d2bbecf2fa11f1d2c423535c7c30851d2d7bb core/src/main/scala/kafka/consumer/TopicCount.scala a3eb53e8262115d1184cd1c7a2b47f21c22c077b core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala c0350cd05cf1b59866a1fedccbeb700b3e828d44 core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/ZookeeperLeaderElector.scala 33b73609b1178c56e692fb60e35aca04ad1af586 core/src/main/scala/kafka/utils/Utils.scala c9ca95f1937d0ef2e64c70e4d811a0d4f358d9db core/src/main/scala/kafka/utils/ZkUtils.scala 856d13605b0b4bf86010571eacbacc0fb0ba7950 Diff: https://reviews.apache.org/r/15665/diff/ Testing --- Verified that zookeeper data looks like the structures defined in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper Thanks, Swapnil Ghike
[jira] [Resolved] (KAFKA-1135) Code cleanup - use Json.encode() to write json data to zk
[ https://issues.apache.org/jira/browse/KAFKA-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1135. Resolution: Fixed Thanks for the patch. +1 and committed to trunk. Code cleanup - use Json.encode() to write json data to zk - Key: KAFKA-1135 URL: https://issues.apache.org/jira/browse/KAFKA-1135 Project: Kafka Issue Type: Bug Reporter: Swapnil Ghike Assignee: Swapnil Ghike Fix For: 0.8.1 Attachments: KAFKA-1135.patch, KAFKA-1135_2013-11-18_19:17:54.patch, KAFKA-1135_2013-11-18_19:20:58.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15659: Patch for KAFKA-1103
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/#review29175 --- Looks good except for the following minor comment. core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala https://reviews.apache.org/r/15659/#comment56316 topicFilter could be a black list too. So we need to fix the logged message. - Jun Rao On Nov. 18, 2013, 11:30 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/ --- (Updated Nov. 18, 2013, 11:30 p.m.) Review request for kafka. Bugs: KAFKA-1103 https://issues.apache.org/jira/browse/KAFKA-1103 Repository: kafka Description --- KAFKA-1103.v1 Diffs - core/src/main/scala/kafka/consumer/TopicFilter.scala cf3853b223095e1fe0921175c407a906828b8113 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala c0350cd05cf1b59866a1fedccbeb700b3e828d44 core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala a67c193df9f7cbfc52f75dc1b71dc017de1b5fe2 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 40a2bf7a9277eb5f94bc07b40d7726d81860cefc Diff: https://reviews.apache.org/r/15659/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 15711: Patch for KAFKA-930
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29177 --- Thanks for the patch. A few comments. core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56317 Instead of delaying by 5 secs, should we just delay by config.leaderImbalanceCheckIntervalSeconds? core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56319 Could we rename updateZk to sth like isTriggeredByCommandLine? core/src/main/scala/kafka/server/KafkaConfig.scala https://reviews.apache.org/r/15711/#comment56318 I am wondering if this config is really necessary. Wouldn't it be simpler to always do the balancing on all partitions that are not already on the preferred replica? - Jun Rao On Nov. 20, 2013, 1:38 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 20, 2013, 1:38 a.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
Re: Review Request 15659: Patch for KAFKA-1103
On Nov. 20, 2013, 6:01 p.m., Jun Rao wrote: core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala, lines 769-770 https://reviews.apache.org/r/15659/diff/1/?file=388148#file388148line769 topicFilter could be a black list too. So we need to fix the logged message. Done. - Guozhang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/#review29175 --- On Nov. 18, 2013, 11:30 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/ --- (Updated Nov. 18, 2013, 11:30 p.m.) Review request for kafka. Bugs: KAFKA-1103 https://issues.apache.org/jira/browse/KAFKA-1103 Repository: kafka Description --- KAFKA-1103.v1 Diffs - core/src/main/scala/kafka/consumer/TopicFilter.scala cf3853b223095e1fe0921175c407a906828b8113 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala c0350cd05cf1b59866a1fedccbeb700b3e828d44 core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala a67c193df9f7cbfc52f75dc1b71dc017de1b5fe2 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 40a2bf7a9277eb5f94bc07b40d7726d81860cefc Diff: https://reviews.apache.org/r/15659/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 15711: Patch for KAFKA-930
On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/controller/KafkaController.scala, lines 260-261 https://reviews.apache.org/r/15711/diff/3/?file=388714#file388714line260 Instead of delaying by 5 secs, should we just delay by config.leaderImbalanceCheckIntervalSeconds? The problem is that leaderImbalanceCheckIntervalSeconds could be set to any value. If it was set to an hour or more than that and you have a controller failover because of an intermittent GC on the prev controller, and you want to immediately run the rebalnce, it will not happen. There are other cases where this is true. It is better to decouple the values. I can make it configurable but did not see a strong reason to do so. On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/controller/KafkaController.scala, lines 745-746 https://reviews.apache.org/r/15711/diff/3/?file=388714#file388714line745 Could we rename updateZk to sth like isTriggeredByCommandLine? I dont like the external usage to leak into the code. I see your intent to make the usage of this flag more explicit. How about isTriggeredByAutoRebalance and not update zk if it is set? On Nov. 20, 2013, 6:30 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaConfig.scala, lines 236-239 https://reviews.apache.org/r/15711/diff/3/?file=388715#file388715line236 I am wondering if this config is really necessary. Wouldn't it be simpler to always do the balancing on all partitions that are not already on the preferred replica? I do think there is value in this. To ensure rebalance happens always we can set it to 0. There are cases where few topic partition movements does not enforce a rebalance and hence cause unavailability. It is useful to have this to operationalize this feature and understand its behavior in production. - Sriram --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29177 --- On Nov. 20, 2013, 1:38 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 20, 2013, 1:38 a.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
Re: Review Request 15659: Incorporate Joel/Jun's comments, MM system test passed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/ --- (Updated Nov. 20, 2013, 8:59 p.m.) Review request for kafka. Summary (updated) - Incorporate Joel/Jun's comments, MM system test passed Bugs: KAFKA-1103 https://issues.apache.org/jira/browse/KAFKA-1103 Repository: kafka Description (updated) --- KAFKA-1103.v2 Dummy KAFKA-1103.v1 Diffs (updated) - core/src/main/scala/kafka/consumer/TopicFilter.scala cf3853b223095e1fe0921175c407a906828b8113 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala c0350cd05cf1b59866a1fedccbeb700b3e828d44 core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala a67c193df9f7cbfc52f75dc1b71dc017de1b5fe2 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 40a2bf7a9277eb5f94bc07b40d7726d81860cefc system_test/migration_tool_testsuite/0.7/config/test-log4j.properties a3ae33f20e4b7cff87d8cf8368d0639b8bea73a6 Diff: https://reviews.apache.org/r/15659/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828107#comment-13828107 ] Guozhang Wang commented on KAFKA-1103: -- Updated reviewboard https://reviews.apache.org/r/15659/ against branch origin/trunk Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1117) tool for checking the consistency among replicas
[ https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828146#comment-13828146 ] Swapnil Ghike commented on KAFKA-1117: -- Hey Jun, after committing this patch, builds with scala 2.10.* are breaking, could you please take a look: [error] /home/sghike/kafka-server/kafka-server_trunk/kafka/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala:364: ambiguous reference to overloaded definition, [error] both constructor FetchResponsePartitionData in class FetchResponsePartitionData of type (messages: kafka.message.MessageSet)kafka.api.FetchResponsePartitionData [error] and constructor FetchResponsePartitionData in class FetchResponsePartitionData of type (error: Short, hw: Long, messages: kafka.message.MessageSet)kafka.api.FetchResponsePartitionData [error] match argument types (messages: kafka.message.ByteBufferMessageSet) and expected result type kafka.api.FetchResponsePartitionData [error] replicaBuffer.addFetchedData(topicAndPartition, sourceBroker.id, new FetchResponsePartitionData(messages = MessageSet.Empty)) [error] tool for checking the consistency among replicas Key: KAFKA-1117 URL: https://issues.apache.org/jira/browse/KAFKA-1117 Project: Kafka Issue Type: New Feature Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.1 Attachments: KAFKA-1117.patch, KAFKA-1117_2013-11-11_08:44:25.patch, KAFKA-1117_2013-11-12_08:34:53.patch, KAFKA-1117_2013-11-14_08:24:41.patch, KAFKA-1117_2013-11-18_09:58:23.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Reopened] (KAFKA-1117) tool for checking the consistency among replicas
[ https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reopened KAFKA-1117: -- tool for checking the consistency among replicas Key: KAFKA-1117 URL: https://issues.apache.org/jira/browse/KAFKA-1117 Project: Kafka Issue Type: New Feature Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.1 Attachments: KAFKA-1117.patch, KAFKA-1117_2013-11-11_08:44:25.patch, KAFKA-1117_2013-11-12_08:34:53.patch, KAFKA-1117_2013-11-14_08:24:41.patch, KAFKA-1117_2013-11-18_09:58:23.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired
[ https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein resolved KAFKA-1131. -- Resolution: Fixed copy some more files into the release tar and zip that are needed/desired - Key: KAFKA-1131 URL: https://issues.apache.org/jira/browse/KAFKA-1131 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Fix For: 0.8, 0.8.1 Attachments: KAFKA-1131.patch e.g. LICENSE, NOTICE, README (or maybe a new quick start file .txt) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-404) When using chroot path, create chroot on startup if it doesn't exist
[ https://issues.apache.org/jira/browse/KAFKA-404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marek Dolgos updated KAFKA-404: --- Attachment: KAFKA-404-auto-create-zookeeper-chroot-on-start-up-i.patch Submitting a patch to auto-create the chroot on Kafka server startup if it does not exist. Please give feedback, I would be happy to make changes until the patch is acceptable. When using chroot path, create chroot on startup if it doesn't exist Key: KAFKA-404 URL: https://issues.apache.org/jira/browse/KAFKA-404 Project: Kafka Issue Type: Improvement Affects Versions: 0.7.1 Environment: CentOS 5.5, Linux 2.6.18-194.32.1.el5 x86_64 GNU/Linux Reporter: Jonathan Creasy Labels: newbie, patch Fix For: 0.8.1 Attachments: KAFKA-404-0.7.1.patch, KAFKA-404-0.8.patch, KAFKA-404-auto-create-zookeeper-chroot-on-start-up-i.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828274#comment-13828274 ] Guozhang Wang commented on KAFKA-1103: -- [~jjkoshy]], can you give another round of review for this one? Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (KAFKA-1103) Consumer uses two zkclients
[ https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828274#comment-13828274 ] Guozhang Wang edited comment on KAFKA-1103 at 11/20/13 11:21 PM: - [~jjkoshy], can you give another round of review for this one? was (Author: guozhang): [~jjkoshy]], can you give another round of review for this one? Consumer uses two zkclients --- Key: KAFKA-1103 URL: https://issues.apache.org/jira/browse/KAFKA-1103 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1103.patch, KAFKA-1103_2013-11-20_12:59:09.patch .. which is very confusing when debugging consumer logs. I don't remember any good reason for this, and we should get rid of the one instantiated in ZookeeperTopicEventWatcher if possible. -- This message was sent by Atlassian JIRA (v6.1#6144)
Patch for KAFKA-404, auto-create chroot on startup
I've submitted a patch for the auto-create of the chroot on server startup. I initially asked about submitting this from my gmail account. There seemed to be some interest, please find it at: https://issues.apache.org/jira/browse/KAFKA-404 Patched against latest in 'trunk' -Marek Dolgos
Review Request 15743: Patch for KAFKA-1133
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15743/ --- Review request for kafka. Bugs: KAFKA-1133 https://issues.apache.org/jira/browse/KAFKA-1133 Repository: kafka Description --- KAFKA-1133 LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven KAFKA-1131 copy license and notice to the release files, reviewed by David Arthur KAFKA-1126 Remove the DISCLAIMER it is left over from incubation reviewed by Neha KAFKA-1094 Configure reviewboard url in kafka-patch-review tool; reviewed by Neha Narkhede KAFKA-1086 Improve GetOffsetShell to find metadata automatically; reviwed by Jun Rao and Joel Koshy KAFKA-1089; Fix run-class and log4j for migration tool system test; reviewed by Jun Rao KAFKA-1087 Empty topic list causes consumer to fetch metadata of all topics; reviewed by Guozhang Wang and Neha Narkhede KAFKA-1081 Remove re-run functionality from start script and use exec to launch commands to avoid spurious .sh process. KAFKA-1018 tidy up the pom, 0.8.0 release, build the test jars now too KAFKA-1008 Unmap offset indexes before resizing. Diffs - DISCLAIMER 950e15d67781b07a837fd8a4c6a4b91c4c02d6f5 bin/kafka-add-partitions.sh c08b837cd0dcb601e2f089336ef07f283e4cf613 bin/kafka-console-consumer.sh 95edc9e5bff1207d92bd55e2f887f83b507bcb6f bin/kafka-console-producer.sh f102c2239a55c898f2a4e0b5154c4d8f5e3037de bin/kafka-consumer-perf-test.sh 39434d162e54063b12ee9fa027326f2ad74d31b1 bin/kafka-create-topic.sh 7a5f29735fceb9e4f87fb8e0b1f910e48bb705c1 bin/kafka-list-topic.sh bb2dde52c0d5775e025854c62e428d0b8cf4492e bin/kafka-preferred-replica-election.sh 6416dc1614002c17fdec94fc21fd88c9163bae0b bin/kafka-producer-perf-test.sh 1935522079cddf7e245ce043ae33e5ad8fda2ad5 bin/kafka-reassign-partitions.sh a6bd148752bb8ab1e46c60f3745bdf3d83b4bcc9 bin/kafka-replay-log-producer.sh 86943eed22e3a3ca9a825367ecda64d0c8d84dd1 bin/kafka-run-class.sh eb6ff1b12e53813622d0dd44588f74c9516d0919 bin/kafka-server-start.sh 4005289428d5d602d12562f834ea78dc79e2fc2b bin/kafka-simple-consumer-perf-test.sh 94c9ed0bfc28353695d0dcd1eec4530c89bf25da bin/kafka-simple-consumer-shell.sh 8d69357838f1bcb4476f70d599eda510cc8abbc7 bin/zookeeper-server-start.sh e4746a84b3b95f3a748762e932b83ceff8fd105f bin/zookeeper-shell.sh e0de33f2e16e5d0057cd710ac5fbd51553d36e4c core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala 8c033089691d30d4176f12fd037a680a4e2fb76d core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 36b167b28ac08286da05017aaf265cdecfbec591 core/src/main/scala/kafka/log/OffsetIndex.scala 9de3d31d7c8e964c835dd71e23bb84e6712a4f5e core/src/main/scala/kafka/server/AbstractFetcherThread.scala a5fc96df72a0dc4d763985eb23b318a932e92a23 core/src/main/scala/kafka/tools/GetOffsetShell.scala 2b9438ae3a4da2c7f97e05fb26bc179fe48f137d core/src/main/scala/kafka/utils/Os.scala PRE-CREATION core/src/main/scala/kafka/utils/Utils.scala e0a5a27c72abf3560f68fc6c2dbfc67d90cc5cd9 core/src/main/scala/kafka/utils/ZkUtils.scala 6eede1bde352a39a622899dc507cc8edfacbdafd core/src/test/scala/unit/kafka/utils/TestUtils.scala ee591d092b95c69917c5a65e2a42e873976edd5c core/src/test/scala/unit/kafka/utils/UtilsTest.scala 6b21554feee092771168d392cb34dd7ee14e708b kafka-patch-review.py f1d5192057755e9cd24861c1f995638e2eada1d1 project/Build.scala 33bb76f4ebb899610e66acd8af1d15c15af1f7cc system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh 8ebe7a7f3eb4a246cb3232a1631eb0e7fa66e976 system_test/migration_tool_testsuite/0.7/config/test-log4j.properties PRE-CREATION system_test/utils/kafka_system_test_utils.py aec31cbf38ca36bff2b6d9d5439b62905e901610 Diff: https://reviews.apache.org/r/15743/diff/ Testing --- Thanks, Joe Stein
[jira] [Commented] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828314#comment-13828314 ] Joe Stein commented on KAFKA-1133: -- Created reviewboard LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1133: - Attachment: KAFKA-1133.patch LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1133: - Attachment: (was: KAFKA-1133.patch) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8.1 This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1133: - Fix Version/s: 0.8 LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8, 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828317#comment-13828317 ] Joe Stein commented on KAFKA-1133: -- Created reviewboard https://reviews.apache.org/r/15744/ against branch trunk LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8, 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1133: - Attachment: KAFKA-1133.patch LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8, 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[ https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1133: - Status: Patch Available (was: Open) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven - Key: KAFKA-1133 URL: https://issues.apache.org/jira/browse/KAFKA-1133 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8.1 Attachments: KAFKA-1133.patch This needs to happen in our Build.scala the sbt package docs http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight forward line of code or ten or whatever to-do this maybe -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1136) Add subAppend in Log4jAppender for generic usage
[ https://issues.apache.org/jira/browse/KAFKA-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated KAFKA-1136: - Status: Patch Available (was: Open) Add subAppend in KafkaLog4jAppender for generic usage Add subAppend in Log4jAppender for generic usage Key: KAFKA-1136 URL: https://issues.apache.org/jira/browse/KAFKA-1136 Project: Kafka Issue Type: Improvement Components: producer Reporter: Jie Huang Assignee: Jun Rao Priority: Trivial KafkaLog4jAppender is quite useful for us to send our log4j logs to the Kafka system with ease. However, according to our experience, it is not so that convenient to customize the message content before emitting it out. Sometimes, we need to decorate the message like adding more system level information before passing it to the producer. I wonder if it is possible to add one subAppend() function, like org.apache.log4j.WriterAppender does. Thus, the end user can customize their message by overwriting the subAppend() only in their own hierarchy class, and re-use all the rest part. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Issue Comment Deleted] (KAFKA-1136) Add subAppend in Log4jAppender for generic usage
[ https://issues.apache.org/jira/browse/KAFKA-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated KAFKA-1136: - Comment: was deleted (was: Add subAppend in KafkaLog4jAppender for generic usage) Add subAppend in Log4jAppender for generic usage Key: KAFKA-1136 URL: https://issues.apache.org/jira/browse/KAFKA-1136 Project: Kafka Issue Type: Improvement Components: producer Reporter: Jie Huang Assignee: Jun Rao Priority: Trivial Attachments: KAFKA-1136.diff KafkaLog4jAppender is quite useful for us to send our log4j logs to the Kafka system with ease. However, according to our experience, it is not so that convenient to customize the message content before emitting it out. Sometimes, we need to decorate the message like adding more system level information before passing it to the producer. I wonder if it is possible to add one subAppend() function, like org.apache.log4j.WriterAppender does. Thus, the end user can customize their message by overwriting the subAppend() only in their own hierarchy class, and re-use all the rest part. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1136) Add subAppend in Log4jAppender for generic usage
[ https://issues.apache.org/jira/browse/KAFKA-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated KAFKA-1136: - Attachment: KAFKA-1136.diff Add subAppend in Log4jAppender for generic usage Key: KAFKA-1136 URL: https://issues.apache.org/jira/browse/KAFKA-1136 Project: Kafka Issue Type: Improvement Components: producer Reporter: Jie Huang Assignee: Jun Rao Priority: Trivial Attachments: KAFKA-1136.diff KafkaLog4jAppender is quite useful for us to send our log4j logs to the Kafka system with ease. However, according to our experience, it is not so that convenient to customize the message content before emitting it out. Sometimes, we need to decorate the message like adding more system level information before passing it to the producer. I wonder if it is possible to add one subAppend() function, like org.apache.log4j.WriterAppender does. Thus, the end user can customize their message by overwriting the subAppend() only in their own hierarchy class, and re-use all the rest part. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15711: Patch for KAFKA-930
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29203 --- core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56352 Another option is to _always_ try and delete the path - although if the znode doesn't exist it will give a spurious warning (deleted during connection loss). core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56351 Any reason to not call onPreferredReplicaElection on the entire set of partitions (instead of one at a time). Doing it all at once within the controller context lock would also prevent a concurrent preferred replica election tool from proceeding into onPreferredReplicaElection (although if this feature is turned on you wouldn't need to use the command-line tool anyway). - Joel Koshy On Nov. 20, 2013, 1:38 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 20, 2013, 1:38 a.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
Re: Review Request 15659: Incorporate Joel/Jun's comments, MM system test passed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/#review29205 --- Ship it! Looks good. - Joel Koshy On Nov. 20, 2013, 8:59 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15659/ --- (Updated Nov. 20, 2013, 8:59 p.m.) Review request for kafka. Bugs: KAFKA-1103 https://issues.apache.org/jira/browse/KAFKA-1103 Repository: kafka Description --- KAFKA-1103.v2 Dummy KAFKA-1103.v1 Diffs - core/src/main/scala/kafka/consumer/TopicFilter.scala cf3853b223095e1fe0921175c407a906828b8113 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala c0350cd05cf1b59866a1fedccbeb700b3e828d44 core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala a67c193df9f7cbfc52f75dc1b71dc017de1b5fe2 core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 40a2bf7a9277eb5f94bc07b40d7726d81860cefc system_test/migration_tool_testsuite/0.7/config/test-log4j.properties a3ae33f20e4b7cff87d8cf8368d0639b8bea73a6 Diff: https://reviews.apache.org/r/15659/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 15744: Patch for KAFKA-1133
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15744/#review29206 --- Ship it! - Joel Koshy On Nov. 21, 2013, midnight, Joe Stein wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15744/ --- (Updated Nov. 21, 2013, midnight) Review request for kafka. Bugs: KAFKA-1133 https://issues.apache.org/jira/browse/KAFKA-1133 Repository: kafka Description --- KAFKA-1133 LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven Diffs - project/Build.scala 500040ba9e69b6f76f4655ce3df9598404737008 Diff: https://reviews.apache.org/r/15744/diff/ Testing --- Thanks, Joe Stein
Re: Review Request 15711: Patch for KAFKA-930
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/#review29207 --- core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56354 Should we always delete the admin path? Because if auto rebalance achieved leader balance, then the manual rebalance has no work to do anyways. core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56356 rename to allReplicasForTopicPartitionsPerBroker? (I saw the per convention used somewhere else) core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56355 rename to topicPartitionsNotLedByPreferredReplica? core/src/main/scala/kafka/controller/KafkaController.scala https://reviews.apache.org/r/15711/#comment56353 we should be able to pass the entire set of partitions in one call, right? core/src/main/scala/kafka/server/KafkaConfig.scala https://reviews.apache.org/r/15711/#comment56357 Will it be simpler to have a per cluster config instead of a per broker config? i cant think of any downsides. - Swapnil Ghike On Nov. 20, 2013, 1:38 a.m., Sriram Subramanian wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15711/ --- (Updated Nov. 20, 2013, 1:38 a.m.) Review request for kafka. Bugs: KAFKA-930 https://issues.apache.org/jira/browse/KAFKA-930 Repository: kafka Description --- commit missing code some more changes fix merge conflicts Add auto leader rebalance support Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Conflicts: core/src/main/scala/kafka/admin/AdminUtils.scala core/src/main/scala/kafka/admin/TopicCommand.scala change comments commit the remaining changes Move AddPartitions into TopicCommand Diffs - core/src/main/scala/kafka/controller/KafkaController.scala 88792c2b2a360e928ab9cd00de151e5d5f94452d core/src/main/scala/kafka/server/KafkaConfig.scala b324344d0a383398db8bfe2cbeec2c1378fe13c9 Diff: https://reviews.apache.org/r/15711/diff/ Testing --- Thanks, Sriram Subramanian
[jira] [Commented] (KAFKA-1137) Problem of FetchRequest API
[ https://issues.apache.org/jira/browse/KAFKA-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828487#comment-13828487 ] Jay Kreps commented on KAFKA-1137: -- One thing to be aware of is that we process requests from a connection sequentially and allow only a single in-flight request per connection to ensure ordering. I can't tell from your description if this is the issue. Specifically say that t1 has no messages ready but t2 has one message ready. Let's say you issue Fetch(topic = t1, minBytes = 1, maxWait = 30) Fetch(topic = t2, minBytes = 1, maxWait = 30) The expected behavior is that the requests will be answered in the order issued so the first request will block for 30 seconds and then respond with no messages and then the second request will immediately respond with the ready message. Is that what you see, it kind of sounds like you are saying the opposite... Problem of FetchRequest API --- Key: KAFKA-1137 URL: https://issues.apache.org/jira/browse/KAFKA-1137 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8 Reporter: aio Assignee: Neha Narkhede We meet this problem when use Fetch API [0] in Node.js client [1]. The detail of the problem as follow: Send 2 FetchRequest in sequence through one long polling socket connection, each request hold one different topic (topic1 and topic2), topic2 is sent after topic1,MaxWaitTime set 30s, MinBytes set 1 byte. As the protocol doc said, when topic has new produced message, Kafka server would write back response to the socket, but the fact is Kafka server write response only when the last topic (topic2) has new message, otherwise we have to wait 30s to get the fetchResponse of topic1. I think Kafka server only approve the the MaxWaitTime in last FetchRequest of one socket connection, but why respond the new message in all topic when the last request have message. [0] : https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchRequest [1] : https://github.com/SOHU-Co/kafka-node -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1138) Remote producer uses the hostname defined in broker
Hyun-Gul Roh created KAFKA-1138: --- Summary: Remote producer uses the hostname defined in broker Key: KAFKA-1138 URL: https://issues.apache.org/jira/browse/KAFKA-1138 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8 Reporter: Hyun-Gul Roh Assignee: Jun Rao When the producer API in the node which is not the broker sends message to a broker, only TopicMetadataRequest is sent, but ProducerRequest is not by observing the log of kafka-request.log According to my analysis, when the producer api sends ProducerRequest, it seems to use the hostname defined in the broker. So, if the hostname is not the one registered in DNS, the producer cannot send the ProducerRequest. I am attaching the log: [2013-11-21 15:28:49,464] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(id:0,host:111.111.111.111,port:9092)] failed (kafka.producer.async.DefaultEventHandler) [2013-11-21 15:28:49,465] INFO Back off for 100 ms before retrying send. Remaining retries = 1 (kafka.producer.async.DefaultEventHandler) [2013-11-21 15:28:49,566] INFO Fetching metadata from broker id:0,host:111.111.111.111,port:9092 with correlation id 6 for 1 topic(s) Set(test) (kafka.client.ClientUtils$) [2013-11-21 15:28:49,819] ERROR Producer connection to 111.111.111.111:9092 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: 연결이 거부됨 at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:146) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) at kafka.producer.SyncProducer.send(SyncProducer.scala:112) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53) at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79) at kafka.utils.Utils$.swallow(Utils.scala:186) at kafka.utils.Logging$class.swallowError(Logging.scala:105) at kafka.utils.Utils$.swallowError(Utils.scala:45) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) at scala.collection.immutable.Stream.foreach(Stream.scala:254) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) [2013-11-21 15:28:49,821] WARN Fetching topic metadata with correlation id 6 for topics [Set(test)] from broker [id:0,host:111.111.111.111,port:9092] failed (kafka.client.ClientUtils$) java.net.ConnectException: 연결이 거부됨 at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:146) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68) at kafka.producer.SyncProducer.send(SyncProducer.scala:112) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53) at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79) at kafka.utils.Utils$.swallow(Utils.scala:186) at kafka.utils.Logging$class.swallowError(Logging.scala:105) at kafka.utils.Utils$.swallowError(Utils.scala:45) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) at
[jira] [Commented] (KAFKA-1137) Problem of FetchRequest API
[ https://issues.apache.org/jira/browse/KAFKA-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828546#comment-13828546 ] aio commented on KAFKA-1137: My apologies, I made mistake. In above situation what you descript is right. Problem of FetchRequest API --- Key: KAFKA-1137 URL: https://issues.apache.org/jira/browse/KAFKA-1137 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8 Reporter: aio Assignee: Neha Narkhede We meet this problem when use Fetch API [0] in Node.js client [1]. The detail of the problem as follow: Send 2 FetchRequest in sequence through one long polling socket connection, each request hold one different topic (topic1 and topic2), topic2 is sent after topic1,MaxWaitTime set 30s, MinBytes set 1 byte. As the protocol doc said, when topic has new produced message, Kafka server would write back response to the socket, but the fact is Kafka server write response only when the last topic (topic2) has new message, otherwise we have to wait 30s to get the fetchResponse of topic1. I think Kafka server only approve the the MaxWaitTime in last FetchRequest of one socket connection, but why respond the new message in all topic when the last request have message. [0] : https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchRequest [1] : https://github.com/SOHU-Co/kafka-node -- This message was sent by Atlassian JIRA (v6.1#6144)
