Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 6, 2013, 9:41 p.m., Guozhang Wang wrote: Topic config change as well as create-topic, add-partition, partition-reassignment and preferred leader election are all asynchronous in the sense that the admin command would return immediately and one has to check himself if the process has finished; I think it is better to add an option to make these commands blocking until the process is done. Guozhang, that is a good suggestion and applies to all tools. Would you mind filing a JIRA for that change? - Neha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28319 --- On Nov. 6, 2013, 6:13 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 6, 2013, 6:13 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 6, 2013, 10:42 p.m., Swapnil Ghike wrote: It will help to add a --deleteConfig option. We can use it as --config config1=newVal --deleteConfig config2. Neha Narkhede wrote: Like we discussed offline, deleteConfig complicates the ability to accept configs easily in bulk and apply those. Actually, I think I misunderstood your suggestion earlier. This makes sense - Neha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28327 --- On Nov. 7, 2013, 4:50 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 4:50 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 5:50 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description (updated) --- Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs (updated) - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1119: - Attachment: KAFKA-1119_2013-11-07_09:50:20.patch Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816177#comment-13816177 ] Neha Narkhede commented on KAFKA-1119: -- Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 6, 2013, 9:41 p.m., Guozhang Wang wrote: Topic config change as well as create-topic, add-partition, partition-reassignment and preferred leader election are all asynchronous in the sense that the admin command would return immediately and one has to check himself if the process has finished; I think it is better to add an option to make these commands blocking until the process is done. Neha Narkhede wrote: Guozhang, that is a good suggestion and applies to all tools. Would you mind filing a JIRA for that change? Done. - Guozhang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28319 --- On Nov. 7, 2013, 5:50 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 5:50 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Created] (KAFKA-1125) Add options to let admin tools blocking until finish
Guozhang Wang created KAFKA-1125: Summary: Add options to let admin tools blocking until finish Key: KAFKA-1125 URL: https://issues.apache.org/jira/browse/KAFKA-1125 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Topic config change as well as create-topic, add-partition, partition-reassignment and preferred leader election are all asynchronous in the sense that the admin command would return immediately and one has to check himself if the process has finished. It is better to add an option to make these commands blocking until the process is done. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 6:17 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description (updated) --- Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs (updated) - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 7, 2013, 5:57 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/TopicCommand.scala, line 153 https://reviews.apache.org/r/15274/diff/2/?file=379572#file379572line153 This convention should be fine, but personally would prefer an unconfig or del-config since people can easily make mistakes and miss the actual config value they intend to alter. Introduced --deleteConfig in the latest patch. On Nov. 7, 2013, 5:57 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/TopicCommand.scala, line 151 https://reviews.apache.org/r/15274/diff/2/?file=379572#file379572line151 As part of this patch, can you change the regex to use the format: i.e., \s*=\s* I think that's clearer in that it avoids double escapes. Also, we should probably trim the split strings. Included the escapes change. Regarding the trim, included it but probably not required since the tool errors out for - nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic shutdown-test3 --alter --config segment.ms = 18 Exception in thread main java.lang.IllegalArgumentException: requirement failed: Invalid topic config: all configs to be added must be in the format key=val. at scala.Predef$.require(Predef.scala:145) at kafka.admin.TopicCommand$.parseTopicConfigs(TopicCommand.scala:156) at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:84) at kafka.admin.TopicCommand$.main(TopicCommand.scala:51) at kafka.admin.TopicCommand.main(TopicCommand.scala) Works for - nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic shutdown-test3 --alter --config retention.ms=18 Updated config for topic shutdown-test3. nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic shutdown-test3 --describe shutdown-test3 configs: retention.ms = 18 partitions: 2 topic: shutdown-test3 partition: 0leader: 0 replicas: 0 isr: 0 topic: shutdown-test3 partition: 1leader: 0 replicas: 0 isr: 0 On Nov. 7, 2013, 5:57 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/AdminUtils.scala, line 225 https://reviews.apache.org/r/15274/diff/2/?file=379571#file379571line225 Would it be better to do validation of all configs (to be removed and altered) earlier - i.e., in parseTopicConfigs? Added it to parseTopicConfigs, but left it in changeTopicConfigs since that is where the zookeeper write happens. - Neha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28385 --- On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 6:17 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1119: - Attachment: KAFKA-1119_2013-11-07_10:17:14.patch Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816200#comment-13816200 ] Neha Narkhede commented on KAFKA-1119: -- Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28393 --- core/src/main/scala/kafka/admin/AdminUtils.scala https://reviews.apache.org/r/15274/#comment55224 Not sure if this is required any more if we're doing validation earlier - since a validation failure would be fatal. core/src/main/scala/kafka/admin/TopicCommand.scala https://reviews.apache.org/r/15274/#comment55223 This shouldn't be called for configsToBeDeleted right? - i.e., validation could fail for deleted configs which have no associated value. - joel koshy On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 6:17 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28395 --- core/src/main/scala/kafka/admin/TopicCommand.scala https://reviews.apache.org/r/15274/#comment55226 Also, if we are adding the deleteConfig option we need not use the same regex and split. i.e., we expect only one argument. - joel koshy On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 6:17 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (72 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1117 tool for checking the consistency among replicas https://issues.apache.org/jira/browse/KAFKA-1117 KAFKA-1116 Need to upgrade sbt-assembly to compile on scala 2.10.2 https://issues.apache.org/jira/browse/KAFKA-1116 KAFKA-1112 broker can not start itself after kafka is killed with -9 https://issues.apache.org/jira/browse/KAFKA-1112 KAFKA-1110 Unable to produce messages with snappy/gzip compression https://issues.apache.org/jira/browse/KAFKA-1110 KAFKA-1109 Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS https://issues.apache.org/jira/browse/KAFKA-1109 KAFKA-1106 HighwaterMarkCheckpoint failure puting broker into a bad state https://issues.apache.org/jira/browse/KAFKA-1106 KAFKA-1093 Log.getOffsetsBefore(t, …) does not return the last confirmed offset before t https://issues.apache.org/jira/browse/KAFKA-1093 KAFKA-1086 Improve GetOffsetShell to find metadata automatically https://issues.apache.org/jira/browse/KAFKA-1086 KAFKA-1082 zkclient dies after UnknownHostException in zk reconnect https://issues.apache.org/jira/browse/KAFKA-1082 KAFKA-1079 Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually https://issues.apache.org/jira/browse/KAFKA-1079 KAFKA-1049 Encoder implementations are required to provide an undocumented constructor. https://issues.apache.org/jira/browse/KAFKA-1049 KAFKA-1032 Messages sent to the old leader will be lost on broker GC resulted failure https://issues.apache.org/jira/browse/KAFKA-1032 KAFKA-1020 Remove getAllReplicasOnBroker from KafkaController https://issues.apache.org/jira/browse/KAFKA-1020 KAFKA-1012 Implement an Offset Manager and hook offset requests to it https://issues.apache.org/jira/browse/KAFKA-1012 KAFKA-1011 Decompression and re-compression on MirrorMaker could result in messages being dropped in the pipeline https://issues.apache.org/jira/browse/KAFKA-1011 KAFKA-1005 kafka.perf.ConsumerPerformance not shutting down consumer https://issues.apache.org/jira/browse/KAFKA-1005 KAFKA-1004 Handle topic event for trivial whitelist topic filters https://issues.apache.org/jira/browse/KAFKA-1004 KAFKA-998 Producer should not retry on non-recoverable error codes https://issues.apache.org/jira/browse/KAFKA-998 KAFKA-997 Provide a strict verification mode when reading configuration properties https://issues.apache.org/jira/browse/KAFKA-997 KAFKA-996 Capitalize first letter for log entries https://issues.apache.org/jira/browse/KAFKA-996 KAFKA-984 Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same https://issues.apache.org/jira/browse/KAFKA-984 KAFKA-976 Order-Preserving Mirror Maker Testcase https://issues.apache.org/jira/browse/KAFKA-976 KAFKA-967 Use key range in ProducerPerformance https://issues.apache.org/jira/browse/KAFKA-967 KAFKA-917 Expose zk.session.timeout.ms in console consumer https://issues.apache.org/jira/browse/KAFKA-917 KAFKA-885 sbt package builds two kafka jars https://issues.apache.org/jira/browse/KAFKA-885 KAFKA-881 Kafka broker not respecting log.roll.hours https://issues.apache.org/jira/browse/KAFKA-881 KAFKA-873 Consider replacing zkclient with curator (with zkclient-bridge) https://issues.apache.org/jira/browse/KAFKA-873 KAFKA-868 System Test - add test case for rolling controlled shutdown https://issues.apache.org/jira/browse/KAFKA-868 KAFKA-863 System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases https://issues.apache.org/jira/browse/KAFKA-863 KAFKA-859 support basic auth protection of mx4j console https://issues.apache.org/jira/browse/KAFKA-859 KAFKA-855 Ant+Ivy build for Kafka https://issues.apache.org/jira/browse/KAFKA-855 KAFKA-854 Upgrade dependencies for 0.8 https://issues.apache.org/jira/browse/KAFKA-854 KAFKA-815 Improve SimpleConsumerShell to take in a max messages config option https://issues.apache.org/jira/browse/KAFKA-815 KAFKA-745 Remove getShutdownReceive() and other kafka specific code from the RequestChannel https://issues.apache.org/jira/browse/KAFKA-745 KAFKA-735 Add looping and JSON output for ConsumerOffsetChecker https://issues.apache.org/jira/browse/KAFKA-735 KAFKA-717 scala 2.10 build support https://issues.apache.org/jira/browse/KAFKA-717 KAFKA-686 0.8 Kafka broker should give a better error message when running against
Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 2
Joe, thanks for doing this. Couple of questions before I can vote because I'm not sure if any of these would block voting: From below: http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to This link is out of date - should be http://svn.apache.org/repos/asf/kafka/KEYS In release notes: http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html KAFKA-296, KAFKA-314, KAFKA-317 were closed as won't fix since we removed non-java clients. Did you use a jira query to get the list? If so, we can filter out anything that is not marked as resolved-fixed. In kafka-0.8.0-src.tgz: The DISCLAIMER file need not be there or needs to be updated (it talks about Kafka being an incubator project). Looks good otherwise! Thanks, Joel On Sun, Nov 03, 2013 at 08:55:35PM -0500, Joe Stein wrote: Hello, This is the second candidate for release of Apache Kafka 0.8.0. The only change to this release candidate from the last is that this builds against Java 6 as expected. Release Notes for the 0.8.0 release http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html *** Please download, test and vote by Thursday November, 7th, 6pm PDT Kafka's KEYS file containing PGP keys we use to sign the release: http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to the md5 and sha1 checksum * Release artifacts to be voted upon (source and binary): http://people.apache.org/~joestein/kafka-0.8.0-candidate2/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ (i.e. in sbt land this can be added to the build.sbt to use Kafka resolvers += Apache Staging at https://repository.apache.org/content/groups/staging/; libraryDependencies += org.apache.kafka % kafka_2.10 % 0.8.0 ) * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
[jira] [Commented] (KAFKA-1036) Unable to rename replication offset checkpoint in windows
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816304#comment-13816304 ] David Lao commented on KAFKA-1036: -- Hi Jay, Can you provide a patch for 0.8 as well? I'm running into similar issue on Windows. Unable to rename replication offset checkpoint in windows - Key: KAFKA-1036 URL: https://issues.apache.org/jira/browse/KAFKA-1036 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Environment: windows Reporter: Timothy Chen Priority: Critical Labels: windows Fix For: 0.8.1 Although there was a fix for checkpoint file renaming in windows that tries to delete the existing checkpoint file if renamed failed, I'm still seeing renaming errors on windows even though the destination file doesn't exist. A bit investigation shows that it wasn't able to rename the file since the kafka jvm still holds a fie lock on the tmp file and wasn't able to rename it. Attaching a patch that calls a explict writer.close so it can release the lock and can able to rename it. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28440 --- core/src/main/scala/kafka/admin/TopicCommand.scala https://reviews.apache.org/r/15274/#comment55256 Should we enforce that configsToBeAdded and configsToBeDeleted should not contain the same config? - Swapnil Ghike On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 7, 2013, 6:17 p.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Commented] (KAFKA-1122) Kafka can log giant log lines
[ https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816805#comment-13816805 ] Jason Rosenberg commented on KAFKA-1122: [~guozhang] I understand that point, but these are actually human readable log files, that we like to scroll through or tail, to make sure things look good during deploys, etc. Can you not have one greppable log line that gives enough reference info, that allows you then to pull all the corresponding log lines that go with it. E.g. in the example above, you can have one log line: Handling LeaderAndIsr request 123456 and LeaderAndIsr for request 123456: Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) - (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2) repeat 700 times Jason Kafka can log giant log lines - Key: KAFKA-1122 URL: https://issues.apache.org/jira/browse/KAFKA-1122 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Jason Rosenberg Priority: Minor There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file. This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example: 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) - (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),. Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer. I'd think these log lines could be separated into a single line per topic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation
Joe Stein created KAFKA-1126: Summary: Remove the DISCLAIMER it is left over from incubation Key: KAFKA-1126 URL: https://issues.apache.org/jira/browse/KAFKA-1126 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1122) Kafka can log giant log lines
[ https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1122: - Attachment: KAFKA-1122.patch Kafka can log giant log lines - Key: KAFKA-1122 URL: https://issues.apache.org/jira/browse/KAFKA-1122 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Jason Rosenberg Priority: Minor Attachments: KAFKA-1122.patch There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file. This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example: 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) - (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),. Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer. I'd think these log lines could be separated into a single line per topic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation
[ https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1126: - Attachment: KAFKA-1122.patch Remove the DISCLAIMER it is left over from incubation - Key: KAFKA-1126 URL: https://issues.apache.org/jira/browse/KAFKA-1126 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8 Attachments: KAFKA-1122.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation
[ https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1126: - Status: Patch Available (was: Open) Remove the DISCLAIMER it is left over from incubation - Key: KAFKA-1126 URL: https://issues.apache.org/jira/browse/KAFKA-1126 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8 Attachments: KAFKA-1122.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1122) Kafka can log giant log lines
[ https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1122: - Attachment: (was: KAFKA-1122.patch) Kafka can log giant log lines - Key: KAFKA-1122 URL: https://issues.apache.org/jira/browse/KAFKA-1122 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Jason Rosenberg Priority: Minor There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file. This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example: 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) - (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),. Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer. I'd think these log lines could be separated into a single line per topic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1122) Kafka can log giant log lines
[ https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816856#comment-13816856 ] Joe Stein commented on KAFKA-1122: -- You are both right, is there some type of patch we could introduce to toggle for both use cases? Perhaps in Logging do some parsing that would output multi line when some default configuration is overridden? Or better yet we have a tool or known command using awk in it to to use when tailing the log? Kafka can log giant log lines - Key: KAFKA-1122 URL: https://issues.apache.org/jira/browse/KAFKA-1122 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Jason Rosenberg Priority: Minor There are a number of log lines that the kafka server, and high-level consumer, can log, that can end up becoming a giant log line. This can be cumbersome to deal with in a log file. This happens in my case as I have have a large number of topics (on the order of 500-700 topics). Typically, these giant log lines will say something separately about every topic on the broker. An example: 2013-11-04 23:28:11,148 INFO [kafka-request-handler-0] server.ReplicaManager - [Replica Manager on Broker 10]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0) - (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),. Imagine that line going on with a separate entry for 700 topics. There are many other examples of this phenomenon in the server, and high-level consumer. I'd think these log lines could be separated into a single line per topic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression
[ https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816862#comment-13816862 ] Joe Stein commented on KAFKA-1110: -- Hey Evan, if you can get it to work that is a good next step, yup. And then once it is working can you suggest what would be the best changes to the WIKI in this ticket for others that are doing what are you doing so as to-do it correctly for how Kafka code is written and working now? Unable to produce messages with snappy/gzip compression --- Key: KAFKA-1110 URL: https://issues.apache.org/jira/browse/KAFKA-1110 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Environment: Kafka version: kafka-0.8.0-beta1 OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64 Reporter: Arup Malakar Attachments: kafka_producer_snappy_pkt_63.pcapng, sarama_producer_snappy_pkt_1.pcapng Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is following the specs as defined in: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol but messages are not getting into the kafka log file and consumers never see them when gzip/snappy is used. Without compression it works fine though. Few observations we made: 1. Kafka service does have required jars to be able to interpret snappy messages. When I modify ConsoleProducer to produce messages using SnappyCompressionCodec instead of default GZip codec. I was able to produce/consume messages. Looking at the kafka log files I see that Snappy Compression was indeed getting used: % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/aruptest-0/.log | tail -1 offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 compresscodec: SnappyCompressionCodec crc: 1602790249 So I don't think it would be a case of missing jars in kafka server. 2. Kafka doesn't return any error if the message doesn't make it to the log file. This seems pretty serious, as I would expect kafka to throw an error if I am using WaitForLocal/WaitForAll. 3. We did an inspection of the tcp packet to see the difference between what ConsoleProducer sends vs what sarama sends (Following is a copy/paste from a github issue): [~eapache] : So I have no idea what the ConsoleProducer is actually sending in this case. The outer protocol layers in both cases look identical, but if you compare the actual message value: a. Sarama sends two bytes of snappy header and then msg-payload (since Snappy decides it's too short to properly encode, so makes it a literal). Pretty straightforward. b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what appears to be a complete embedded produce request without any compression. This is neither valid snappy nor valid Kafka according to anything I've seen, so I'm pretty confused. It looks almost like an incorrect version of [1] but it's missing several key fields and the case of the identifying string is wrong. 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt Let us know if recent changes in the codebase makes the protocol page obsolete, in that case if the protocol page is updated we could update our client to use the new spec. More information could be found in the following github issue: https://github.com/Shopify/sarama/issues/32 -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description (updated) --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs (updated) - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816877#comment-13816877 ] Neha Narkhede commented on KAFKA-1119: -- Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch, KAFKA-1119_2013-11-07_17:07:18.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes
[ https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1119: - Attachment: KAFKA-1119_2013-11-07_17:07:18.patch Kafka 0.8.1 overwrites previous per topic config changes Key: KAFKA-1119 URL: https://issues.apache.org/jira/browse/KAFKA-1119 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.1 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch, KAFKA-1119_2013-11-07_17:07:18.patch kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 7, 2013, 6:31 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/TopicCommand.scala, line 153 https://reviews.apache.org/r/15274/diff/4/?file=380482#file380482line153 Also, if we are adding the deleteConfig option we need not use the same regex and split. i.e., we expect only one argument. That's true but the most common mistake while using this tool could be to use deleteConfig like config. So I included the extra check in order to give a relevant error message. - Neha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28395 --- On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 7, 2013, 6:29 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/AdminUtils.scala, line 224 https://reviews.apache.org/r/15274/diff/4/?file=380481#file380481line224 Not sure if this is required any more if we're doing validation earlier - since a validation failure would be fatal. The reason I kept it in the internal API (changeTopicConfigs) is if we use it in places other than tools, it is better to double check the values written to Zookeeper instead of ending up with corrupted topic configs On Nov. 7, 2013, 6:29 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/TopicCommand.scala, line 164 https://reviews.apache.org/r/15274/diff/4/?file=380482#file380482line164 This shouldn't be called for configsToBeDeleted right? - i.e., validation could fail for deleted configs which have no associated value. Refactored this in the latest patch and simplified the logic. Could you take another look please? - Neha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28393 --- On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Commented] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation
[ https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816883#comment-13816883 ] Neha Narkhede commented on KAFKA-1126: -- +1 Remove the DISCLAIMER it is left over from incubation - Key: KAFKA-1126 URL: https://issues.apache.org/jira/browse/KAFKA-1126 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8 Attachments: KAFKA-1122.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation
[ https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1126: - Resolution: Fixed Status: Resolved (was: Patch Available) Remove the DISCLAIMER it is left over from incubation - Key: KAFKA-1126 URL: https://issues.apache.org/jira/browse/KAFKA-1126 Project: Kafka Issue Type: Bug Reporter: Joe Stein Fix For: 0.8 Attachments: KAFKA-1122.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 7, 2013, 6:31 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/TopicCommand.scala, line 153 https://reviews.apache.org/r/15274/diff/4/?file=380482#file380482line153 Also, if we are adding the deleteConfig option we need not use the same regex and split. i.e., we expect only one argument. Neha Narkhede wrote: That's true but the most common mistake while using this tool could be to use deleteConfig like config. So I included the extra check in order to give a relevant error message. Ok - although I think the config validation would catch that if it is used like config (since there would be whitespace in the option value). - joel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28395 --- On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
On Nov. 7, 2013, 6:29 p.m., joel koshy wrote: core/src/main/scala/kafka/admin/AdminUtils.scala, line 224 https://reviews.apache.org/r/15274/diff/4/?file=380481#file380481line224 Not sure if this is required any more if we're doing validation earlier - since a validation failure would be fatal. Neha Narkhede wrote: The reason I kept it in the internal API (changeTopicConfigs) is if we use it in places other than tools, it is better to double check the values written to Zookeeper instead of ending up with corrupted topic configs Good point - makes sense. - joel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28393 --- On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
Re: Review Request 15274: Patch for KAFKA-1119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/#review28481 --- Ship it! - joel koshy On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15274/ --- (Updated Nov. 8, 2013, 1:07 a.m.) Review request for kafka. Bugs: KAFKA-1119 https://issues.apache.org/jira/browse/KAFKA-1119 Repository: kafka Description --- Improved the patch to simplify the usage of the internal changeTopicConfigs API. It now includes only the final list of configs to be applied to a topic. Any additions/deletions should be done prior to invoking this API Addressed Joel's review comments Add an explicit --deleteConfig option more cleanup Code cleanup Fixed bug that now allows removing all the topic overrides correctly and falling back to the defaults 1. Change the --config to diff the previous configs 2. Add the ability to remove per topic overrides if config is specified without a value Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala 8107a64cf1ef1cac763e152bae9f835411c9d3f3 core/src/main/scala/kafka/admin/TopicCommand.scala 56f3177e28a34df0ace1d192aef0060cb5e235df core/src/main/scala/kafka/log/LogConfig.scala 51ec796e9e6a10b76daefbd9aea02121fc1d573a core/src/main/scala/kafka/server/TopicConfigManager.scala 56cae58f2a216dcba88b2656fd4a490f11461270 Diff: https://reviews.apache.org/r/15274/diff/ Testing --- Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting config 4. Deleting all config overrides Thanks, Neha Narkhede
[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression
[ https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816949#comment-13816949 ] Evan Huus commented on KAFKA-1110: -- I don't know how to make it work. The provided ConsoleProducer appears, for all intents and purposes, to be sending garbage and yet it is accepted for some reason. I was hoping somebody who knows the protocol could clarify the structure, then I could match it in Go. I really don't want to have to reverse-engineer the protocol from scratch. Unable to produce messages with snappy/gzip compression --- Key: KAFKA-1110 URL: https://issues.apache.org/jira/browse/KAFKA-1110 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Environment: Kafka version: kafka-0.8.0-beta1 OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64 Reporter: Arup Malakar Attachments: kafka_producer_snappy_pkt_63.pcapng, sarama_producer_snappy_pkt_1.pcapng Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is following the specs as defined in: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol but messages are not getting into the kafka log file and consumers never see them when gzip/snappy is used. Without compression it works fine though. Few observations we made: 1. Kafka service does have required jars to be able to interpret snappy messages. When I modify ConsoleProducer to produce messages using SnappyCompressionCodec instead of default GZip codec. I was able to produce/consume messages. Looking at the kafka log files I see that Snappy Compression was indeed getting used: % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/aruptest-0/.log | tail -1 offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 compresscodec: SnappyCompressionCodec crc: 1602790249 So I don't think it would be a case of missing jars in kafka server. 2. Kafka doesn't return any error if the message doesn't make it to the log file. This seems pretty serious, as I would expect kafka to throw an error if I am using WaitForLocal/WaitForAll. 3. We did an inspection of the tcp packet to see the difference between what ConsoleProducer sends vs what sarama sends (Following is a copy/paste from a github issue): [~eapache] : So I have no idea what the ConsoleProducer is actually sending in this case. The outer protocol layers in both cases look identical, but if you compare the actual message value: a. Sarama sends two bytes of snappy header and then msg-payload (since Snappy decides it's too short to properly encode, so makes it a literal). Pretty straightforward. b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what appears to be a complete embedded produce request without any compression. This is neither valid snappy nor valid Kafka according to anything I've seen, so I'm pretty confused. It looks almost like an incorrect version of [1] but it's missing several key fields and the case of the identifying string is wrong. 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt Let us know if recent changes in the codebase makes the protocol page obsolete, in that case if the protocol page is updated we could update our client to use the new spec. More information could be found in the following github issue: https://github.com/Shopify/sarama/issues/32 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
Raymond Liu created KAFKA-1127: -- Summary: kafka and zookeeper server should start in daemon mode and log to correct position Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Reporter: Raymond Liu Priority: Minor Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
[ https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Liu updated KAFKA-1127: --- Attachment: kafka-1127.patch kafka and zookeeper server should start in daemon mode and log to correct position -- Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9 Reporter: Raymond Liu Priority: Minor Attachments: kafka-1127.patch Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
[ https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Liu updated KAFKA-1127: --- Affects Version/s: 0.9 Status: Patch Available (was: Open) kafka and zookeeper server should start in daemon mode and log to correct position -- Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9 Reporter: Raymond Liu Priority: Minor Attachments: kafka-1127.patch Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression
[ https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816991#comment-13816991 ] Joe Stein commented on KAFKA-1110: -- Hey Evan, if the issue is not occurring without compression then as I said the issue is not with the wire protocol. Compression is handled in the message structure as I posted above. Is this happening with gzip also? If not then maybe the issue is in fact with the Go Snappy library (though I still think this is likely not the issue but if gzip works then maybe it is). Maybe you can take a look at the C++ implementation from Adobe https://github.com/adobe-research/libkafka/blob/535860eafce32d9e7e454176770219abf473a344/lib/src/Message.cc#L129 which is just like the Scala code I posted above. Also, I don't know what 0x07 means is in your code you posted attributes := int8(m.Codec) 0x07 pe.putInt8(attributes) have you thought of just trying attributes := int8(m.Codec) pe.putInt8(attributes) again, I don't know Go so I am at a disadvantage to help even though I am trying to-do so to make it work with how it is Unable to produce messages with snappy/gzip compression --- Key: KAFKA-1110 URL: https://issues.apache.org/jira/browse/KAFKA-1110 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Environment: Kafka version: kafka-0.8.0-beta1 OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64 Reporter: Arup Malakar Attachments: kafka_producer_snappy_pkt_63.pcapng, sarama_producer_snappy_pkt_1.pcapng Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is following the specs as defined in: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol but messages are not getting into the kafka log file and consumers never see them when gzip/snappy is used. Without compression it works fine though. Few observations we made: 1. Kafka service does have required jars to be able to interpret snappy messages. When I modify ConsoleProducer to produce messages using SnappyCompressionCodec instead of default GZip codec. I was able to produce/consume messages. Looking at the kafka log files I see that Snappy Compression was indeed getting used: % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/aruptest-0/.log | tail -1 offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 compresscodec: SnappyCompressionCodec crc: 1602790249 So I don't think it would be a case of missing jars in kafka server. 2. Kafka doesn't return any error if the message doesn't make it to the log file. This seems pretty serious, as I would expect kafka to throw an error if I am using WaitForLocal/WaitForAll. 3. We did an inspection of the tcp packet to see the difference between what ConsoleProducer sends vs what sarama sends (Following is a copy/paste from a github issue): [~eapache] : So I have no idea what the ConsoleProducer is actually sending in this case. The outer protocol layers in both cases look identical, but if you compare the actual message value: a. Sarama sends two bytes of snappy header and then msg-payload (since Snappy decides it's too short to properly encode, so makes it a literal). Pretty straightforward. b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what appears to be a complete embedded produce request without any compression. This is neither valid snappy nor valid Kafka according to anything I've seen, so I'm pretty confused. It looks almost like an incorrect version of [1] but it's missing several key fields and the case of the identifying string is wrong. 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt Let us know if recent changes in the codebase makes the protocol page obsolete, in that case if the protocol page is updated we could update our client to use the new spec. More information could be found in the following github issue: https://github.com/Shopify/sarama/issues/32 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
[ https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13816995#comment-13816995 ] Guozhang Wang commented on KAFKA-1127: -- Thanks for the patch. Could this jira be merged with KAFKA-1109? kafka and zookeeper server should start in daemon mode and log to correct position -- Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9 Reporter: Raymond Liu Priority: Minor Attachments: kafka-1127.patch Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression
[ https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817016#comment-13817016 ] Arup Malakar commented on KAFKA-1110: - May be Evan would be able to provide more information. But gzip is not working either. Unable to produce messages with snappy/gzip compression --- Key: KAFKA-1110 URL: https://issues.apache.org/jira/browse/KAFKA-1110 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Environment: Kafka version: kafka-0.8.0-beta1 OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64 Reporter: Arup Malakar Attachments: kafka_producer_snappy_pkt_63.pcapng, sarama_producer_snappy_pkt_1.pcapng Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is following the specs as defined in: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol but messages are not getting into the kafka log file and consumers never see them when gzip/snappy is used. Without compression it works fine though. Few observations we made: 1. Kafka service does have required jars to be able to interpret snappy messages. When I modify ConsoleProducer to produce messages using SnappyCompressionCodec instead of default GZip codec. I was able to produce/consume messages. Looking at the kafka log files I see that Snappy Compression was indeed getting used: % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/aruptest-0/.log | tail -1 offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 compresscodec: SnappyCompressionCodec crc: 1602790249 So I don't think it would be a case of missing jars in kafka server. 2. Kafka doesn't return any error if the message doesn't make it to the log file. This seems pretty serious, as I would expect kafka to throw an error if I am using WaitForLocal/WaitForAll. 3. We did an inspection of the tcp packet to see the difference between what ConsoleProducer sends vs what sarama sends (Following is a copy/paste from a github issue): [~eapache] : So I have no idea what the ConsoleProducer is actually sending in this case. The outer protocol layers in both cases look identical, but if you compare the actual message value: a. Sarama sends two bytes of snappy header and then msg-payload (since Snappy decides it's too short to properly encode, so makes it a literal). Pretty straightforward. b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what appears to be a complete embedded produce request without any compression. This is neither valid snappy nor valid Kafka according to anything I've seen, so I'm pretty confused. It looks almost like an incorrect version of [1] but it's missing several key fields and the case of the identifying string is wrong. 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt Let us know if recent changes in the codebase makes the protocol page obsolete, in that case if the protocol page is updated we could update our client to use the new spec. More information could be found in the following github issue: https://github.com/Shopify/sarama/issues/32 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
[ https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817020#comment-13817020 ] Raymond Liu edited comment on KAFKA-1127 at 11/8/13 5:50 AM: - So v2 patch is combined patch with KAFKA-1109 was (Author: colorant): So this one is combined patch with KAFKA-1109 kafka and zookeeper server should start in daemon mode and log to correct position -- Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9 Reporter: Raymond Liu Priority: Minor Attachments: kafka-1127-v2.patch, kafka-1127.patch Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position
[ https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Liu updated KAFKA-1127: --- Attachment: kafka-1127-v2.patch So this one is combined patch with KAFKA-1109 kafka and zookeeper server should start in daemon mode and log to correct position -- Key: KAFKA-1127 URL: https://issues.apache.org/jira/browse/KAFKA-1127 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.9 Reporter: Raymond Liu Priority: Minor Attachments: kafka-1127-v2.patch, kafka-1127.patch Hi At present, the kafka-server-start.sh and zookeeper-server-start.sh will start kafka broker and zookeeper in fg mode, though daemon parameter is added. you need to add to make it bg. But the console log will still output in fg. Should make it fully run in bg and redirect console log to files. And log file position should not depends on where the cmd is running. Thus it will be more friendly for scripts to launch server on nodes by batch. -- This message was sent by Atlassian JIRA (v6.1#6144)
