[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156023#comment-13156023 ] Neha Narkhede commented on KAFKA-176: - Thanks. Just committed this. I'm thinking there will be more improvements to these perf tools as part of KAFKA-175 Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: KAFKA-176-v2.patch, kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155674#comment-13155674 ] Jay Kreps commented on KAFKA-176: - +1 Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: KAFKA-176-v2.patch, kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13140895#comment-13140895 ] Jay Kreps commented on KAFKA-176: - tryCleanupZookeeper looks cut-and-pasted from place to place. It shows up in ConsoleConsumer.scala, ReplayLogProducer.scala, and ConsumerPerformance.scala. We should not do that. Can we make some kind of utility function for that? Also there is a PerfConfig class, which is a great idea as a way to normalize some of the config options we are using between all the tools. But it looks like the class is just duplicated between the tools. Can this be shared? I would like to cleanup the scripts used to run these things so that we get rid of all the silly ancient ones (*simple-perf-test.sh and *shell.sh), but I think I will open a seperate ticket for that since it is unrelated to your changes. Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139844#comment-13139844 ] Jun Rao commented on KAFKA-176: --- 1. It can be in the code. Maybe we just need a command line option to control whether to display header or not? 2. I am suggesting that we alway show aggregated output. So there won't be a -aggregate option. Instead, have a --showDetails option to enable/disable detailed stats in each thread, probably with default set to false. 3. Some system tests can use a producer that generates variable-sized random messages. Instead of having another producer tool, it would be good if we can just allow such option in this tool. Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139845#comment-13139845 ] Neha Narkhede commented on KAFKA-176: - 1. I think we can add a command line option --header that will control the display of the header 2. Good suggestion. I think what you are saying is that aggregate stats should be the default, and instead of having --aggregate option, we should have --showDetailedStats option. I think that is a good idea. 3. OK. I'll take a stab at that, and upload a new patch. 4. KAFKA-175 will need to be updated as well. It would be great if people can look at that too. Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-176) Fix existing perf tools
[ https://issues.apache.org/jira/browse/KAFKA-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136761#comment-13136761 ] Jun Rao commented on KAFKA-176: --- ProducerPerformance: 1. The following code seems to be used to enable/disable the header. Is it better to control that in config instead using debug logging (so that it's not mixed with other debug logging)? Also, the header info is not complete, missing the first few fields. The header is probably useful for stats printed out periodically in each thread. So it should be printed out early, if enabled. if(logger.isDebugEnabled) logger.debug(message size, batch size, total data sent in MB, MB/sec, total data sent in nMsg, nMsg/sec) 2. Whether avgPerf is specified or not, the user is probably always interested in the aggregated numbers across all threads. How about we always print it out and have a config option showDetails to enable/disable periodic reporting in each thread. Ditto in other perf tools. 3. ProducerThread has multiple bugs: 3.1. Variable-sized messages are not picked up in Async mode 3.2. In Sync mode, messageSet needs to be reset for each batch, if messages are of variable size (seems to be an existing bug) 4. It's better not to duplicate the following code. Defining it once in a static method seems better. println((%s, %d, %d, %d, %d, %.2f, %.4f, %d, %.4f).format(formattedReportTime, config.compressionCodec.codec, threadId, config.messageSize, config.batchSize, (bytesSent*1.0)/(1024 * 1024), mbPerSec, nSends, numMessagesPerSec)) Fix existing perf tools --- Key: KAFKA-176 URL: https://issues.apache.org/jira/browse/KAFKA-176 Project: Kafka Issue Type: Sub-task Reporter: Neha Narkhede Assignee: Neha Narkhede Fix For: 0.8 Attachments: kafka-176.patch The existing perf tools - ProducerPerformance.scala, ConsumerPerformance.scala and SimpleConsumerPerformance.scala are slightly buggy. It will be good to - 1. move them to a perf directory from the existing kafka/tools location 2. fix the bugs, so that they measure throughput correctly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
