Re: Review Request 19957: Patch for KAFKA-1356
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19957/ --- (Updated April 8, 2014, 8:38 a.m.) Review request for kafka. Bugs: KAFKA-1356 https://issues.apache.org/jira/browse/KAFKA-1356 Repository: kafka Description --- KAFKA-1356 Improve topic metadata handling in kafka api Diffs (updated) - core/src/main/scala/kafka/api/TopicMetadata.scala 0513a59ed94e556894b3515dc38666ee9a66ae3d core/src/main/scala/kafka/controller/KafkaController.scala c8c02ced543a6278ba5b1edfa73a370f1edb1891 core/src/main/scala/kafka/server/KafkaApis.scala 705d87ed874f2363e6d9d6ea1143bc6035a0a9d6 core/src/test/scala/unit/kafka/admin/AdminTest.scala d5644ea40ec7678b975c4775546b79fcfa9f64b7 core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 22bb6f2847b8895f8fbba6c531963ebb0fffe2ca core/src/test/scala/unit/kafka/utils/TestUtils.scala 2054c25bbced6bd90c092a1974975732ad346146 Diff: https://reviews.apache.org/r/19957/diff/ Testing --- Thanks, Timothy Chen
[jira] [Commented] (KAFKA-1356) Topic metadata requests takes too long to process
[ https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962707#comment-13962707 ] Timothy Chen commented on KAFKA-1356: - Updated reviewboard https://reviews.apache.org/r/19957/ against branch origin/trunk Topic metadata requests takes too long to process - Key: KAFKA-1356 URL: https://issues.apache.org/jira/browse/KAFKA-1356 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Timothy Chen Attachments: KAFKA-1356.patch, KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, KAFKA-1356_2014-04-08_01:38:23.patch Currently we're seeing slow response times in handling get topic metadata requests. Local testing shows that even locally it takes 300 avg ms to respond, even though it's not doing any IO operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1356) Topic metadata requests takes too long to process
[ https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1356: Attachment: KAFKA-1356_2014-04-08_01:38:23.patch Topic metadata requests takes too long to process - Key: KAFKA-1356 URL: https://issues.apache.org/jira/browse/KAFKA-1356 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Timothy Chen Attachments: KAFKA-1356.patch, KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, KAFKA-1356_2014-04-08_01:38:23.patch Currently we're seeing slow response times in handling get topic metadata requests. Local testing shows that even locally it takes 300 avg ms to respond, even though it's not doing any IO operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[GitHub] kafka pull request: Fix to run hadoop-consumer with hadoop 2
Github user mvalleavila closed the pull request at: https://github.com/apache/kafka/pull/18 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (KAFKA-1368) Upgrade log4j
Vladislav Pernin created KAFKA-1368:
---
Summary: Upgrade log4j
Key: KAFKA-1368
URL: https://issues.apache.org/jira/browse/KAFKA-1368
Project: Kafka
Issue Type: Improvement
Affects Versions: 0.8.0
Reporter: Vladislav Pernin
Upgrade log4j to at least 1.2.16 ou 1.2.17.
Usage of EnhancedPatternLayout will be possible.
It allows to set delimiters around the full log, stacktrace included, making
log messages collection easier with tools like Logstash.
Example : [%d{}]...[%t] %m%throwable%n
[2014-04-08 11:07:20,360] ERROR [KafkaApi-1] Error when processing fetch
request for partition [X,6] offset 700 from consumer with correlation id 0
(kafka.server.KafkaApis)
kafka.common.OffsetOutOfRangeException: Request for offset 700 but we only have
log segments in the range 16021 to 16021.
at kafka.log.Log.read(Log.scala:429)
...
at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1210) Windows Bat files are not working properly
[ https://issues.apache.org/jira/browse/KAFKA-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962749#comment-13962749 ] Stevo Slavic commented on KAFKA-1210: - kafka-run-class.bat changes from the patch unfortunatelly do not apply to current 0.8.1 branch head. Befor trying to apply patch file, on Windows, I had to first convert patch file encoding to ANSI and EOL to Unix format, like explained in [this SO question|http://stackoverflow.com/q/13675782/381140]. Windows Bat files are not working properly -- Key: KAFKA-1210 URL: https://issues.apache.org/jira/browse/KAFKA-1210 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Environment: Windows Reporter: Håkan Canberger Priority: Trivial Fix For: 0.8.2 Attachments: 1210-v1.patch The bat files are not working properly. The paths in them are invalid. They have not been updated to reflect the changes made to the shell scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1195) windows kafka-run-class.bat not updated
[ https://issues.apache.org/jira/browse/KAFKA-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962751#comment-13962751 ] Stevo Slavic commented on KAFKA-1195: - Duplicate of KAFKA-1210 ? windows kafka-run-class.bat not updated --- Key: KAFKA-1195 URL: https://issues.apache.org/jira/browse/KAFKA-1195 Project: Kafka Issue Type: Bug Components: config Affects Versions: 0.8.0 Environment: windows 2008R2 Reporter: Amir Gershman Priority: Minor Labels: newbie The kafka-run-class.bat in the windows directory doesn't work. base dir is wrongly defined. ivy jars names don't match. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1370) Gradle startup script for Windows
Stevo Slavic created KAFKA-1370: --- Summary: Gradle startup script for Windows Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[GitHub] kafka pull request: KAFKA-1370 Added Gradle startup script for Win...
GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/22 KAFKA-1370 Added Gradle startup script for Windows This patch adds Gradle startup script for Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1370 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit 2d47e088d7f47fd06e7842b444b9039715616358 Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:02:16Z KAFKA-1370 Added Gradle startup script for Windows --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-1370) Gradle startup script for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962800#comment-13962800 ] ASF GitHub Bot commented on KAFKA-1370: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/22 KAFKA-1370 Added Gradle startup script for Windows This patch adds Gradle startup script for Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1370 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit 2d47e088d7f47fd06e7842b444b9039715616358 Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:02:16Z KAFKA-1370 Added Gradle startup script for Windows Gradle startup script for Windows - Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: gradle Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1370) Gradle startup script for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962801#comment-13962801 ] Stevo Slavic commented on KAFKA-1370: - Created pull request with this change (see [here|https://github.com/apache/kafka/pull/21]) Gradle startup script for Windows - Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: gradle Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1371) Ignore build output dirs
Stevo Slavic created KAFKA-1371: --- Summary: Ignore build output dirs Key: KAFKA-1371 URL: https://issues.apache.org/jira/browse/KAFKA-1371 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial After a clean clone and project build, build output directories get reported as changes/new. They should be ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1371) Ignore build output dirs
[ https://issues.apache.org/jira/browse/KAFKA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962803#comment-13962803 ] Stevo Slavic commented on KAFKA-1371: - Created pull request with fix for this (see [here|https://github.com/apache/kafka/pull/21]) Ignore build output dirs Key: KAFKA-1371 URL: https://issues.apache.org/jira/browse/KAFKA-1371 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: git After a clean clone and project build, build output directories get reported as changes/new. They should be ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1372) Upgrade to Gradle 1.11
Stevo Slavic created KAFKA-1372: --- Summary: Upgrade to Gradle 1.11 Key: KAFKA-1372 URL: https://issues.apache.org/jira/browse/KAFKA-1372 Project: Kafka Issue Type: Task Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Minor Currently used version of Gradle wrapper is 1.6 while 1.11 is available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[GitHub] kafka pull request: KAFKA-1372 Upgrade to Gradle 1.11
GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/23 KAFKA-1372 Upgrade to Gradle 1.11 This patch upgrades Gradle wrapper from 1.6 to 1.11 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1372 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/23.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23 commit 9f23afe3ef194dd61bb7ba36b28df98f3490e10b Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:19:45Z KAFKA-1372 Upgrade to Gradle 1.11 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-1372) Upgrade to Gradle 1.11
[ https://issues.apache.org/jira/browse/KAFKA-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962809#comment-13962809 ] ASF GitHub Bot commented on KAFKA-1372: --- GitHub user sslavic opened a pull request: https://github.com/apache/kafka/pull/23 KAFKA-1372 Upgrade to Gradle 1.11 This patch upgrades Gradle wrapper from 1.6 to 1.11 You can merge this pull request into a Git repository by running: $ git pull https://github.com/sslavic/kafka KAFKA-1372 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/23.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23 commit 9f23afe3ef194dd61bb7ba36b28df98f3490e10b Author: Stevo Slavic | hybris stevo.sla...@hybris.com Date: 2014-04-08T12:19:45Z KAFKA-1372 Upgrade to Gradle 1.11 Upgrade to Gradle 1.11 -- Key: KAFKA-1372 URL: https://issues.apache.org/jira/browse/KAFKA-1372 Project: Kafka Issue Type: Task Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Minor Labels: gradle Currently used version of Gradle wrapper is 1.6 while 1.11 is available. -- This message was sent by Atlassian JIRA (v6.2#6252)
Build failed in Jenkins: Kafka-trunk #151
See https://builds.apache.org/job/Kafka-trunk/151/ -- Started by user junrao Building remotely on solaris1 (Solaris) in workspace https://builds.apache.org/job/Kafka-trunk/ws/ Cloning the remote Git repository Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Error performing command: git fetch --tags --progress https://git-wip-us.apache.org/repos/asf/kafka.git +refs/heads/*:refs/remotes/origin/* at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1190) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1053) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:88) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:219) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:370) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Cannot run program git (in directory https://builds.apache.org/job/Kafka-trunk/ws/;): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) at hudson.Proc$LocalProc.init(Proc.java:244) at hudson.Proc$LocalProc.init(Proc.java:216) at hudson.Launcher$LocalLauncher.launch(Launcher.java:773) at hudson.Launcher$ProcStarter.start(Launcher.java:353) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1179) ... 15 more Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.init(UNIXProcess.java:53) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) ... 20 more ERROR: null Retrying after 10 seconds ERROR: Workspace has a .git repository, but it appears to be corrupt. hudson.plugins.git.GitException: Error performing command: git rev-parse --is-inside-work-tree at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1190) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1160) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1156) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:972) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:982) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.hasGitRepo(CliGitAPIImpl.java:136) at hudson.plugins.git.GitAPI.hasGitRepo(GitAPI.java:186) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:299) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:280) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:239) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
Re: Review Request 20038: Patch for KAFKA-1355
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20038/#review39798 --- core/src/main/scala/kafka/controller/ControllerChannelManager.scala https://reviews.apache.org/r/20038/#comment72399 This comment seems outdated. Could you make it up to date? - Jun Rao On April 4, 2014, 8:51 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20038/ --- (Updated April 4, 2014, 8:51 p.m.) Review request for kafka. Bugs: KAFKA-1355 https://issues.apache.org/jira/browse/KAFKA-1355 Repository: kafka Description --- Avoid sending all topic metadata on state changes. Diffs - core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala 03117377e1cb2ad63e4c7740d97ca9a4f20abeeb core/src/main/scala/kafka/controller/ControllerChannelManager.scala f17d97603798e803d035f313f0165f11f6d0f2c0 core/src/main/scala/kafka/controller/KafkaController.scala fcabd0da201ccd478f5884bcca9b75a8c8ea8373 core/src/main/scala/kafka/controller/TopicDeletionManager.scala 09f54acc733d49a9be9a9c6633271c190dea1bf0 core/src/main/scala/kafka/server/KafkaApis.scala c068ef69207c351eec413a595f1747c59f8b3983 Diff: https://reviews.apache.org/r/20038/diff/ Testing --- Thanks, Joel Koshy
Re: Review Request 20109: Fix KAFKA-1366
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20109/#review39795 --- Ship it! Ship It! - Neha Narkhede On April 7, 2014, 11:12 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20109/ --- (Updated April 7, 2014, 11:12 p.m.) Review request for kafka. Bugs: KAFKA-1366 https://issues.apache.org/jira/browse/KAFKA-1366 Repository: kafka Description --- Use config names from ProducerConfig KAFKA-1366.v1 Diffs - clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 9ba7ee734d9a26ab71b192ea23d0e09b67a0da15 core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala 0067a5375ed33d7831548450d80bee13d0196555 core/src/test/scala/unit/kafka/log4j/KafkaLog4jAppenderTest.scala 67497dd042dfd4ad54875fe29200d67f59137264 Diff: https://reviews.apache.org/r/20109/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Resolved] (KAFKA-1366) Multiple Unit Test failures with new producer
[ https://issues.apache.org/jira/browse/KAFKA-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1366. -- Resolution: Fixed Committed to trunk Multiple Unit Test failures with new producer - Key: KAFKA-1366 URL: https://issues.apache.org/jira/browse/KAFKA-1366 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1366.patch, KAFKA-1366_2014-04-07_16:12:00.patch Current known failed tests include 1. Log4jAppenderTest 2. ProducerFailureHandlingTest These tests failed mainly due to the config changes in KAFKA-1337, some others due to the async mode of the new producer. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20034: Fix KAFKA-1353
On April 8, 2014, 4:53 p.m., Jun Rao wrote: core/src/main/scala/kafka/network/RequestChannel.scala, lines 163-165 https://reviews.apache.org/r/20034/diff/1/?file=549058#file549058line163 Do we still need this? This is still used in some unit tests, in which they do need blocking behavior. On April 8, 2014, 4:53 p.m., Jun Rao wrote: core/src/main/scala/kafka/network/SocketServer.scala, lines 274-275 https://reviews.apache.org/r/20034/diff/1/?file=549059#file549059line274 Idle time should just be the time in doing socket select. It actually is just recording the socket select() function, no? - Guozhang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/#review39792 --- On April 4, 2014, 5:44 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/ --- (Updated April 4, 2014, 5:44 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20034/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 20034: Fix KAFKA-1353: incorporate Jun's comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/ --- (Updated April 8, 2014, 5:41 p.m.) Review request for kafka. Summary (updated) - Fix KAFKA-1353: incorporate Jun's comments Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description (updated) --- KAFKA-1353.v2 KAFKA-1353.v1 Diffs (updated) - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20034/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Updated] (KAFKA-1353) report capacity used by request thread pool and network thread pool
[ https://issues.apache.org/jira/browse/KAFKA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1353: - Attachment: KAFKA-1353_2014-04-08_10:41:15.patch report capacity used by request thread pool and network thread pool --- Key: KAFKA-1353 URL: https://issues.apache.org/jira/browse/KAFKA-1353 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-1353.patch, KAFKA-1353_2014-04-08_10:41:15.patch For monitoring and alerting, it would be very useful for the broker to report the percentage of the request thread pool and network thread pool used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1353) report capacity used by request thread pool and network thread pool
[ https://issues.apache.org/jira/browse/KAFKA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963212#comment-13963212 ] Guozhang Wang commented on KAFKA-1353: -- Updated reviewboard https://reviews.apache.org/r/20034/ against branch origin/trunk report capacity used by request thread pool and network thread pool --- Key: KAFKA-1353 URL: https://issues.apache.org/jira/browse/KAFKA-1353 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-1353.patch, KAFKA-1353_2014-04-08_10:41:15.patch For monitoring and alerting, it would be very useful for the broker to report the percentage of the request thread pool and network thread pool used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (KAFKA-1356) Topic metadata requests takes too long to process
[ https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1356. -- Resolution: Fixed Thanks for the patches! Committed to trunk Topic metadata requests takes too long to process - Key: KAFKA-1356 URL: https://issues.apache.org/jira/browse/KAFKA-1356 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Timothy Chen Assignee: Timothy Chen Attachments: KAFKA-1356.patch, KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, KAFKA-1356_2014-04-08_01:38:23.patch Currently we're seeing slow response times in handling get topic metadata requests. Local testing shows that even locally it takes 300 avg ms to respond, even though it's not doing any IO operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Closed] (KAFKA-1356) Topic metadata requests takes too long to process
[ https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1356. Topic metadata requests takes too long to process - Key: KAFKA-1356 URL: https://issues.apache.org/jira/browse/KAFKA-1356 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Timothy Chen Assignee: Timothy Chen Attachments: KAFKA-1356.patch, KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, KAFKA-1356_2014-04-08_01:38:23.patch Currently we're seeing slow response times in handling get topic metadata requests. Local testing shows that even locally it takes 300 avg ms to respond, even though it's not doing any IO operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1356) Topic metadata requests takes too long to process
[ https://issues.apache.org/jira/browse/KAFKA-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1356: - Assignee: Timothy Chen Topic metadata requests takes too long to process - Key: KAFKA-1356 URL: https://issues.apache.org/jira/browse/KAFKA-1356 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Timothy Chen Assignee: Timothy Chen Attachments: KAFKA-1356.patch, KAFKA-1356_2014-04-02_18:39:36.patch, KAFKA-1356_2014-04-04_14:40:18.patch, KAFKA-1356_2014-04-04_17:45:37.patch, KAFKA-1356_2014-04-06_01:45:47.patch, KAFKA-1356_2014-04-08_01:38:23.patch Currently we're seeing slow response times in handling get topic metadata requests. Local testing shows that even locally it takes 300 avg ms to respond, even though it's not doing any IO operations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1210) Windows Bat files are not working properly
[ https://issues.apache.org/jira/browse/KAFKA-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963238#comment-13963238 ] John Arnold commented on KAFKA-1210: Is there a branch that just works on windows? I tried the 0.8.1 tarball with windows batch file fixes and had issues with loading the classes, even after setting static paths in the batch files. PS C:\kafka .\bin\windows\kafka-create-topic.bat --zookeeper localhost:2181 --replica 1 --partition 1 --topic test C:\kafka C:\kafka\libs classpath=C:\kafka\libs;C:\kafka\target\scala-2.9.2\kafka_2.9.2-0.1-SNAPSHOT.jar;C:\kafka\target\scala-2.9.2\kafka_2.9.2-0.1-SNAPSHOT.jar;C:\kafka\libs\jopt-simple-3.2.jar;C:\kafka\libs\kafka_2.9.2-0.8.1.jar;C:\kafka\libs\log4j-1.2.15.jar;C:\kafka\libs\metrics-annotation-2.2.0.jar;C:\kafka\libs\metrics-core-2.2.0.jar;C:\kafka\libs\scala-library-2.9.2.jar;C:\kafka\libs\slf4j-api-1.7.2.jar;C:\kafka\libs\slf4j-log4j12-1.7.6.jar;C:\kafka\libs\snappy-java-1.0.5.jar;C:\kafka\libs\zkclient-0.3.jar;C:\kafka\libs\zookeeper-3.3.4.jar C:\java\bin\java -Xmx256M -server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dlog4j.configuration=file:C:\kafka\config\tools-log4j.properties -cp C:\kafka\libs;C:\kafka\target\scala-2.9.2\kafka_2.9.2-0.1-SNAPSHOT.jar;C:\kafka\target\scala-2.9.2\kafka_2.9.2-0.1-SNAPSHOT.jar;C:\kafka\libs\jopt-simple-3.2.jar;C:\kafka\libs\kafka_2.9.2-0.8.1 .jar;C:\kafka\libs\log4j-1.2.15.jar;C:\kafka\libs\metrics-annotation-2.2.0.jar;C:\kafka\libs\metrics-core-2.2.0.jar;C:\kafka\libs\scala-library-2.9.2.jar;C:\kafka\libs\slf4j-api-1.7.2.jar;C:\kafka\libs\slf4j-log4j12-1.7.6.jar;C:\kafka\libs\snappy-java-1.0.5.jar;C:\kafka\libs\zkclient-0.3.jar;C:\kafka\libs\zookeeper-3.3.4.jar kafka.admin.CreateTopicCommand --zookeeper localhost:2181 --replica 1 --partition 1 --topic test Error: Could not find or load main class kafka.admin.CreateTopicCommand PS C:\kafka Help? Windows Bat files are not working properly -- Key: KAFKA-1210 URL: https://issues.apache.org/jira/browse/KAFKA-1210 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Environment: Windows Reporter: Håkan Canberger Priority: Trivial Fix For: 0.8.2 Attachments: 1210-v1.patch The bat files are not working properly. The paths in them are invalid. They have not been updated to reflect the changes made to the shell scripts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1364) ReplicaManagerTest hard-codes log dir
[ https://issues.apache.org/jira/browse/KAFKA-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1364: -- Fix Version/s: 0.8.1.1 ReplicaManagerTest hard-codes log dir - Key: KAFKA-1364 URL: https://issues.apache.org/jira/browse/KAFKA-1364 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Fix For: 0.8.2, 0.8.1.1 Saw the following unit test failure. This is probably due to that the hard-coded log.dir conflicts with the default config when running a standalone sack. kafka.server.ReplicaManagerTest testHighwaterMarkDirectoryMapping FAILED java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/tmp/kafka-logs/test-topic-1/.index) has non-zero size but the last offset is 0 and the base offset is 0 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.server.ReplicaManagerTest.testHighwaterMarkDirectoryMapping(ReplicaManagerTest.scala:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (KAFKA-1364) ReplicaManagerTest hard-codes log dir
[ https://issues.apache.org/jira/browse/KAFKA-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reassigned KAFKA-1364: - Assignee: Guozhang Wang ReplicaManagerTest hard-codes log dir - Key: KAFKA-1364 URL: https://issues.apache.org/jira/browse/KAFKA-1364 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Guozhang Wang Fix For: 0.8.2, 0.8.1.1 Saw the following unit test failure. This is probably due to that the hard-coded log.dir conflicts with the default config when running a standalone sack. kafka.server.ReplicaManagerTest testHighwaterMarkDirectoryMapping FAILED java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/tmp/kafka-logs/test-topic-1/.index) has non-zero size but the last offset is 0 and the base offset is 0 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.server.ReplicaManagerTest.testHighwaterMarkDirectoryMapping(ReplicaManagerTest.scala:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs
[ https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reassigned KAFKA-1363: - Assignee: Timothy Chen testTopicConfigChangesDuringDeleteTopic hangs - Key: KAFKA-1363 URL: https://issues.apache.org/jira/browse/KAFKA-1363 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Timothy Chen Fix For: 0.8.1.1 Saw the following deadlock during shutting down the delete topic manager. delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on condition [0x7fd53d16] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b41d6318 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition [0x7fd570a87000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b5b6f580 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36) at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.shutdown(KafkaController.scala:646) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:166) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:45) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:45) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at kafka.admin.DeleteTopicTest.testTopicConfigChangesDuringDeleteTopic(DeleteTopicTest.scala:362) -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20128: Fix KAFKA-1364
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20128/ --- Review request for kafka. Bugs: KAFKA-1364 https://issues.apache.org/jira/browse/KAFKA-1364 Repository: kafka Description --- KAFKA-1364.v1 Diffs - core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala b5936d4101b513baa805ab26361fe965bdf980aa Diff: https://reviews.apache.org/r/20128/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1364) ReplicaManagerTest hard-codes log dir
[ https://issues.apache.org/jira/browse/KAFKA-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963433#comment-13963433 ] Guozhang Wang commented on KAFKA-1364: -- Created reviewboard https://reviews.apache.org/r/20128/ against branch origin/trunk ReplicaManagerTest hard-codes log dir - Key: KAFKA-1364 URL: https://issues.apache.org/jira/browse/KAFKA-1364 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Guozhang Wang Fix For: 0.8.2, 0.8.1.1 Attachments: KAFKA-1364.patch Saw the following unit test failure. This is probably due to that the hard-coded log.dir conflicts with the default config when running a standalone sack. kafka.server.ReplicaManagerTest testHighwaterMarkDirectoryMapping FAILED java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/tmp/kafka-logs/test-topic-1/.index) has non-zero size but the last offset is 0 and the base offset is 0 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.server.ReplicaManagerTest.testHighwaterMarkDirectoryMapping(ReplicaManagerTest.scala:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1364) ReplicaManagerTest hard-codes log dir
[ https://issues.apache.org/jira/browse/KAFKA-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1364: - Attachment: KAFKA-1364.patch ReplicaManagerTest hard-codes log dir - Key: KAFKA-1364 URL: https://issues.apache.org/jira/browse/KAFKA-1364 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Guozhang Wang Fix For: 0.8.2, 0.8.1.1 Attachments: KAFKA-1364.patch Saw the following unit test failure. This is probably due to that the hard-coded log.dir conflicts with the default config when running a standalone sack. kafka.server.ReplicaManagerTest testHighwaterMarkDirectoryMapping FAILED java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/tmp/kafka-logs/test-topic-1/.index) has non-zero size but the last offset is 0 and the base offset is 0 at scala.Predef$.require(Predef.scala:145) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.log.Log.loadSegments(Log.scala:158) at kafka.log.Log.init(Log.scala:64) at kafka.server.ReplicaManagerTest.testHighwaterMarkDirectoryMapping(ReplicaManagerTest.scala:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20034: Fix KAFKA-1353: incorporate Jun's comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/#review39819 --- core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20034/#comment72478 Is there value in including Broker-id in the bean name? How about NetworkProcessorAvgIdlePercent and NetworkProcessor-id-IdlePercent? core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20034/#comment72476 No longer used. core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20034/#comment72477 The way that we use meter is a bit tricky. Could you add some comments on the logic behind it and why we need to do the division? core/src/main/scala/kafka/server/KafkaRequestHandler.scala https://reviews.apache.org/r/20034/#comment72479 The idle time needs to be computed inside the while loop. Otherwise, if there is no new request, the metrics is not updated. core/src/main/scala/kafka/server/KafkaRequestHandler.scala https://reviews.apache.org/r/20034/#comment72480 Same comment on broker id. For the name, will RequestHandlerAvgIdlePercent be better? - Jun Rao On April 8, 2014, 5:41 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/ --- (Updated April 8, 2014, 5:41 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20034/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 20034: Fix KAFKA-1353: incorporate Jun's comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/#review39821 --- core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20034/#comment72482 Perhaps aggregateIdleMeter will be better? core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20034/#comment72481 Perhaps rename to totalProcessorThreads? - Jun Rao On April 8, 2014, 5:41 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20034/ --- (Updated April 8, 2014, 5:41 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20034/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Created] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
Joel Koshy created KAFKA-1373: - Summary: LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20130: Patch for KAFKA-1373
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/ --- Review request for kafka. Bugs: KAFKA-1373 https://issues.apache.org/jira/browse/KAFKA-1373 Repository: kafka Description --- fix Diffs - core/src/main/scala/kafka/log/LogCleanerManager.scala 6a981345fd7dc711e6c4f058a42bd7f8f9d350fe core/src/main/scala/kafka/server/OffsetCheckpoint.scala 19f61a9718a7f8e6f9bf743ac9eb2118d0fdd9b0 Diff: https://reviews.apache.org/r/20130/diff/ Testing --- Thanks, Joel Koshy
[jira] [Commented] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[ https://issues.apache.org/jira/browse/KAFKA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963461#comment-13963461 ] Joel Koshy commented on KAFKA-1373: --- Created reviewboard https://reviews.apache.org/r/20130/ against branch origin/trunk LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 Attachments: KAFKA-1373.patch If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[ https://issues.apache.org/jira/browse/KAFKA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1373: -- Attachment: KAFKA-1373.patch LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 Attachments: KAFKA-1373.patch If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
Joel Koshy created KAFKA-1374: - Summary: LogCleaner (compaction) does not support compressed topics Key: KAFKA-1374 URL: https://issues.apache.org/jira/browse/KAFKA-1374 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Fix For: 0.8.2 This is a known issue, but opening a ticket to track. If you try to compact a topic that has compressed messages you will run into various exceptions - typically because during iteration we advance the position based on the decompressed size of the message. I have a bunch of stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1353) report capacity used by request thread pool and network thread pool
[ https://issues.apache.org/jira/browse/KAFKA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1353: - Attachment: KAFKA-1353.patch report capacity used by request thread pool and network thread pool --- Key: KAFKA-1353 URL: https://issues.apache.org/jira/browse/KAFKA-1353 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-1353.patch, KAFKA-1353.patch, KAFKA-1353_2014-04-08_10:41:15.patch For monitoring and alerting, it would be very useful for the broker to report the percentage of the request thread pool and network thread pool used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1353) report capacity used by request thread pool and network thread pool
[ https://issues.apache.org/jira/browse/KAFKA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963467#comment-13963467 ] Guozhang Wang commented on KAFKA-1353: -- Created reviewboard https://reviews.apache.org/r/20131/ against branch origin/trunk report capacity used by request thread pool and network thread pool --- Key: KAFKA-1353 URL: https://issues.apache.org/jira/browse/KAFKA-1353 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-1353.patch, KAFKA-1353.patch, KAFKA-1353_2014-04-08_10:41:15.patch For monitoring and alerting, it would be very useful for the broker to report the percentage of the request thread pool and network thread pool used. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20131: Fix KAFKA-1353: incorporate Jun's comment round two
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/ --- Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- Remove unused imports KAFKA-1353.v3 KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20131/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 20131: Fix KAFKA-1353: incorporate Jun's comment round two
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/ --- (Updated April 8, 2014, 9:56 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- Remove unused imports KAFKA-1353.v3 KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20131/diff/ Testing (updated) --- Local test with 9 new producer performance and one handler thread: in 3 minutes, the io threads idle time drops to 10%, but after that remains vibrating, the network threads keeps dropping but much slower, only to about 85% in 3 minutes. Adding to 15 producers, the io thread starts to drop again, until close to 2% where it starts vibrating again, whereas the network threads drops to about 80%. Thanks, Guozhang Wang
Re: Review Request 20050: Fix KAFKA-1359
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20050/#review39825 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java https://reviews.apache.org/r/20050/#comment72489 Perhaps these can all be wrapped in the same if statement in line 815 to save the unnecessary overhead? clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java https://reviews.apache.org/r/20050/#comment72490 Ditto as the above. clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java https://reviews.apache.org/r/20050/#comment72494 I am wondering if this is the right place to do the registration. Since this is only called when we have producer request, if the first request sent to a node is not a produce request, we won't have pre-created the node level sensor. I think this can happen during a broker failure when the first request to a node could be a metadata request. So, we probably have to register this in Selector. This may also solve your other problem that a metric is not registered in unit test. - Jun Rao On April 4, 2014, 9:33 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20050/ --- (Updated April 4, 2014, 9:33 p.m.) Review request for kafka. Bugs: KAFKA-1359 https://issues.apache.org/jira/browse/KAFKA-1359 Repository: kafka Description --- KAFKA-1359.v1 Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 855ae84f14aa91653b3fa855c2af40560323f42a clients/src/main/java/org/apache/kafka/common/network/Selector.java 558f8b47638b354f9c1a30be5d45dc7b61131bea Diff: https://reviews.apache.org/r/20050/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 20131: Fix KAFKA-1353: incorporate Jun's comment round two
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/#review39831 --- core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/20131/#comment72514 How about using the following comment? We use a single meter for aggregate idle percentage for the thread pool. Since meter is calculated as total_recorded_value / time_window and time_window is independent of the number of threads, each recorded idle time should be discounted by # threads. core/src/main/scala/kafka/server/KafkaRequestHandler.scala https://reviews.apache.org/r/20131/#comment72506 Perhaps use totalHandlerThreads? - Jun Rao On April 8, 2014, 9:56 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/ --- (Updated April 8, 2014, 9:56 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- Remove unused imports KAFKA-1353.v3 KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20131/diff/ Testing --- Local test with 9 new producer performance and one handler thread: in 3 minutes, the io threads idle time drops to 10%, but after that remains vibrating, the network threads keeps dropping but much slower, only to about 85% in 3 minutes. Adding to 15 producers, the io thread starts to drop again, until close to 2% where it starts vibrating again, whereas the network threads drops to about 80%. Thanks, Guozhang Wang
Re: Review Request 20130: Patch for KAFKA-1373
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/ --- (Updated April 8, 2014, 11:18 p.m.) Review request for kafka. Bugs: KAFKA-1373 https://issues.apache.org/jira/browse/KAFKA-1373 Repository: kafka Description (updated) --- Set first dirty (uncompacted) offset to first offset of the log if no checkpoint exists. Diffs (updated) - core/src/main/scala/kafka/log/LogCleanerManager.scala 6a981345fd7dc711e6c4f058a42bd7f8f9d350fe core/src/main/scala/kafka/server/KafkaConfig.scala b0506d4881176cd95914847c4275e428a8d2ca10 core/src/main/scala/kafka/server/OffsetCheckpoint.scala 19f61a9718a7f8e6f9bf743ac9eb2118d0fdd9b0 core/src/main/scala/kafka/server/OffsetManager.scala 89a88a7e4055f804429b64e85b2f65312d1e2155 core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 9aeb69d4934b9aa94e0899af14fc27e7c20f039f Diff: https://reviews.apache.org/r/20130/diff/ Testing --- Thanks, Joel Koshy
[jira] [Commented] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[ https://issues.apache.org/jira/browse/KAFKA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963605#comment-13963605 ] Joel Koshy commented on KAFKA-1373: --- Updated reviewboard https://reviews.apache.org/r/20130/ against branch origin/trunk LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 Attachments: KAFKA-1373.patch, KAFKA-1373_2014-04-08_16:18:22.patch If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[ https://issues.apache.org/jira/browse/KAFKA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1373: -- Attachment: KAFKA-1373_2014-04-08_16:18:22.patch LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 Attachments: KAFKA-1373.patch, KAFKA-1373_2014-04-08_16:18:22.patch If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1373) LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[ https://issues.apache.org/jira/browse/KAFKA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963608#comment-13963608 ] Joel Koshy commented on KAFKA-1373: --- Updated the RB with an additional fix: KAFKA-1289 changed the compaction config from dedupe to compact. That needed to be updated in a couple of other places as well. Ideally it should have been some global constant string but this is fine for now. LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint Key: KAFKA-1373 URL: https://issues.apache.org/jira/browse/KAFKA-1373 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1 Reporter: Joel Koshy Assignee: Joel Koshy Fix For: 0.8.1.1 Attachments: KAFKA-1373.patch, KAFKA-1373_2014-04-08_16:18:22.patch If you try enabling the compaction policy on a topic that already exists and if its first segment starts with anything other than zero you will see something like this: java.lang.IllegalArgumentException: requirement failed: Last clean offset is 0 but segment base offset is 2722629 for log test-0. at scala.Predef$.require(Predef.scala:145) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:489) at kafka.log.Cleaner.clean(LogCleaner.scala:287) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:188) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20050: Fix KAFKA-1359
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20050/#review39836 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java https://reviews.apache.org/r/20050/#comment72523 I agree, it is better to register this in Selector. This would not solve the unit tests though since there are Sender unit tests that will be broken then.. - Guozhang Wang On April 4, 2014, 9:33 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20050/ --- (Updated April 4, 2014, 9:33 p.m.) Review request for kafka. Bugs: KAFKA-1359 https://issues.apache.org/jira/browse/KAFKA-1359 Repository: kafka Description --- KAFKA-1359.v1 Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 855ae84f14aa91653b3fa855c2af40560323f42a clients/src/main/java/org/apache/kafka/common/network/Selector.java 558f8b47638b354f9c1a30be5d45dc7b61131bea Diff: https://reviews.apache.org/r/20050/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 20131: Fix KAFKA-1353: incorporate Jun's comment round three
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/ --- (Updated April 8, 2014, 11:31 p.m.) Review request for kafka. Summary (updated) - Fix KAFKA-1353: incorporate Jun's comment round three Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description (updated) --- KAFKA-1353.v4 Remove unused imports KAFKA-1353.v3 KAFKA-1353.v2 KAFKA-1353.v1 Diffs (updated) - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaConfig.scala b0506d4881176cd95914847c4275e428a8d2ca10 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20131/diff/ Testing --- Local test with 9 new producer performance and one handler thread: in 3 minutes, the io threads idle time drops to 10%, but after that remains vibrating, the network threads keeps dropping but much slower, only to about 85% in 3 minutes. Adding to 15 producers, the io thread starts to drop again, until close to 2% where it starts vibrating again, whereas the network threads drops to about 80%. Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1353) report capacity used by request thread pool and network thread pool
[ https://issues.apache.org/jira/browse/KAFKA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963617#comment-13963617 ] Guozhang Wang commented on KAFKA-1353: -- Updated reviewboard https://reviews.apache.org/r/20131/ against branch origin/trunk report capacity used by request thread pool and network thread pool --- Key: KAFKA-1353 URL: https://issues.apache.org/jira/browse/KAFKA-1353 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-1353.patch, KAFKA-1353.patch, KAFKA-1353_2014-04-08_10:41:15.patch, KAFKA-1353_2014-04-08_16:31:45.patch For monitoring and alerting, it would be very useful for the broker to report the percentage of the request thread pool and network thread pool used. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19957: Patch for KAFKA-1356
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19957/#review39840 --- Thanks for the patch. Some additional comments. 1. The introduction of MetadataCache is a good idea and provides better code isolation. 1.1 Could we move aliveBroker in it too? 1.2 Could we make it a private class? 1.3 Could we more the readwriteLock and methods that access the lock inside this class too? This way, the usage of the lock is only limited inside the class. 2. I am concerned of the overhead of the lock in ensureTopicExists(). This method is called in every produce and fetch request and is a global lock. I think the purpose is to disable access to a topic when a topic is deleted. However, even if we don't call ensureTopicExists(), once a replica is deleted by the ReplicaManager, all requests will get the UnknownTopicOrPartitonException. ReplicaManager may find out that a partition is being deleted a bit later. But this is not a big issue since we just accumulated a bit more data which will be deleted soon. - Jun Rao On April 8, 2014, 8:38 a.m., Timothy Chen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19957/ --- (Updated April 8, 2014, 8:38 a.m.) Review request for kafka. Bugs: KAFKA-1356 https://issues.apache.org/jira/browse/KAFKA-1356 Repository: kafka Description --- KAFKA-1356 Improve topic metadata handling in kafka api Diffs - core/src/main/scala/kafka/api/TopicMetadata.scala 0513a59ed94e556894b3515dc38666ee9a66ae3d core/src/main/scala/kafka/controller/KafkaController.scala c8c02ced543a6278ba5b1edfa73a370f1edb1891 core/src/main/scala/kafka/server/KafkaApis.scala 705d87ed874f2363e6d9d6ea1143bc6035a0a9d6 core/src/test/scala/unit/kafka/admin/AdminTest.scala d5644ea40ec7678b975c4775546b79fcfa9f64b7 core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 22bb6f2847b8895f8fbba6c531963ebb0fffe2ca core/src/test/scala/unit/kafka/utils/TestUtils.scala 2054c25bbced6bd90c092a1974975732ad346146 Diff: https://reviews.apache.org/r/19957/diff/ Testing --- Thanks, Timothy Chen
Re: Review Request 20131: Fix KAFKA-1353: incorporate Jun's comment round three
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/#review39841 --- core/src/main/scala/kafka/server/KafkaConfig.scala https://reviews.apache.org/r/20131/#comment72554 This should be reverted. - Jun Rao On April 8, 2014, 11:31 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20131/ --- (Updated April 8, 2014, 11:31 p.m.) Review request for kafka. Bugs: KAFKA-1353 https://issues.apache.org/jira/browse/KAFKA-1353 Repository: kafka Description --- KAFKA-1353.v4 Remove unused imports KAFKA-1353.v3 KAFKA-1353.v2 KAFKA-1353.v1 Diffs - core/src/main/scala/kafka/network/RequestChannel.scala a6ec970d72fc8adc56bc6e200d1ff5bbf0d2b748 core/src/main/scala/kafka/network/SocketServer.scala dcfca3f4156957b0006ba9bfdeca848c115825a8 core/src/main/scala/kafka/server/KafkaConfig.scala b0506d4881176cd95914847c4275e428a8d2ca10 core/src/main/scala/kafka/server/KafkaRequestHandler.scala 871212bf9110ca95ca7d2109be070f69d0f90247 Diff: https://reviews.apache.org/r/20131/diff/ Testing --- Local test with 9 new producer performance and one handler thread: in 3 minutes, the io threads idle time drops to 10%, but after that remains vibrating, the network threads keeps dropping but much slower, only to about 85% in 3 minutes. Adding to 15 producers, the io thread starts to drop again, until close to 2% where it starts vibrating again, whereas the network threads drops to about 80%. Thanks, Guozhang Wang
Re: Review Request 20128: Fix KAFKA-1364
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20128/#review39843 --- core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala https://reviews.apache.org/r/20128/#comment72557 Using File.separator instead of / will be more portable. - Jun Rao On April 8, 2014, 8:56 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20128/ --- (Updated April 8, 2014, 8:56 p.m.) Review request for kafka. Bugs: KAFKA-1364 https://issues.apache.org/jira/browse/KAFKA-1364 Repository: kafka Description --- KAFKA-1364.v1 Diffs - core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala b5936d4101b513baa805ab26361fe965bdf980aa Diff: https://reviews.apache.org/r/20128/diff/ Testing --- Thanks, Guozhang Wang
Review Request 20143: Patch for KAFKA-1363
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20143/ --- Review request for kafka. Bugs: KAFKA-1363 https://issues.apache.org/jira/browse/KAFKA-1363 Repository: kafka Description --- KAFKA-1363 Avoid delete topic deadlock by checking thread run status Diffs - core/src/main/scala/kafka/controller/TopicDeletionManager.scala d29e556232ae549545bde1b6c5c9fabd5fa67bf5 Diff: https://reviews.apache.org/r/20143/diff/ Testing --- Thanks, Timothy Chen
[jira] [Updated] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs
[ https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1363: Attachment: KAFKA-1363.patch testTopicConfigChangesDuringDeleteTopic hangs - Key: KAFKA-1363 URL: https://issues.apache.org/jira/browse/KAFKA-1363 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Timothy Chen Fix For: 0.8.1.1 Attachments: KAFKA-1363.patch Saw the following deadlock during shutting down the delete topic manager. delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on condition [0x7fd53d16] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b41d6318 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition [0x7fd570a87000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b5b6f580 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36) at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.shutdown(KafkaController.scala:646) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:166) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:45) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:45) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at kafka.admin.DeleteTopicTest.testTopicConfigChangesDuringDeleteTopic(DeleteTopicTest.scala:362) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs
[ https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963673#comment-13963673 ] Timothy Chen commented on KAFKA-1363: - Created reviewboard https://reviews.apache.org/r/20143/ against branch origin/trunk testTopicConfigChangesDuringDeleteTopic hangs - Key: KAFKA-1363 URL: https://issues.apache.org/jira/browse/KAFKA-1363 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Timothy Chen Fix For: 0.8.1.1 Attachments: KAFKA-1363.patch Saw the following deadlock during shutting down the delete topic manager. delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on condition [0x7fd53d16] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b41d6318 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition [0x7fd570a87000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b5b6f580 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36) at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.shutdown(KafkaController.scala:646) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:166) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:45) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:45) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at kafka.admin.DeleteTopicTest.testTopicConfigChangesDuringDeleteTopic(DeleteTopicTest.scala:362) -- This
Re: Review Request 20143: Patch for KAFKA-1363
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20143/ --- (Updated April 9, 2014, 12:52 a.m.) Review request for kafka. Bugs: KAFKA-1363 https://issues.apache.org/jira/browse/KAFKA-1363 Repository: kafka Description --- KAFKA-1363 Avoid delete topic deadlock by checking thread run status Diffs (updated) - core/src/main/scala/kafka/controller/TopicDeletionManager.scala d29e556232ae549545bde1b6c5c9fabd5fa67bf5 Diff: https://reviews.apache.org/r/20143/diff/ Testing --- Thanks, Timothy Chen
[jira] [Updated] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs
[ https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1363: Attachment: KAFKA-1363_2014-04-08_17:52:17.patch testTopicConfigChangesDuringDeleteTopic hangs - Key: KAFKA-1363 URL: https://issues.apache.org/jira/browse/KAFKA-1363 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Timothy Chen Fix For: 0.8.1.1 Attachments: KAFKA-1363.patch, KAFKA-1363_2014-04-08_17:52:17.patch Saw the following deadlock during shutting down the delete topic manager. delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on condition [0x7fd53d16] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b41d6318 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition [0x7fd570a87000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b5b6f580 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36) at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.shutdown(KafkaController.scala:646) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:166) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:45) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:45) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at kafka.admin.DeleteTopicTest.testTopicConfigChangesDuringDeleteTopic(DeleteTopicTest.scala:362) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1363) testTopicConfigChangesDuringDeleteTopic hangs
[ https://issues.apache.org/jira/browse/KAFKA-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963675#comment-13963675 ] Timothy Chen commented on KAFKA-1363: - Updated reviewboard https://reviews.apache.org/r/20143/ against branch origin/trunk testTopicConfigChangesDuringDeleteTopic hangs - Key: KAFKA-1363 URL: https://issues.apache.org/jira/browse/KAFKA-1363 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Timothy Chen Fix For: 0.8.1.1 Attachments: KAFKA-1363.patch, KAFKA-1363_2014-04-08_17:52:17.patch Saw the following deadlock during shutting down the delete topic manager. delete-topics-thread prio=10 tid=0x7fd50c003800 nid=0x7d9 waiting on condition [0x7fd53d16] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b41d6318 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:363) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Test worker prio=10 tid=0x7fd578928800 nid=0x763d waiting on condition [0x7fd570a87000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006b5b6f580 (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36) at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:100) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:345) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:341) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:341) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply$mcV$sp(KafkaController.scala:648) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.controller.KafkaController$$anonfun$shutdown$1.apply(KafkaController.scala:646) at kafka.utils.Utils$.inLock(Utils.scala:537) at kafka.controller.KafkaController.shutdown(KafkaController.scala:646) at kafka.server.KafkaServer$$anonfun$shutdown$9.apply$mcV$sp(KafkaServer.scala:242) at kafka.utils.Utils$.swallow(Utils.scala:166) at kafka.utils.Logging$class.swallowWarn(Logging.scala:92) at kafka.utils.Utils$.swallowWarn(Utils.scala:45) at kafka.utils.Logging$class.swallow(Logging.scala:94) at kafka.utils.Utils$.swallow(Utils.scala:45) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:242) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at kafka.admin.DeleteTopicTest$$anonfun$testTopicConfigChangesDuringDeleteTopic$1.apply(DeleteTopicTest.scala:362) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at
[jira] [Resolved] (KAFKA-1352) Reduce logging on the server
[ https://issues.apache.org/jira/browse/KAFKA-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1352. Resolution: Fixed Fix Version/s: 0.8.2 Thanks for the patch. +1 and committed to trunk after rebasing. Reduce logging on the server Key: KAFKA-1352 URL: https://issues.apache.org/jira/browse/KAFKA-1352 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.0, 0.8.1 Reporter: Neha Narkhede Assignee: Ivan Lyutov Labels: newbie, usability Fix For: 0.8.2 Attachments: KAFKA-1352.patch, KAFKA-1352_2014-04-04_21:20:31.patch We have excessive logging in the server, making the logs unreadable and also affecting the performance of the server in practice. We need to clean the logs to address these issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1371) Ignore build output dirs
[ https://issues.apache.org/jira/browse/KAFKA-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963679#comment-13963679 ] Jun Rao commented on KAFKA-1371: Thanks for filing the jira. Could you attach the patch to the jira? We need that to know that you grant the code change to Apache. Ignore build output dirs Key: KAFKA-1371 URL: https://issues.apache.org/jira/browse/KAFKA-1371 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: git After a clean clone and project build, build output directories get reported as changes/new. They should be ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1372) Upgrade to Gradle 1.11
[ https://issues.apache.org/jira/browse/KAFKA-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963677#comment-13963677 ] Jun Rao commented on KAFKA-1372: Could you attach the change as a patch? I think we need that to take care of Apache licensing issue. Upgrade to Gradle 1.11 -- Key: KAFKA-1372 URL: https://issues.apache.org/jira/browse/KAFKA-1372 Project: Kafka Issue Type: Task Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Minor Labels: gradle Currently used version of Gradle wrapper is 1.6 while 1.11 is available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1370) Gradle startup script for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963680#comment-13963680 ] Jun Rao commented on KAFKA-1370: Could you attach the change as a patch? I think we need that to take care of Apache licensing issue. Gradle startup script for Windows - Key: KAFKA-1370 URL: https://issues.apache.org/jira/browse/KAFKA-1370 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1 Reporter: Stevo Slavic Priority: Trivial Labels: gradle Please provide Gradle startup script for Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20083: Patch for KAFKA-1351
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20083/#review39852 --- Thanks for the patch. Do you have any performance numbers? - Jun Rao On April 7, 2014, 3:02 p.m., Ivan Lyutov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20083/ --- (Updated April 7, 2014, 3:02 p.m.) Review request for kafka. Bugs: KAFKA-1351 https://issues.apache.org/jira/browse/KAFKA-1351 Repository: kafka Description --- KAFKA-1351 - Implemented more efficient substitution for String.format in Scala. Diffs - core/src/main/scala/kafka/utils/Utils.scala 6bfbac16e2f8d68b8c711a0336c698aa6f610ae8 core/src/test/scala/unit/kafka/utils/UtilsTest.scala 920f318c85a747b689f97c0e73e4234f20b1460a Diff: https://reviews.apache.org/r/20083/diff/ Testing --- Thanks, Ivan Lyutov
Build failed in Jenkins: Kafka-trunk #154
See https://builds.apache.org/job/Kafka-trunk/154/changes Changes: [junrao] kafka-1352; Reduce logging on the server; patched by Ivan Lyutovg; reviewed by Guozhang Wang and Jun Rao -- [...truncated 397 lines...] kafka.log.CleanerTest testCleaningWithDeletes PASSED kafka.log.CleanerTest testCleanSegmentsWithAbort PASSED kafka.log.CleanerTest testSegmentGrouping PASSED kafka.log.CleanerTest testBuildOffsetMap PASSED kafka.log.LogTest testTimeBasedLogRoll PASSED kafka.log.LogTest testSizeBasedLogRoll PASSED kafka.log.LogTest testLoadEmptyLog PASSED kafka.log.LogTest testAppendAndReadWithSequentialOffsets PASSED kafka.log.LogTest testAppendAndReadWithNonSequentialOffsets PASSED kafka.log.LogTest testReadAtLogGap PASSED kafka.log.LogTest testReadOutOfRange PASSED kafka.log.LogTest testLogRolls PASSED kafka.log.LogTest testCompressedMessages PASSED kafka.log.LogTest testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED kafka.log.LogTest testMessageSizeCheck PASSED kafka.log.LogTest testLogRecoversToCorrectOffset PASSED kafka.log.LogTest testIndexRebuild PASSED kafka.log.LogTest testTruncateTo PASSED kafka.log.LogTest testIndexResizingAtTruncation PASSED kafka.log.LogTest testBogusIndexSegmentsAreRemoved PASSED kafka.log.LogTest testReopenThenTruncate PASSED kafka.log.LogTest testAsyncDelete PASSED kafka.log.LogTest testOpenDeletesObsoleteFiles PASSED kafka.log.LogTest testAppendMessageWithNullPayload PASSED kafka.log.LogTest testCorruptLog PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.FileMessageSetTest testWrittenEqualsRead PASSED kafka.log.FileMessageSetTest testIteratorIsConsistent PASSED kafka.log.FileMessageSetTest testSizeInBytes PASSED kafka.log.FileMessageSetTest testWriteTo PASSED kafka.log.FileMessageSetTest testTruncate PASSED kafka.log.FileMessageSetTest testFileSize PASSED kafka.log.FileMessageSetTest testIterationOverPartialAndTruncation PASSED kafka.log.FileMessageSetTest testIterationDoesntChangePosition PASSED kafka.log.FileMessageSetTest testRead PASSED kafka.log.FileMessageSetTest testSearch PASSED kafka.log.FileMessageSetTest testIteratorWithLimits PASSED kafka.log.LogManagerTest testCreateLog PASSED kafka.log.LogManagerTest testGetNonExistentLog PASSED kafka.log.LogManagerTest testCleanupExpiredSegments PASSED kafka.log.LogManagerTest testCleanupSegmentsToMaintainSize PASSED kafka.log.LogManagerTest testTimeBasedFlush PASSED kafka.log.LogManagerTest testLeastLoadedAssignment PASSED kafka.log.LogManagerTest testTwoLogManagersUsingSameDirFails PASSED kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.OffsetIndexTest truncate PASSED kafka.log.OffsetIndexTest randomLookupTest PASSED kafka.log.OffsetIndexTest lookupExtremeCases PASSED kafka.log.OffsetIndexTest appendTooMany PASSED kafka.log.OffsetIndexTest appendOutOfOrder PASSED kafka.log.OffsetIndexTest testReopen PASSED kafka.metrics.KafkaTimerTest testKafkaTimer PASSED kafka.network.SocketServerTest simpleRequest PASSED kafka.network.SocketServerTest tooBigRequestIsRejected PASSED kafka.network.SocketServerTest testNullResponse PASSED kafka.network.SocketServerTest testSocketsCloseOnShutdown PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.utils.UtilsTest testSwallow PASSED kafka.utils.UtilsTest testCircularIterator PASSED kafka.utils.UtilsTest testReadBytes PASSED kafka.utils.UtilsTest testReplaceSuffix PASSED kafka.utils.UtilsTest testReadInt PASSED kafka.utils.UtilsTest testCsvList PASSED kafka.utils.UtilsTest testInLock PASSED kafka.utils.JsonTest testJsonEncoding PASSED kafka.api.RequestResponseSerializationTest testSerializationAndDeserialization PASSED kafka.api.ProducerFailureHandlingTest testInvalidPartition PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckZero PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckOne PASSED kafka.api.ProducerFailureHandlingTest testNonExistTopic PASSED kafka.api.ProducerFailureHandlingTest testWrongBrokerList PASSED kafka.api.ProducerFailureHandlingTest testNoResponse PASSED kafka.api.ProducerFailureHandlingTest testSendAfterClosed PASSED kafka.api.ProducerFailureHandlingTest testBrokerFailure PASSED kafka.api.ApiUtilsTest testShortStringNonASCII PASSED kafka.api.ApiUtilsTest testShortStringASCII PASSED kafka.api.test.ProducerSendTest testSendOffset PASSED kafka.api.test.ProducerSendTest testClose PASSED kafka.api.test.ProducerSendTest testSendToPartition PASSED kafka.api.test.ProducerSendTest
[jira] [Commented] (KAFKA-1360) Configure continuous integration (CI) for Kafka branches under active development
[ https://issues.apache.org/jira/browse/KAFKA-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963752#comment-13963752 ] Jun Rao commented on KAFKA-1360: A Jenkins job is now set up on for trunk and will be triggered on every commit (within an hour). https://builds.apache.org/job/Kafka-trunk/ Configure continuous integration (CI) for Kafka branches under active development - Key: KAFKA-1360 URL: https://issues.apache.org/jira/browse/KAFKA-1360 Project: Kafka Issue Type: Improvement Components: packaging Reporter: Clark Breyman Priority: Minor Labels: build, ci Original Estimate: 72h Remaining Estimate: 72h Having continuous integration configured for Kafka would help the project by - ensuring that developers are notified when the build or tests break - automates the publication of test coverage and performance information - simplifies the process of publishing nightlies to both maven central and the package download site (which also makes it easier to get feedback on pre-release builds) See http://ci.apache.org for Apache managed options. Jenkins seems the most appealing. -- This message was sent by Atlassian JIRA (v6.2#6252)
Kafka Jenkins job for unit tests
We now have a Jenkins job set up for trunk. It will run testAll and jarAll after every commit (within an hour). If the test fails, an email will be sent to the dev mailing list. https://builds.apache.org/job/Kafka-trunk/ The job now fails due to unit test failures. Thanks, Jun
Re: Kafka Jenkins job for unit tests
Super cool. Thanks for setting this up, Jun! On Tue, Apr 8, 2014 at 8:23 PM, Jun Rao jun...@gmail.com wrote: We now have a Jenkins job set up for trunk. It will run testAll and jarAll after every commit (within an hour). If the test fails, an email will be sent to the dev mailing list. https://builds.apache.org/job/Kafka-trunk/ The job now fails due to unit test failures. Thanks, Jun
Re: Review Request 20130: Patch for KAFKA-1373
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/#review39862 --- Ship it! Ship It! - Neha Narkhede On April 8, 2014, 11:18 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/ --- (Updated April 8, 2014, 11:18 p.m.) Review request for kafka. Bugs: KAFKA-1373 https://issues.apache.org/jira/browse/KAFKA-1373 Repository: kafka Description --- Set first dirty (uncompacted) offset to first offset of the log if no checkpoint exists. Diffs - core/src/main/scala/kafka/log/LogCleanerManager.scala 6a981345fd7dc711e6c4f058a42bd7f8f9d350fe core/src/main/scala/kafka/server/KafkaConfig.scala b0506d4881176cd95914847c4275e428a8d2ca10 core/src/main/scala/kafka/server/OffsetCheckpoint.scala 19f61a9718a7f8e6f9bf743ac9eb2118d0fdd9b0 core/src/main/scala/kafka/server/OffsetManager.scala 89a88a7e4055f804429b64e85b2f65312d1e2155 core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 9aeb69d4934b9aa94e0899af14fc27e7c20f039f Diff: https://reviews.apache.org/r/20130/diff/ Testing --- Thanks, Joel Koshy
Re: Review Request 20130: Patch for KAFKA-1373
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/#review39863 --- Ship it! Ship It! - Timothy Chen On April 8, 2014, 11:18 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20130/ --- (Updated April 8, 2014, 11:18 p.m.) Review request for kafka. Bugs: KAFKA-1373 https://issues.apache.org/jira/browse/KAFKA-1373 Repository: kafka Description --- Set first dirty (uncompacted) offset to first offset of the log if no checkpoint exists. Diffs - core/src/main/scala/kafka/log/LogCleanerManager.scala 6a981345fd7dc711e6c4f058a42bd7f8f9d350fe core/src/main/scala/kafka/server/KafkaConfig.scala b0506d4881176cd95914847c4275e428a8d2ca10 core/src/main/scala/kafka/server/OffsetCheckpoint.scala 19f61a9718a7f8e6f9bf743ac9eb2118d0fdd9b0 core/src/main/scala/kafka/server/OffsetManager.scala 89a88a7e4055f804429b64e85b2f65312d1e2155 core/src/test/scala/unit/kafka/log/LogCleanerIntegrationTest.scala 9aeb69d4934b9aa94e0899af14fc27e7c20f039f Diff: https://reviews.apache.org/r/20130/diff/ Testing --- Thanks, Joel Koshy
