High CPU Usage 0.8.1.2 on a kafka client
System: jdk-7u76 (Oracle), on Ubuntu 14.04.1 LTS, Trusty Tahr, Kernel: 3.13.0-44-generic Kafka Version: 0.8.1.2 Broker Cfg: Three brokers, Zookeeper Cfg: Three brokers, Replication Factor: 2 Client Machine Configuration:: - Memory: 4G - CPU: 2 (Xeon @ 2.50GHz) - Java args:: -server -Xmx2G -Xms2G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dorg.jboss.netty.epollBugWorkaround -Xrunhprof:cpu=samples,depth=30,file=/tmp/java.hprof Folks, I'll appreciate your help in trying to understand high CPU usage by our kafka clients. We are running four simple java consumers within the same java process. One of these consumers uses ten streams (off 50 topics/partitions) consumed by ten worker threads and the rest of the consumers (three with four independent topics/partitions each) draw three individual streams fed into three independent worker threads respectively. The overall messaging traffic is very low (around two messages/sec). The process runs fine for some time but after few minutes of normal run it escalates into a high CPU mode. Following is a 10 minutes of trace showing the reason for high CPU -8- CPU SAMPLES BEGIN (total = 1426307) Fri May 1 18:34:37 2015 rank selfaccum count trace method 1 50.26% 50.26% 716803 306229 sun.nio.ch.EPollArrayWrapper.epollWait (kafka consumer see TRACE # below) 2 22.27% 72.53% 317632 302942 sun.nio.ch.EPollArrayWrapper.epollWait (netty related see TRACE # below) 3 21.85% 94.38% 311678 302855 sun.nio.ch.EPollArrayWrapper.epollWait (zookeeper client see TRACE # below) 4 2.78% 97.16%39709 302957 sun.nio.ch.EPollArrayWrapper.epollWait 5 2.78% 99.95%39706 304668 sun.nio.ch.EPollArrayWrapper.epollWait Where:: TRACE 306229: sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown line) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:221) sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) kafka.utils.Utils$.read(Utils.scala:375) kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) kafka.network.Receive$class.readCompletely(Transmission.scala:56) kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) kafka.network.BlockingChannel.receive(BlockingChannel.scala:100) kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73) kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71) kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109) kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109) kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109) kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108) kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108) kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108) kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107) kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96) kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88) kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) TRACE 302855: sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown line) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1134) TRACE 302942: sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown line) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select
Re: 0.8.1.2
Does 0.8.2 includes new producer which mentioned at the documentation of kafka? If not, which version will include it? Thanks, Mungeol On Wed, Oct 22, 2014 at 11:21 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Shlomi, As Jun mentioned, we are voting on a 0.8.2 beta release now. Are you suggesting there be an 0.8.1.2 release in addition to that? We can take a quick vote from the community to see how many people prefer to have this and why. Thanks, Neha On Tue, Oct 21, 2014 at 6:03 PM, Jun Rao jun...@gmail.com wrote: We are voting an 0.8.2 beta release right now. Thanks, Jun On Tue, Oct 21, 2014 at 11:17 AM, Shlomi Hazan shl...@viber.com wrote: Hi All, Will version 0.8.1.2 happen? Shlomi
Re: 0.8.1.2
Yes, 0.8.2 includes the new producer. 0.8.2 will have a lot of new features which will take time to stabilize. If people want 0.8.1.2 for some critical bug fixes, we can discuss the feasibility of doing the release. On Wed, Oct 22, 2014 at 1:39 AM, Shlomi Hazan shl...@viber.com wrote: at the time I thought it was a good idea but if I understand correctly what Jun is saying is that 0.8.1.2 will not happen. I assume Jun sees 0.8.2 coming soon enough to remove any added value from 0.8.1.2. Shlomi On Wed, Oct 22, 2014 at 5:21 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Shlomi, As Jun mentioned, we are voting on a 0.8.2 beta release now. Are you suggesting there be an 0.8.1.2 release in addition to that? We can take a quick vote from the community to see how many people prefer to have this and why. Thanks, Neha On Tue, Oct 21, 2014 at 6:03 PM, Jun Rao jun...@gmail.com wrote: We are voting an 0.8.2 beta release right now. Thanks, Jun On Tue, Oct 21, 2014 at 11:17 AM, Shlomi Hazan shl...@viber.com wrote: Hi All, Will version 0.8.1.2 happen? Shlomi
0.8.1.2
Hi All, Will version 0.8.1.2 happen? Shlomi
Re: 0.8.1.2
We are voting an 0.8.2 beta release right now. Thanks, Jun On Tue, Oct 21, 2014 at 11:17 AM, Shlomi Hazan shl...@viber.com wrote: Hi All, Will version 0.8.1.2 happen? Shlomi
Re: [DISCUSS] 0.8.1.2 Release
Having the same question: what happened to 0.8.2 release, when it's supposed to happen? Thanks. On Tue, Sep 30, 2014 at 12:49 PM, Jonathan Weeks jonathanbwe...@gmail.com wrote: I was one asking for 0.8.1.2 a few weeks back, when 0.8.2 was at least 6-8 weeks out. If we truly believe that 0.8.2 will go “golden” and stable in 2-3 weeks, I, for one, don’t need a 0.8.1.2, but it depends on the confidence in shipping 0.8.2 soonish. YMMV, -Jonathan On Sep 30, 2014, at 12:37 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein joe.st...@stealth.ly wrote: Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
Re: [DISCUSS] 0.8.1.2 Release
We already cut an 0.8.2 release branch. The plan is to have the remaining blockers resolved before releasing it. Hopefully this will just take a couple of weeks. https://issues.apache.org/jira/browse/KAFKA-1663?filter=-4jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%200.8.2%20ORDER%20BY%20createdDate%20DESC Thanks, Jun On Thu, Oct 2, 2014 at 11:28 AM, Kane Kane kane.ist...@gmail.com wrote: Having the same question: what happened to 0.8.2 release, when it's supposed to happen? Thanks. On Tue, Sep 30, 2014 at 12:49 PM, Jonathan Weeks jonathanbwe...@gmail.com wrote: I was one asking for 0.8.1.2 a few weeks back, when 0.8.2 was at least 6-8 weeks out. If we truly believe that 0.8.2 will go “golden” and stable in 2-3 weeks, I, for one, don’t need a 0.8.1.2, but it depends on the confidence in shipping 0.8.2 soonish. YMMV, -Jonathan On Sep 30, 2014, at 12:37 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein joe.st...@stealth.ly wrote: Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
[DISCUSS] 0.8.1.2 Release
Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
Re: [DISCUSS] 0.8.1.2 Release
Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein joe.st...@stealth.ly wrote: Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
RE: [DISCUSS] 0.8.1.2 Release
In DISH we are having issues in 0.8-beta version used in PROD, it's crashing every 2 days and becoming a blocker for us. It would be great if we get 0.8.2 or 0.8.1.2 whichever is faster as we can't wait for 3 weeks as our new Order Management system is going to sit on top of Kafka. -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Tuesday, September 30, 2014 1:37 PM To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject: Re: [DISCUSS] 0.8.1.2 Release Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein joe.st...@stealth.ly wrote: Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
Re: [DISCUSS] 0.8.1.2 Release
I was one asking for 0.8.1.2 a few weeks back, when 0.8.2 was at least 6-8 weeks out. If we truly believe that 0.8.2 will go “golden” and stable in 2-3 weeks, I, for one, don’t need a 0.8.1.2, but it depends on the confidence in shipping 0.8.2 soonish. YMMV, -Jonathan On Sep 30, 2014, at 12:37 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 0.8.2? On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein joe.st...@stealth.ly wrote: Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release. Here are the JIRAs I would like to propose to back port a patch (if not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty) https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala 2.11) https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on partition state update failures) https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial setup output from source distribution) https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src release) If the community and committers can comment on the patches proposed that would be great. If I missed any bring them up or if you think any I have proposed shouldn't be int he release bring that up too please. Once we have consensus on this thread my thought was that I would apply and commit the agreed to tickets to the 0.8.1 branch. If any tickets don't apply of course a back port patch has to happen through our standard process (not worried about that we have some engineering cycles to contribute to making that happen). Once that is all done, I will build 0.8.1.2 release artifacts and call a VOTE for RC1. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
