Hi Jiangjie, Resending profiling screenshot for High CPU usage on Idle kafka broker.
Appreciate your help. Thanks, Anuja On Mon, Jun 8, 2015 at 1:45 PM, pundlik.anuja <pundlik.an...@gmail.com> wrote: > Yes, partly it seems to be due to log cleaner thread. > > I changed log cleaning settings for each broker (server*.properties) file. > > > # The minimum age of a log file to be eligible for deletion > log.retention.hours=1 > > # A size-based retention policy for logs. Segments are pruned from the log > as long as the remaining > # segments don't drop below log.retention.bytes. > #log.retention.bytes=1073741824 > > # The maximum size of a log segment file. When this size is reached a new > log segment will be created. > log.segment.bytes=536870912 > > # The interval at which log segments are checked to see if they can be > deleted according > # to the retention policies > log.retention.check.interval.ms=10000 (****** This was 1 ms before) > > # By default the log cleaner is disabled and the log retention policy will > default to just delete segments after their retention expires. > # If log.cleaner.enable=true is set the cleaner will be enabled and > individual logs can then be marked for log compaction. > log.cleaner.enable=false (***** this was true before) > > > > *Question:* > Per my understanding, setting log.cleaner.enable=false will cause deletion > of logs instead of compaction. Is that correct? Or will it cause the logs > to keep on accumulating and filling up the disk space? > > > THanks, > Anuja > > On Mon, Jun 8, 2015 at 1:24 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Could it also be that the log cleaner is running? This will definitely use >> some CPU while the cleaning is occurring (it would attempt to use one cpu >> per log cleaner thread you configure). >> >> -Jay >> >> On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <j...@linkedin.com.invalid> >> wrote: >> >> > It seems the attachments are lost. But high CPU for ReplicaRetcherThread >> > might be related to KAFKA-1461. Can you try to apply that patch and see >> if >> > it solves problem? >> > >> > Thanks, >> > >> > Jiangjie (Becket) Qin >> > >> > From: "pundlik.anuja" <pundlik.an...@gmail.com<mailto: >> > pundlik.an...@gmail.com>> >> > Reply-To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" < >> > users@kafka.apache.org<mailto:users@kafka.apache.org>> >> > Date: Monday, June 8, 2015 at 12:20 PM >> > To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" < >> > users@kafka.apache.org<mailto:users@kafka.apache.org>> >> > Subject: Re: High CPU usage for idle kafka server >> > >> > Seems to be LogCleaner and FetchRequest threads. >> > Attached is the profiling screenshot >> > >> > On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <j...@linkedin.com.invalid >> > <mailto:j...@linkedin.com.invalid>> wrote: >> > Has this to do with KAFKA-1461? >> > Can you see which thread is taking a lot of cpu? Some jconsole plugin >> can >> > get that information. >> > >> > Jiangjie (Becket) Qin >> > >> > On 6/5/15, 2:57 PM, "pundlik.anuja" <pundlik.an...@gmail.com<mailto: >> > pundlik.an...@gmail.com>> wrote: >> > >> > >Hi Jay, >> > > >> > >Good to hear from you. I met you at the kafka meetup at linkedin. >> > > >> > >- No, I am running kafka_2.11-0.8.2.1 >> > > >> > > >> > >Are there any logs/ any info that I can provide that will help you >> > >understand what could be the issue? >> > > >> > >Thanks, >> > >Anuja >> > > >> > >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <jay.kr...@gmail.com<mailto: >> > jay.kr...@gmail.com>> wrote: >> > > >> > >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are >> > >>running >> > >> that pre-release version is there? >> > >> >> > >> -Jay >> > >> >> > >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) < >> > >> apund...@cisco.com<mailto:apund...@cisco.com> >> > >> > wrote: >> > >> >> > >> > Hi, >> > >> > >> > >> > I am using Kafka 0.8.2.1. >> > >> > We have 1 zookeeper, 3 kafka brokers. >> > >> > We have 9 topics, out of which 1 topic has 18 partitions, while >> > >>another >> > >> > has 12 partitions. All other topics have 1 partition each. >> > >> > >> > >> > We see that idle kafka brokers (not carrying any message) are using >> > >>more >> > >> > than 50% of CPU. See top output below. >> > >> > >> > >> > Is this a known issue? >> > >> > >> > >> > >> > >> > Thanks >> > >> > >> > >> > >> > >> > >> > >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92 >> > >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 >> zombie >> > >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, >> 0.1%si, >> > >> > 0.4%st >> > >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k >> > >>buffers >> > >> > Swap: 0k total, 0k used, 0k free, 13382460k >> > >>cached >> > >> > >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> > >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java >> > >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java >> > >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java >> > >> > >> > >> >> > >> > >> > >> > >
