Kafka documentation recommends <0.3 for above metric. I assume processor is
busier if this goes below 0.3 and obviously it being < 0.3 for long does
not seem to be a good sign.
What should be our criteria to raise an alert, I though it should be when
its value goes below 0.3. However, the value s
Could you try the 0.8.2.1 release being voted on now? It fixes a CPU issue
and should reduce the CPU load in network thread.
Thanks,
Jun
On Thu, Feb 19, 2015 at 11:54 AM, Zakee wrote:
> Kafka documentation recommends <0.3 for above metric. I assume processor is
> busier if this goes below 0.3
Jun,
I am already using the latest release 0.8.2.1.
-Zakee
On Thu, Feb 19, 2015 at 2:46 PM, Jun Rao wrote:
> Could you try the 0.8.2.1 release being voted on now? It fixes a CPU issue
> and should reduce the CPU load in network thread.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 19, 2015 at 11:54 AM,
What kind of load do you have on the brokers? On an idle cluster (just
fetch requests from the follower replicas), I
saw NetworkProcessorAvgIdlePercent at about 97%.
Thanks,
Jun
On Thu, Feb 19, 2015 at 5:19 PM, Zakee wrote:
> Jun,
>
> I am already using the latest release 0.8.2.1.
&
Hi Jun,
With ~100G of data being pushed per hour across 35 topics
(replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly
showing below 0.5 sometimes when the producers send on a high rate.
Thanks
-Zakee
On Sun, Feb 22, 2015 at 10:29 PM, Jun Rao wrote:
> What kind of load do
What about RequestHandlerAvgIdlePercent?
Thanks,
Jun
On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
> Hi Jun,
>
> With ~100G of data being pushed per hour across 35 topics
> (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly
> showing below 0.5 sometimes whe
0G of data being pushed per hour across 35 topics
> > (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly
> > showing below 0.5 sometimes when the producers send on a high rate.
> >
> > Thanks
> > -Zakee
> >
> > On Sun, Feb 22, 2015 at 10:
gt;
> > What about RequestHandlerAvgIdlePercent?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
> >
> > > Hi Jun,
> > >
> > > With ~100G of data being pushed per hour across 35 topics
> > > (replicat
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > With ~100G of data being pushed per hour across 35 topics
> > > >
gt; > >
> > > > Jun
> > > >
> > > > On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > With ~100G of data being pushed per hour across 35 to
gt;> On Tue, Feb 24, 2015 at 2:43 PM, Jun Rao wrote:
>>>>
>>>>> What about RequestHandlerAvgIdlePercent?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Mon, Feb 23, 2015
hreads.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Tue, Feb 24, 2015 at 7:48 PM, Zakee wrote:
> >>>
> >>>> Similar pattern for that too. Mostly hovering below.
> >>>>
> >>>&g
PM, Jun Rao wrote:
>>>>
>>>>> Then you may want to consider increasing num.io.threads
>>>>> and num.network.threads.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>&g
This is my problem
1.The request queue is always at 500
2.NetworkProcessorAvgIdlePercent is lower than 0.2
This is my BROKER configuration
num.io.threads=32
num.network.threads=64
How have I identified the cause and how to optimize my KAFKA cluster?
THKS。
This is my problem
1.The request queue is always at 500
2.There are 130 machines in the cluster, and the network idle rate of 30
machines is less than 20.
This is my BROKER configuration
num.io.threads=32
num.network.threads=64
How should I locate the problem? I tried to increase the parameters
You should investigate the cause of request-queue full situation first.
Since I guess low network idle ratio is the consequence of that.
(Network-threads would block on queueing when request-queue is full)
I recommend running async-profiler to take the profile of the broker
process if possible (wi
I have a question: why does the overall CPU of the cluster decrease when
the KAFKA cluster traffic increases, the request queue is full, and the
idle rate is low?
Haruki Okada 于2024年1月15日周一 21:56写道：
> You should investigate the cause of request-queue full situation first.
> Since I guess low net
This is just a conjecture though, one possible reason I can imagine is
traffic increase => request queue full => request latency increase => more
batching on client-side => overall request-count decrease => CPU-usage
decrease.
You should check request-count metric.
At any rate, full-request-queue
18 matches
Mail list logo
