Brokers may have temporary problems catching up with the leaders. So I
would not worry about it if it happens only once a while and goes away.
Occasionally we have seen under replicated topics for long time, which
might be caused by ZooKeeper session problem as indicated by such log
messages:
[in
We restarted the Kafka brokers this morning. That fixed the issue.
Gene Robichaux
Manager, Database Operations
Match.com
8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
-Original Message-
From: Allen Wang [mailto:aw...@netflix.com.INVALID]
Sent: Tuesday, December 30, 2014 1:38 PM
To:
The produce to the local log is the time taken to write to the local
filesystem. If you see that spike up then that is the culprit.
What I was referring to is the line in kafka-request.log that looks
something like:
"Completed request:%s from client
%s;totalTime:%d,requestQueueTime:%d,localTime:%d
We have seen cases with 0.8.1 when, under load, replication threads would hang
up and not transfer data any longer. Restarting clears this.
I haven't found a way to monitor for this in a nice way, other than seeing
partitions stay under-replicated for long periods of time.
Sent from my BlackBe
Thanks for the response. We are using python to grab the JMX values and stuff
them into graphite. We noticed on some graphs that we had a server with 2
underreplicated partitions. The restarted fixed it.
Gene Robichaux
Manager, Database Operations
Match.com
8300 Douglas Avenue I Suite 800 I Dall
I am still trying to find a way to detect how far behind a replica is, nicely,
so I can differentiate between "10 offsets" and "1 offsets" behind.
This would help with problems like this one, as we often have replicas that are
just slightly behind, due to Bursty traffic, but the ones
It sounds like you are looking for FetcherLagMetrics. See
https://kafka.apache.org/documentation.html#monitoring you can (if you find
your ISR shrink/growth rate flapping) increase the max lag setting
replica.lag.max.messages (default is 4000) depending on where you see this
lag often hovering at a
I saw the same issue, here's the solution... escape the double quotes...
java -d64 -Xmx64M -jar ./jmxcmd.jar - 192.168.2.222:9005
"\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"AllTopicsMessagesInPerSec\""
Count RateUnit FiveMinuteRate FifteenMinuteRate OneMinuteRate MeanRate
Wrapping wit
Got it. I had to enable to see the logs. Here are two files with the
timings I got from the period I had logging enabled:
Producer Requests :
https://docs.google.com/spreadsheets/d/1spVFWsf7T8ZmwM0JMYnkBoyYufRhHav50sueFA9ztP8/edit?usp=sharing
Fetch Requests:
https://docs.google.com/spreadsheets/d/
Edit: I had to set kafka.request.logger=TRACE to see the request timings.
On Tue, Dec 30, 2014 at 4:37 PM, Rajiv Kurian wrote:
> Got it. I had to enable to see the logs. Here are two files with the
> timings I got from the period I had logging enabled:
> Producer Requests :
> https://docs.google
Yes, the fetch request blocks until there is data for you, so you should
see high remote time there.
For the producer you should only see remote time if you have acks=-1. This
represents the time it takes the message to replicate to other in-sync
replicas.
I agree if there are no slow requests on
My broker logs say acks = 1 unless this is some other ack:
Completed request:Name: ProducerRequest; Version: 0; CorrelationId: 858424;
ClientId: ; *RequiredAcks: 1*; AckTimeoutMs: 3 ms; TopicAndPartition:
I'll try to get a stand alone test going, and try to narrow down the
problem. Thanks for
Oh, thanks. Both solutions work.
2014-12-31 7:03 GMT+08:00 Chris Cordray :
> I saw the same issue, here's the solution... escape the double quotes...
>
> java -d64 -Xmx64M -jar ./jmxcmd.jar - 192.168.2.222:9005
> "\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"AllTopicsMessagesInPerSec\""
>
13 matches
Mail list logo
