Hi Rajiv,
Did you ever find out what was causing this issue?
I notice something similar on my Kafka Cluster only that the 95th percentile
of the log flush goes above 10 or 20 Min, and then I start to see under
replicated partitions.
Besides the difference on time, the scenario is quite the
If you've noticed the default values of the above configuration, it's
Long.MAX_VALUE.
This is set to discourage the users not to edit / re-configure it. The
above configuration
is to flush the messages from the cache to the disk (fsync). Kafka
delegates the task of
flushing the messages to disk to
During my Kafka installation, I got some questions with some of the
parameter configurations
I see that log.flush.interval.messages and log.flush.interval.ms are
commented out in the default kafka server properties file. I read two
conflicting statements about these parameters. In one place, I r
Hi,
I've been reading on log flush recommendations today and I have a
question:
Up until now I've been basing my production configuration for the log
flush on this: http://kafka.apache.org/documentation/#prodconfig
It works fine, but then I saw here
http://kafka.apache.org/doc
We monitor the log flush latency p95 on all our Kafka nodes and
occasionally we see it creep up from the regular figure of under 15 ms to
above 150 ms.
Restarting the node usually doesn't help. It seems to fix itself over time
but we are not quite sure about the underlying reason. It'
Thanks Steve. I followed your suggestion to get the topics.
What is weird is that the bad broker does not get any more traffic
(messages or bytes) when this happens. Also I have more than 2 G (out of
28G) free memory according to collectd and running vmstat on the box, so I
hope that things don't
There may be more elegant ways to do this, but I'd think that you could just
ls all the directories specified in log.dirs in your server.properties file for
Kafka. You should see directories for each topicname-partitionnumber there.
Offhand it sounds to me like maybe something's evicting
I have a particular broker(version 0.8.2.1) in a cluster receiving about
15000 messages/second of around 100 bytes each (bytes-in / messages-in).
This broker has bursts of really high log flush latency p95s. The latency
sometimes goes to above 1.5 seconds from a steady state of < 20 ms.
Runn
Just an update. We moved all the partitions from the one topic that
generated most of the 545 thousand messages/second to their own set of
brokers. The old set of 9 brokers now only get 135 thousand
messages/second or 15 thousand messages/sec/broker. We are still seeing the
same log flush time
and kafka-messages-in are pretty much the same across
servers which seems to suggest that there isn't much imbalance. Our average
message size (kafka-bytes-in / kafka-messages-in) is around 80 bytes for
all brokers.
The 95th percentile of the log flush times on each broker is usually 20
tiple redundant replicas to manage your durability concerns, if you can.
>
> B
>
> > On 7 Aug 2015, at 05:49, Tao Feng wrote:
> >
> > Hi ,
> >
> > I am trying to understand the Kafka log flush behavior. My understanding
> is
> > when the broker spec
.messages. I should add that this isn’t generally the best
approach. You’ll get much better performance if you use multiple redundant
replicas to manage your durability concerns, if you can.
B
> On 7 Aug 2015, at 05:49, Tao Feng wrote:
>
> Hi ,
>
> I am trying to understand the
Hi ,
I am trying to understand the Kafka log flush behavior. My understanding is
when the broker specifies broker config param "log.flush.interval.ms", it
will specify log config param "flush.ms" internally. In logManager logic,
when the log exceed flush.ms, it will call Log.f
I have a single broker in a cluster of 9 brokers that has a
log-flush-time-99th of 260 ms or more. Other brokers have
a log-flush-time-99th of less than 30 ms. The misbehaving broker is running
on the same kind of machine (c3.4x on Ec2) that the other ones are running
on. It's bytes-in, byte
Filed: https://issues.apache.org/jira/browse/KAFKA-839
By the way, the jira queue needs to be updated to know that 0.7.2 is now a
released version, etc.
Jason
On Thu, Mar 28, 2013 at 7:58 AM, Jun Rao wrote:
> Jason,
>
> Could you file a jira so that we can track it?
>
> Thanks,
>
> Jun
>
> O
Jason,
Could you file a jira so that we can track it?
Thanks,
Jun
On Thu, Mar 28, 2013 at 12:43 AM, Jason Rosenberg wrote:
> It looks like there is a race condition between the settings for the 2
> properties: log.default.flush.scheduler.interval.ms &
> log.default.flush.interval.ms. I'm us
It looks like there is a race condition between the settings for the 2
properties: log.default.flush.scheduler.interval.ms &
log.default.flush.interval.ms. I'm using 0.7.2.
By default, both of these get set to 3000ms (and in the docs, it
recommends setting flushInterval to be a multiple of the
f
17 matches
Mail list logo
