Quotas feature Kafka 0.9.0.1

[Liju John]

Hi ,

We are exploring the new quotas feature with Kafka 0.9.01.
Could you please let me know if quotas feature works for fetch follower as
well ?
We see that when a broker is down for a long time and brought back , the
replica catches up aggressively , impacting the whole cluster.
Would it be possible to throttle Fetch follower as well with quotas?


Regards,
Liju John

Kafka RecordAccumulator size grows beyond configured buffer.memory size

[Liju John]

Hi,

We are experiencing an issue where the RecordAccumulator size is growing
beyond the configured buffer.memory size .

Below is the sequence of events when this issue occured -

1. One of the broker out of 10 brokers  was brought down at 15:10 PM
2.Till 17:11 PM (same date) the kafka producer instance was in healthy state
3. The same down broker was brought up at ~17:11PM ( same date)
4. Observed UnknownTopicOrPartitionException for some messages
5. HeapDump on  17:14 shows the RecordAccumulator size around ~2.9 GB way
above the configured buffer.size =120MB


Other Details  -

kafka version = 0.8.2.1

Below is the kafka producer properties configured  -

kafka.bootstrap.servers=qa-vip-url
kafka.acks=1
kafka.buffer.memory=125829120
kafka.compression.type=none
kafka.retries=3
kafka.batch.size=200
kafka.client.id=service-instance-id
kafka.linger.ms=0
kafka.max.request.size=5242880
kafka.receive.buffer.bytes=32768
kafka.send.buffer.bytes=131072
kafka.timeout.ms=3000
kafka.block.on.buffer.full=false
kafka.metadata.fetch.timeout.ms=6
kafka.metadata.max.age.ms=30
kafka.reconnect.backoff.ms=30
kafka.retry.backoff.ms=3000
kafka.key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
kafka.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer


Please help in identifing the root cause which is causing this memory leak
for RecordAccumulator.

Attached is the HeapDump overview for reference

Let me know if you need any other information

Regards,
Liju John

Ques regarding topic partition offset

[Liju John]

Hi ,

I have query regarding partition offset .

While running kafka cluster for some time ,I noticed that the partition
offset keeps on increasing and at some point the offset decreased by some
number .
In what  scenarios does the offset of a topic partition reduces ?

The problem I am facing is that my consumers are pulling the msgs from the
topic but at somepoint in time it throws exception that the current offset
is greater that the latest offset of the partition .
Is it because of retension the latest offset gets reset ? How can I handle
this scenarios

Regards,
Liju John
S/W developer

Query regarding serialization

[Liju John]

Hi,

I am new to kafka and still learning .I have a query .. As per my
understanding the serialization is happening before the partitioning and
grouping of messages per broker . Is my understanding correct and what is
the reason for the same?

Regards,
Liju John

Query regarding Kafka publishing

[Liju John]

Hi ,

I have a general query -

As per the code in Kafka producer  the serialization happens before
partitioning , Is my understanding correct ? If yes whats the reason for it
?

Regards,
Liju John