Quotas feature Kafka 0.9.0.1
Hi , We are exploring the new quotas feature with Kafka 0.9.01. Could you please let me know if quotas feature works for fetch follower as well ? We see that when a broker is down for a long time and brought back , the replica catches up aggressively , impacting the whole cluster. Would it be possible to throttle Fetch follower as well with quotas? Regards, Liju John
Kafka RecordAccumulator size grows beyond configured buffer.memory size
Hi, We are experiencing an issue where the RecordAccumulator size is growing beyond the configured buffer.memory size . Below is the sequence of events when this issue occured - 1. One of the broker out of 10 brokers was brought down at 15:10 PM 2.Till 17:11 PM (same date) the kafka producer instance was in healthy state 3. The same down broker was brought up at ~17:11PM ( same date) 4. Observed UnknownTopicOrPartitionException for some messages 5. HeapDump on 17:14 shows the RecordAccumulator size around ~2.9 GB way above the configured buffer.size =120MB Other Details - kafka version = 0.8.2.1 Below is the kafka producer properties configured - kafka.bootstrap.servers=qa-vip-url kafka.acks=1 kafka.buffer.memory=125829120 kafka.compression.type=none kafka.retries=3 kafka.batch.size=200 kafka.client.id=service-instance-id kafka.linger.ms=0 kafka.max.request.size=5242880 kafka.receive.buffer.bytes=32768 kafka.send.buffer.bytes=131072 kafka.timeout.ms=3000 kafka.block.on.buffer.full=false kafka.metadata.fetch.timeout.ms=6 kafka.metadata.max.age.ms=30 kafka.reconnect.backoff.ms=30 kafka.retry.backoff.ms=3000 kafka.key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer kafka.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer Please help in identifing the root cause which is causing this memory leak for RecordAccumulator. Attached is the HeapDump overview for reference Let me know if you need any other information Regards, Liju John
Ques regarding topic partition offset
Hi , I have query regarding partition offset . While running kafka cluster for some time ,I noticed that the partition offset keeps on increasing and at some point the offset decreased by some number . In what scenarios does the offset of a topic partition reduces ? The problem I am facing is that my consumers are pulling the msgs from the topic but at somepoint in time it throws exception that the current offset is greater that the latest offset of the partition . Is it because of retension the latest offset gets reset ? How can I handle this scenarios Regards, Liju John S/W developer
Query regarding serialization
Hi, I am new to kafka and still learning .I have a query .. As per my understanding the serialization is happening before the partitioning and grouping of messages per broker . Is my understanding correct and what is the reason for the same? Regards, Liju John
Query regarding Kafka publishing
Hi , I have a general query - As per the code in Kafka producer the serialization happens before partitioning , Is my understanding correct ? If yes whats the reason for it ? Regards, Liju John