It may be possibly caused by this issue https://issues.apache.org/jira/browse/KAFKA-2281. Can you apply the patch and try again?
On Thu, 6 Aug 2015 at 04:03 Liju John <lijubj...@gmail.com> wrote: > Just to add more info - > > Our message size = 1.5MB , so effectively there was no batching as our > batch size is 200. > > Is there any case when the RecordAccumulator can grow beyond the configured > buffer.memory ? > > Regards, > Liju John > > On Wed, Aug 5, 2015 at 12:41 PM, Liju John <lijubj...@gmail.com> wrote: > > > Hi, > > > > We are experiencing an issue where the RecordAccumulator size is growing > > beyond the configured buffer.memory size . > > > > Below is the sequence of events when this issue occured - > > > > 1. One of the broker out of 10 brokers was brought down at 15:10 PM > > 2.Till 17:11 PM (same date) the kafka producer instance was in healthy > > state > > 3. The same down broker was brought up at ~17:11PM ( same date) > > 4. Observed UnknownTopicOrPartitionException for some messages > > 5. HeapDump on 17:14 shows the RecordAccumulator size around ~2.9 GB way > > above the configured buffer.size =120MB > > > > > > Other Details - > > > > kafka version = 0.8.2.1 > > > > Below is the kafka producer properties configured - > > > > kafka.bootstrap.servers=qa-vip-url > > kafka.acks=1 > > kafka.buffer.memory=125829120 > > kafka.compression.type=none > > kafka.retries=3 > > kafka.batch.size=200 > > kafka.client.id=service-instance-id > > kafka.linger.ms=0 > > kafka.max.request.size=5242880 > > kafka.receive.buffer.bytes=32768 > > kafka.send.buffer.bytes=131072 > > kafka.timeout.ms=3000 > > kafka.block.on.buffer.full=false > > kafka.metadata.fetch.timeout.ms=60000 > > kafka.metadata.max.age.ms=300000 > > kafka.reconnect.backoff.ms=30 > > kafka.retry.backoff.ms=3000 > > > > > kafka.key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer > > > > > kafka.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer > > > > > > Please help in identifing the root cause which is causing this memory > leak > > for RecordAccumulator. > > > > Attached is the HeapDump overview for reference > > > > Let me know if you need any other information > > > > Regards, > > Liju John > > >
