I am seeing this in error in producer logs after upgrading broker to
0.10.0 from 0.8 version , In producer side i still use 0..8 version of
producer api. Any suggestions to fix this. ?.
2016-08-19 22:18:30.598 c.y.a.t.s.o.KafkaMessageSender pool-10-thread-5
[WARN] caught exception when sending msg to kafka
kafka.common.QueueFullException: Event queue is full of unsent messages,
could not send event: KeyedMessage(test_stats_raw,null,null,{"event_time":
1471645110598, "source": "test", "entity": "SEGMENT", "entity_id":
[4343434343], "colo": "aaa", "phase": "OUTPUT", "debug_info": null,
"user_id": "ewewew", "user_id_type": "cxcx", "count": 2})
at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Producer.scala:113)
~[stormjar.jar:0.4.429]
at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Producer.scala:91)
~[stormjar.jar:0.4.429]
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
~[stormjar.jar:0.4.429]
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
~[stormjar.jar:0.4.429]
at kafka.producer.Producer.asyncSend(Producer.scala:91)
~[stormjar.jar:0.4.429]
at kafka.producer.Producer.send(Producer.scala:78) ~[stormjar.jar:0.4.429]
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
~[stormjar.jar:0.4.429]
at
com.go.ads.targeting.stats.output.KafkaMessageSender.write(KafkaMessageSender.java:69)
[stormjar.jar:0.4.429]
which parameter i need to tune in broker config to fix this.
I use all default values in broker configs.
broker.id=1
log.message.format.version=0.9.0.X
inter.broker.protocol.version=0.9.0.X
listeners=SSL://:9093
num.network.threads=24
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/home/y/logs/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=4
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.retention.bytes=1073741824
log.segment.bytes=1288490188
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=hostname:2181
zookeeper.connection.timeout.ms=120000
delete.topic.enable=false
default.replication.factor=1
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=300
leader.imbalance.per.broker.percentage=10
broker.rack=abc1
broker.id.generation.enable=true
controlled.shutdown.enable=true
controlled.shutdown.max.retries=3
controlled.shutdown.retry.backoff.ms=5000
Thanks,
Bastian
