The kafka brokers have a maximum message size limit, this is a protection measure and avoids sending monster messages to kafka.
You have two options: 1. On the brokers, increase the max.request.size, default is at ~2mb, making it 5 or even 10 is not an issue normally. Java applications can happily handle this size. 2. breakup you're records into smaller size. If you haven't enabled compression, enable compression in your client configuration. You should also probably inspect this particular message that goes over 2mb, if its web data sometimes bogus hrefs with looong query strings that make no sense can be dropped in a pre processor before sending to kafka. If its real custom data, you should try to rethink your data model and spend some time on finding the right data format that provides best compression and speed (e.g GPB, Avro, JSON + GZIP), csv is almost never what you want. Hope this helps. Regards, Gerrit On Fri, Jan 6, 2017 at 8:12 AM, zhilong wang <wangzhilong0...@gmail.com> wrote: > flume-kafka-sink send message to kafka-0.9 > > > > > 06 Jan 2017 13:13:36,595 ERROR > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.SinkRunner$PollingRunner.run:158) - Unable to deliver > event. Exception follows > org.apache.flume.EventDeliveryException: Failed to publish events > at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:252) > at > org.apache.flume.sink.DefaultSinkProcessor.process( > DefaultSinkProcessor.java:67) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.RecordTooLargeException: The message is > 2742363 bytes when serialized which is larger than th > e maximum request size you have configured with the max.request.size > configuration. > at > org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>( > KafkaProducer.java:686) > at > org.apache.kafka.clients.producer.KafkaProducer.send( > KafkaProducer.java:449) > at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:212) > ... 3 more > Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The > message is 2742363 bytes when serialized which is larger than the maximum > request size you have configure > d with the max.request.size configuration. >
