You need to configure the Kafka broker to allow you to send larger messages.
The relevant parameters are:
message.max.bytes (default:1000000) – Maximum size of a message the
broker will accept. This has to be smaller than the consumer
fetch.message.max.bytes, or the broker will have messages that can’t
be consumed, causing consumers to hang.
replica.fetch.max.bytes (default: 1MB) – Maximum size of data that a
broker can replicate. This has to be larger than message.max.bytes, or
a broker will accept messages and fail to replicate them. Leading to
potential data loss.
Gwen
On Sat, Jul 11, 2015 at 9:08 PM, David Montgomery
<davidmontgom...@gmail.com> wrote:
> I cant send this soooo simple payload using python.
>
> topic: topic-test-development
> payload: {"utcdt": "2015-07-12T03:59:36", "ghznezzhmx": "apple"}
>
>
> No handlers could be found for logger "kafka.conn"
> Traceback (most recent call last):
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 81,
> in <module>
> test_send_data_to_realtimenode()
> File "/home/ubuntu/workspace/feed-tests/tests/druid-adstar.py", line 38,
> in test_send_data_to_realtimenode
> response = producer.send_messages(test_topic,test_payload)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/simple.py",
> line 54, in send_messages
> topic, partition, *msg
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 349, in send_messages
> return self._send_messages(topic, partition, *msg)
> File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py",
> line 390, in _send_messages
> fail_on_error=self.sync_fail_on_error
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 480,
> in send_produce_request
> (not fail_on_error or not self._raise_on_response_error(resp))]
> File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247,
> in _raise_on_response_error
> raise resp
> kafka.common.FailedPayloadsError
>
> Here is what is in my logs
> [2015-07-12 03:29:58,103] INFO Closing socket connection to
> /xxx.xxx.xxx.xxx due to invalid request: Request of length 1550939497 is
> not valid, it is larger than the maximum size of 104857600 bytes.
> (kafka.network.Processor)
>
>
>
> Server is 4 gigs of ram.
>
> I used export KAFKA_HEAP_OPTS=-Xmx256M -Xms128M in kafka-server-start.sh
>
> So.....why?
