[
https://issues.apache.org/jira/browse/KAFKA-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877616#comment-15877616
]
Huadong Liu commented on KAFKA-4762:
------------------------------------
Hi [~huxi_2b], you are right. The RecordTooLargeException happened on our
0.9.0.1 consumer. How did you tell that compression is enabled on the producer?
Our python producer derives from the SimpleProducer. It does not enable
compression
https://github.com/dpkp/kafka-python/blob/8fde79dbb5a3793b1a9ebd10e032d5f3dd535645/kafka/producer/base.py#L281.
The log dump [~neoeahit] posted shows "compresscodec: NoCompressionCodec".
What did you mean by "the whole compressed message is often much larger than a
single one"? Were you talking about batching? Thanks for your insights on this.
> Consumer throwing RecordTooLargeException even when messages are not that
> large
> -------------------------------------------------------------------------------
>
> Key: KAFKA-4762
> URL: https://issues.apache.org/jira/browse/KAFKA-4762
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.1
> Reporter: Vipul Singh
>
> We were just recently hit by a weird error.
> Before going in any further, explaining of our service setup. we have a
> producer which produces messages not larger than 256 kb of messages( we have
> an explicit check about this on the producer side) and on the client side we
> have a fetch limit of 512kb(max.partition.fetch.bytes is set to 524288 bytes)
> Recently our client started to see this error:
> {quote}
> org.apache.kafka.common.errors.RecordTooLargeException: There are some
> messages at [Partition=Offset]: {topic_name-0=9925056036} whose size is
> larger than the fetch size 524288 and hence cannot be ever returned. Increase
> the fetch size, or decrease the maximum message size the broker will allow.
> {quote}
> We tried consuming messages with another consumer, without any
> max.partition.fetch.bytes limit, and it consumed fine. The messages were
> small, and did not seem to be greater than 256 kb
> We took a log dump, and the log size looked fine.
> {quote}
> mpresscodec: NoCompressionCodec crc: 2473548911 keysize: 8
> offset: 9925056032 position: 191380053 isvalid: true payloadsize: 539 magic:
> 0 compresscodec: NoCompressionCodec crc: 1656420267 keysize: 8
> offset: 9925056033 position: 191380053 isvalid: true payloadsize: 1551 magic:
> 0 compresscodec: NoCompressionCodec crc: 2398479758 keysize: 8
> offset: 9925056034 position: 191380053 isvalid: true payloadsize: 1307 magic:
> 0 compresscodec: NoCompressionCodec crc: 2845554215 keysize: 8
> offset: 9925056035 position: 191380053 isvalid: true payloadsize: 1520 magic:
> 0 compresscodec: NoCompressionCodec crc: 3106984195 keysize: 8
> offset: 9925056036 position: 191713371 isvalid: true payloadsize: 1207 magic:
> 0 compresscodec: NoCompressionCodec crc: 3462154435 keysize: 8
> offset: 9925056037 position: 191713371 isvalid: true payloadsize: 418 magic:
> 0 compresscodec: NoCompressionCodec crc: 1536701802 keysize: 8
> offset: 9925056038 position: 191713371 isvalid: true payloadsize: 299 magic:
> 0 compresscodec: NoCompressionCodec crc: 4112567543 keysize: 8
> offset: 9925056039 position: 191713371 isvalid: true payloadsize: 1571 magic:
> 0 compresscodec: NoCompressionCodec crc: 3696994307 keysize: 8
> {quote}
> Has anyone seen something similar? or any points to troubleshoot this further
> Please Note: To overcome this issue, we deployed a new consumer, without this
> limit of max.partition.fetch.bytes, and it worked fine.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
