[
https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177478#comment-14177478
]
Jun Rao commented on KAFKA-1718:
--------------------------------
Another thing that we do on the broker is to assign a new offset to each
(uncompressed) message and recompress those messages again. It may be possible
that because of the newly assigned offsets, the new message set doesn't
compress as well as before and thus exceeds the limit.
> "Message Size Too Large" error when only small messages produced with Snappy
> ----------------------------------------------------------------------------
>
> Key: KAFKA-1718
> URL: https://issues.apache.org/jira/browse/KAFKA-1718
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.1.1
> Reporter: Evan Huus
> Priority: Critical
>
> I'm the primary author of the Go bindings, and while I originally received
> this as a bug against my bindings, I'm coming to the conclusion that it's a
> bug in the broker somehow.
> Specifically, take a look at the last two kafka packets in the following
> packet capture: https://dl.dropboxusercontent.com/u/171647/kafka.pcapng (you
> will need a trunk build of Wireshark to fully decode the kafka part of the
> packets).
> The produce request contains two partitions on one topic. Each partition has
> one message set (sizes 977205 bytes and 967362 bytes respectively). Each
> message set is a sequential collection of snappy-compressed messages, each
> message of size 46899. When uncompressed, each message contains a message set
> of 999600 bytes, containing a sequence of uncompressed 1024-byte messages.
> However, the broker responds to this with a MessageSizeTooLarge error, full
> stacktrace from the broker logs being:
> kafka.common.MessageSizeTooLargeException: Message size is 1070127 bytes
> which exceeds the maximum configured message size of 1000012.
> at kafka.log.Log$$anonfun$append$1.apply(Log.scala:267)
> at kafka.log.Log$$anonfun$append$1.apply(Log.scala:265)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
> at kafka.log.Log.append(Log.scala:265)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:354)
> at
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:376)
> at
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:366)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:366)
> at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:292)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:185)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:695)
> Since as far as I can tell none of the sizes in the actual produced packet
> exceed the defined maximum, I can only assume that the broker is
> miscalculating something somewhere and throwing the exception improperly.
> ---
> This issue can be reliably reproduced using an out-of-the-box binary download
> of 0.8.1.1 and the following gist:
> https://gist.github.com/eapache/ce0f15311c605a165ce7 (you will need to use
> the `producer-ng` branch of the Sarama library).
> ---
> I am happy to provide any more information you might need, or to do relevant
> experiments etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
