Well, the batch size is decided by the value set for the property : "batch.size"; "The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records."
Also it may happen that message size may increase due to compression which is kind of weird. Thanks, Mayuresh On Tue, May 12, 2015 at 4:40 PM, Jamie X <[email protected]> wrote: > Hi, > > I'm wondering when you call kafka.javaapi.Producer.send() with a list of > messages, and also have compression on (snappy in this case), how does it > decide how many messages to put together as one? > > The reason I'm asking is that even though my messages are only 70kb > uncompressed, the broker complains that I'm hitting the 1mb message limit > such as: > > > kafka.common.MessageSizeTooLargeException: Message size is 1035608 bytes > which exceeds the maximum configured message size of 1000012. > at > kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:378) > at > kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:361) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) > at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:361) > at kafka.log.Log.append(Log.scala:257) > at > > kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379) > at > > kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365) > at kafka.utils.Utils$.inLock(Utils.scala:535) > at kafka.utils.Utils$.inReadLock(Utils.scala:541) > at > kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365) > at > > kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291) > at > > kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > > Thanks, > Jamie > -- -Regards, Mayuresh R. Gharat (862) 250-7125
