Mayuresh, this is about the old producer instead of the new Java producer. Jamie, In the old producer, if you use sync mode, the list of message will be sent as a batch. On the other hand, if you are using async mode, the messages are just put into the queue and batched with other messages. Notice that the old producer uses number of messages as batch limitation instead of number of bytes.
But in your case, it seems you have a single message whose compressed size is larger than the max message size Kafka broker accepts. Any idea why? Thanks. Jiangjie (Becket) Qin On 5/12/15, 9:11 PM, "Mayuresh Gharat" <[email protected]> wrote: >Well, the batch size is decided by the value set for the property : > > "batch.size"; > "The producer will attempt to batch records together into fewer requests >whenever multiple records are being sent to the same partition. This helps >performance on both the client and the server. This configuration controls >the default batch size in bytes. No attempt will be made to batch records >larger than this size. Requests sent to brokers will contain multiple >batches, one for each partition with data available to be sent. A small >batch size will make batching less common and may reduce throughput (a >batch size of zero will disable batching entirely). A very large batch >size >may use memory a bit more wastefully as we will always allocate a buffer >of >the specified batch size in anticipation of additional records." > >Also it may happen that message size may increase due to compression which >is kind of weird. > >Thanks, > >Mayuresh > >On Tue, May 12, 2015 at 4:40 PM, Jamie X <[email protected]> wrote: > >> Hi, >> >> I'm wondering when you call kafka.javaapi.Producer.send() with a list of >> messages, and also have compression on (snappy in this case), how does >>it >> decide how many messages to put together as one? >> >> The reason I'm asking is that even though my messages are only 70kb >> uncompressed, the broker complains that I'm hitting the 1mb message >>limit >> such as: >> >> >> kafka.common.MessageSizeTooLargeException: Message size is 1035608 bytes >> which exceeds the maximum configured message size of 1000012. >> at >> >>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:378 >>) >> at >> >>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:361 >>) >> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >> at >>kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) >> at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:361) >> at kafka.log.Log.append(Log.scala:257) >> at >> >> >>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition >>.scala:379) >> at >> >> >>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition >>.scala:365) >> at kafka.utils.Utils$.inLock(Utils.scala:535) >> at kafka.utils.Utils$.inReadLock(Utils.scala:541) >> at >> kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365) >> at >> >> >>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala: >>291) >> at >> >> >>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala: >>282) >> at >> >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at >> >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at >> >>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9 >>8) >> at >> >>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9 >>8) >> at >> >>scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226 >>) >> at >>scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> >> Thanks, >> Jamie >> > > > >-- >-Regards, >Mayuresh R. Gharat >(862) 250-7125
