If you are sending in sync mode, producer will just group by partition the list of messages you provided as argument of send() and send them out. You don¹t need to worry about batch.num.messages.
There is a potential that compressed message is even bigger than uncompressed message, though. I¹m not sure if this is the case you see there if you are sure your message size is only 70KB while the limitation is ~1MB. Can you send the messages one by one use the batch send - each MessageSet only has one message - and see which message caused the problem? Jiangjie (Becket) Qin On 5/13/15, 9:57 AM, "Jamie X" <[email protected]> wrote: >(sorry if this messes up the mailing list, I didn't seem to get replies in >my inbox) > >Jiangjie, I am indeed using the old producer, and on sync mode. > >> Notice that the old producer uses number of messages as batch limitation >instead of number of bytes. > >Can you clarify this? I see a setting batch.num.messages but it is only >for >async and wouldn't affect sync mode. > >> But in your case, it seems you have a single message whose compressed >size is larger than the max message size Kafka broker accepts. Any idea >why? > >I don't think this is the case as my messages are at most 70kb when >uncompressed. (I checked the message sizes it was trying to send) > >When you say >> the list of message will be sent as a batch > >does that mean that the producer would group the messages by partition, >and >for each partition, it would batch all the messages for that partition >together, regardless of whether it would exceed a size limit? If so that >may explain things. > >Thanks, >Jamie > >On Tue, May 12, 2015 at 4:40 PM, Jamie X <[email protected]> wrote: > >> Hi, >> >> I'm wondering when you call kafka.javaapi.Producer.send() with a list of >> messages, and also have compression on (snappy in this case), how does >>it >> decide how many messages to put together as one? >> >> The reason I'm asking is that even though my messages are only 70kb >> uncompressed, the broker complains that I'm hitting the 1mb message >>limit >> such as: >> >> >> kafka.common.MessageSizeTooLargeException: Message size is 1035608 bytes >> which exceeds the maximum configured message size of 1000012. >> at >> >>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:378 >>) >> at >> >>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:361 >>) >> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >> at >>kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) >> at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:361) >> at kafka.log.Log.append(Log.scala:257) >> at >> >>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition >>.scala:379) >> at >> >>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition >>.scala:365) >> at kafka.utils.Utils$.inLock(Utils.scala:535) >> at kafka.utils.Utils$.inReadLock(Utils.scala:541) >> at >> kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365) >> at >> >>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala: >>291) >> at >> >>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala: >>282) >> at >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at >> >>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9 >>8) >> at >> >>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9 >>8) >> at >> >>scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226 >>) >> at >>scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> >> Thanks, >> Jamie >>
