If you are sending in sync mode, producer will just group by partition the
list of messages you provided as argument of send() and send them out. You
don¹t need to worry about batch.num.messages.

There is a potential that compressed message is even bigger than
uncompressed message, though. I¹m not sure if this is the case you see
there if you are sure your message size is only 70KB while the limitation
is ~1MB.

Can you send the messages one by one use the batch send - each MessageSet
only has one message - and see which message caused the problem?

Jiangjie (Becket) Qin

On 5/13/15, 9:57 AM, "Jamie X" <[email protected]> wrote:

>(sorry if this messes up the mailing list, I didn't seem to get replies in
>my inbox)
>
>Jiangjie, I am indeed using the old producer, and on sync mode.
>
>> Notice that the old producer uses number of messages as batch limitation
>instead of number of bytes.
>
>Can you clarify this? I see a setting batch.num.messages but it is only
>for
>async and wouldn't affect sync mode.
>
>> But in your case, it seems you have a single message whose compressed
>size is larger than the max message size Kafka broker accepts. Any idea
>why?
>
>I don't think this is the case as my messages are at most 70kb when
>uncompressed. (I checked the message sizes it was trying to send)
>
>When you say
>> the list of message will be sent as a batch
>
>does that mean that the producer would group the messages by partition,
>and
>for each partition, it would batch all the messages for that partition
>together, regardless of whether it would exceed a size limit? If so that
>may explain things.
>
>Thanks,
>Jamie
>
>On Tue, May 12, 2015 at 4:40 PM, Jamie X <[email protected]> wrote:
>
>> Hi,
>>
>> I'm wondering when you call kafka.javaapi.Producer.send() with a list of
>> messages, and also have compression on (snappy in this case), how does
>>it
>> decide how many messages to put together as one?
>>
>> The reason I'm asking is that even though my messages are only 70kb
>> uncompressed, the broker complains that I'm hitting the 1mb message
>>limit
>> such as:
>>
>>
>> kafka.common.MessageSizeTooLargeException: Message size is 1035608 bytes
>> which exceeds the maximum configured message size of 1000012.
>>         at
>> 
>>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:378
>>)
>>         at
>> 
>>kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:361
>>)
>>         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>         at 
>>kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
>>         at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:361)
>>         at kafka.log.Log.append(Log.scala:257)
>>         at
>> 
>>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition
>>.scala:379)
>>         at
>> 
>>kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition
>>.scala:365)
>>         at kafka.utils.Utils$.inLock(Utils.scala:535)
>>         at kafka.utils.Utils$.inReadLock(Utils.scala:541)
>>         at
>> kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
>>         at
>> 
>>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:
>>291)
>>         at
>> 
>>kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:
>>282)
>>         at
>> 
>>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca
>>la:244)
>>         at
>> 
>>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca
>>la:244)
>>         at
>> 
>>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9
>>8)
>>         at
>> 
>>scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:9
>>8)
>>         at
>> 
>>scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226
>>)
>>         at 
>>scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>>
>> Thanks,
>> Jamie
>>

Reply via email to