am I miss reading this loop: https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/log/Log.scala#L265-L269
it seems like all messages from `validMessages` (which is ByteBufferMessageSet) are NOT appended if one of the message size exceeds the limit. I hope I'm missing something. On Wed, Aug 27, 2014 at 9:38 PM, Alexis Midon < alexis.mi...@airbedandbreakfast.com> wrote: > Hi Jun, > > thanks for you answer. > Unfortunately the size won't help much, I'd like to see the actual message > data. > > By the way what are the things to consider when deciding on > `message.max.bytes` value? > > > > > > > On Wed, Aug 27, 2014 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote: > >> The message size check is currently only done on the broker. If you enable >> trace level logging in RequestChannel, you will see the produce request, >> which includes the size of each partition. >> >> Thanks, >> >> Jun >> >> >> On Wed, Aug 27, 2014 at 4:40 PM, Alexis Midon < >> alexis.mi...@airbedandbreakfast.com> wrote: >> >> > Hello, >> > >> > my brokers are reporting that some received messages exceed the >> > `message.max.bytes` value. >> > I'd like to know what producers are at fault but It is pretty much >> > impossible: >> > - the brokers don't log the content of the rejected messages >> > - the log messages do not contain the IP of the producers >> > - on the consumer side, no exception is thrown (afaik it is because >> Ack-0 >> > is used). The only kind of notification is to closed the connection. >> > >> > [1] Do you have any suggestions to track down the guilty producers or >> find >> > out the message content? >> > >> > Even though it makes total sense to have the limit defined and applied >> on >> > the brokers, I was thinking that this check could also be applied by the >> > producers. Some google results suggest that `message.max.bytes` might be >> > used by the producers but I can't find any trace of that behavior in the >> > code. >> > >> > The closest thing I have is >> > >> > >> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/SyncProducer.scala#L67 >> > but it simply logs the message size and content and the log level is >> trace. >> > >> > [2] could you please confirm if such a producer-side check exists? >> > >> > >> > thanks! >> > >> > Alexis >> > >> > >
