Setting max.request.size does not prevent RecordTooLargeException from being thrown. Thanks for the header info.
9. Jul 2018 09:20 by [email protected] <mailto:[email protected]>.INVALID: > you can configure max.request.size > <> https://kafka.apache.org/documentation/#configuration > <https://kafka.apache.org/documentation/#configuration>> > when you create > Kafka producer, about the second questions, Where is the key size + value > size + header size calculated? you can check ` > > org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound() > {...} > ` > > Ty > > > On Fri, Jul 6, 2018 at 12:42 AM, Jerry Richardson < > [email protected] <mailto:[email protected]>> > wrote: > >> My administrator will not allow messages larger than 1MB to be stored in >> Kafka. >> How can I limit the size of my messages to 1MB? If I have a message larger >> than 1MB, I want to truncate or throw away the message to avoid the >> RecordTooLargeException. What is the max size of the headers? Where is >> the key size + value size + header size calculated? >> >> 5. Jul 2018 14:15 by >> [email protected] >> <mailto:[email protected]>>> <>> mailto:[email protected] >> <mailto:mailto:[email protected]> >> >.INVALID: >> >> >> > sorry, just saw your email a little late. >> > >> > I am confused with why you said `I cannot increase the max size of >> messages >> > stored in Kafka` ? can we add some explanation ? >> > >> > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson < >> > >> [email protected] <mailto:[email protected]>>> >> > >> <>> mailto:[email protected] >> > >> <mailto:mailto:[email protected]>>> >> > >> wrote: >> > >> >> >> >> What class is this in? What's the maximum header size? Is there >> >> documentation on this? >> >> >> >> 2. Jul 2018 18:29 by >> >> [email protected] >> >> <mailto:[email protected]>>> <mailto: >> [email protected] <mailto:[email protected]>>> >>> <>> >> >> mailto:[email protected] <mailto:mailto:[email protected]>>> >> <mailto: >> mailto:[email protected] <mailto:mailto:[email protected]>>> > >> >> >.INVALID: >> >> >> >> >> >> > You can consider increase `max.request.size` a little big (the >> default >> >> > value is `1048576`.), after checking Kafka client source code, they >> count >> >> > [`key size` + `value size` + `header size` + others] together, so it's >> >> > possible the calculated size is a little bigger than the default >> value. >> >> > >> >> > please check with > >> >> https://kafka.apache.org >> >> > <https://kafka.apache.org>>> < >> https://kafka.apache.org <https://kafka.apache.org/>>> > >> >> documentation/#configuration <>> >> https://kafka.apache.org >> >> <https://kafka.apache.org>>> < >> https://kafka.apache.org <https://kafka.apache.org/>>> > >> >> documentation/#configuration>> . >> >> > >> >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> >> [email protected] >> >> > <mailto:[email protected]> >> <>> mailto:[email protected] >> <mailto:mailto:[email protected]>>> >>> <mailto: >> >> >> [email protected] <mailto:[email protected]>>> >> >> >> <>> mailto:[email protected] >> >> >> <mailto:mailto:[email protected]>>> >>> >> >> > wrote: >> >> > >> >> >> Hi all, >> >> >> >> >> >> I get this error even when my records are smaller than the 1000012 >> byte >> >> >> limit: >> >> >> >> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request >> >> >> included a message larger than the max message size the server will >> >> accept. >> >> >> >> >> >> How do I ensure that my producer doesn't send records that are too >> >> large? >> >> >> >> >> >> Thanks in advance for any suggestions and help. >> >> >> >> >> >>
