Again, we haven't changed the default message size, I believe this exception is a red herring.
On Tue, 2020-07-28 at 17:38 +0000, manoj.agraw...@cognizant.com wrote: [EXTERNAL EMAIL] Attention: This email was sent from outside Xperi. DO NOT CLICK any links or attachments unless you expected them. ________________________________ Hi , You also make to change at producer and consumer side as well server.properties: message.max.bytes=15728640 replica.fetch.max.bytes=15728640 max.request.size=15728640 fetch.message.max.bytes=15728640 and producer.properties: max.request.size=15728640 consumer max.partition.fetch.bytes On 7/28/20, 9:51 AM, "Thomas Becker" <thomas.bec...@xperi.com<mailto:thomas.bec...@xperi.com>> wrote: [External] We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message: org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size, or decrease the maximum message size the broker will allow. We have not increased the maximum message size on either the broker nor topic level, and I'm quite confident no messages approaching that size are in the topic. Further, even if I increase the max.partition.fetch.bytes to a very large value such as Integer.MAX_VALUE, the error still occurs. I stumbled across https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-4762&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=%2FBPbP0E2r7XkMihp%2Btnu19s6OTfV655VS9%2BTRQ%2F6il4%3D&reserved=0 which seems to match what we're seeing, but our messages are not compressed. But sure enough, a test application using the 0.10.1.0 consumer is able to consume the topic with no issues. Unfortunately upgrading our legacy applications is difficult for other reasons. Any ideas what's happening here? -- [https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdts-web-images.s3.amazonaws.com%2FImages%2Femail%2Bsignatures%2Fxperi_117.png&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=6gCzFkW2s9X%2FvRZTs7Zzh1FJzFLNZk46k1%2BvZUkugD4%3D&reserved=0] Tommy Becker Principal Engineer Pronouns: he/him/his O: 919.460.4747 E: thomas.bec...@xperi.com<mailto:thomas.bec...@xperi.com> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.xperi.com%2F&data=02%7C01%7CThomas.Becker%40tivo.com%7Cfd376717d3824bbde22d08d8331d40ae%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637315548016939647&sdata=aND7C18XgVspkuEh6aRURyljGG4Qo3I8K8Ji5Lge9Jo%3D&reserved=0 ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. -- [cid:f5f2359b5a2deb6470a7af30f647ae6ee88ae657.camel@tivo.com] Tommy Becker Principal Engineer Personalized Content Discovery O +1 919.460.4747 tivo.com<http://www.tivo.com/> ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.
