Thank you for the reference but I don't see how it answers my question.
Basically, I am looking for a Java call that I can make from the producer to determine the maximum message size set on the broker. Assume a process is starting and it has the connection information for the broker so it can create a producer. While the producer can assume that the max buffersize is the same as the default this may not be the case. The buffer may have been increased or decreased. My process would like to know what the limit is so I can determine how to split very data streams being serialized through kafka. Is there a way to do this without reading the broker configuration (which is probably not available to my process anyway) Claude ________________________________ From: Satish Duggana <[email protected]> Sent: Monday, September 3, 2018 11:03:32 AM To: [email protected] Subject: Re: Determining broker max buffer size from producer. ** This mail has been sent from an external source. Treat hyperlinks and attachments in this email with caution** Are you asking about the maximum message size that can be sent from a producer? max.message.bytes is used to control that at broker/topic level. You can find more details at the below links. *"The largest record batch size allowed by Kafka. If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that the they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case."* https://kafka.apache.org/documentation/#topicconfigs https://kafka.apache.org/documentation/#brokerconfigs Thanks, Satish. On Mon, Sep 3, 2018 at 2:12 PM, [email protected] < [email protected]> wrote: > Greetings, > > > I have spent several hours looking through documentation and historical > email chains and have been unable to find a solution to my problem. > > > I have a case where I need to construct a producer and I need to know the > broker max buffer size so that I don't get any nasty buffer too big errors > from the server when serializing large objects. I have been unable to find > any way to do this short of reading the broker configuration and I can't do > that from a remote call (as far as I can tell). Does anybody have a > solution? > > > Claude > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com<http://www.wipro.com> > The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
