You can use AdminClient describeConfigs API to fetch topic/broker configs. https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html
On Tue, Sep 4, 2018 at 1:47 PM claude.war...@wipro.com < claude.war...@wipro.com> wrote: > Thank you for the reference but I don't see how it answers my question. > > > Basically, I am looking for a Java call that I can make from the producer > to determine the maximum message size set on the broker. > > > Assume a process is starting and it has the connection information for the > broker so it can create a producer. While the producer can assume that the > max buffersize is the same as the default this may not be the case. The > buffer may have been increased or decreased. My process would like to know > what the limit is so I can determine how to split very data streams being > serialized through kafka. Is there a way to do this without reading the > broker configuration (which is probably not available to my process anyway) > > > Claude > > ________________________________ > From: Satish Duggana <satish.dugg...@gmail.com> > Sent: Monday, September 3, 2018 11:03:32 AM > To: users@kafka.apache.org > Subject: Re: Determining broker max buffer size from producer. > > ** This mail has been sent from an external source. Treat hyperlinks and > attachments in this email with caution** > > Are you asking about the maximum message size that can be sent from a > producer? > max.message.bytes is used to control that at broker/topic level. You can > find more details > at the below links. > > *"The largest record batch size allowed by Kafka. If this is increased and > there are consumers older than 0.10.2, the consumers' fetch size must also > be increased so that the they can fetch record batches this large. In the > latest message format version, records are always grouped into batches for > efficiency. In previous message format versions, uncompressed records are > not grouped into batches and this limit only applies to a single record in > that case."* > > > https://kafka.apache.org/documentation/#topicconfigs > https://kafka.apache.org/documentation/#brokerconfigs > > Thanks, > Satish. > > On Mon, Sep 3, 2018 at 2:12 PM, claude.war...@wipro.com < > claude.war...@wipro.com> wrote: > > > Greetings, > > > > > > I have spent several hours looking through documentation and historical > > email chains and have been unable to find a solution to my problem. > > > > > > I have a case where I need to construct a producer and I need to know the > > broker max buffer size so that I don't get any nasty buffer too big > errors > > from the server when serializing large objects. I have been unable to > find > > any way to do this short of reading the broker configuration and I can't > do > > that from a remote call (as far as I can tell). Does anybody have a > > solution? > > > > > > Claude > > > > The information contained in this electronic message and any attachments > > to this message are intended for the exclusive use of the addressee(s) > and > > may contain proprietary, confidential or privileged information. If you > are > > not the intended recipient, you should not disseminate, distribute or > copy > > this e-mail. Please notify the sender immediately and destroy all copies > of > > this message and any attachments. WARNING: Computer viruses can be > > transmitted via email. The recipient should check this email and any > > attachments for the presence of viruses. The company accepts no liability > > for any damage caused by any virus transmitted by this email. > > www.wipro.com<http://www.wipro.com> > > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com >
