Re: Determining broker max buffer size from producer.
You can use AdminClient describeConfigs API to fetch topic/broker configs. https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html On Tue, Sep 4, 2018 at 1:47 PM claude.war...@wipro.com < claude.war...@wipro.com> wrote: > Thank you for the reference but I don't see how it answers my question. > > > Basically, I am looking for a Java call that I can make from the producer > to determine the maximum message size set on the broker. > > > Assume a process is starting and it has the connection information for the > broker so it can create a producer. While the producer can assume that the > max buffersize is the same as the default this may not be the case. The > buffer may have been increased or decreased. My process would like to know > what the limit is so I can determine how to split very data streams being > serialized through kafka. Is there a way to do this without reading the > broker configuration (which is probably not available to my process anyway) > > > Claude > > > From: Satish Duggana > Sent: Monday, September 3, 2018 11:03:32 AM > To: users@kafka.apache.org > Subject: Re: Determining broker max buffer size from producer. > > ** This mail has been sent from an external source. Treat hyperlinks and > attachments in this email with caution** > > Are you asking about the maximum message size that can be sent from a > producer? > max.message.bytes is used to control that at broker/topic level. You can > find more details > at the below links. > > *"The largest record batch size allowed by Kafka. If this is increased and > there are consumers older than 0.10.2, the consumers' fetch size must also > be increased so that the they can fetch record batches this large. In the > latest message format version, records are always grouped into batches for > efficiency. In previous message format versions, uncompressed records are > not grouped into batches and this limit only applies to a single record in > that case."* > > > https://kafka.apache.org/documentation/#topicconfigs > https://kafka.apache.org/documentation/#brokerconfigs > > Thanks, > Satish. > > On Mon, Sep 3, 2018 at 2:12 PM, claude.war...@wipro.com < > claude.war...@wipro.com> wrote: > > > Greetings, > > > > > > I have spent several hours looking through documentation and historical > > email chains and have been unable to find a solution to my problem. > > > > > > I have a case where I need to construct a producer and I need to know the > > broker max buffer size so that I don't get any nasty buffer too big > errors > > from the server when serializing large objects. I have been unable to > find > > any way to do this short of reading the broker configuration and I can't > do > > that from a remote call (as far as I can tell). Does anybody have a > > solution? > > > > > > Claude > > > > The information contained in this electronic message and any attachments > > to this message are intended for the exclusive use of the addressee(s) > and > > may contain proprietary, confidential or privileged information. If you > are > > not the intended recipient, you should not disseminate, distribute or > copy > > this e-mail. Please notify the sender immediately and destroy all copies > of > > this message and any attachments. WARNING: Computer viruses can be > > transmitted via email. The recipient should check this email and any > > attachments for the presence of viruses. The company accepts no liability > > for any damage caused by any virus transmitted by this email. > > www.wipro.com<http://www.wipro.com> > > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com >
Re: Determining broker max buffer size from producer.
Thank you for the reference but I don't see how it answers my question. Basically, I am looking for a Java call that I can make from the producer to determine the maximum message size set on the broker. Assume a process is starting and it has the connection information for the broker so it can create a producer. While the producer can assume that the max buffersize is the same as the default this may not be the case. The buffer may have been increased or decreased. My process would like to know what the limit is so I can determine how to split very data streams being serialized through kafka. Is there a way to do this without reading the broker configuration (which is probably not available to my process anyway) Claude From: Satish Duggana Sent: Monday, September 3, 2018 11:03:32 AM To: users@kafka.apache.org Subject: Re: Determining broker max buffer size from producer. ** This mail has been sent from an external source. Treat hyperlinks and attachments in this email with caution** Are you asking about the maximum message size that can be sent from a producer? max.message.bytes is used to control that at broker/topic level. You can find more details at the below links. *"The largest record batch size allowed by Kafka. If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that the they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case."* https://kafka.apache.org/documentation/#topicconfigs https://kafka.apache.org/documentation/#brokerconfigs Thanks, Satish. On Mon, Sep 3, 2018 at 2:12 PM, claude.war...@wipro.com < claude.war...@wipro.com> wrote: > Greetings, > > > I have spent several hours looking through documentation and historical > email chains and have been unable to find a solution to my problem. > > > I have a case where I need to construct a producer and I need to know the > broker max buffer size so that I don't get any nasty buffer too big errors > from the server when serializing large objects. I have been unable to find > any way to do this short of reading the broker configuration and I can't do > that from a remote call (as far as I can tell). Does anybody have a > solution? > > > Claude > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com<http://www.wipro.com> > The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: Determining broker max buffer size from producer.
Are you asking about the maximum message size that can be sent from a producer? max.message.bytes is used to control that at broker/topic level. You can find more details at the below links. *"The largest record batch size allowed by Kafka. If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that the they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case."* https://kafka.apache.org/documentation/#topicconfigs https://kafka.apache.org/documentation/#brokerconfigs Thanks, Satish. On Mon, Sep 3, 2018 at 2:12 PM, claude.war...@wipro.com < claude.war...@wipro.com> wrote: > Greetings, > > > I have spent several hours looking through documentation and historical > email chains and have been unable to find a solution to my problem. > > > I have a case where I need to construct a producer and I need to know the > broker max buffer size so that I don't get any nasty buffer too big errors > from the server when serializing large objects. I have been unable to find > any way to do this short of reading the broker configuration and I can't do > that from a remote call (as far as I can tell). Does anybody have a > solution? > > > Claude > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.wipro.com >
Determining broker max buffer size from producer.
Greetings, I have spent several hours looking through documentation and historical email chains and have been unable to find a solution to my problem. I have a case where I need to construct a producer and I need to know the broker max buffer size so that I don't get any nasty buffer too big errors from the server when serializing large objects. I have been unable to find any way to do this short of reading the broker configuration and I can't do that from a remote call (as far as I can tell). Does anybody have a solution? Claude The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
