RecordTooLargeException in Spark *Structured* Streaming
I keep getting this error message: *The message is 1169350 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.* As indicated in other posts, I am trying to set the “max.request.size” configuration in the Producer as follows: - .writeStream .format(*"kafka"*) .option( *"kafka.bootstrap.servers"*, conig.outputBootstrapServer ) .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"1000"*) - But this is not working. Am I setting this correctly? Is there a different way to set this property under Spark Structured Streaming? Please help. Thanks.
Re: RecordTooLargeException in Spark *Structured* Streaming
Hi, You need to add the prefix "kafka." for the configurations which should be propagated to the Kafka. Others will be used in Spark data source itself. (Kafka connector in this case) https://spark.apache.org/docs/2.4.5/structured-streaming-kafka-integration.html#kafka-specific-configurations Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Tue, May 26, 2020 at 6:42 AM Something Something < mailinglist...@gmail.com> wrote: > I keep getting this error message: > > > *The message is 1169350 bytes when serialized which is larger than the > maximum request size you have configured with the max.request.size > configuration.* > > > > As indicated in other posts, I am trying to set the “max.request.size” > configuration in the Producer as follows: > > > - > > .writeStream > > .format(*"kafka"*) > > .option( > > *"kafka.bootstrap.servers"*, > > conig.outputBootstrapServer > > ) > > .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"1000"*) > > - > > > > But this is not working. Am I setting this correctly? Is there a different > way to set this property under Spark Structured Streaming? > > > Please help. Thanks. > > >
Re: RecordTooLargeException in Spark *Structured* Streaming
Thanks. Missed that part of documentation. Appreciate your help. Regards. On Mon, May 25, 2020 at 10:42 PM Jungtaek Lim wrote: > Hi, > > You need to add the prefix "kafka." for the configurations which should be > propagated to the Kafka. Others will be used in Spark data source > itself. (Kafka connector in this case) > > > https://spark.apache.org/docs/2.4.5/structured-streaming-kafka-integration.html#kafka-specific-configurations > > Hope this helps. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > > On Tue, May 26, 2020 at 6:42 AM Something Something < > mailinglist...@gmail.com> wrote: > >> I keep getting this error message: >> >> >> *The message is 1169350 bytes when serialized which is larger than the >> maximum request size you have configured with the max.request.size >> configuration.* >> >> >> >> As indicated in other posts, I am trying to set the “max.request.size” >> configuration in the Producer as follows: >> >> >> - >> >> .writeStream >> >> .format(*"kafka"*) >> >> .option( >> >> *"kafka.bootstrap.servers"*, >> >> conig.outputBootstrapServer >> >> ) >> >> .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"1000"*) >> >> - >> >> >> >> But this is not working. Am I setting this correctly? Is there a >> different way to set this property under Spark Structured Streaming? >> >> >> Please help. Thanks. >> >> >>