Hi, You need to add the prefix "kafka." for the configurations which should be propagated to the Kafka. Others will be used in Spark data source itself. (Kafka connector in this case)
https://spark.apache.org/docs/2.4.5/structured-streaming-kafka-integration.html#kafka-specific-configurations Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Tue, May 26, 2020 at 6:42 AM Something Something < mailinglist...@gmail.com> wrote: > I keep getting this error message: > > > *The message is 1169350 bytes when serialized which is larger than the > maximum request size you have configured with the max.request.size > configuration.* > > > > As indicated in other posts, I am trying to set the “max.request.size” > configuration in the Producer as follows: > > > --------------------- > > .writeStream > > .format(*"kafka"*) > > .option( > > *"kafka.bootstrap.servers"*, > > conig.outputBootstrapServer > > ) > > .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"10000000"*) > > --------------------- > > > > But this is not working. Am I setting this correctly? Is there a different > way to set this property under Spark Structured Streaming? > > > Please help. Thanks. > > >
