Re: [Spark Kafka] How to update batch size of input dynamically for spark kafka consumer?

You can't change the batch time, but you can limit the number of items in the batch http://spark.apache.org/docs/latest/configuration.html spark.streaming.backpressure.enabled spark.streaming.kafka.maxRatePerPartition On Tue, Jan 3, 2017 at 4:00 AM, 周家帅 wrote: > Hi, > > I am

[Spark Kafka] How to update batch size of input dynamically for spark kafka consumer?

Hi, I am an intermediate spark user and have some experience in large data processing. I post this question in StackOverflow but receive no response. My problem is as follows: I use createDirectStream in my spark streaming application. I set the batch interval to 7 seconds and most of the time