You can throttle the no receiver direct Kafka stream using
spark.streaming.kafka.maxRatePerPartition
http://spark.apache.org/docs/latest/configuration.html#spark-streaming
On Wed, May 27, 2015 at 4:34 PM, Ted Yu yuzhih...@gmail.com wrote:
Have you seen
Got it, thank you, Tathagata and Ted.
Could you comment on my other question
http://apache-spark-user-list.1001560.n3.nabble.com/Autoscaling-Spark-cluster-based-on-topic-sizes-rate-of-growth-in-Kafka-or-Spark-s-metrics-tt23062.html
as well? Basically, I'm trying to get a handle on a good
Have you seen
http://stackoverflow.com/questions/29051579/pausing-throttling-spark-spark-streaming-application
?
Cheers
On Wed, May 27, 2015 at 4:11 PM, dgoldenberg dgoldenberg...@gmail.com
wrote:
Hi,
With the no receivers approach to streaming from Kafka, is there a way to
set something
Hi,
With the no receivers approach to streaming from Kafka, is there a way to
set something like spark.streaming.receiver.maxRate so as not to overwhelm
the Spark consumers?
What would be some of the ways to throttle the streamed messages so that the
consumers don't run out of memory?
--
