Also look at back pressure enabled. Both of these can be used to limit the rate
Sent from my iPhone > On May 10, 2016, at 8:02 AM, chandan prakash <chandanbaran...@gmail.com> > wrote: > > Hi, > I am using Spark Streaming with Direct kafka approach. > Want to limit number of event records coming in my batches. > Have question regarding following 2 parameters : > 1. spark.streaming.receiver.maxRate > 2. spark.streaming.kafka.maxRatePerPartition > > > The documentation > (http://spark.apache.org/docs/latest/streaming-programming-guide.html#deploying-applications > ) says ..... > " spark.streaming.receiver.maxRate for receivers and > spark.streaming.kafka.maxRatePerPartition for Direct Kafka approach " > > Does it mean that spark.streaming.receiver.maxRate is valid only for > Receiver based approach only ? (not the DirectKafkaApproach as well) > > If yes, then how do we control total number of records/sec in DirectKafka > ?.....because spark.streaming.kafka.maxRatePerPartition only controls max > rate per partition and not whole records . There might be many partitions > some with very fast rate and some with very slow rate. > > Regards, > Chandan > > > > -- > Chandan Prakash >
