maxRate is not used by the direct stream. Significant skew in rate across different partitions for the same topic is going to cause you all kinds of problems, not just with spark streaming.
You can turn on backpressure, but you're better off addressing the underlying issue if you can. On Tue, May 10, 2016 at 8:08 AM, Soumitra Siddharth Johri <soumitra.siddha...@gmail.com> wrote: > Also look at back pressure enabled. Both of these can be used to limit the > rate > > Sent from my iPhone > > On May 10, 2016, at 8:02 AM, chandan prakash <chandanbaran...@gmail.com> > wrote: > > Hi, > I am using Spark Streaming with Direct kafka approach. > Want to limit number of event records coming in my batches. > Have question regarding following 2 parameters : > 1. spark.streaming.receiver.maxRate > 2. spark.streaming.kafka.maxRatePerPartition > > > The documentation > (http://spark.apache.org/docs/latest/streaming-programming-guide.html#deploying-applications > ) says ..... > " spark.streaming.receiver.maxRate for receivers and > spark.streaming.kafka.maxRatePerPartition for Direct Kafka approach " > > Does it mean that spark.streaming.receiver.maxRate is valid only for > Receiver based approach only ? (not the DirectKafkaApproach as well) > > If yes, then how do we control total number of records/sec in DirectKafka > ?.....because spark.streaming.kafka.maxRatePerPartition only controls max > rate per partition and not whole records . There might be many partitions > some with very fast rate and some with very slow rate. > > Regards, > Chandan > > > > -- > Chandan Prakash > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
