Hi, I am using Spark Streaming with Direct kafka approach. Want to limit number of event records coming in my batches. Have question regarding following 2 parameters : 1. spark.streaming.receiver.maxRate 2. spark.streaming.kafka.maxRatePerPartition
The documentation ( http://spark.apache.org/docs/latest/streaming-programming-guide.html#deploying-applications ) says ..... " spark.streaming.receiver.maxRate for receivers and spark.streaming.kafka.maxRatePerPartition for Direct Kafka approach " *Does it mean that spark.streaming.receiver.maxRate is valid only for Receiver based approach only ? (not the DirectKafkaApproach as well)* *If yes, then how do we control total number of records/sec in DirectKafka ?.....because spark.streaming.kafka.maxRatePerPartition only controls max rate per partition and not whole records . There might be many partitions some with very fast rate and some with very slow rate.* Regards, Chandan -- Chandan Prakash
