[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravinder Matte updated SPARK-23294: ----------------------------------- Description: Using following configs while building SparkSession ("spark.streaming.backpressure.enabled", "true") ("spark.streaming.receiver.maxRate", "100") ("spark.streaming.backpressure.initialRate", "100") Source: Rate Source with following options. rowsPerSecond=100000 Sink: Console Sink. I am expecting the process rate to limit at 100 Rows per Second, but maxRate is ignored and streaming job is processing at 100000 rate. was: I am calling spark-submit passing maxRate, I have a single kinesis receiver, and batches of 1s spark-submit --conf spark.streaming.receiver.maxRate=10 .... however a single batch can greatly exceed the stablished maxRate. i.e: Im getting 300 records. it looks like Kinesis is completely ignoring the spark.streaming.receiver.maxRate configuration. If you look inside KinesisReceiver.onStart, you see: val kinesisClientLibConfiguration = new KinesisClientLibConfiguration(checkpointAppName, streamName, awsCredProvider, workerId) .withKinesisEndpoint(endpointUrl) .withInitialPositionInStream(initialPositionInStream) .withTaskBackoffTimeMillis(500) .withRegionName(regionName) This constructor ends up calling another constructor which has a lot of default values for the configuration. One of those values is DEFAULT_MAX_RECORDS which is constantly set to 10,000 records. > Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated > --------------------------------------------------------------------------- > > Key: SPARK-23294 > URL: https://issues.apache.org/jira/browse/SPARK-23294 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.2.1 > Reporter: Ravinder Matte > Assignee: Takeshi Yamamuro > Priority: Minor > Labels: ConsoleSink, RateSource > > Using following configs while building SparkSession > ("spark.streaming.backpressure.enabled", "true") > ("spark.streaming.receiver.maxRate", "100") > ("spark.streaming.backpressure.initialRate", "100") > > Source: Rate Source with following options. > rowsPerSecond=100000 > Sink: Console Sink. > > I am expecting the process rate to limit at 100 Rows per Second, but maxRate > is ignored and streaming job is processing at 100000 rate. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org