[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravinder Matte updated SPARK-23294: ----------------------------------- Component/s: (was: DStreams) Structured Streaming > Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated > --------------------------------------------------------------------------- > > Key: SPARK-23294 > URL: https://issues.apache.org/jira/browse/SPARK-23294 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.2.1 > Reporter: Ravinder Matte > Assignee: Takeshi Yamamuro > Priority: Minor > Labels: ConsoleSink, RateSource > > I am calling spark-submit passing maxRate, I have a single kinesis receiver, > and batches of 1s > spark-submit --conf spark.streaming.receiver.maxRate=10 .... > however a single batch can greatly exceed the stablished maxRate. i.e: Im > getting 300 records. > it looks like Kinesis is completely ignoring the > spark.streaming.receiver.maxRate configuration. > If you look inside KinesisReceiver.onStart, you see: > val kinesisClientLibConfiguration = > new KinesisClientLibConfiguration(checkpointAppName, streamName, > awsCredProvider, workerId) > .withKinesisEndpoint(endpointUrl) > .withInitialPositionInStream(initialPositionInStream) > .withTaskBackoffTimeMillis(500) > .withRegionName(regionName) > This constructor ends up calling another constructor which has a lot of > default values for the configuration. One of those values is > DEFAULT_MAX_RECORDS which is constantly set to 10,000 records. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org