david przybill created SPARK-18620: -------------------------------------- Summary: Spark Streaming + Kinesis : Receiver MaxRate is violated Key: SPARK-18620 URL: https://issues.apache.org/jira/browse/SPARK-18620 Project: Spark Issue Type: Bug Components: DStreams Affects Versions: 2.0.2 Reporter: david przybill Priority: Minor
I am calling spark-submit passing maxRate, I have a single kinesis receiver, and batches of 1s spark-submit --conf spark.streaming.receiver.maxRate=10 .... however a single batch can greatly exceed the stablished maxRate. i.e: Im getting 300 records. it looks like Kinesis is completely ignoring the spark.streaming.receiver.maxRate configuration. If you look inside KinesisReceiver.onStart, you see: val kinesisClientLibConfiguration = new KinesisClientLibConfiguration(checkpointAppName, streamName, awsCredProvider, workerId) .withKinesisEndpoint(endpointUrl) .withInitialPositionInStream(initialPositionInStream) .withTaskBackoffTimeMillis(500) .withRegionName(regionName) This constructor ends up calling another constructor which has a lot of default values for the configuration. One of those values is DEFAULT_MAX_RECORDS which is constantly set to 10,000 records. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org