[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Burak Yavuz reassigned SPARK-20140: ----------------------------------- Assignee: Yash Sharma > Remove hardcoded kinesis retry wait and max retries > --------------------------------------------------- > > Key: SPARK-20140 > URL: https://issues.apache.org/jira/browse/SPARK-20140 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 2.1.0 > Reporter: Yash Sharma > Assignee: Yash Sharma > Labels: kinesis, recovery > > The pull requests proposes to remove the hardcoded values for Amazon Kinesis > - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES. > This change is critical for kinesis checkpoint recovery when the kinesis > backed rdd is huge. > Following happens in a typical kinesis recovery : > - kinesis throttles large number of requests while recovering > - retries in case of throttling are not able to recover due to the small wait > period > - kinesis throttles per second, the wait period should be configurable for > recovery > The patch picks the spark kinesis configs from: > - spark.streaming.kinesis.retry.wait.time > - spark.streaming.kinesis.retry.max.attempts -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org