[jira] [Commented] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013191#comment-16013191 ] Burak Yavuz commented on SPARK-20140: - resolved by https://github.com/apache/spark/pull/17467 > Remove hardcoded kinesis retry wait and max retries > --- > > Key: SPARK-20140 > URL: https://issues.apache.org/jira/browse/SPARK-20140 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.1.0 >Reporter: Yash Sharma >Assignee: Yash Sharma > Labels: kinesis, recovery > > The pull requests proposes to remove the hardcoded values for Amazon Kinesis > - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES. > This change is critical for kinesis checkpoint recovery when the kinesis > backed rdd is huge. > Following happens in a typical kinesis recovery : > - kinesis throttles large number of requests while recovering > - retries in case of throttling are not able to recover due to the small wait > period > - kinesis throttles per second, the wait period should be configurable for > recovery > The patch picks the spark kinesis configs from: > - spark.streaming.kinesis.retry.wait.time > - spark.streaming.kinesis.retry.max.attempts -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947099#comment-15947099 ] Apache Spark commented on SPARK-20140: -- User 'yssharma' has created a pull request for this issue: https://github.com/apache/spark/pull/17467 > Remove hardcoded kinesis retry wait and max retries > --- > > Key: SPARK-20140 > URL: https://issues.apache.org/jira/browse/SPARK-20140 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.1.0 >Reporter: Yash Sharma > Labels: kinesis, recovery > > The pull requests proposes to remove the hardcoded values for Amazon Kinesis > - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES. > This change is critical for kinesis checkpoint recovery when the kinesis > backed rdd is huge. > Following happens in a typical kinesis recovery : > - kinesis throttles large number of requests while recovering > - retries in case of throttling are not able to recover due to the small wait > period > - kinesis throttles per second, the wait period should be configurable for > recovery > The patch picks the spark kinesis configs from: > - spark.streaming.kinesis.retry.wait.time > - spark.streaming.kinesis.retry.max.attempts -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947015#comment-15947015 ] Yash Sharma commented on SPARK-20140: - Proposing : https://github.com/apache/spark/pull/17467 Please review. > Remove hardcoded kinesis retry wait and max retries > --- > > Key: SPARK-20140 > URL: https://issues.apache.org/jira/browse/SPARK-20140 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.1.0 >Reporter: Yash Sharma > Labels: kinesis, recovery > > The pull requests proposes to remove the hardcoded values for Amazon Kinesis > - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES. > This change is critical for kinesis checkpoint recovery when the kinesis > backed rdd is huge. > Following happens in a typical kinesis recovery : > - kinesis throttles large number of requests while recovering > - retries in case of throttling are not able to recover due to the small wait > period > - kinesis throttles per second, the wait period should be configurable for > recovery > The patch picks the spark kinesis configs from: > - spark.streaming.kinesis.retry.wait.time > - spark.streaming.kinesis.retry.max.attempts -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
