[jira] [Commented] (SPARK-14230) Config the start time (jitter) for streaming jobs
[ https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218938#comment-15218938 ] Davies Liu commented on SPARK-14230: For non-window batch, could be supported via trigger, see https://github.com/apache/spark/pull/11976/files > Config the start time (jitter) for streaming jobs > - > > Key: SPARK-14230 > URL: https://issues.apache.org/jira/browse/SPARK-14230 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Liyin Tang > > Currently, RecurringTimer will normalize the start time. For instance, if > batch duration is 1 min, all the job will start exactly at 1 min boundary. > This actually adds some burden to the streaming source. Assuming the source > is Kafka, and there is a list of streaming jobs with 1 min batch duration, > then at first few seconds of each min, high network traffic will be observed > in Kafka. This makes Kafka capacity planning tricky. > It will be great to have an option in the streaming context to set the job > start time. In this way, user can add a jitter for the start time for each, > and make Kafka fetch_request much smooth across the duration window. > {code} > class RecurringTimer { > def getStartTime(): Long = { > (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period + > jitter > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14230) Config the start time (jitter) for streaming jobs
[ https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218672#comment-15218672 ] Liyin Tang commented on SPARK-14230: [~davies], thanks for the response. If I understand it correctly, this PR you pointed is to set a start time for window function. What about non-window function, do we also have a way to specify the start time ? > Config the start time (jitter) for streaming jobs > - > > Key: SPARK-14230 > URL: https://issues.apache.org/jira/browse/SPARK-14230 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Liyin Tang > > Currently, RecurringTimer will normalize the start time. For instance, if > batch duration is 1 min, all the job will start exactly at 1 min boundary. > This actually adds some burden to the streaming source. Assuming the source > is Kafka, and there is a list of streaming jobs with 1 min batch duration, > then at first few seconds of each min, high network traffic will be observed > in Kafka. This makes Kafka capacity planning tricky. > It will be great to have an option in the streaming context to set the job > start time. In this way, user can add a jitter for the start time for each, > and make Kafka fetch_request much smooth across the duration window. > {code} > class RecurringTimer { > def getStartTime(): Long = { > (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period + > jitter > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14230) Config the start time (jitter) for streaming jobs
[ https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218657#comment-15218657 ] Davies Liu commented on SPARK-14230: This will be supported in structured streaming: see https://github.com/apache/spark/pull/12008/files#diff-80a6da9ac9681594543c70c837b12641R2597 > Config the start time (jitter) for streaming jobs > - > > Key: SPARK-14230 > URL: https://issues.apache.org/jira/browse/SPARK-14230 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Liyin Tang > > Currently, RecurringTimer will normalize the start time. For instance, if > batch duration is 1 min, all the job will start exactly at 1 min boundary. > This actually adds some burden to the streaming source. Assuming the source > is Kafka, and there is a list of streaming jobs with 1 min batch duration, > then at first few seconds of each min, high network traffic will be observed > in Kafka. This makes Kafka capacity planning tricky. > It will be great to have an option in the streaming context to set the job > start time. In this way, user can add a jitter for the start time for each, > and make Kafka fetch_request much smooth across the duration window. > {code} > class RecurringTimer { > def getStartTime(): Long = { > (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period + > jitter > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14230) Config the start time (jitter) for streaming jobs
[ https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215403#comment-15215403 ] Apache Spark commented on SPARK-14230: -- User 'liyintang' has created a pull request for this issue: https://github.com/apache/spark/pull/12026 > Config the start time (jitter) for streaming jobs > - > > Key: SPARK-14230 > URL: https://issues.apache.org/jira/browse/SPARK-14230 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Liyin Tang > > Currently, RecurringTimer will normalize the start time. For instance, if > batch duration is 1 min, all the job will start exactly at 1 min boundary. > This actually adds some burden to the streaming source. Assuming the source > is Kafka, and there is a list of streaming jobs with 1 min batch duration, > then at first few seconds of each min, high network traffic will be observed > in Kafka. This makes Kafka capacity planning tricky. > It will be great to have an option in the streaming context to set the job > start time. In this way, user can add a jitter for the start time for each, > and make Kafka fetch_request much smooth across the duration window. > {code} > class RecurringTimer { > def getStartTime(): Long = { > (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period + > jitter > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org