[
https://issues.apache.org/jira/browse/SPARK-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218672#comment-15218672
]
Liyin Tang commented on SPARK-14230:
------------------------------------
[~davies], thanks for the response. If I understand it correctly, this PR you
pointed is to set a start time for window function. What about non-window
function, do we also have a way to specify the start time ?
> Config the start time (jitter) for streaming jobs
> -------------------------------------------------
>
> Key: SPARK-14230
> URL: https://issues.apache.org/jira/browse/SPARK-14230
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Liyin Tang
>
> Currently, RecurringTimer will normalize the start time. For instance, if
> batch duration is 1 min, all the job will start exactly at 1 min boundary.
> This actually adds some burden to the streaming source. Assuming the source
> is Kafka, and there is a list of streaming jobs with 1 min batch duration,
> then at first few seconds of each min, high network traffic will be observed
> in Kafka. This makes Kafka capacity planning tricky.
> It will be great to have an option in the streaming context to set the job
> start time. In this way, user can add a jitter for the start time for each,
> and make Kafka fetch_request much smooth across the duration window.
> {code}
> class RecurringTimer {
> def getStartTime(): Long = {
> (math.floor(clock.currentTime.toDouble / period) + 1).toLong * period +
> jitter
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
