[
https://issues.apache.org/jira/browse/SPARK-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-13618:
------------------------------------
Assignee: (was: Apache Spark)
> Make Streaming web UI display rate-limit lines in the statistics graph
> ----------------------------------------------------------------------
>
> Key: SPARK-13618
> URL: https://issues.apache.org/jira/browse/SPARK-13618
> Project: Spark
> Issue Type: Improvement
> Components: Streaming, Web UI
> Affects Versions: 1.6.0, 2.0.0
> Reporter: Liwei Lin
> Attachments: 1.png, 2.png
>
>
> This JIRA propose to make Streaming web UI display rate-limit lines in the
> statistics graph.
> Specifically, this JIRA propose to:
> 1. add in `RateLimiter` a data structure keeping history of rate limit
> changes, so that calculating the upper bound of how many records we can
> receive in a block interval is possible;
> 2. add the `numRecordsLimit` information into the path from `BlockGenerator`
> generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can
> be transferred on wire to the driver side's `ReceivedBlockTracker`);
> 3. make changes in `StreamingJobProgressListener` and related places, so that
> the aggregated `numRecordsLimit` information for every batch can be
> calculated;
> 4. make changes in `StreamingPage` and related places, so two or more lines
> can be drawn on a single statistics graph.
> [Screenshots]
> without back pressure:
> !https://issues.apache.org/jira/secure/attachment/12790928/1.png!
> with back pressure:
> !https://issues.apache.org/jira/secure/attachment/12790928/2.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
