[
https://issues.apache.org/jira/browse/SPARK-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liwei Lin updated SPARK-13618:
------------------------------
Description:
This JIRA propose to make Streaming web UI display rate-limit lines in the
statistics graph.
[Screenshots]
without back pressure:
!https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png!
with back pressure:
!https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png!
was:
This JIRA propose to make Streaming web UI display rate-limit lines in the
statistics graph.
Specifically, this JIRA propose to:
1. add in `RateLimiter` a data structure keeping history of rate limit changes,
so that calculating the upper bound of how many records we can receive in a
block interval is possible;
2. add the `numRecordsLimit` information into the path from `BlockGenerator`
generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can
be transferred on wire to the driver side's `ReceivedBlockTracker`);
3. make changes in `StreamingJobProgressListener` and related places, so that
the aggregated `numRecordsLimit` information for every batch can be calculated;
4. make changes in `StreamingPage` and related places, so two or more lines can
be drawn on a single statistics graph.
[Screenshots]
without back pressure:
!https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png!
with back pressure:
!https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png!
> Make Streaming web UI display rate-limit lines in the statistics graph
> ----------------------------------------------------------------------
>
> Key: SPARK-13618
> URL: https://issues.apache.org/jira/browse/SPARK-13618
> Project: Spark
> Issue Type: Improvement
> Components: Streaming, Web UI
> Affects Versions: 1.6.0, 2.0.0
> Reporter: Liwei Lin
>
> This JIRA propose to make Streaming web UI display rate-limit lines in the
> statistics graph.
> [Screenshots]
> without back pressure:
> !https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png!
> with back pressure:
> !https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
