Liwei Lin created SPARK-13618:
---------------------------------
Summary: Make Streaming web UI display rate-limit lines in the
statistics graph
Key: SPARK-13618
URL: https://issues.apache.org/jira/browse/SPARK-13618
Project: Spark
Issue Type: Improvement
Components: Streaming, Web UI
Affects Versions: 1.6.0, 2.0.0
Reporter: Liwei Lin
This JIRA propose to make Streaming web UI display rate-limit lines in the
statistics graph.
Specifically, this JIRA propose to:
1. add in `RateLimiter` a data structure keeping history of rate limit changes,
so that calculating the upper bound of how many records we can receive in a
block interval is possible;
2. add the `numRecordsLimit` information into the path from `BlockGenerator`
generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can
be transferred on wire to the driver side's `ReceivedBlockTracker`);
3. make changes in `StreamingJobProgressListener` and related places, so that
the aggregated `numRecordsLimit` information for every batch can be calculated;
4. make changes in `StreamingPage` and related places, so two or more lines can
be drawn on a single statistics graph.
[Screenshots]
without back pressure:
with back pressure:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
