[ https://issues.apache.org/jira/browse/SPARK-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liwei Lin updated SPARK-13618: ------------------------------ Description: This JIRA propose to make Streaming web UI display rate-limit lines in the statistics graph. [Screenshots] without back pressure: !https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png! with back pressure: !https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png! was: This JIRA propose to make Streaming web UI display rate-limit lines in the statistics graph. Specifically, this JIRA propose to: 1. add in `RateLimiter` a data structure keeping history of rate limit changes, so that calculating the upper bound of how many records we can receive in a block interval is possible; 2. add the `numRecordsLimit` information into the path from `BlockGenerator` generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can be transferred on wire to the driver side's `ReceivedBlockTracker`); 3. make changes in `StreamingJobProgressListener` and related places, so that the aggregated `numRecordsLimit` information for every batch can be calculated; 4. make changes in `StreamingPage` and related places, so two or more lines can be drawn on a single statistics graph. [Screenshots] without back pressure: !https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png! with back pressure: !https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png! > Make Streaming web UI display rate-limit lines in the statistics graph > ---------------------------------------------------------------------- > > Key: SPARK-13618 > URL: https://issues.apache.org/jira/browse/SPARK-13618 > Project: Spark > Issue Type: Improvement > Components: Streaming, Web UI > Affects Versions: 1.6.0, 2.0.0 > Reporter: Liwei Lin > > This JIRA propose to make Streaming web UI display rate-limit lines in the > statistics graph. > [Screenshots] > without back pressure: > !https://cloud.githubusercontent.com/assets/15843379/13664195/d2264c48-e6e0-11e5-85e6-f13187d4cbde.png! > with back pressure: > !https://cloud.githubusercontent.com/assets/15843379/13664196/d2549c7e-e6e0-11e5-9f62-d7f1458f1c27.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org