Liwei Lin created SPARK-13618: --------------------------------- Summary: Make Streaming web UI display rate-limit lines in the statistics graph Key: SPARK-13618 URL: https://issues.apache.org/jira/browse/SPARK-13618 Project: Spark Issue Type: Improvement Components: Streaming, Web UI Affects Versions: 1.6.0, 2.0.0 Reporter: Liwei Lin
This JIRA propose to make Streaming web UI display rate-limit lines in the statistics graph. Specifically, this JIRA propose to: 1. add in `RateLimiter` a data structure keeping history of rate limit changes, so that calculating the upper bound of how many records we can receive in a block interval is possible; 2. add the `numRecordsLimit` information into the path from `BlockGenerator` generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can be transferred on wire to the driver side's `ReceivedBlockTracker`); 3. make changes in `StreamingJobProgressListener` and related places, so that the aggregated `numRecordsLimit` information for every batch can be calculated; 4. make changes in `StreamingPage` and related places, so two or more lines can be drawn on a single statistics graph. [Screenshots] without back pressure: with back pressure: -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org