[ https://issues.apache.org/jira/browse/SPARK-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-13618: ------------------------------------ Assignee: Apache Spark > Make Streaming web UI display rate-limit lines in the statistics graph > ---------------------------------------------------------------------- > > Key: SPARK-13618 > URL: https://issues.apache.org/jira/browse/SPARK-13618 > Project: Spark > Issue Type: Improvement > Components: Streaming, Web UI > Affects Versions: 1.6.0, 2.0.0 > Reporter: Liwei Lin > Assignee: Apache Spark > Attachments: 1.png, 2.png > > > This JIRA propose to make Streaming web UI display rate-limit lines in the > statistics graph. > Specifically, this JIRA propose to: > 1. add in `RateLimiter` a data structure keeping history of rate limit > changes, so that calculating the upper bound of how many records we can > receive in a block interval is possible; > 2. add the `numRecordsLimit` information into the path from `BlockGenerator` > generates a `Block` to the `ReceivedBlockInfo` (so that `numRecordsLimit` can > be transferred on wire to the driver side's `ReceivedBlockTracker`); > 3. make changes in `StreamingJobProgressListener` and related places, so that > the aggregated `numRecordsLimit` information for every batch can be > calculated; > 4. make changes in `StreamingPage` and related places, so two or more lines > can be drawn on a single statistics graph. > [Screenshots] > without back pressure: > !https://issues.apache.org/jira/secure/attachment/12790928/1.png! > with back pressure: > !https://issues.apache.org/jira/secure/attachment/12790928/2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org