Albert Cheng created SPARK-15429:
------------------------------------
Summary: When `spark.streaming.concurrentJobs > 1`,
PIDRateEstimator cannot estimate the receiving rate accurately.
Key: SPARK-15429
URL: https://issues.apache.org/jira/browse/SPARK-15429
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.6.1
Reporter: Albert Cheng
When `spark.streaming.concurrentJobs > 1`, PIDRateEstimator cannot estimate the
receiving rate accurately.
For example, if the batch duration is set to 10 seconds, each rdd in the
dstream will take 20s to compute. By changing
`spark.streaming.concurrentJobs=2`, each rdd in the dstream still takes 20s to
consume the data, which leads to poor estimation of backpressure by
PIDRateEstimator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
