[
https://issues.apache.org/jira/browse/SPARK-12073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated SPARK-12073:
---------------------------------
Affects Version/s: 1.6.1
1.6.0
> Backpressure causes individual Kafka partitions to lag
> ------------------------------------------------------
>
> Key: SPARK-12073
> URL: https://issues.apache.org/jira/browse/SPARK-12073
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.2, 1.6.0, 1.6.1
> Reporter: Jason White
> Assignee: Jason White
> Fix For: 2.0.0
>
>
> We're seeing a growing lag on (2) individual Kafka partitions, on a topic
> with 32 partitions. Our individual batch sessions are completing in 5-7s,
> with a batch window of 30s, so there's plenty of room for Streaming to catch
> up, but it looks to be intentionally limiting itself. These partitions are
> experiencing unbalanced load (higher than most of the others)
> What I believe is happening is that maxMessagesPerPartition calculates an
> appropriate limit for the message rate from all partitions, and then divides
> by the number of partitions to determine how many messages to retrieve per
> partition. The problem with this approach is that when one partition is
> behind by millions of records (due to random Kafka issues) or is experiencing
> heavy load, the number of messages to be retrieved shouldn't be evenly split
> among the partitions. In this scenario, if the rate estimator calculates only
> 100k total messages can be retrieved, each partition (out of say 32) only
> retrieves max 100k/32=3125 messages.
> Under some conditions, this results in the backpressure keeping the lagging
> partition from recovering. The PIDRateEstimator doesn't increase the number
> of messages to retrieve enough to recover, and we stabilize at a point where
> these individual partitions slowly grow.
> I have a PR on our fork in progress to allocate the maxMessagesPerPartition
> by weighting the number to be retrieved on the current lag each partition is
> currently experiencing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
