L. C. Hsieh created SPARK-33962:
-----------------------------------
Summary: Fix incorrect min partition condition in getRanges
Key: SPARK-33962
URL: https://issues.apache.org/jira/browse/SPARK-33962
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
When calculating offset ranges, we consider minPartitions configuration. If
minPartitions is not set or is less than or equal the size of given ranges, it
means there are enough partitions at Kafka so we don't need to split offsets to
satisfy min partition requirement. But the current condition is
offsetRanges.size > minPartitions.get and is not correct. Currently getRanges
will split offsets in unnecessary case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
