L. C. Hsieh created SPARK-33962: ----------------------------------- Summary: Fix incorrect min partition condition in getRanges Key: SPARK-33962 URL: https://issues.apache.org/jira/browse/SPARK-33962 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 3.2.0 Reporter: L. C. Hsieh Assignee: L. C. Hsieh
When calculating offset ranges, we consider minPartitions configuration. If minPartitions is not set or is less than or equal the size of given ranges, it means there are enough partitions at Kafka so we don't need to split offsets to satisfy min partition requirement. But the current condition is offsetRanges.size > minPartitions.get and is not correct. Currently getRanges will split offsets in unnecessary case. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org