Hi all, I'd like to start a discussion about releasing Apache Spark 4.2.1.
I'm particularly interested in getting SPARK-59572 <https://issues.apache.org/jira/browse/SPARK-59572> into 4.2.1. It fixes a correctness issue introduced in 4.2.0 where failures to evaluate a DSv2 PartitionPredicate can be treated as matches, leading to incorrect query results. It's been over two months since Spark 4.2.0 was released in July. The changes on branch-4.2 since 4.2.0 <https://github.com/apache/spark/compare/v4.2.0...branch-4.2> now include 99 PRs covering 97 JIRA issues. Besides SPARK-59572, these include correctness and stability fixes across SQL, Core, Streaming, PySpark, and Kubernetes, with several addressing storage-partitioned joins. I'm happy to volunteer to drive the release to learn the process! Would folks be in favor of moving forward with a 4.2.1 release? Are there any other fixes or blockers we should address first? Thanks, Szehon
