Will this NPE make Spark 4.2 completely unusable? If not, dependency upgrades theoretically should not be backported to the 4.2.x branch, as third-party dependencies for the entire 4.x line have been frozen, correct?
Jie Yang On 2026/06/30 02:35:43 huaxin gao wrote: > Thanks Cheng for flagging this! > > I prefer not to delay RC5 (which is for the WKB security fix) just to wait > for an outside dependency. Also, 2.26.1 just started voting, so we can't > use it until it is on Maven Central (about 4-5 days). So I will cut RC5 > with the current log4j. If RC5 fails, we can include 2.26.1, otherwise > let's include it in 4.2.1. > > Thanks, > > Huaxin > > On Mon, Jun 29, 2026 at 6:57 PM Cheng Pan <[email protected]> wrote: > > > Can we include the log4j 2.26.1 upgrading in next RC? > > > > TL:DR, the current log4j2 2.25.x used by branch-4.2 may throw NPE on > > logging, downgrading to 2.24.x can fix the issue but bring some CVEs back, > > log4j2 2.26.1 is in the voting phase and likely available soon. See full > > context at [1] > > > > [1] https://github.com/apache/spark/pull/51719#issuecomment-3341344974 > > > > Thanks, > > Cheng Pan > > > > > > > > On Jun 30, 2026, at 08:52, huaxin gao <[email protected]> wrote: > > > > Thanks Szehon for reporting the problem. I'm failing RC4, and will roll > > RC5 once the fix <https://github.com/apache/spark/pull/56875> is merged > > and backported to branch-4.2. > > > > Thanks to everyone who tested RC4! > > > > Huaxin > > > > On Mon, Jun 29, 2026 at 5:14 PM Szehon Ho <[email protected]> wrote: > > > >> I made a fix for this, it would be great to get it into the release. > >> > >> https://github.com/apache/spark/pull/56875 > >> > >> Thanks Huaxin! > >> Szehon > >> > >> > >> On Mon, Jun 29, 2026 at 4:54 PM Szehon Ho <[email protected]> > >> wrote: > >> > >>> Sorry I was informed of a potential vulnerability/problem in new fromWKB > >>> in Spark 4.2, looking now at it. > >>> > >>> Thanks > >>> Szehon > >>> > >>> On Mon, Jun 29, 2026 at 1:05 PM Allison Wang <[email protected]> > >>> wrote: > >>> > >>>> +1 > >>>> > >>>> On Mon, Jun 29, 2026 at 10:34 AM Max Gekk <[email protected]> wrote: > >>>> > >>>>> +1 > >>>>> > >>>>> On Mon, Jun 29, 2026 at 6:32 PM Xiao Li <[email protected]> wrote: > >>>>> > > >>>>> > +1 (binding) > >>>>> > > >>>>> > Verified on Linux (Ubuntu, JDK 17, Python 3.11): > >>>>> > > >>>>> > - Signatures: every artifact (source, the three binary > >>>>> distributions, the pyspark/pyspark_client/pyspark_connect tarballs, and > >>>>> SparkR) has a good GPG signature from the RM key > >>>>> 709226B910E0F10917123B6259B586ADA5A538D1, which is in KEYS. > >>>>> > - Checksums: SHA512 matches for all artifacts.- Tag/commit: > >>>>> v4.2.0-rc4 resolves to f92a807c06b, and the RELEASE metadata embedded in > >>>>> the binary distributions records the same git revision. > >>>>> > - Source build: compiled and packaged spark-core and its module > >>>>> dependencies from the source tarball with ./build/mvn on JDK 17 (BUILD > >>>>> SUCCESS). > >>>>> > - Binary distribution: ran SparkPi and a spark-shell Scala job > >>>>> (range aggregation + Spark SQL) — results correct. > >>>>> > - PySpark: pip-installed pyspark-4.2.0.tar.gz and ran DataFrame, > >>>>> Spark SQL, and a Python UDF in local mode — results correct. > >>>>> > > >>>>> > Thanks Huaxin for driving the release! > >>>>> > > >>>>> > Xiao > >>>>> > > >>>>> > Uroš Bojanić <[email protected]> 于2026年6月29日周一 09:12写道： > >>>>> >> > >>>>> >> +1 (non-binding) > >>>>> >> > >>>>> >> verified RC4 (tag v4.2.0-rc4, commit f92a807c06b) on macOS/arm64. > >>>>> >> > >>>>> >> - Good GPG signatures and SHA512 sums on all artifacts (source, > >>>>> three binaries, the PySpark/SparkR tarballs) against KEYS. > >>>>> >> - Built the full source tree from the tag with -Phive > >>>>> -Phive-thriftserver, got a clean BUILD SUCCESS across all 39 modules. > >>>>> >> - Ran quick smoke tests across every API surface: Java, Scala, SQL, > >>>>> and PySpark job from the bundled distro; all look good. > >>>>> >> - Sanity-checked the binary dist (RELEASE, LICENSE/NOTICE/licenses > >>>>> all present) and ran dev/check-license; RAT passes. > >>>>> >> - Diffed the docs against 4.1.0 and analyzed the changes (new > >>>>> pages, migration guides and version refs); all look good. > >>>>> >> > >>>>> >> Thank you Huaxin Gao! > >>>>> >> > >>>>> >> On 2026/06/27 00:21:32 [email protected] wrote: > >>>>> >> > Please vote on releasing the following candidate as Apache Spark > >>>>> version 4.2.0. > >>>>> >> > > >>>>> >> > The vote is open until Mon, 29 Jun 2026 18:21:31 PDT and passes > >>>>> if a majority +1 PMC votes are cast, with > >>>>> >> > a minimum of 3 +1 votes. > >>>>> >> > > >>>>> >> > [ ] +1 Release this package as Apache Spark 4.2.0 > >>>>> >> > [ ] -1 Do not release this package because ... > >>>>> >> > > >>>>> >> > To learn more about Apache Spark, please see > >>>>> https://spark.apache.org/ > >>>>> >> > > >>>>> >> > The tag to be voted on is v4.2.0-rc4 (commit f92a807c06b): > >>>>> >> > https://github.com/apache/spark/tree/v4.2.0-rc4 > >>>>> >> > > >>>>> >> > The release files, including signatures, digests, etc. can be > >>>>> found at: > >>>>> >> > https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-bin/ > >>>>> >> > > >>>>> >> > Signatures used for Spark RCs can be found in this file: > >>>>> >> > https://downloads.apache.org/spark/KEYS > >>>>> >> > > >>>>> >> > The staging repository for this release can be found at: > >>>>> >> > > >>>>> https://repository.apache.org/content/repositories/orgapachespark-1524/ > >>>>> >> > > >>>>> >> > The documentation corresponding to this release can be found at: > >>>>> >> > https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-docs/ > >>>>> >> > > >>>>> >> > The list of bug fixes going into 4.2.0 can be found at the > >>>>> following URL: > >>>>> >> > https://issues.apache.org/jira/projects/SPARK/versions/12356380 > >>>>> >> > > >>>>> >> > FAQ > >>>>> >> > > >>>>> >> > ========================= > >>>>> >> > How can I help test this release? > >>>>> >> > ========================= > >>>>> >> > > >>>>> >> > If you are a Spark user, you can help us test this release by > >>>>> taking > >>>>> >> > an existing Spark workload and running on this release candidate, > >>>>> then > >>>>> >> > reporting any regressions. > >>>>> >> > > >>>>> >> > If you're working in PySpark you can set up a virtual env and > >>>>> install > >>>>> >> > the current RC via "pip install > >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-bin/pyspark-4.2.0.tar.gz > >>>>> " > >>>>> >> > and see if anything important breaks. > >>>>> >> > In the Java/Scala, you can add the staging repository to your > >>>>> project's resolvers and test > >>>>> >> > with the RC (make sure to clean up the artifact cache > >>>>> before/after so > >>>>> >> > you don't end up building with an out of date RC going forward). > >>>>> >> > > >>>>> >> > > >>>>> --------------------------------------------------------------------- > >>>>> >> > To unsubscribe e-mail: [email protected] > >>>>> >> > > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> --------------------------------------------------------------------- > >>>>> >> To unsubscribe e-mail: [email protected] > >>>>> >> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe e-mail: [email protected] > >>>>> > >>>>> > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
