I don't know whether it is related but Scala 2.12.17 is fine for the Spark 3 family (compile and run) . I spent a day compiling Spark 3.4.0 code against Scala 2.13.8 with maven and was getting all sorts of weird and wonderful errors at runtime.
HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Tue, 30 May 2023 at 01:59, Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Shall we initiate a new discussion thread for Scala 2.13 by default? While > I'm not an expert on this area, it sounds like the change is major and > (probably) breaking. It seems to be worth having a separate > discussion thread rather than just treat it like one of 25 items. > > On Tue, May 30, 2023 at 9:54 AM Sean Owen <sro...@gmail.com> wrote: > >> It does seem risky; there are still likely libs out there that don't >> cross compile for 2.13. I would make it the default at 4.0, myself. >> >> On Mon, May 29, 2023 at 7:16 PM Hyukjin Kwon <gurwls...@apache.org> >> wrote: >> >>> While I support going forward with a higher version, actually using >>> Scala 2.13 by default is a big deal especially in a way that: >>> >>> - Users would likely download the built-in version assuming that >>> it’s backward binary compatible. >>> - PyPI doesn't allow specifying the Scala version, meaning that >>> users wouldn’t have a way to 'pip install pyspark' based on Scala 2.12. >>> >>> I wonder if it’s safer to do it in Spark 4 (which I believe will be >>> discussed soon). >>> >>> >>> On Mon, 29 May 2023 at 13:21, Jia Fan <fan...@apache.org> wrote: >>> >>>> Thanks Dongjoon! >>>> There are some ticket I want to share. >>>> SPARK-39420 Support ANALYZE TABLE on v2 tables >>>> SPARK-42750 Support INSERT INTO by name >>>> SPARK-43521 Support CREATE TABLE LIKE FILE >>>> >>>> Dongjoon Hyun <dongj...@apache.org> 于2023年5月29日周一 08:42写道： >>>> >>>>> Hi, All. >>>>> >>>>> Apache Spark 3.5.0 is scheduled for August (1st Release Candidate) and >>>>> currently a few notable things are under discussions in the mailing list. >>>>> >>>>> I believe it's a good time to share a short summary list (containing >>>>> both completed and in-progress items) to give a highlight in advance and >>>>> to >>>>> collect your targets too. >>>>> >>>>> Please share your expectations or working items if you want to >>>>> prioritize them more in the community in Apache Spark 3.5.0 timeframe. >>>>> >>>>> (Sorted by ID) >>>>> SPARK-40497 Upgrade Scala 2.13.11 >>>>> SPARK-42452 Remove hadoop-2 profile from Apache Spark 3.5.0 >>>>> SPARK-42913 Upgrade to Hadoop 3.3.5 (aws-java-sdk-bundle: 1.12.262 -> >>>>> 1.12.316) >>>>> SPARK-43024 Upgrade Pandas to 2.0.0 >>>>> SPARK-43200 Remove Hadoop 2 reference in docs >>>>> SPARK-43347 Remove Python 3.7 Support >>>>> SPARK-43348 Support Python 3.8 in PyPy3 >>>>> SPARK-43351 Add Spark Connect Go prototype code and example >>>>> SPARK-43379 Deprecate old Java 8 versions prior to 8u371 >>>>> SPARK-43394 Upgrade to Maven 3.8.8 >>>>> SPARK-43436 Upgrade to RocksDbjni 8.1.1.1 >>>>> SPARK-43446 Upgrade to Apache Arrow 12.0.0 >>>>> SPARK-43447 Support R 4.3.0 >>>>> SPARK-43489 Remove protobuf 2.5.0 >>>>> SPARK-43519 Bump Parquet to 1.13.1 >>>>> SPARK-43581 Upgrade kubernetes-client to 6.6.2 >>>>> SPARK-43588 Upgrade to ASM 9.5 >>>>> SPARK-43600 Update K8s doc to recommend K8s 1.24+ >>>>> SPARK-43738 Upgrade to DropWizard Metrics 4.2.18 >>>>> SPARK-43831 Build and Run Spark on Java 21 >>>>> SPARK-43832 Upgrade to Scala 2.12.18 >>>>> SPARK-43836 Make Scala 2.13 as default in Spark 3.5 >>>>> SPARK-43842 Upgrade gcs-connector to 2.2.14 >>>>> SPARK-43844 Update to ORC 1.9.0 >>>>> UMBRELLA: Add SQL functions into Scala, Python and R API >>>>> >>>>> Thanks, >>>>> Dongjoon. >>>>> >>>>> PS. The above is not a list of release blockers. Instead, it could be >>>>> a nice-to-have from someone's perspective. >>>>> >>>>
