[
https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841287#comment-16841287
]
Lukas Rytz commented on SPARK-25075:
------------------------------------
In our own interest of testing Scala the 2.13 RCs, I took a stab at compiling
spark core on 2.13.
My goal was *not* to get something running, just compiling and seeing what kind
of breaking changes there are. So I chose a slightly adventurous methodology: I
updated the {{scalaVersion}} to 2.13.0-pre-06392a5-SNAPSHOT (a very recent
local build), but forced the {{scalaBinaryVersion}} to 2.12, so that the 2.12
dependencies end up on the classpath. That way I didn't have to worry about
missing / incompatible dependencies.
The first step was to avoid using {{scala.Seq}}, I used scalafix to rewrite all
references of the type {{scala.Seq}} to {{scala.collection.Seq}}. As discussed
on https://issues.apache.org/jira/browse/SPARK-27681, this is not necessarily
the best solution, but the easiest.
Here's a list of other breaking changes:
* {{foo(someMutableOrGenericCollection: _*)}} no longer works, because varargs
de-sugars to {{scala.Seq}}, so an immutable collection is now required. Calling
{{.toSeq}} works, but is inefficient. Better build an immutable collection from
the beginning. For arrays, {{immutable.ArraySeq.unsafeWrapArray}} can be used.
Maybe the standard library should provide an unsafe {{immutable.SeqWrapper}}
that wraps a {{collection.Seq}} for the cases when the users are certain it's
safe.
* Views are quite different in 2.13, for example {{seq.view}} is no longer a
{{Seq}}, views are a separate hierarchy. This needs some adjustments, not too
difficult.
* Parallel collections are a separate module now
([https://github.com/scala/scala-parallel-collections]), no longer in the
standard library. However, you might want to use StreamConverters instead to do
parallel processing via a Java stream
([https://github.com/scala/scala/blob/2.13.x/src/library/scala/jdk/StreamConverters.scala]).
* Subclasses of collections need some adjustments: {{BoundedPriorityQueue}},
{{TimeStampedHashMap}}. For example, {{++=}} cannot be overridden anymore, as
it's a final alias for {{addAll}} now.
The other changes are relatively minor. The brannch is here - for reference,
too hacky to be actually useful:
[https://github.com/lrytz/spark/commits/2.13-experiment]. {{sbt core/compile}}
passes (except for scalastyle).
Overall, this is more or less what we expected in terms of breaking changes. We
definietly want to use the time between now and 2.13.0 final to improve
migration documentation
([https://docs.scala-lang.org/overviews/core/collections-migration-213.html])
and scalafix rules ([https://github.com/scala/scala-rewrites] is in its early
days).
> Build and test Spark against Scala 2.13
> ---------------------------------------
>
> Key: SPARK-25075
> URL: https://issues.apache.org/jira/browse/SPARK-25075
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Project Infra
> Affects Versions: 3.0.0
> Reporter: Guillaume Massé
> Priority: Major
>
> This umbrella JIRA tracks the requirements for building and testing Spark
> against the current Scala 2.13 milestone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
