[
https://issues.apache.org/jira/browse/SPARK-34507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291311#comment-17291311
]
Guillaume Martres commented on SPARK-34507:
-------------------------------------------
> I think scala-2.13 profile should overrdie this property:
* What decide which profile gets used?
* If the profile is indeed used, then why does
[https://github.com/apache/spark/blob/8f994cbb4a18558c2e81516ef1e339d9c8fa0d41/dev/change-scala-version.sh#L65]
also change the default value of scala.binary.version but not scala.version,
shouldn't they match?
> are you really facing some issues when you are pulling Scala 2.13 artifacts
> from the Maven?
Yes, here's a simple reproducer, using [coursier|https://get-coursier.io/] to
fetch spark I get the following jars:
{code:java}
% cs fetch -r https://repository.apache.org/content/repositories/snapshots
org.apache.spark:spark-tags_2.13:3.2.0-SNAPSHOT
/home/smarter/.cache/coursier/v1/https/repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-tags_2.13/3.2.0-SNAPSHOT/spark-tags_2.13-3.2.0-20210225.010548-31.jar
/home/smarter/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.10/scala-library-2.12.10.jar
/home/smarter/.cache/coursier/v1/https/repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
{code}
Coursier is used by sbt to resolve dependencies, so I assume this affects at
least all sbt projects, but I just found out this is a known issue in coursier:
[https://github.com/coursier/coursier/issues/973]
> Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12
> -------------------------------------------------------------------------
>
> Key: SPARK-34507
> URL: https://issues.apache.org/jira/browse/SPARK-34507
> Project: Spark
> Issue Type: Sub-task
> Components: Build
> Affects Versions: 3.2.0
> Reporter: Guillaume Martres
> Priority: Major
>
> Snapshots of Spark 3.2 built against Scala 2.13 are available at
> [https://repository.apache.org/content/repositories/snapshots/org/apache/spark/,]
> but they seem to depend on Scala 2.12. Specifically if I look at
> [https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-parent_2.13/3.2.0-SNAPSHOT/spark-parent_2.13-3.2.0-20210223.010629-29.pom]
> I see:
> {code:java}
> <scala.version>2.12.10</scala.version>
> <scala.binary.version>2.13</scala.binary.version{code}
> It looks like
> [https://github.com/apache/spark/blob/8f994cbb4a18558c2e81516ef1e339d9c8fa0d41/dev/change-scala-version.sh#L65]
> needs to be updated to also change the `scala.version` and not just the
> `scala.binary.version`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
