Sergey Torgashov created SPARK-27864:
----------------------------------------
Summary: spark-submit 2.4 cannot run apps compiled with Scala 2.12
Key: SPARK-27864
URL: https://issues.apache.org/jira/browse/SPARK-27864
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.4.3
Environment: * {{MacOS Mojave 10.14.5}}
** {{Spark 2.4.3 (installed with homebrew)}}
** {{OracleJDK (}}java version "1.8.0_202")
* Windows 10 Pro 1809
** Spark 2.4.3 (installed with scoop)
** Oracle JDK (java version "1.8.0_201")
Reporter: Sergey Torgashov
Installed: Spark 2.4.3 (checked on both Mac OS and Windows)
build.sbt:
{code:java}
name := "test-submit"
version := "1"
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"
{code}
SubmitApp.scala:
{code:java}
import org.apache.spark.sql.SparkSession
object SubmitApp {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().master("local").getOrCreate()
spark.sparkContext.parallelize(Seq(1, 2, 3)).collect().foreach(println)
}
}
{code}
Commands to reproduce:
{code:java}
$ sbt package
$ spark-submit --class <package-to>.SubmitApp
./target/scala-2.12/test-submit_2.12-1.jar
{code}
Expected result: {{spark-submit}} executes successfully.
Actual result: {{spark-submit}} fails to execute with the following error:
{code}
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Predef$.intArrayOps([I)[I
at <package-to>.SubmitApp$.main(SubmitApp.scala:8)
at <package-to>.SubmitApp.main(SubmitApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
