[
https://issues.apache.org/jira/browse/SPARK-27864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854418#comment-16854418
]
Sergey Torgashov commented on SPARK-27864:
------------------------------------------
[~dongjoon], I tried the binary that you suggested. It doesn't work for me as
well, but in a different way:
I unpacked *spark-2.4.3-bin-without-hadoop-scala-2.12.tgz* into a separate
directory,
Then I inserted a path to this unpacked version into the first place in the
{{PATH}} env var:
{{export
PATH=<path-to-spark-2.4.3-bin-without-hadoop-scala-2.12>/bin:$\{PATH}}}
I also configured {{SPARK_HOME}} to
{{<path-to-unpacked-spark-2.4.3-bin-without-hadoop-scala-2.12>}}
Now when I try to run *spark-submit* or *spark-shell*, they both fail with the
following error:
{{Error: A JNI error has occurred, please check your installation and try
again}}
{{Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger}}
{{ at java.lang.Class.getDeclaredMethods0(Native Method)}}
{{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}}
{{ at java.lang.Class.privateGetMethodRecursive(Class.java:3048)}}
{{ at java.lang.Class.getMethod0(Class.java:3018)}}
{{ at java.lang.Class.getMethod(Class.java:1784)}}
{{ at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)}}
{{ at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)}}
{{Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger}}
{{ at java.net.URLClassLoader.findClass(URLClassLoader.java:382)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}}
{{ at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}}
{{ ... 7 more}}
I guess, I did something wrong, but I'm not sure, what exactly. Could you
suggest, please?
> spark-submit 2.4 cannot run apps compiled with Scala 2.12
> ---------------------------------------------------------
>
> Key: SPARK-27864
> URL: https://issues.apache.org/jira/browse/SPARK-27864
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.4.3
> Environment: * {{MacOS Mojave 10.14.5}}
> ** {{Spark 2.4.3 (installed with homebrew)}}
> ** Oracle JDK (java version "1.8.0_202")
> * Windows 10 Pro 1809
> ** Spark 2.4.3 (installed with scoop)
> ** Oracle JDK (java version "1.8.0_201")
> Reporter: Sergey Torgashov
> Priority: Major
>
> Installed: Spark 2.4.3 (checked on both Mac OS and Windows)
> build.sbt:
> {code:java}
> name := "test-submit"
> version := "1"
> scalaVersion := "2.12.8"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"
> {code}
> SubmitApp.scala:
> {code:java}
> import org.apache.spark.sql.SparkSession
> object SubmitApp {
> def main(args: Array[String]): Unit = {
> val spark = SparkSession.builder().master("local").getOrCreate()
> spark.sparkContext.parallelize(Seq(1, 2, 3)).collect().foreach(println)
> }
> }
> {code}
> Commands to reproduce:
> {code:java}
> $ sbt package
> $ spark-submit --class <package-to>.SubmitApp
> ./target/scala-2.12/test-submit_2.12-1.jar
> {code}
> Expected result: {{spark-submit}} executes successfully.
> Actual result: {{spark-submit}} fails to execute with the following error:
> {code}
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.Predef$.intArrayOps([I)[I
> at <package-to>.SubmitApp$.main(SubmitApp.scala:8)
> at <package-to>.SubmitApp.main(SubmitApp.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
