<scala.version>2.13.8</scala.version> <https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/pom.xml#LL3606C46-L3606C46>
you must change 2.13.6 to 2.13.8 man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh <mich.talebza...@gmail.com >: > Thanks everyone. Still not much progress :(. It is becoming a bit > confusing as I am getting this error > > Compiling ReduceByKey > [INFO] Scanning for projects... > [INFO] > [INFO] -------------------------< spark:ReduceByKey > >-------------------------- > [INFO] Building ReduceByKey 3.0 > [INFO] from pom.xml > [INFO] --------------------------------[ jar > ]--------------------------------- > [INFO] > [INFO] --- resources:3.3.0:resources (default-resources) @ ReduceByKey --- > [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy > filtered resources, i.e. build is platform dependent! > [INFO] skip non existing resourceDirectory > /data6/hduser/scala/ReduceByKey/src/main/resources > [INFO] > [INFO] --- compiler:3.10.1:compile (default-compile) @ ReduceByKey --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- scala:2.15.2:compile (default) @ ReduceByKey --- > [INFO] Checking for multiple versions of scala > [WARNING] Expected all dependencies to require Scala version: 2.13.8 > [WARNING] spark:ReduceByKey:3.0 requires scala version: 2.13.8 > [WARNING] org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4 > requires scala version: 2.13.6 > [WARNING] Multiple versions of scala libraries detected! > [INFO] includes = [**/*.java,**/*.scala,] > [INFO] excludes = [] > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- resources:3.3.0:testResources (default-testResources) @ > ReduceByKey --- > [WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy > filtered resources, i.e. build is platform dependent! > [INFO] skip non existing resourceDirectory > /data6/hduser/scala/ReduceByKey/src/test/resources > [INFO] > [INFO] --- compiler:3.10.1:testCompile (default-testCompile) @ ReduceByKey > --- > [INFO] No sources to compile > [INFO] > [INFO] --- surefire:3.0.0:test (default-test) @ ReduceByKey --- > [INFO] No tests to run. > [INFO] > [INFO] --- jar:3.3.0:jar (default-jar) @ ReduceByKey --- > [INFO] Building jar: /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar > [INFO] > [INFO] --- shade:1.6:shade (default) @ ReduceByKey --- > [INFO] Including org.scala-lang:scala-library:jar:2.13.8 in the shaded jar. > [INFO] Replacing original artifact with shaded artifact. > [INFO] Replacing /data6/hduser/scala/ReduceByKey/target/ReduceByKey.jar > with /data6/hduser/scala/ReduceByKey/target/ReduceByKey-3.0-shaded.jar > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 3.321 s > [INFO] Finished at: 2023-05-29T16:54:47+01:00 > [INFO] > ------------------------------------------------------------------------ > [WARNING] > [WARNING] Plugin validation issues were detected in 4 plugin(s) > [WARNING] > [WARNING] * org.scala-tools:maven-scala-plugin:2.15.2 > [WARNING] * org.apache.maven.plugins:maven-compiler-plugin:3.10.1 > [WARNING] * org.apache.maven.plugins:maven-shade-plugin:1.6 > [WARNING] * org.apache.maven.plugins:maven-resources-plugin:3.3.0 > [WARNING] > [WARNING] For more or less details, use 'maven.plugin.validation' property > with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE] > [WARNING] > Completed compiling > Mon May 29 16:54:47 BST 2023 , Running in **local mode** > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > > Application is ReduceByKey > > Exception in thread "main" java.lang.NoSuchMethodError: > scala.package$.Seq()Lscala/collection/immutable/Seq$; > at ReduceByKey$.main(ReduceByKey.scala:23) > at ReduceByKey.main(ReduceByKey.scala) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at org.apache.spark.deploy.SparkSubmit.org > $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020) > at > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) > at > org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) > at > org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111) > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > > Part of pom.xml is here > > > <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi=" > https://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 > https://maven.apache.org/maven-v4_0_0.xsd";> > <modelVersion>4.0.0</modelVersion> > <groupId>spark</groupId> > <version>3.0</version> > <artifactId>ReduceByKey</artifactId> > <name>${project.artifactId}</name> > > <properties> > <maven.compiler.source>11.0.1</maven.compiler.source> > <maven.compiler.target>11.0.1</maven.compiler.target> > <encoding>UTF-8</encoding> > <scala.version>2.13.8</scala.version> <!-- Is this correct for spark 3.4? > --> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> > </properties> > > <dependencies> > <dependency> > <groupId>org.scala-lang</groupId> > <artifactId>scala-library</artifactId> > <version>2.13.8</version> <!-- Is this correct for spark 3.4? --> > </dependency> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core_2.13</artifactId> <!-- Is this correct for > spark 3.4? --> > <version>3.4.0</version> > <scope>provided</scope> > </dependency> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-sql_2.13</artifactId> > <version>3.4.0</version> > <scope>provided</scope> > </dependency> > > > Thanks > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 29 May 2023 at 13:44, Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> Change >> >> <dependencies> >> <dependency> >> <groupId>org.scala-lang</groupId> >> <artifactId>scala-library</artifactId> >> <version>2.13.11-M2</version> >> </dependency> >> <dependency> >> >> to >> >> <dependencies> >> <dependency> >> <groupId>org.scala-lang</groupId> >> <artifactId>scala-library</artifactId> >> <version>${scala.version}</version> >> </dependency> >> >> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <lingzhe....@hirain.com>: >> >>> Hi Mich, >>> >>> Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8 >>> <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>. >>> Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to >>> the major(13) and the minor version(8). Not using any of these may cause >>> unexpected behaviour(though scala claims compatibility among minor version >>> changes, I've encountered problem using the scala package with the same >>> major version and different minor version. That may due to bug fixes and >>> upgrade of scala itself.). >>> And although I did not encountered such problem, this >>> <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for >>> you. >>> >>> ------------------------------ >>> Best Regards! >>> >>> ........................................................................... >>> Lingzhe Sun >>> Hirain Technology >>> >>> >>> *From:* Mich Talebzadeh <mich.talebza...@gmail.com> >>> *Date:* 2023-05-29 17:55 >>> *To:* Bjørn Jørgensen <bjornjorgen...@gmail.com> >>> *CC:* user @spark <user@spark.apache.org> >>> *Subject:* Re: maven with Spark 3.4.0 fails compilation >>> Thanks for your helpful comments Bjorn. >>> >>> I managed to compile the code with maven but when it run it fails with >>> >>> Application is ReduceByKey >>> >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> scala.package$.Seq()Lscala/collection/immutable/Seq$; >>> at ReduceByKey$.main(ReduceByKey.scala:23) >>> at ReduceByKey.main(ReduceByKey.scala) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) >>> at org.apache.spark.deploy.SparkSubmit.org >>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020) >>> at >>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) >>> at >>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) >>> at >>> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) >>> at >>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111) >>> at >>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> I attach the pom.xml and the sample scala code is self contained and >>> basic. Again it runs with SBT with no issues. >>> >>> FYI, my scala version on host is >>> >>> scala -version >>> Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and >>> Lightbend, Inc. >>> >>> I think I have a scala incompatible somewhere again >>> >>> Cheers >>> >>> >>> Mich Talebzadeh, >>> Lead Solutions Architect/Engineering Lead >>> Palantir Technologies Limited >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bjornjorgen...@gmail.com> >>> wrote: >>> >>>> From chatgpt4 >>>> >>>> >>>> The problem appears to be that there is a mismatch between the version >>>> of Scala used by the Scala Maven plugin and the version of the Scala >>>> library defined as a dependency in your POM. You've defined your Scala >>>> version in your properties as `2.12.17` but you're pulling in >>>> `scala-library` version `2.13.6` as a dependency. >>>> >>>> The Scala Maven plugin will be using the Scala version defined in the >>>> `scala.version` property for compilation, but then it tries to load classes >>>> from a different Scala version, hence the error. >>>> >>>> To resolve this issue, make sure the `scala.version` property matches >>>> the version of `scala-library` defined in your dependencies. In your case, >>>> you may want to change `scala.version` to `2.13.6`. >>>> >>>> Here's the corrected part of your POM: >>>> >>>> ```xml >>>> <properties> >>>> <maven.compiler.source>1.7</maven.compiler.source> >>>> <maven.compiler.target>1.7</maven.compiler.target> >>>> <encoding>UTF-8</encoding> >>>> <scala.version>2.13.6</scala.version> <!-- Here's the change --> >>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>>> </properties> >>>> ``` >>>> >>>> Additionally, ensure that the Scala versions in the Spark dependencies >>>> match the `scala.version` property as well. If you've updated the Scala >>>> version to `2.13.6`, the artifactIds for Spark dependencies should be >>>> `spark-core_2.13` and `spark-sql_2.13`. >>>> >>>> Another thing to consider: your Java version defined in >>>> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is >>>> quite outdated and might not be compatible with the latest versions of >>>> these libraries. Consider updating to a more recent version of Java, such >>>> as Java 8 or above, depending on the requirements of the libraries you're >>>> using. >>>> >>>> >>>> >>>> The same problem persists in this updated POM file - there's a mismatch >>>> in the Scala version declared in the properties and the version used in >>>> your dependencies. Here's what you need to update: >>>> >>>> 1. Update the Scala version in your properties to match the Scala >>>> library and your Spark dependencies: >>>> >>>> ```xml >>>> <properties> >>>> <maven.compiler.source>1.7</maven.compiler.source> >>>> <maven.compiler.target>1.7</maven.compiler.target> >>>> <encoding>UTF-8</encoding> >>>> <scala.version>2.13.6</scala.version> <!-- updated Scala version --> >>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>>> </properties> >>>> ``` >>>> >>>> 2. Make sure all your Spark dependencies use the same Scala version. In >>>> this case, I see `spark-streaming-kafka_2.11` which should be >>>> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`. >>>> >>>> ```xml >>>> <dependency> >>>> <groupId>org.apache.spark</groupId> >>>> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to >>>> 2.13 --> >>>> <version>1.6.3</version> <!-- this might also need to be updated as >>>> this is a very old version --> >>>> <scope>provided</scope> >>>> </dependency> >>>> ``` >>>> >>>> 3. As mentioned in the previous message, your Java version >>>> (`maven.compiler.source` and `maven.compiler.target`) is also quite >>>> outdated. Depending on the requirements of the libraries you're using, you >>>> might want to update this to a newer version, such as Java 8 or above. >>>> >>>> Finally, ensure that the correct versions of these libraries exist in >>>> your Maven repository or are available in the central Maven repository. If >>>> the versions don't match, Maven will not be able to find and download the >>>> correct dependencies, which can lead to problems. >>>> >>>> Please note that it's crucial to maintain consistency in your Scala and >>>> Java versions across your project and its dependencies to avoid these kinds >>>> of issues. Mixing versions can lead to binary incompatibility errors, such >>>> as the one you're seeing. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> The differences in behavior between SBT and Maven when resolving Scala >>>> dependencies can be attributed to how they each handle Scala binary >>>> versions. >>>> >>>> SBT is specifically designed for Scala projects and has built-in >>>> support for handling Scala's binary compatibility issues. When you define a >>>> Scala library dependency in SBT, you can specify the Scala binary version >>>> as "_2.12", "_2.13", etc. in the library artifact name. SBT will then >>>> automatically replace this with the actual Scala binary version defined in >>>> your SBT configuration. So, if you've defined different Scala versions for >>>> your project and for a specific dependency, SBT will handle this >>>> gracefully. >>>> >>>> Maven, on the other hand, is more generic and does not have the same >>>> level of built-in support for Scala's binary versions. In Maven, the Scala >>>> version is typically hardcoded in the artifact name, and if this doesn't >>>> match the actual Scala version used in your project, it can lead to binary >>>> compatibility issues. >>>> >>>> This difference between SBT and Maven means that a project can work >>>> fine when built with SBT but fail when built with Maven, due to these Scala >>>> version discrepancies. To avoid these issues in Maven, you need to ensure >>>> that the Scala version is consistent across your project configuration and >>>> all your dependencies. >>>> >>>> Also, another reason for this can be because SBT often fetches >>>> dependencies on the fly, while Maven requires them to be explicitly >>>> declared. SBT can download and link the correct Scala library version even >>>> if it's not explicitly declared, while Maven will only use the versions >>>> that have been specified in the POM file. >>>> >>>> To summarize, SBT is more flexible and Scala-oriented than Maven, which >>>> can lead to different behavior when handling Scala dependencies. When using >>>> Maven for Scala projects, it's essential to ensure that all Scala versions >>>> match across the project. >>>> >>>> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh < >>>> mich.talebza...@gmail.com>: >>>> >>>>> This compilation works fine with SBT but fails with maven! >>>>> >>>>> Spark version 3.4.0 >>>>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c) >>>>> Java version: 11.0.1, vendor: Oracle Corporation, runtime: >>>>> /opt/jdk-11.0.1 >>>>> >>>>> This from the pom.xml file >>>>> >>>>> <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi=" >>>>> https://www.w3.org/2001/XMLSchema-instance"; >>>>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 >>>>> https://maven.apache.org/maven-v4_0_0.xsd";> >>>>> <modelVersion>4.0.0</modelVersion> >>>>> <groupId>spark</groupId> >>>>> <version>3.0</version> >>>>> <name>${project.artifactId}</name> >>>>> >>>>> <properties> >>>>> <maven.compiler.source>1.7</maven.compiler.source> >>>>> <maven.compiler.target>1.7</maven.compiler.target> >>>>> <encoding>UTF-8</encoding> >>>>> <scala.version>2.12.17</scala.version> >>>>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>>>> </properties> >>>>> <dependencies> >>>>> <dependency> >>>>> <groupId>org.scala-lang</groupId> >>>>> <artifactId>scala-library</artifactId> >>>>> <version>2.13.6</version> >>>>> </dependency> >>>>> <dependency> >>>>> <groupId>org.apache.spark</groupId> >>>>> <artifactId>spark-core_2.13</artifactId> >>>>> <version>3.4.0</version> >>>>> <scope>provided</scope> >>>>> </dependency> >>>>> <dependency> >>>>> <groupId>org.apache.spark</groupId> >>>>> <artifactId>spark-sql_2.13</artifactId> >>>>> <version>3.4.0</version> >>>>> <scope>provided</scope> >>>>> >>>>> The pom file is attached. These are the errors I am getting >>>>> >>>>> [ERROR] error: error while loading package, class file >>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)' >>>>> is broken >>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala >>>>> signature of package.class: Scala signature package has wrong version >>>>> [ERROR] error: error while loading package, class file >>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)' >>>>> is broken >>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala >>>>> signature of package.class: Scala signature package has wrong version >>>>> [ERROR] error: error while loading package, class file >>>>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)' >>>>> is broken >>>>> [ERROR] (class java.lang.RuntimeException/error reading Scala >>>>> signature of package.class: Scala signature package has wrong version >>>>> [ERROR] three errors found >>>>> [ERROR] Failed to execute goal >>>>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project >>>>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with >>>>> an error: 1(Exit value: 1) -> [Help 1] >>>>> >>>>> Any ideas will be appreciated. >>>>> >>>>> Mich Talebzadeh, >>>>> Lead Solutions Architect/Engineering Lead >>>>> Palantir Technologies Limited >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> >>>> >>>> -- >>>> Bjørn Jørgensen >>>> Vestre Aspehaug 4, 6010 Ålesund >>>> Norge >>>> >>>> +47 480 94 297 >>>> >>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
