Change <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.13.11-M2</version> </dependency> <dependency>
to <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun <lingzhe....@hirain.com>: > Hi Mich, > > Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8 > <https://github.com/apache/spark/blob/88f69d6f92860823b1a90bc162ebca2b7c8132fc/pom.xml#L170>. > Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to > the major(13) and the minor version(8). Not using any of these may cause > unexpected behaviour(though scala claims compatibility among minor version > changes, I've encountered problem using the scala package with the same > major version and different minor version. That may due to bug fixes and > upgrade of scala itself.). > And although I did not encountered such problem, this > <https://stackoverflow.com/a/26411339/19476830>can be a a pitfall for you. > > ------------------------------ > Best Regards! > ........................................................................... > Lingzhe Sun > Hirain Technology > > > *From:* Mich Talebzadeh <mich.talebza...@gmail.com> > *Date:* 2023-05-29 17:55 > *To:* Bjørn Jørgensen <bjornjorgen...@gmail.com> > *CC:* user @spark <user@spark.apache.org> > *Subject:* Re: maven with Spark 3.4.0 fails compilation > Thanks for your helpful comments Bjorn. > > I managed to compile the code with maven but when it run it fails with > > Application is ReduceByKey > > Exception in thread "main" java.lang.NoSuchMethodError: > scala.package$.Seq()Lscala/collection/immutable/Seq$; > at ReduceByKey$.main(ReduceByKey.scala:23) > at ReduceByKey.main(ReduceByKey.scala) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at org.apache.spark.deploy.SparkSubmit.org > $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020) > at > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) > at > org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) > at > org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111) > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > I attach the pom.xml and the sample scala code is self contained and > basic. Again it runs with SBT with no issues. > > FYI, my scala version on host is > > scala -version > Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and > Lightbend, Inc. > > I think I have a scala incompatible somewhere again > > Cheers > > > Mich Talebzadeh, > Lead Solutions Architect/Engineering Lead > Palantir Technologies Limited > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Sun, 28 May 2023 at 20:29, Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> From chatgpt4 >> >> >> The problem appears to be that there is a mismatch between the version of >> Scala used by the Scala Maven plugin and the version of the Scala library >> defined as a dependency in your POM. You've defined your Scala version in >> your properties as `2.12.17` but you're pulling in `scala-library` version >> `2.13.6` as a dependency. >> >> The Scala Maven plugin will be using the Scala version defined in the >> `scala.version` property for compilation, but then it tries to load classes >> from a different Scala version, hence the error. >> >> To resolve this issue, make sure the `scala.version` property matches the >> version of `scala-library` defined in your dependencies. In your case, you >> may want to change `scala.version` to `2.13.6`. >> >> Here's the corrected part of your POM: >> >> ```xml >> <properties> >> <maven.compiler.source>1.7</maven.compiler.source> >> <maven.compiler.target>1.7</maven.compiler.target> >> <encoding>UTF-8</encoding> >> <scala.version>2.13.6</scala.version> <!-- Here's the change --> >> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >> </properties> >> ``` >> >> Additionally, ensure that the Scala versions in the Spark dependencies >> match the `scala.version` property as well. If you've updated the Scala >> version to `2.13.6`, the artifactIds for Spark dependencies should be >> `spark-core_2.13` and `spark-sql_2.13`. >> >> Another thing to consider: your Java version defined in >> `maven.compiler.source` and `maven.compiler.target` is `1.7`, which is >> quite outdated and might not be compatible with the latest versions of >> these libraries. Consider updating to a more recent version of Java, such >> as Java 8 or above, depending on the requirements of the libraries you're >> using. >> >> >> >> The same problem persists in this updated POM file - there's a mismatch >> in the Scala version declared in the properties and the version used in >> your dependencies. Here's what you need to update: >> >> 1. Update the Scala version in your properties to match the Scala library >> and your Spark dependencies: >> >> ```xml >> <properties> >> <maven.compiler.source>1.7</maven.compiler.source> >> <maven.compiler.target>1.7</maven.compiler.target> >> <encoding>UTF-8</encoding> >> <scala.version>2.13.6</scala.version> <!-- updated Scala version --> >> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >> </properties> >> ``` >> >> 2. Make sure all your Spark dependencies use the same Scala version. In >> this case, I see `spark-streaming-kafka_2.11` which should be >> `spark-streaming-kafka_2.13` if you're using Scala `2.13.6`. >> >> ```xml >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-streaming-kafka_2.13</artifactId> <!-- updated to >> 2.13 --> >> <version>1.6.3</version> <!-- this might also need to be updated as >> this is a very old version --> >> <scope>provided</scope> >> </dependency> >> ``` >> >> 3. As mentioned in the previous message, your Java version >> (`maven.compiler.source` and `maven.compiler.target`) is also quite >> outdated. Depending on the requirements of the libraries you're using, you >> might want to update this to a newer version, such as Java 8 or above. >> >> Finally, ensure that the correct versions of these libraries exist in >> your Maven repository or are available in the central Maven repository. If >> the versions don't match, Maven will not be able to find and download the >> correct dependencies, which can lead to problems. >> >> Please note that it's crucial to maintain consistency in your Scala and >> Java versions across your project and its dependencies to avoid these kinds >> of issues. Mixing versions can lead to binary incompatibility errors, such >> as the one you're seeing. >> >> >> >> >> >> >> >> The differences in behavior between SBT and Maven when resolving Scala >> dependencies can be attributed to how they each handle Scala binary >> versions. >> >> SBT is specifically designed for Scala projects and has built-in support >> for handling Scala's binary compatibility issues. When you define a Scala >> library dependency in SBT, you can specify the Scala binary version as >> "_2.12", "_2.13", etc. in the library artifact name. SBT will then >> automatically replace this with the actual Scala binary version defined in >> your SBT configuration. So, if you've defined different Scala versions for >> your project and for a specific dependency, SBT will handle this gracefully. >> >> Maven, on the other hand, is more generic and does not have the same >> level of built-in support for Scala's binary versions. In Maven, the Scala >> version is typically hardcoded in the artifact name, and if this doesn't >> match the actual Scala version used in your project, it can lead to binary >> compatibility issues. >> >> This difference between SBT and Maven means that a project can work fine >> when built with SBT but fail when built with Maven, due to these Scala >> version discrepancies. To avoid these issues in Maven, you need to ensure >> that the Scala version is consistent across your project configuration and >> all your dependencies. >> >> Also, another reason for this can be because SBT often fetches >> dependencies on the fly, while Maven requires them to be explicitly >> declared. SBT can download and link the correct Scala library version even >> if it's not explicitly declared, while Maven will only use the versions >> that have been specified in the POM file. >> >> To summarize, SBT is more flexible and Scala-oriented than Maven, which >> can lead to different behavior when handling Scala dependencies. When using >> Maven for Scala projects, it's essential to ensure that all Scala versions >> match across the project. >> >> søn. 28. mai 2023 kl. 21:03 skrev Mich Talebzadeh < >> mich.talebza...@gmail.com>: >> >>> This compilation works fine with SBT but fails with maven! >>> >>> Spark version 3.4.0 >>> Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c) >>> Java version: 11.0.1, vendor: Oracle Corporation, runtime: >>> /opt/jdk-11.0.1 >>> >>> This from the pom.xml file >>> >>> <project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi=" >>> https://www.w3.org/2001/XMLSchema-instance" >>> xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 >>> https://maven.apache.org/maven-v4_0_0.xsd"> >>> <modelVersion>4.0.0</modelVersion> >>> <groupId>spark</groupId> >>> <version>3.0</version> >>> <name>${project.artifactId}</name> >>> >>> <properties> >>> <maven.compiler.source>1.7</maven.compiler.source> >>> <maven.compiler.target>1.7</maven.compiler.target> >>> <encoding>UTF-8</encoding> >>> <scala.version>2.12.17</scala.version> >>> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >>> </properties> >>> <dependencies> >>> <dependency> >>> <groupId>org.scala-lang</groupId> >>> <artifactId>scala-library</artifactId> >>> <version>2.13.6</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-core_2.13</artifactId> >>> <version>3.4.0</version> >>> <scope>provided</scope> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-sql_2.13</artifactId> >>> <version>3.4.0</version> >>> <scope>provided</scope> >>> >>> The pom file is attached. These are the errors I am getting >>> >>> [ERROR] error: error while loading package, class file >>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/reflect/package.class)' >>> is broken >>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature >>> of package.class: Scala signature package has wrong version >>> [ERROR] error: error while loading package, class file >>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/package.class)' >>> is broken >>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature >>> of package.class: Scala signature package has wrong version >>> [ERROR] error: error while loading package, class file >>> '/home/hduser/.m2/repository/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar(scala/collection/package.class)' >>> is broken >>> [ERROR] (class java.lang.RuntimeException/error reading Scala signature >>> of package.class: Scala signature package has wrong version >>> [ERROR] three errors found >>> [ERROR] Failed to execute goal >>> org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project >>> scala: wrap: org.apache.commons.exec.ExecuteException: Process exited with >>> an error: 1(Exit value: 1) -> [Help 1] >>> >>> Any ideas will be appreciated. >>> >>> Mich Talebzadeh, >>> Lead Solutions Architect/Engineering Lead >>> Palantir Technologies Limited >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297