Anyway I did google on your error and found this scala project maven sync failed <https://youtrack.jetbrains.com/issue/SCL-18941> Is this the same as the one you are getting?
søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > Thanks Bjorn. I am aware of that. I just really wanted to create the uber > jar files with both sbt and maven in Intellij. > > > cheers > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark >> <https://spark.apache.org/docs/latest/building-spark.html> "Spark >> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark >> 3.0.0." >> >> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh < >> mich.talebza...@gmail.com>: >> >>> OK I decided to give a try to maven. >>> >>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip >>> apache-maven-3.8.4-bin.zip >>> >>> Then added to Windows env variable as MVN_HOME and added the bin >>> directory to path in windows. Restart intellij to pick up the correct path. >>> >>> Again on the command line in intellij do >>> >>> *mvn -v* >>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) >>> Maven home: d:\temp\apache-maven-3.8.4 >>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program >>> Files\Java\jdk1.8.0_73\jre >>> Default locale: en_GB, platform encoding: Cp1252 >>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" >>> >>> in Intellij add maven support to your project. Follow this link Add >>> Maven support to an existing project >>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html> >>> >>> There will be a pom.xml file under project directory >>> >>> [image: image.png] >>> >>> Edit that pom.xml file and add the following >>> >>> <?xml version="1.0" encoding="UTF-8"?> >>> <project xmlns="http://maven.apache.org/POM/4.0.0"; >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>> http://maven.apache.org/xsd/maven-4.0.0.xsd";> >>> <modelVersion>4.0.0</modelVersion> >>> >>> <groupId>spark</groupId> >>> <artifactId>MichTest</artifactId> >>> <version>1.0</version> >>> >>> <properties> >>> <maven.compiler.source>8</maven.compiler.source> >>> <maven.compiler.target>8</maven.compiler.target> >>> </properties> >>> <dependencies> >>> <dependency> >>> <groupId>org.scala-lang</groupId> >>> <artifactId>scala-library</artifactId> >>> <version>2.11.7</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-core_2.10</artifactId> >>> <version>2.0.0</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-sql_2.10</artifactId> >>> <version>2.0.0</version> >>> </dependency> >>> </dependencies> >>> </project> >>> >>> In intellij open a Terminal under project sub-directory where the pom >>> file is created and you edited. >>> >>> >>> *mvn clean* >>> >>> [INFO] Scanning for projects... >>> >>> [INFO] >>> >>> [INFO] ---------------------------< spark:MichTest >>> >--------------------------- >>> >>> [INFO] Building MichTest 1.0 >>> >>> [INFO] --------------------------------[ jar >>> ]--------------------------------- >>> >>> [INFO] >>> >>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest --- >>> >>> [INFO] Deleting D:\temp\intellij\MichTest\target >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> [INFO] BUILD SUCCESS >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> [INFO] Total time: 4.451 s >>> >>> [INFO] Finished at: 2022-02-27T19:37:57Z >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> *mvn compile* >>> >>> [INFO] Scanning for projects... >>> >>> [INFO] >>> >>> [INFO] ---------------------------< spark:MichTest >>> >--------------------------- >>> >>> [INFO] Building MichTest 1.0 >>> >>> [INFO] --------------------------------[ jar >>> ]--------------------------------- >>> >>> [INFO] >>> >>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >>> MichTest --- >>> >>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>> resources, i.e. build is platform dependent! >>> >>> [INFO] Copying 0 resource >>> >>> [INFO] >>> >>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ >>> MichTest --- >>> >>> [INFO] Nothing to compile - all classes are up to date >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> [INFO] BUILD SUCCESS >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> [INFO] Total time: 1.242 s >>> >>> [INFO] Finished at: 2022-02-27T19:38:58Z >>> >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> Now create the package >>> >>> *mvn package* >>> [INFO] Scanning for projects... >>> [INFO] >>> [INFO] ---------------------------< spark:MichTest >>> >--------------------------- >>> [INFO] Building MichTest 1.0 >>> [INFO] --------------------------------[ jar >>> ]--------------------------------- >>> [INFO] >>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >>> MichTest --- >>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>> resources, i.e. build is platform dependent! >>> [INFO] Copying 0 resource >>> [INFO] >>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ >>> MichTest --- >>> [INFO] Nothing to compile - all classes are up to date >>> [INFO] >>> [INFO] --- maven-resources-plugin:2.6:testResources >>> (default-testResources) @ MichTest --- >>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>> resources, i.e. build is platform dependent! >>> [INFO] skip non existing resourceDirectory >>> D:\temp\intellij\MichTest\src\test\resources >>> [INFO] >>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ >>> MichTest --- >>> [INFO] Nothing to compile - all classes are up to date >>> [INFO] >>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest >>> --- >>> [INFO] No tests to run. >>> [INFO] >>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest --- >>> [INFO] *Building jar: >>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar* >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] BUILD SUCCESS >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] Total time: 1.511 s >>> [INFO] Finished at: 2022-02-27T19:40:22Z >>> [INFO] >>> ------------------------------------------------------------------------ >>> >>> sbt should work from the same directory as well. >>> >>> I find it easier* to package from the command line in Intellij >>> Terminal. * >>> >>> Let me know if this is a wrong approach or if there are better ways of >>> doing it. >>> >>> HTH >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mich.talebza...@gmail.com> >>> wrote: >>> >>>> Got curious with this intellij stuff. >>>> >>>> I recall using sbt rather than MVN so go to terminal in your intellij >>>> and verify what is installed >>>> >>>> sbt -version >>>> sbt version in this project: 1.3.4 >>>> sbt script version: 1.3.4 >>>> >>>> scala -version >>>> >>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL >>>> >>>> java --version >>>> openjdk 11.0.7 2020-04-14 >>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10) >>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode) >>>> >>>> For now in the directory where you have Main.scala create build.sbt file >>>> >>>> // The simplest possible sbt build file is just one line: >>>> >>>> scalaVersion := "2.11.7" >>>> // That is, to create a valid sbt build, all you've got to do is define the >>>> // version of Scala you'd like your project to use. >>>> >>>> / To learn more about multi-project builds, head over to the official sbt >>>> // documentation at http://www.scala-sbt.org/documentation.html >>>> libraryDependencies += "org.scala-lang.modules" %% >>>> "scala-parser-combinators" % "1.1.2" >>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0" >>>> >>>> >>>> This is my Main.scala file a copy from sparkbyexample >>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/> >>>> >>>> >>>> package org.example >>>> import org.apache.spark.sql.SparkSession >>>> object SparkSessionTest extends App{ >>>> val spark = SparkSession.builder() >>>> .master("local[1]") >>>> .appName("SparkByExample") >>>> .getOrCreate(); >>>> >>>> println("First SparkContext:") >>>> println("APP Name :"+spark.sparkContext.appName); >>>> println("Deploy Mode :"+spark.sparkContext.deployMode); >>>> println("Master :"+spark.sparkContext.master); >>>> >>>> val sparkSession2 = SparkSession.builder() >>>> .master("local[1]") >>>> .appName("SparkByExample-test") >>>> .getOrCreate(); >>>> >>>> println("Second SparkContext:") >>>> println("APP Name :"+sparkSession2.sparkContext.appName); >>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode); >>>> println("Master :"+sparkSession2.sparkContext.master); >>>> } >>>> >>>> Go back to Terminal under directory where you have both files >>>> built.sbt and Main.scala >>>> >>>> >>>> *sbt clean* >>>> >>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>> >>>> [info] Loading project definition from >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>> >>>> [info] Loading settings for project scala from build.sbt ... >>>> >>>> [info] Set current project to MichTest (in build >>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>> >>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM >>>> >>>> *sbt compile* >>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>> [info] Loading project definition from >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>> [info] Loading settings for project scala from build.sbt ... >>>> [info] Set current project to MichTest (in build >>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>> [info] Executing in batch mode. For better performance use sbt's shell >>>> [warn] There may be incompatibilities among your library dependencies; >>>> run 'evicted' to see detailed eviction warnings. >>>> [info] Compiling 1 Scala source to >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes >>>> ... >>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM >>>> >>>> *sbt package* >>>> >>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>> >>>> [info] Loading project definition from >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>> >>>> [info] Loading settings for project scala from build.sbt ... >>>> >>>> [info] Set current project to MichTest (in build >>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>> >>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM >>>> >>>> *ls* >>>> >>>> >>>> Directory: >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala >>>> >>>> >>>> Mode LastWriteTime Length Name >>>> >>>> ---- ------------- ------ ---- >>>> >>>> d----- 2/27/2022 7:04 AM null >>>> >>>> d----- 2/27/2022 8:33 AM project >>>> >>>> d----- 2/27/2022 9:08 AM spark-warehouse >>>> >>>> d----- 2/27/2022 9:55 AM target >>>> >>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt >>>> >>>> Note that you have target directory and underneath scala-2.11 (in my >>>> case) and the uber jar file michtest_2.11-1.0.jar >>>> >>>> *ls* >>>> Directory: >>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11 >>>> Mode LastWriteTime Length Name >>>> ---- ------------- ------ ---- >>>> d----- 2/27/2022 9:55 AM classes >>>> d----- 2/27/2022 9:55 AM update >>>> -a---- 2/27/2022 9:56 AM 3938 >>>> *michtest_2.11-1.0.jar* >>>> >>>> >>>> These are old stuff but still shows how to create a jar file with sbt >>>> >>>> HTH >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sro...@gmail.com> wrote: >>>> >>>>> I don't think any of that is related, no. >>>>> How are you dependencies set up? manually with IJ, or in a build file >>>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken >>>>> care >>>>> of for you, but you app would definitely have to express a dependency on >>>>> Scala libs. >>>>> >>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bit...@bitfox.top> wrote: >>>>> >>>>>> Java SDK installed? >>>>>> >>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka < >>>>>> connectsac...@gmail.com> wrote: >>>>>> >>>>>>> Hello , >>>>>>> >>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ >>>>>>> first then also it's giving same error >>>>>>> >>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp >>>>>>> >>>>>>> Thanks >>>>>>> Rajat >>>>>>> >>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bit...@bitfox.top> wrote: >>>>>>> >>>>>>>> You need to install scala first, the current version for spark is >>>>>>>> 2.12.15 >>>>>>>> I would suggest you install scala by sdk which works great. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar < >>>>>>>> kumar.rajat20...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello Users, >>>>>>>>> >>>>>>>>> I am trying to create spark application using Scala(Intellij). >>>>>>>>> I have installed Scala plugin in intelliJ still getting below >>>>>>>>> error:- >>>>>>>>> >>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp >>>>>>>>> >>>>>>>>> >>>>>>>>> Could anyone please help what I am doing wrong? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Rajat >>>>>>>>> >>>>>>>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
