Finally I was able to build Livy using Scala 2.12 and Spark 3.x without using the container (Docker). Here are the steps:
$ cat /etc/fedora-release Fedora release 39 (Thirty Nine) $ conda create -n livy python=2.7 $ conda activate livy (liby) cat .sdkmanrc java=8.0.302-open maven=3.8.2 (livy) mvn clean package -Pthriftserver -Pscala-2.12 -Pspark3 -DskipTests -Dmaven.javadoc.skip=true [...] [INFO] livy-python-api .................................... SUCCESS [ 0.580 s] [INFO] livy-beeline ....................................... SUCCESS [ 3.828 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:12 min [INFO] Finished at: 2024-04-24T19:31:35+02:00 [INFO] ------------------------------------------------------------------------ On Sun, Apr 21, 2024 at 10:43 AM Javi Roman <javiro...@apache.org> wrote: > Hi all! > > I am in the process of studying Apache Livy in depth (in the hope that I > can help). The first steps are to familiarise myself with the project build > process. I'm having the following problems and I don't know if it's due to > lack of knowledge or if there really is a problem. > > In order to build the project I'm using the provided Docker file with a > command like this (according to the documentation): > > podman run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci > mvn package > or the same command with explicitly using the default versions: > podman run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci > mvn clean package -Pscala-2.11 -Pspark2 > > The build process works fine and I'm getting the binary release: > assembly/target/apache-livy-0.9.0-incubating-SNAPSHOT_2.11-bin.zip > > The problem is that I trying to build the project based on Scala 2.12 (I > understand Scala 2.11 and 1.12 are not binary compatibles so we are using > two binary releases of Apache Livy): > > podman run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci > mvn clean package -Pscala-2.12 > > The build system throws an error related with the maven assembly plugin: > > [INFO] livy-server ........................................ SUCCESS [ > 38.707 s] > [INFO] livy-assembly ...................................... FAILURE [ > 0.075 s] > [INFO] livy-client-http ................................... SKIPPED > > [INFO] livy-scala-api-parent .............................. SKIPPED > > [INFO] livy-scala-api_2.12 ................................ SKIPPED > > [INFO] livy-integration-test .............................. SKIPPED > > [INFO] livy-coverage-report ............................... SKIPPED > > [INFO] livy-examples ...................................... SKIPPED > > [INFO] livy-python-api .................................... SKIPPED > > [INFO] > ------------------------------------------------------------------------ > > [INFO] BUILD FAILURE > > [INFO] > ------------------------------------------------------------------------ > > [INFO] Total time: 01:36 min > > [INFO] Finished at: 2024-04-21T08:39:18+00:00 > > [INFO] Final Memory: 230M/3924M > > [INFO] > ------------------------------------------------------------------------ > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process > (process-resource-bundles) on project livy-assem > bly: Failed to resolve dependencies for one or more projects in the > reactor. Reason: Missing: > > [ERROR] ---------- > > [ERROR] 1) org.apache.livy:livy-core_2.11:jar:0.9.0-incubating-SNAPSHOT > > > The same problem was reported at Github here [1]. > > Any help about it? Am I doing something wrong? > > Many thanks! > > > [1] https://github.com/apache/incubator-livy/issues/422 > >
