Hi guys
sorry to annoy you on this but i am getting nowhere. So far i have tried
to build spark 2.0 on my local laptop with no success.... so i blamed my
laptop poor performance
So today i fired off an EC2 Ubuntu 16.06 Instance and installed the
following (i copy paste commands here)
ubuntu@ip-172-31-40-104:~/spark$ history
sudo apt-get install -y python-software-properties
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get install -y oracle-java8-installer
sudo apt-get install -y git
git clone git://github.com/apache/spark.git
cd spark
Then launched the following commands:
First thi sone, with only yarn, as i dont need hadoop
./build/mvn -Pyarn -DskipTests clean package
This failed. after kicking off same command with -X i got this
ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first)
on project spark-core_2.11: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile
(scala-compile-first) on project spark-core_2.11: Execution
scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: Compile failed via zinc server
at
sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136)
at
sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86)
at
scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303)
at
scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119)
at
scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99)
at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
... 21 more
Then i tried to use hadoop even though i dont need hadoop for my code
./build/mvn -Pyarn -Phadoop-2.4 -DskipTests clean package
This failed again, at exactly the same point, with the same error
Then i thought maybe i used an old version of hadoop, so i tried to use 2.7
./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package
Nada. same error.
I have tried sbt, no luck
Could anyone suggest me a mvn proflie to use to build spark 2.0? i am
starting to suspect (before arguing that there's something wrong with spark
git) that i am using somehow wrong parameters..... or perhaps i should
install scala 2.11 before i install spark? or Maven ?
kr
marco
On Fri, Sep 30, 2016 at 8:23 PM, Marco Mistroni <mmistr...@gmail.com> wrote:
> Hi all
> this problem is still bothering me.
> Here's my setup
> - Ubuntu 16.06
> - Java 8
> - Spark 2.0
> - have launched following command: ./build/mvn -X -Pyarn -Phadoop-2.7
> -DskipTests clean package
> and i am gettign this exception:
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> (scala-test-compile-first) on project spark-mllib_2.10: Execution
> scala-test-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> failed.
>
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:212)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:116)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:80)
> at org.apache.maven.lifecycle.internal.builder.singlethreaded.
> SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.
> execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> launchEnhanced(Launcher.java:289)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> launch(Launcher.java:229)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> mainWithExitCode(Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.
> main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> scala-test-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> failed.
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
> DefaultBuildPluginManager.java:145)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:207)
> ... 20 more
> Caused by: Compile failed via zinc server
> at sbt_inc.SbtIncrementalCompiler.zincCompile(
> SbtIncrementalCompiler.java:136)
> at sbt_inc.SbtIncrementalCompiler.compile(
> SbtIncrementalCompiler.java:86)
> at scala_maven.ScalaCompilerSupport.incrementalCompile(
> ScalaCompilerSupport.java:303)
> at scala_maven.ScalaCompilerSupport.compile(
> ScalaCompilerSupport.java:119)
> at scala_maven.ScalaCompilerSupport.doExecute(
> ScalaCompilerSupport.java:99)
> at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
> at scala_maven.ScalaTestCompileMojo.execute(
> ScalaTestCompileMojo.java:48)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
> DefaultBuildPluginManager.java:134)
>
> anyone found a similar error?
>
> kr
>
>
>
>
> On Sat, Sep 3, 2016 at 2:54 PM, Marco Mistroni <mmistr...@gmail.com>
> wrote:
>
>> hi all
>>
>> i am getting failures when building spark 2.0 on Ubuntu 16.06
>> Here's details of what i have installed on the ubuntu host
>> - java 8
>> - scala 2.11
>> - git
>>
>> When i launch the command
>>
>> ./build/mvn -Pyarn -Phadoop-2.7 -DskipTests clean package
>>
>> everything compiles sort of fine and at the end i get this exception
>>
>> INFO] BUILD FAILURE
>> [INFO] ------------------------------------------------------------
>> ------------
>> [INFO] Total time: 01:05 h
>> [INFO] Finished at: 2016-09-03T13:25:27+00:00
>> [INFO] Final Memory: 57M/208M
>> [INFO] ------------------------------------------------------------
>> ------------
>> [ERROR] Failed to execute goal
>> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
>> (scala-test-compile-first) on project spark-streaming_2.11: Execution
>> scala-test-compile-first of goal
>> net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
>> failed. CompileFailed -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1] http://cwiki.apache.org/conflu
>> ence/display/MAVEN/PluginExecutionException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR] mvn <goals> -rf :spark-streaming_2.11
>>
>> could anyone help?
>>
>> kr
>>
>
>
