[jira] [Created] (SPARK-1481) Add Naive Bayes to MLlib documentation
Matei Zaharia created SPARK-1481: Summary: Add Naive Bayes to MLlib documentation Key: SPARK-1481 URL: https://issues.apache.org/jira/browse/SPARK-1481 Project: Spark Issue Type: Sub-task Components: MLlib Reporter: Matei Zaharia Fix For: 1.0.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-1479: -- Attachment: mvn.log export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.4.0 -Dyarn.version=2.0.0-cdh4.4.0 -DskipTests clean package > building spark on 2.0.0-cdh4.4.0 failed > --- > > Key: SPARK-1479 > URL: https://issues.apache.org/jira/browse/SPARK-1479 > Project: Spark > Issue Type: Question > Environment: 2.0.0-cdh4.4.0 > Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL > spark 0.9.1 > java version "1.6.0_32" >Reporter: jackielihf > Attachments: mvn.log > > > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on > project spark-yarn-alpha_2.10: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> > [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile > (scala-compile-first) on project spark-yarn-alpha_2.10: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > ... 19 more > Caused by: Compilation failed > at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) > at > sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) > at sbt.inc.Incremental$.cycle(Incremental.scala:40) > at sbt.inc.Incremental$.compile(Incremental.s
[jira] [Updated] (SPARK-1386) Spark Streaming UI
[ https://issues.apache.org/jira/browse/SPARK-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1386: --- Fix Version/s: 1.0.0 > Spark Streaming UI > -- > > Key: SPARK-1386 > URL: https://issues.apache.org/jira/browse/SPARK-1386 > Project: Spark > Issue Type: New Feature > Components: Streaming >Affects Versions: 0.9.0 >Reporter: Tathagata Das >Priority: Blocker > Fix For: 1.0.0 > > > When debugging Spark Streaming applications it is necessary to monitor > certain metrics that are not shown in the Spark application UI. For example, > what is average processing time of batches? What is the scheduling delay? Is > the system able to process as fast as it is receiving data? How many records > I am receiving through my receivers? > While the StreamingListener interface introduced in the 0.9 provided some of > this information, it could only be accessed programmatically. A UI that shows > information specific to the streaming applications is necessary for easier > debugging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (SPARK-1403) Spark on Mesos does not set Thread's context class loader
[ https://issues.apache.org/jira/browse/SPARK-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1403. Resolution: Fixed > Spark on Mesos does not set Thread's context class loader > - > > Key: SPARK-1403 > URL: https://issues.apache.org/jira/browse/SPARK-1403 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.0.0 > Environment: ubuntu 12.04 on vagrant >Reporter: Bharath Bhushan >Priority: Blocker > Fix For: 1.0.0 > > > I can run spark 0.9.0 on mesos but not spark 1.0.0. This is because the spark > executor on mesos slave throws a java.lang.ClassNotFoundException for > org.apache.spark.serializer.JavaSerializer. > The lengthy discussion is here: > http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-td3510.html#a3513 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (SPARK-1386) Spark Streaming UI
[ https://issues.apache.org/jira/browse/SPARK-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1386. Resolution: Fixed Assignee: Tathagata Das > Spark Streaming UI > -- > > Key: SPARK-1386 > URL: https://issues.apache.org/jira/browse/SPARK-1386 > Project: Spark > Issue Type: New Feature > Components: Streaming >Affects Versions: 0.9.0 >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Blocker > Fix For: 1.0.0 > > > When debugging Spark Streaming applications it is necessary to monitor > certain metrics that are not shown in the Spark application UI. For example, > what is average processing time of batches? What is the scheduling delay? Is > the system able to process as fast as it is receiving data? How many records > I am receiving through my receivers? > While the StreamingListener interface introduced in the 0.9 provided some of > this information, it could only be accessed programmatically. A UI that shows > information specific to the streaming applications is necessary for easier > debugging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1403) Spark on Mesos does not set Thread's context class loader
[ https://issues.apache.org/jira/browse/SPARK-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1403: --- Fix Version/s: 1.0.0 > Spark on Mesos does not set Thread's context class loader > - > > Key: SPARK-1403 > URL: https://issues.apache.org/jira/browse/SPARK-1403 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.0.0 > Environment: ubuntu 12.04 on vagrant >Reporter: Bharath Bhushan >Priority: Blocker > Fix For: 1.0.0 > > > I can run spark 0.9.0 on mesos but not spark 1.0.0. This is because the spark > executor on mesos slave throws a java.lang.ClassNotFoundException for > org.apache.spark.serializer.JavaSerializer. > The lengthy discussion is here: > http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-td3510.html#a3513 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1480) Choose classloader consistently inside of Spark codebase
[ https://issues.apache.org/jira/browse/SPARK-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1480: --- Summary: Choose classloader consistently inside of Spark codebase (was: Chose classloader consistently inside of Spark codebase) > Choose classloader consistently inside of Spark codebase > > > Key: SPARK-1480 > URL: https://issues.apache.org/jira/browse/SPARK-1480 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Patrick Wendell >Assignee: Patrick Wendell >Priority: Blocker > Fix For: 1.1.0 > > > The Spark codebase is not always consistent on which class loader it uses > when classlaoders are explicitly passed to things like serializers. > In most cases what we want is the following behavior: > 1. If there is a context classloader on the thread, use that. > 2. Otherwise use the classloader that loaded Spark. > We should just have a utility function for this and call that function > whenever we need to get a classloader. > Note that SPARK-1403 is a workaround for this exact problem (it sets the > context class loader because downstream code assumes it is set). Once this > gets fixed in a more general way SPARK-1403 can be reverted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (SPARK-1435) Don't assume context class loader is set when creating classes via reflection
[ https://issues.apache.org/jira/browse/SPARK-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1435. Resolution: Duplicate > Don't assume context class loader is set when creating classes via reflection > - > > Key: SPARK-1435 > URL: https://issues.apache.org/jira/browse/SPARK-1435 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.0.0 >Reporter: Patrick Wendell >Assignee: Patrick Wendell >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1480) Chose classloader consistently inside of Spark codebase
[ https://issues.apache.org/jira/browse/SPARK-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1480: --- Priority: Blocker (was: Major) > Chose classloader consistently inside of Spark codebase > --- > > Key: SPARK-1480 > URL: https://issues.apache.org/jira/browse/SPARK-1480 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Patrick Wendell >Assignee: Patrick Wendell >Priority: Blocker > Fix For: 1.1.0 > > > The Spark codebase is not always consistent on which class loader it uses > when classlaoders are explicitly passed to things like serializers. > In most cases what we want is the following behavior: > 1. If there is a context classloader on the thread, use that. > 2. Otherwise use the classloader that loaded Spark. > We should just have a utility function for this and call that function > whenever we need to get a classloader. > Note that SPARK-1403 is a workaround for this exact problem (it sets the > context class loader because downstream code assumes it is set). Once this > gets fixed in a more general way SPARK-1403 can be reverted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SPARK-1480) Chose classloader consistently inside of Spark codebase
Patrick Wendell created SPARK-1480: -- Summary: Chose classloader consistently inside of Spark codebase Key: SPARK-1480 URL: https://issues.apache.org/jira/browse/SPARK-1480 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Patrick Wendell Assignee: Patrick Wendell Fix For: 1.1.0 The Spark codebase is not always consistent on which class loader it uses when classlaoders are explicitly passed to things like serializers. In most cases what we want is the following behavior: 1. If there is a context classloader on the thread, use that. 2. Otherwise use the classloader that loaded Spark. We should just have a utility function for this and call that function whenever we need to get a classloader. Note that SPARK-1403 is a workaround for this exact problem (it sets the context class loader because downstream code assumes it is set). Once this gets fixed in a more general way SPARK-1403 can be reverted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967723#comment-13967723 ] Matei Zaharia commented on SPARK-1476: -- I agree, would be good to understand what kind of operations this arises in. Do you have cached RDD partitions that are this large, or is it shuffle blocks? Is it skew in the shuffle data? The main concern I see with this is that it would complicate the deserializer and block sender code paths, but maybe it's worth it. > 2GB limit in spark for blocks > - > > Key: SPARK-1476 > URL: https://issues.apache.org/jira/browse/SPARK-1476 > Project: Spark > Issue Type: Bug > Components: Spark Core > Environment: all >Reporter: Mridul Muralidharan >Priority: Critical > Fix For: 1.1.0 > > > The underlying abstraction for blocks in spark is a ByteBuffer : which limits > the size of the block to 2GB. > This has implication not just for managed blocks in use, but also for shuffle > blocks (memory mapped blocks are limited to 2gig, even though the api allows > for long), ser-deser via byte array backed outstreams (SPARK-1391), etc. > This is a severe limitation for use of spark when used on non trivial > datasets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (SPARK-1476) 2GB limit in spark for blocks
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967720#comment-13967720 ] Patrick Wendell edited comment on SPARK-1476 at 4/13/14 3:41 AM: - This says it's a "severe limitation" - but why not just use more, smaller blocks? I think Spark's design assumes in various places that block's are not extremely large. Think of it like e.g. the HDFS block size... it can't be arbitrary large. The answer here might be to use multiple blocks in the case of something like a shuffle where the size can get really large. was (Author: pwendell): This says it's a "severe limitation" - but why not just use more, smaller blocks? I think Spark's design assumes in various places that block's are not extremely large. Think of it like e.g. the HDFS block size... it can't be arbitrary large. > 2GB limit in spark for blocks > - > > Key: SPARK-1476 > URL: https://issues.apache.org/jira/browse/SPARK-1476 > Project: Spark > Issue Type: Bug > Components: Spark Core > Environment: all >Reporter: Mridul Muralidharan >Priority: Critical > Fix For: 1.1.0 > > > The underlying abstraction for blocks in spark is a ByteBuffer : which limits > the size of the block to 2GB. > This has implication not just for managed blocks in use, but also for shuffle > blocks (memory mapped blocks are limited to 2gig, even though the api allows > for long), ser-deser via byte array backed outstreams (SPARK-1391), etc. > This is a severe limitation for use of spark when used on non trivial > datasets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (SPARK-1476) 2GB limit in spark for blocks
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967720#comment-13967720 ] Patrick Wendell edited comment on SPARK-1476 at 4/13/14 3:35 AM: - This says it's a "severe limitation" - but why not just use more, smaller blocks? I think Spark's design assumes in various places that block's are not extremely large. Think of it like e.g. the HDFS block size... it can't be arbitrary large. was (Author: pwendell): This says it's a "severe limitation" - but why not just use more, smaller blocks? I Spark's design assumes in various places that block's are not extremely large. Think of it like e.g. the HDFS block size... it can't be arbitrary large. > 2GB limit in spark for blocks > - > > Key: SPARK-1476 > URL: https://issues.apache.org/jira/browse/SPARK-1476 > Project: Spark > Issue Type: Bug > Components: Spark Core > Environment: all >Reporter: Mridul Muralidharan >Priority: Critical > Fix For: 1.1.0 > > > The underlying abstraction for blocks in spark is a ByteBuffer : which limits > the size of the block to 2GB. > This has implication not just for managed blocks in use, but also for shuffle > blocks (memory mapped blocks are limited to 2gig, even though the api allows > for long), ser-deser via byte array backed outstreams (SPARK-1391), etc. > This is a severe limitation for use of spark when used on non trivial > datasets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967720#comment-13967720 ] Patrick Wendell commented on SPARK-1476: This says it's a "severe limitation" - but why not just use more, smaller blocks? I Spark's design assumes in various places that block's are not extremely large. Think of it like e.g. the HDFS block size... it can't be arbitrary large. > 2GB limit in spark for blocks > - > > Key: SPARK-1476 > URL: https://issues.apache.org/jira/browse/SPARK-1476 > Project: Spark > Issue Type: Bug > Components: Spark Core > Environment: all >Reporter: Mridul Muralidharan >Priority: Critical > Fix For: 1.1.0 > > > The underlying abstraction for blocks in spark is a ByteBuffer : which limits > the size of the block to 2GB. > This has implication not just for managed blocks in use, but also for shuffle > blocks (memory mapped blocks are limited to 2gig, even though the api allows > for long), ser-deser via byte array backed outstreams (SPARK-1391), etc. > This is a severe limitation for use of spark when used on non trivial > datasets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-1479: -- Issue Type: Question (was: Bug) > building spark on 2.0.0-cdh4.4.0 failed > --- > > Key: SPARK-1479 > URL: https://issues.apache.org/jira/browse/SPARK-1479 > Project: Spark > Issue Type: Question > Environment: 2.0.0-cdh4.4.0 > Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL > spark 0.9.1 > java version "1.6.0_32" >Reporter: jackielihf > > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on > project spark-yarn-alpha_2.10: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> > [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile > (scala-compile-first) on project spark-yarn-alpha_2.10: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > ... 19 more > Caused by: Compilation failed > at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) > at > sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) > at sbt.inc.Incremental$.cycle(Incremental.scala:40) > at sbt.inc.Incremental$.compile(Incremental.scala:25) > at sbt.inc.IncrementalCompile$.apply(Compile.scala:20) > at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:96) > at sbt.compiler.AggressiveCompile.compile1(Aggre
[jira] [Commented] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[
https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967719#comment-13967719
]
witgo commented on SPARK-1479:
--
You can give all of the logs?
{code}
mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.4.0 -Dyarn.version=2.0.0-chd4.4.0
-DskipTests clean package -X > mvn.log
{code}
> building spark on 2.0.0-cdh4.4.0 failed
> ---
>
> Key: SPARK-1479
> URL: https://issues.apache.org/jira/browse/SPARK-1479
> Project: Spark
> Issue Type: Bug
> Environment: 2.0.0-cdh4.4.0
> Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL
> spark 0.9.1
> java version "1.6.0_32"
>Reporter: jackielihf
>
> [INFO]
>
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on
> project spark-yarn-alpha_2.10: Execution scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed ->
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile
> (scala-compile-first) on project spark-yarn-alpha_2.10: Execution
> scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed.
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> ... 19 more
> Caused by: Compilation failed
> at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76)
> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35)
> at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71)
> at
> sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88)
> at
> sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60)
> at
> sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24)
> at
> sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22)
> at sbt.inc.Incremental$.cycle(Incremental.scala:40)
> at sbt.inc.Incremental$.compile(Incremental.scala:25)
> at sbt.inc.Incremental
[jira] [Updated] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-1479: -- Description: [INFO] [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: Compilation failed at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) at sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) at sbt.inc.Incremental$.cycle(Incremental.scala:40) at sbt.inc.Incremental$.compile(Incremental.scala:25) at sbt.inc.IncrementalCompile$.apply(Compile.scala:20) at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:96) at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:44) at com.typesafe.zinc.Compiler.compile(Compiler.scala:158) at com.typesafe.zinc.Compiler.compile(Compiler.scala:142) at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:77) at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:296) at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:123) at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:104) at scala_maven.ScalaMojoSup
[jira] [Updated] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-1479: -- Description: [INFO] [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: Compilation failed at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) at sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) at sbt.inc.Incremental$.cycle(Incremental.scala:40) at sbt.inc.Incremental$.compile(Incremental.scala:25) at sbt.inc.IncrementalCompile$.apply(Compile.scala:20) at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:96) at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:44) at com.typesafe.zinc.Compiler.compile(Compiler.scala:158) at com.typesafe.zinc.Compiler.compile(Compiler.scala:142) at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:77) at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:296) at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:123) at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:104) at scala_maven.ScalaMojoSup
[jira] [Updated] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
[ https://issues.apache.org/jira/browse/SPARK-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-1479: -- Environment: 2.0.0-cdh4.4.0 Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL spark 0.9.1 java version "1.6.0_32" was: 2.0.0-cdh4.4.0 Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL spark 0.9.1 > building spark on 2.0.0-cdh4.4.0 failed > --- > > Key: SPARK-1479 > URL: https://issues.apache.org/jira/browse/SPARK-1479 > Project: Spark > Issue Type: Bug > Environment: 2.0.0-cdh4.4.0 > Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL > spark 0.9.1 > java version "1.6.0_32" >Reporter: jackielihf > > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on > project spark-yarn-alpha_2.10: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> > [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile > (scala-compile-first) on project spark-yarn-alpha_2.10: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > ... 19 more > Caused by: Compilation failed > at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) > at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) > at > sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) > at > sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) > at > sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) > at > sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) > at sbt.inc.Incremental$.cycle(Incremental.scala:40) > at sbt.inc.Incremental$.compile(Incremental.scala:25) >
[jira] [Created] (SPARK-1479) building spark on 2.0.0-cdh4.4.0 failed
jackielihf created SPARK-1479: - Summary: building spark on 2.0.0-cdh4.4.0 failed Key: SPARK-1479 URL: https://issues.apache.org/jira/browse/SPARK-1479 Project: Spark Issue Type: Bug Environment: 2.0.0-cdh4.4.0 Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL spark 0.9.1 Reporter: jackielihf [INFO] [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on project spark-yarn-alpha_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: Compilation failed at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:76) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:35) at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:29) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile$$anonfun$4$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:71) at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:101) at sbt.compiler.AggressiveCompile$$anonfun$4.compileScala$1(AggressiveCompile.scala:70) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:88) at sbt.compiler.AggressiveCompile$$anonfun$4.apply(AggressiveCompile.scala:60) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:24) at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:22) at sbt.inc.Incremental$.cycle(Incremental.scala:40) at sbt.inc.Incremental$.compile(Incremental.scala:25) at sbt.inc.IncrementalCompile$.apply(Compile.scala:20) at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:96) at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:44) at com.typesafe.zinc.Compiler.compile(Compiler.scala:158) at com.typesafe.zinc.Compiler.compile(Compiler.scala:142) at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:77) at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCom
[jira] [Commented] (SPARK-1216) Add a OneHotEncoder for handling categorical features
[ https://issues.apache.org/jira/browse/SPARK-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967631#comment-13967631 ] Martin Jaggi commented on SPARK-1216: - could merge these two issues: https://issues.apache.org/jira/browse/SPARK-1303 > Add a OneHotEncoder for handling categorical features > - > > Key: SPARK-1216 > URL: https://issues.apache.org/jira/browse/SPARK-1216 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 0.9.0 >Reporter: Sandy Pérez González >Assignee: Sandy Ryza > > It would be nice to add something to MLLib to make it easy to do one-of-K > encoding of categorical features. > Something like: > http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1303) Added discretization capability to MLlib.
[ https://issues.apache.org/jira/browse/SPARK-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967630#comment-13967630 ] Martin Jaggi commented on SPARK-1303: - Discretization: see also https://issues.apache.org/jira/browse/SPARK-1216 Can you link the pull request here as well please? Feature selection: see also https://issues.apache.org/jira/browse/SPARK-1473 > Added discretization capability to MLlib. > - > > Key: SPARK-1303 > URL: https://issues.apache.org/jira/browse/SPARK-1303 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: LIDIAgroup > Fix For: 1.0.0 > > > Some time ago, we have commented with Ameet Talwalkar the possibilty of > including both Feature Selection and Discretization algorithms to MLlib. > In this patch we've implemented Entropy Minimization Discretization following > the algorithm described in the paper "Multi-interval discretization of > continuous-valued attributes for classification learning" by Fayyad and Irani > (1993). This is one of the most used Discretizers and is already included in > most libraries like Weka, etc. This can be used as base for FS algorims and > the NaiveBayes already included in MLlib. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1473) Feature selection for high dimensional datasets
[ https://issues.apache.org/jira/browse/SPARK-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967629#comment-13967629 ] Martin Jaggi commented on SPARK-1473: - MLlib already implements the Lasso (l1-regularized least squares regression), which is widely used for feature selection, and scales relatively well. Just as a baseline method to keep in mind when comparing quality and scalability. > Feature selection for high dimensional datasets > --- > > Key: SPARK-1473 > URL: https://issues.apache.org/jira/browse/SPARK-1473 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Ignacio Zendejas >Priority: Minor > Labels: features > Fix For: 1.1.0 > > > For classification tasks involving large feature spaces in the order of tens > of thousands or higher (e.g., text classification with n-grams, where n > 1), > it is often useful to rank and filter features that are irrelevant thereby > reducing the feature space by at least one or two orders of magnitude without > impacting performance on key evaluation metrics (accuracy/precision/recall). > A feature evaluation interface which is flexible needs to be designed and at > least two methods should be implemented with Information Gain being a > priority as it has been shown to be amongst the most reliable. > Special consideration should be taken in the design to account for wrapper > methods (see research papers below) which are more practical for lower > dimensional data. > Relevant research: > * Brown, G., Pocock, A., Zhao, M. J., & Luján, M. (2012). Conditional > likelihood maximisation: a unifying framework for information theoretic > feature selection.*The Journal of Machine Learning Research*, *13*, 27-66. > * Forman, George. "An extensive empirical study of feature selection metrics > for text classification." The Journal of machine learning research 3 (2003): > 1289-1305. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1478) Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915
[ https://issues.apache.org/jira/browse/SPARK-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967525#comment-13967525 ] Ted Malaska commented on SPARK-1478: After reviewing the code and I found that Spark Streaming is focused on Flume 1.2.0 and Avro-ipc 1.6.3. Cloudera, Hortenworks, and even MapR are using Flume 1.4.0 in their current releases and that version of Flume uses Avro-ipc 1.7.3 which allows for us to apply our own ChannelPipeLine, which allows us to use compression and encryption. I don't think I can complete do this Jira until I get permission to up the versions of Flume in the pom file from 1.2.0 to at least 1.3.0. Flume version 1.3 is more then 2 years old so that should be ok for Spark Users and Flume 1.3.0 uses Avro-ipc 1.7.3, which is what we need. As for now, I will continue as if I had the permission, and I will do a pull request with the change from Flume from 1.2.0 to 1.3.0. Thanks > Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915 > --- > > Key: SPARK-1478 > URL: https://issues.apache.org/jira/browse/SPARK-1478 > Project: Spark > Issue Type: Improvement > Components: Streaming >Reporter: Ted Malaska >Priority: Minor > > Flume-1915 added support for compression over the wire from avro sink to avro > source. I would like to add this functionality to the FlumeReceiver. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1437) Jenkins should build with Java 6
[ https://issues.apache.org/jira/browse/SPARK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967522#comment-13967522 ] Sean Owen commented on SPARK-1437: -- Pardon my boldness in pushing this onto your plate pwendell, but might be a very quick fix in Jenkins. If Travis CI is going to be activated, it can definitely be configured to build with Java 6 and 7 both. > Jenkins should build with Java 6 > > > Key: SPARK-1437 > URL: https://issues.apache.org/jira/browse/SPARK-1437 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 0.9.0 >Reporter: Sean Owen >Assignee: Patrick Wendell >Priority: Minor > Labels: javac, jenkins > Attachments: Screen Shot 2014-04-07 at 22.53.56.png > > > Apologies if this was already on someone's to-do list, but I wanted to track > this, as it bit two commits in the last few weeks. > Spark is intended to work with Java 6, and so compiles with source/target > 1.6. Java 7 can correctly enforce Java 6 language rules and emit Java 6 > bytecode. However, unless otherwise configured with -bootclasspath, javac > will use its own (Java 7) library classes. This means code that uses classes > in Java 7 will be allowed to compile, but the result will fail when run on > Java 6. > This is why you get warnings like ... > Using /usr/java/jdk1.7.0_51 as default JAVA_HOME. > ... > [warn] warning: [options] bootstrap class path not set in conjunction with > -source 1.6 > The solution is just to tell Jenkins to use Java 6. This may be stating the > obvious, but it should just be a setting under "Configure" for > SparkPullRequestBuilder. In our Jenkinses, JDK 6/7/8 are set up; if it's not > an option already I'm guessing it's not too hard to get Java 6 configured on > the Amplab machines. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SPARK-1437) Jenkins should build with Java 6
[ https://issues.apache.org/jira/browse/SPARK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-1437: - Assignee: Patrick Wendell > Jenkins should build with Java 6 > > > Key: SPARK-1437 > URL: https://issues.apache.org/jira/browse/SPARK-1437 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 0.9.0 >Reporter: Sean Owen >Assignee: Patrick Wendell >Priority: Minor > Labels: javac, jenkins > Attachments: Screen Shot 2014-04-07 at 22.53.56.png > > > Apologies if this was already on someone's to-do list, but I wanted to track > this, as it bit two commits in the last few weeks. > Spark is intended to work with Java 6, and so compiles with source/target > 1.6. Java 7 can correctly enforce Java 6 language rules and emit Java 6 > bytecode. However, unless otherwise configured with -bootclasspath, javac > will use its own (Java 7) library classes. This means code that uses classes > in Java 7 will be allowed to compile, but the result will fail when run on > Java 6. > This is why you get warnings like ... > Using /usr/java/jdk1.7.0_51 as default JAVA_HOME. > ... > [warn] warning: [options] bootstrap class path not set in conjunction with > -source 1.6 > The solution is just to tell Jenkins to use Java 6. This may be stating the > obvious, but it should just be a setting under "Configure" for > SparkPullRequestBuilder. In our Jenkinses, JDK 6/7/8 are set up; if it's not > an option already I'm guessing it's not too hard to get Java 6 configured on > the Amplab machines. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (SPARK-1458) Add programmatic way to determine Spark version
[ https://issues.apache.org/jira/browse/SPARK-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-1458. - Resolution: Fixed Fix Version/s: 1.0.0 This issue appears to be resolved in [this merge|https://github.com/apache/spark/pull/204/files#diff-364713d7776956cb8b0a771e9b62f82dR779]. > Add programmatic way to determine Spark version > --- > > Key: SPARK-1458 > URL: https://issues.apache.org/jira/browse/SPARK-1458 > Project: Spark > Issue Type: New Feature > Components: PySpark, Spark Core >Affects Versions: 0.9.0 >Reporter: Nicholas Chammas >Priority: Minor > Fix For: 1.0.0 > > > As discussed > [here|http://apache-spark-user-list.1001560.n3.nabble.com/programmatic-way-to-tell-Spark-version-td1929.html], > I think it would be nice if there was a way to programmatically determine > what version of Spark you are running. > The potential use cases are not that important, but they include: > # Branching your code based on what version of Spark is running. > # Checking your version without having to quit and restart the Spark shell. > Right now in PySpark, I believe the only way to determine your version is by > firing up the Spark shell and looking at the startup banner. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SPARK-1478) Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915
Ted Malaska created SPARK-1478: -- Summary: Upgrade FlumeInputDStream's FlumeReceiver to support FLUME-1915 Key: SPARK-1478 URL: https://issues.apache.org/jira/browse/SPARK-1478 Project: Spark Issue Type: Improvement Components: Streaming Reporter: Ted Malaska Priority: Minor Flume-1915 added support for compression over the wire from avro sink to avro source. I would like to add this functionality to the FlumeReceiver. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SPARK-1477) Add the lifecycle interface
witgo created SPARK-1477: Summary: Add the lifecycle interface Key: SPARK-1477 URL: https://issues.apache.org/jira/browse/SPARK-1477 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.0.0 Reporter: witgo Now the Spark in the code, there are a lot of interface or class defines the stop and start method,eg:[SchedulerBackend|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala],[HttpServer|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/HttpServer.scala],[ContextCleaner|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ContextCleaner.scala] . we should use a life cycle interface improve the code -- This message was sent by Atlassian JIRA (v6.2#6252)
