Oh, I forgot: I am using the “yarn” maven profile to target yarn 2.2
Alex Cozzi
[email protected]
On Jan 15, 2014, at 11:41 PM, Alex Cozzi <[email protected]> wrote:
> Just testing out the rc1. I create a dependent project (using maven) and I
> copied the HdfsTest.scala test, but I added a single line to save the file
> back to disk:
>
> package org.apache.spark.examples
>
> import org.apache.spark._
>
> object HdfsTest {
> def main(args: Array[String]) {
> val sc = new SparkContext(args(0), "HdfsTest",
> System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass))
> val file = sc.textFile(args(1))
> val mapped = file.map(s => s.length).cache()
> for (iter <- 1 to 10) {
> val start = System.currentTimeMillis()
> for (x <- mapped) { x + 2 }
> // println("Processing: " + x)
> val end = System.currentTimeMillis()
> println("Iteration " + iter + " took " + (end-start) + " ms")
> mapped.saveAsTextFile("out")
> }
> System.exit(0)
> }
> }
>
> and this my pom file:
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>my.examples</groupId>
> <artifactId>spark-samples</artifactId>
> <version>0.0.1-SNAPSHOT</version>
> <inceptionYear>2014</inceptionYear>
>
> <properties>
> <maven.compiler.source>1.6</maven.compiler.source>
> <maven.compiler.target>1.6</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.tools.version>2.10</scala.tools.version>
> <scala.version>2.10.0</scala.version>
> </properties>
>
> <repositories>
> <repository>
> <id>spark staging</id>
>
> <url>https://repository.apache.org/content/repositories/orgapachespark-1001</url>
> </repository>
> </repositories>
>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>${scala.version}</version>
> </dependency>
>
> <dependency>
> <groupId>org.apache.spark</groupId>
>
> <artifactId>spark-core_${scala.tools.version}</artifactId>
> <version>0.9.0-incubating</version>
> </dependency>
>
> <!-- Test -->
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.specs2</groupId>
> <artifactId>specs2_${scala.tools.version}</artifactId>
> <version>1.13</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.scalatest</groupId>
>
> <artifactId>scalatest_${scala.tools.version}</artifactId>
> <version>2.0.M6-SNAP8</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
>
> <build>
> <sourceDirectory>src/main/scala</sourceDirectory>
> <testSourceDirectory>src/test/scala</testSourceDirectory>
> <plugins>
> <plugin>
> <!-- see
> http://davidb.github.com/scala-maven-plugin -->
> <groupId>net.alchim31.maven</groupId>
> <artifactId>scala-maven-plugin</artifactId>
> <version>3.1.6</version>
> <configuration>
>
> <scalaCompatVersion>2.10</scalaCompatVersion>
> <jvmArgs>
> <jvmArg>-Xms128m</jvmArg>
> <jvmArg>-Xmx2048m</jvmArg>
> </jvmArgs>
> </configuration>
> <executions>
> <execution>
> <goals>
> <goal>compile</goal>
> <goal>testCompile</goal>
> </goals>
> <configuration>
> <args>
>
> <arg>-make:transitive</arg>
>
> <arg>-dependencyfile</arg>
>
> <arg>${project.build.directory}/.scala_dependencies</arg>
> </args>
> </configuration>
> </execution>
> </executions>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.13</version>
> <configuration>
> <useFile>false</useFile>
>
> <disableXmlReport>true</disableXmlReport>
> <!-- If you have classpath issue like
> NoDefClassError,... -->
> <!--
> useManifestOnlyJar>false</useManifestOnlyJar -->
> <includes>
> <include>**/*Test.*</include>
> <include>**/*Suite.*</include>
> </includes>
> </configuration>
> </plugin>
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>exec-maven-plugin</artifactId>
> <version>1.2.1</version>
> <executions>
> <execution>
> <goals>
> <goal>exec</goal>
> </goals>
> </execution>
> </executions>
> <configuration>
>
> <mainClass>org.apache.spark.examples.HdfsTest</mainClass>
> <arguments>
> <argument>local</argument>
> <argument>pom.xml</argument>
> </arguments>
> </configuration>
> </plugin>
> </plugins>
> </build>
> </project>
>
>
> now, when I run it either in eclipse or using "mvn exec:java" I get the
> following error:
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ spark-samples ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/acozzi/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/acozzi/.m2/repository/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/01/15 23:37:57 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/01/15 23:37:57 INFO Remoting: Starting remoting
> 14/01/15 23:37:57 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://[email protected]:53682]
> 14/01/15 23:37:57 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://[email protected]:53682]
> 14/01/15 23:37:57 INFO spark.SparkEnv: Registering BlockManagerMaster
> 14/01/15 23:37:57 INFO storage.DiskBlockManager: Created local directory at
> /var/folders/mm/4qxz27w91p96v2zp5f9ncmqm38ychm/T/spark-local-20140115233757-7a41
> 14/01/15 23:37:57 INFO storage.MemoryStore: MemoryStore started with capacity
> 1218.8 MB.
> 14/01/15 23:37:57 INFO network.ConnectionManager: Bound socket to port 53683
> with id = ConnectionManagerId(10.0.1.10,53683)
> 14/01/15 23:37:57 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 14/01/15 23:37:57 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
> Registering block manager 10.0.1.10:53683 with 1218.8 MB RAM
> 14/01/15 23:37:57 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/01/15 23:37:57 INFO spark.HttpServer: Starting HTTP Server
> 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
> 14/01/15 23:37:57 INFO server.AbstractConnector: Started
> [email protected]:53684
> 14/01/15 23:37:57 INFO broadcast.HttpBroadcast: Broadcast server started at
> http://10.0.1.10:53684
> 14/01/15 23:37:57 INFO spark.SparkEnv: Registering MapOutputTracker
> 14/01/15 23:37:57 INFO spark.HttpFileServer: HTTP File server directory is
> /var/folders/mm/4qxz27w91p96v2zp5f9ncmqm38ychm/T/spark-e9304513-3714-430f-aa14-1a430a915d98
> 14/01/15 23:37:57 INFO spark.HttpServer: Starting HTTP Server
> 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
> 14/01/15 23:37:57 INFO server.AbstractConnector: Started
> [email protected]:53685
> 14/01/15 23:37:57 INFO server.Server: jetty-7.6.8.v20121106
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/storage/rdd,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/storage,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/stages/stage,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/stages/pool,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/stages,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/environment,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/executors,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/metrics/json,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/static,null}
> 14/01/15 23:37:57 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/,null}
> 14/01/15 23:37:57 INFO server.AbstractConnector: Started
> [email protected]:4040
> 14/01/15 23:37:57 INFO ui.SparkUI: Started Spark Web UI at
> http://10.0.1.10:4040
> 2014-01-15 23:37:57.929 java[34819:1020b] Unable to load realm mapping info
> from SCDynamicStore
> 14/01/15 23:37:58 INFO storage.MemoryStore: ensureFreeSpace(35456) called
> with curMem=0, maxMem=1278030643
> 14/01/15 23:37:58 INFO storage.MemoryStore: Block broadcast_0 stored as
> values to memory (estimated size 34.6 KB, free 1218.8 MB)
> 14/01/15 23:37:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/01/15 23:37:58 WARN snappy.LoadSnappy: Snappy native library not loaded
> 14/01/15 23:37:58 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> 14/01/15 23:37:58 INFO spark.SparkContext: Starting job: foreach at
> HdfsTest.scala:30
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Got job 0 (foreach at
> HdfsTest.scala:30) with 1 output partitions (allowLocal=false)
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Final stage: Stage 0 (foreach
> at HdfsTest.scala:30)
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Parents of final stage: List()
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Missing parents: List()
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Submitting Stage 0
> (MappedRDD[2] at map at HdfsTest.scala:27), which has no missing parents
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
> from Stage 0 (MappedRDD[2] at map at HdfsTest.scala:27)
> 14/01/15 23:37:58 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with
> 1 tasks
> 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0
> on executor localhost: localhost (PROCESS_LOCAL)
> 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as
> 1778 bytes in 5 ms
> 14/01/15 23:37:58 INFO executor.Executor: Running task ID 0
> 14/01/15 23:37:58 INFO storage.BlockManager: Found block broadcast_0 locally
> 14/01/15 23:37:58 INFO spark.CacheManager: Partition rdd_2_0 not found,
> computing it
> 14/01/15 23:37:58 INFO rdd.HadoopRDD: Input split:
> file:/Users/acozzi/Documents/workspace/spark-samples/pom.xml:0+4092
> 14/01/15 23:37:58 INFO storage.MemoryStore: ensureFreeSpace(2853) called with
> curMem=35456, maxMem=1278030643
> 14/01/15 23:37:58 INFO storage.MemoryStore: Block rdd_2_0 stored as values to
> memory (estimated size 2.8 KB, free 1218.8 MB)
> 14/01/15 23:37:58 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
> Added rdd_2_0 in memory on 10.0.1.10:53683 (size: 2.8 KB, free: 1218.8 MB)
> 14/01/15 23:37:58 INFO storage.BlockManagerMaster: Updated info of block
> rdd_2_0
> 14/01/15 23:37:58 INFO executor.Executor: Serialized size of result for 0 is
> 525
> 14/01/15 23:37:58 INFO executor.Executor: Sending result for 0 directly to
> driver
> 14/01/15 23:37:58 INFO executor.Executor: Finished task ID 0
> 14/01/15 23:37:58 INFO scheduler.TaskSetManager: Finished TID 0 in 61 ms on
> localhost (progress: 0/1)
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Completed ResultTask(0, 0)
> 14/01/15 23:37:58 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 0.0 from
> pool
> 14/01/15 23:37:58 INFO scheduler.DAGScheduler: Stage 0 (foreach at
> HdfsTest.scala:30) finished in 0.071 s
> 14/01/15 23:37:58 INFO spark.SparkContext: Job finished: foreach at
> HdfsTest.scala:30, took 0.151199 s
> Iteration 1 took 189 ms
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:171)
> at
> org.apache.hadoop.mapred.SparkHadoopMapRedUtil$class.firstAvailableClass(SparkHadoopMapRedUtil.scala:48)
> at
> org.apache.hadoop.mapred.SparkHadoopMapRedUtil$class.newJobContext(SparkHadoopMapRedUtil.scala:23)
> at
> org.apache.hadoop.mapred.SparkHadoopWriter.newJobContext(SparkHadoopWriter.scala:40)
> at
> org.apache.hadoop.mapred.SparkHadoopWriter.getJobContext(SparkHadoopWriter.scala:149)
> at
> org.apache.hadoop.mapred.SparkHadoopWriter.preSetup(SparkHadoopWriter.scala:64)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:713)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:686)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:572)
> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:894)
> at
> org.apache.spark.examples.HdfsTest$$anonfun$main$1.apply$mcVI$sp(HdfsTest.scala:34)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
> at org.apache.spark.examples.HdfsTest$.main(HdfsTest.scala:28)
> at org.apache.spark.examples.HdfsTest.main(HdfsTest.scala)
> ... 6 more
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 3.224s
> [INFO] Finished at: Wed Jan 15 23:37:58 PST 2014
> [INFO] Final Memory: 12M/81M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default-cli) on project spark-samples: An exception occured while executing
> the Java class. null: InvocationTargetException: Implementing class -> [Help
> 1]
>
>
> Alex Cozzi
> [email protected]
> On Jan 15, 2014, at 5:48 PM, Patrick Wendell <[email protected]> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark
>> (incubating) version 0.9.0.
>>
>> A draft of the release notes along with the changes file is attached
>> to this e-mail.
>>
>> The tag to be voted on is v0.9.0-incubating (commit 7348893):
>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=7348893f0edd96dacce2f00970db1976266f7008
>>
>> The release files, including signatures, digests, etc can be found at:
>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc1/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1001/
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc1-docs/
>>
>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>
>> The vote is open until Sunday, January 19, at 02:00 UTC
>> and passes if a majority of at least 3 +1 PPMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see
>> http://spark.incubator.apache.org/
>
