sparkR - is it possible to run sparkR on yarn?
Hi all, I searched around, but fail to find anything that says about running sparkR on YARN. so, is it possible to run sparkR with yarn ? either with yarn-standalone or yarn-client mode. if so, is there any document that could guide me through the build & setup processes? I am desparate for some answers, so please help!
Re: SparkException: env SPARK_YARN_APP_JAR is not set
according to the code, SPARK_YARN_APP_JAR is retrieved from system
variables.
and the key-value pairs you pass through to JavaSparkContext is isolated
from system variables.
so, you maybe should try setting it through System.setProperty().
thanks
On Wed, Apr 23, 2014 at 6:05 PM, 肥肥 <19934...@qq.com> wrote:
> I have a small program, which I can launch successfully by yarn client
> with yarn-standalon mode.
>
> the command look like this:
> (javac javac -classpath .:jars/spark-assembly-0.9.1-hadoop2.2.0.jar
> LoadTest.java)
> (jar cvf loadtest.jar LoadTest.class)
> SPARK_JAR=assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.2.0.jar
> ./bin/spark-class org.apache.spark.deploy.yarn.Client --jar
> /opt/mytest/loadtest.jar --class LoadTest --args yarn-standalone
> --num-workers 2 --master-memory 2g --worker-memory 2g --worker-cores 1
>
> the program LoadTest.java:
> public class LoadTest {
> static final String USER = "root";
> public static void main(String[] args) {
> System.setProperty("user.name", USER);
> System.setProperty("HADOOP_USER_NAME", USER);
> System.setProperty("spark.executor.memory", "7g");
> JavaSparkContext sc = new JavaSparkContext(args[0], "LoadTest",
> System.getenv("SPARK_HOME"), JavaSparkContext.jarOfClass(LoadTest.class));
> String file = "file:/opt/mytest/123.data";
> JavaRDD data1 = sc.textFile(file, 2);
> long c1=data1.count();
> System.out.println("1"+c1);
> }
> }
>
> BUT due to my other pragram's need, I must have it run with command of
> "java". So I add “environment” parameter to JavaSparkContext(). Followed is
> The ERROR I get:
> Exception in thread "main" org.apache.spark.SparkException: env
> SPARK_YARN_APP_JAR is not set
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:49)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:125)
> at org.apache.spark.SparkContext.(SparkContext.scala:200)
> at org.apache.spark.SparkContext.(SparkContext.scala:100)
> at
> org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:93)
> at LoadTest.main(LoadTest.java:37)
>
> the program LoadTest.java:
> public class LoadTest {
>
> static final String USER = "root";
> public static void main(String[] args) {
> System.setProperty("user.name", USER);
> System.setProperty("HADOOP_USER_NAME", USER);
> System.setProperty("spark.executor.memory", "7g");
>
> Map env = new HashMap();
> env.put("SPARK_YARN_APP_JAR", "file:/opt/mytest/loadtest.jar");
> env.put("SPARK_WORKER_INSTANCES", "2" );
> env.put("SPARK_WORKER_CORES", "1");
> env.put("SPARK_WORKER_MEMORY", "2G");
> env.put("SPARK_MASTER_MEMORY", "2G");
> env.put("SPARK_YARN_APP_NAME", "LoadTest");
> env.put("SPARK_YARN_DIST_ARCHIVES",
> "file:/opt/test/spark-0.9.1-bin-hadoop1/assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.2.0.jar");
> JavaSparkContext sc = new JavaSparkContext("yarn-client",
> "LoadTest", System.getenv("SPARK_HOME"),
> JavaSparkContext.jarOfClass(LoadTest.class), env);
> String file = "file:/opt/mytest/123.dna";
> JavaRDD data1 = sc.textFile(file, 2);//.cache();
>
> long c1=data1.count();
> System.out.println("1"+c1);
> }
> }
>
> the command:
> javac -classpath .:jars/spark-assembly-0.9.1-hadoop2.2.0.jar LoadTest.java
> jar cvf loadtest.jar LoadTest.class
> nohup java -classpath .:jars/spark-assembly-0.9.1-hadoop2.2.0.jar LoadTest
> >> loadTest.log 2>&1 &
>
> What did I miss?? Or I did it in wrong way??
>
Re: "sbt/sbt run" command returns a JVM problem
the total memory of your machine is 2G right?
then how much memory is left free? wouldn`t ubuntu take up quite a big
portion of 2G?
just a guess!
On Sat, May 3, 2014 at 8:15 PM, Carter wrote:
> Hi, thanks for all your help.
> I tried your setting in the sbt file, but the problem is still there.
>
> The Java setting in my sbt file is:
> java \
> -Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=256m \
> -jar ${JAR} \
> "$@"
>
> I have tried to set these 3 parameters bigger and smaller, but nothing
> works. Did I change the right thing?
>
> Thank you very much.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/sbt-sbt-run-command-returns-a-JVM-problem-tp5157p5267.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
what`s the meaning of primitive in "gradient descent primitive"?
Hi all, I am reading the doc of spark ( http://spark.apache.org/docs/0.9.0/mllib-guide.html#gradient-descent-primitive). I am trying to translate the doc into Chinese, and there it talks about gradient descent primitive, and but i am not quite sure what it mean by primitive? I know gradient descent, but I am not sure what he is referring to by gradient descent primitive? does it mean the most basic form of gradient descent? please enlighten me. thank you in advance.
Re: spark 0.9.1: ClassNotFoundException
check if the jar file that includes your example code is under examples/target/scala-2.10/. On Sat, May 3, 2014 at 5:58 AM, SK wrote: > I am using Spark 0.9.1 in standalone mode. In the > SPARK_HOME/examples/src/main/scala/org/apache/spark/ folder, I created my > directory called "mycode" in which I have placed some standalone scala > code. > I was able to compile. I ran the code using: > > ./bin/run-example org.apache.spark.mycode.MyClass local > > However, I get a ClassNotFound exception, although I do see the compiled > classes in > examples/target/scala-2.10/classes/org/apache/spark/mycode > > When I place the same code in the same folder structure in the spark 0.9.0 > version, I am able to run it. Where should I place my standalone code with > respect to SPARK_HOME, in spark0.9.1 so that the classes can be found? > > thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-0-9-1-ClassNotFoundException-tp5256.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: How to use spark-submit
I used spark-submit to run the MovieLensALS example from the examples module. here is the command: $spark-submit --master local /home/phoenix/spark/spark-dev/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop1.0.4.jar --class org.apache.spark.examples.mllib.MovieLensALS u.data also, you could check the parameters of spark-submit by $spark-submit --h hope this helps! On Wed, May 7, 2014 at 9:27 AM, Tathagata Das wrote: > Doesnt the run-example script work for you? Also, are you on the latest > commit of branch-1.0 ? > > TD > > > On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta > wrote: > >> >> >> Yes, I'm struggling with a similar problem where my class are not found >> on the worker nodes. I'm using 1.0.0_SNAPSHOT. I would really appreciate >> if someone can provide some documentation on the usage of spark-submit. >> >> Thanks >> >> > On May 5, 2014, at 10:24 PM, Stephen Boesch wrote: >> > >> > >> > I have a spark streaming application that uses the external streaming >> modules (e.g. kafka, mqtt, ..) as well. It is not clear how to properly >> invoke the spark-submit script: what are the ---driver-class-path and/or >> -Dspark.executor.extraClassPath parameters required? >> > >> > For reference, the following error is proving difficult to resolve: >> > >> > java.lang.ClassNotFoundException: >> org.apache.spark.streaming.examples.StreamingExamples >> > >> > >
spark on yarn-standalone, throws StackOverflowError and fails somtimes and succeed for the rest
Hi all, My spark code is running on yarn-standalone. the last three lines of the code as below, val result = model.predict(prdctpairs) result.map(x => x.user+","+x.product+","+x.rating).saveAsTextFile(output) sc.stop() the same code, sometimes be able to run successfully and could give out the right result, while from time to time, it throws StackOverflowError and fail. and I don`t have a clue how I should debug. below is the error, (the start and end portion to be exact): 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 44 to sp...@rxx43.mc10.site.net:43885 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] MapOutputTrackerMaster: Size of output statuses for shuffle 44 is 148 bytes 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-35] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 45 to sp...@rxx43.mc10.site.net:43885 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-35] MapOutputTrackerMaster: Size of output statuses for shuffle 45 is 453 bytes 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-20] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 44 to sp...@rxx43.mc10.site.net:56767 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 45 to sp...@rxx43.mc10.site.net:56767 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 44 to sp...@rxx43.mc10.site.net:49879 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 45 to sp...@rxx43.mc10.site.net:49879 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] TaskSetManager: Starting task 946.0:17 as TID 146 on executor 6: rx15.mc10.site.net (PROCESS_LOCAL) 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] TaskSetManager: Serialized task 946.0:17 as 6414 bytes in 0 ms 14-05-09 17:55:51 WARN [Result resolver thread-0] TaskSetManager: Lost TID 133 (task 946.0:4) 14-05-09 17:55:51 WARN [Result resolver thread-0] TaskSetManager: Loss was due to java.lang.StackOverflowError java.lang.StackOverflowError at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-5] TaskSetManager: Starting task 946.0:4 as TID 147 on executor 6: r15.mc10.site.net (PROCESS_LOCAL) 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-5] TaskSetManager: Serialized task 946.0:4 as 6414 bytes in 0 ms 14-05-09 17:55:51 WARN [Result resolver thread-1] TaskSetManager: Lost TID 139 (task 946.0:10) 14-05-09 17:55:51 INFO [Result resolver thread-1] TaskSetManager: Loss was due to java.lang.StackOverflowError [duplicate 1] 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-5] CoarseGrainedSchedulerBackend: Executor 4 disconnected, so removing it 14-05-09 17:55:51 ERROR [spark-akka.actor.default-dis
Re: spark on yarn-standalone, throws StackOverflowError and fails somtimes and succeed for the rest
after a couple of tests, I find that, if I use: val result = model.predict(prdctpairs) result.map(x => x.user+","+x.product+","+x.rating).saveAsTextFile(output) it always fails with above error and the exception seems iterative. but if I do: val result = model.predict(prdctpairs) result.cach() result.map(x => x.user+","+x.product+","+x.rating).saveAsTextFile(output) it succeeds. could anyone help explain why the cach() is necessary? thanks On Fri, May 9, 2014 at 6:45 PM, phoenix bai wrote: > Hi all, > > My spark code is running on yarn-standalone. > > the last three lines of the code as below, > > val result = model.predict(prdctpairs) > result.map(x => > x.user+","+x.product+","+x.rating).saveAsTextFile(output) > sc.stop() > > the same code, sometimes be able to run successfully and could give out > the right result, while from time to time, it throws StackOverflowError and > fail. > > and I don`t have a clue how I should debug. > > below is the error, (the start and end portion to be exact): > > > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 44 to sp...@rxx43.mc10.site.net:43885 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] > MapOutputTrackerMaster: Size of output statuses for shuffle 44 is 148 bytes > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-35] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 45 to sp...@rxx43.mc10.site.net:43885 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-35] > MapOutputTrackerMaster: Size of output statuses for shuffle 45 is 453 bytes > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-20] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 44 to sp...@rxx43.mc10.site.net:56767 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 45 to sp...@rxx43.mc10.site.net:56767 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 44 to sp...@rxx43.mc10.site.net:49879 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-29] > MapOutputTrackerMasterActor: Asked to send map output locations for shuffle > 45 to sp...@rxx43.mc10.site.net:49879 > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] > TaskSetManager: Starting task 946.0:17 as TID 146 on executor 6: > rx15.mc10.site.net (PROCESS_LOCAL) > 14-05-09 17:55:51 INFO [spark-akka.actor.default-dispatcher-17] > TaskSetManager: Serialized task 946.0:17 as 6414 bytes in 0 ms > 14-05-09 17:55:51 WARN [Result resolver thread-0] TaskSetManager: Lost TID > 133 (task 946.0:4) > 14-05-09 17:55:51 WARN [Result resolver thread-0] TaskSetManager: Loss was > due to java.lang.StackOverflowError > java.lang.StackOverflowError > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > > > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:187
Re: new error for me
I am currently facing the same problem. error snapshot as below: 14-07-24 19:15:30 WARN [pool-3-thread-1] SendingConnection: Error finishing connection to r64b22034.tt.net/10.148.129.84:47525 java.net.ConnectException: Connection timed out at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:203) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 14-07-24 19:15:30 INFO [pool-3-thread-1] ConnectionManager: Handling connection error on connection to ConnectionManagerId(r64b22034.tt.net,47525) 14-07-24 19:15:30 INFO [pool-3-thread-1] ConnectionManager: Removing SendingConnection to ConnectionManagerId(r64b22034.tt.net,47525) 14-07-24 19:15:30 INFO [pool-3-thread-1] ConnectionManager: Notifying org.apache.spark.network.ConnectionManager$MessageStatus@1704ebb could anyone help shed a light on this? thanks On Tue, Jul 22, 2014 at 11:35 AM, Nathan Kronenfeld < nkronenf...@oculusinfo.com> wrote: > Does anyone know what this error means: > 14/07/21 23:07:22 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks > 14/07/21 23:07:22 INFO TaskSetManager: Starting task 3.0:0 as TID 1620 on > executor 27: r104u05.oculus.local (PROCESS_LOCAL) > 14/07/21 23:07:22 INFO TaskSetManager: Serialized task 3.0:0 as 8620 bytes > in 1 ms > 14/07/21 23:07:36 INFO BlockManagerInfo: Added taskresult_1620 in memory > on r104u05.oculus.local:50795 (size: 64.9 MB, free: 18.3 GB) > 14/07/21 23:07:36 INFO SendingConnection: Initiating connection to > [r104u05.oculus.local/192.168.0.105:50795] > 14/07/21 23:07:57 INFO ConnectionManager: key already cancelled ? > sun.nio.ch.SelectionKeyImpl@1d86a150 > java.nio.channels.CancelledKeyException > at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) > at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) > at > org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:265) > at > org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:115) > 14/07/21 23:07:57 WARN SendingConnection: Error finishing connection to > r104u05.oculus.local/192.168.0.105:50795 > java.net.ConnectException: Connection timed out > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) > at > org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318) > at > org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:202) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > 14/07/21 23:07:57 INFO ConnectionManager: Handling connection error on > connection to ConnectionManagerId(r104u05.oculus.local,50795) > 14/07/21 23:07:57 INFO ConnectionManager: Removing SendingConnection to > ConnectionManagerId(r104u05.oculus.local,50795) > 14/07/21 23:07:57 INFO ConnectionManager: Notifying > org.apache.spark.network.ConnectionManager$MessageStatus@13ad274d > 14/07/21 23:07:57 INFO ConnectionManager: Handling connection error on > connection to ConnectionManagerId(r104u05.oculus.local,50795) > 14/07/21 23:07:57 INFO ConnectionManager: Removing SendingConnection to > ConnectionManagerId(r104u05.oculus.local,50795) > 14/07/21 23:07:57 INFO ConnectionManager: Removing SendingConnection to > ConnectionManagerId(r104u05.oculus.local,50795) > 14/07/21 23:07:57 WARN TaskSetManager: Lost TID 1620 (task 3.0:0) > 14/07/21 23:07:57 WARN TaskSetManager: Lost result for TID 1620 on host > r104u05.oculus.local > > I've never seen this one before, and now it's coming up consistently. > > Thanks, > -Nathan > >
Matrix multiplication in spark
there is RowMatrix implemented in spark. and I check for a while but failed to find any matrix operations (like multiplication etc) are defined in the class yet. so, my question is, if I want to do matrix multiplication, (to do vector x matrix multiplication to be precise), need to convert the vector/matrix to the the matrix type defined in breeze package right? thanks
