Hi All,
We've started with deploying spark on Hadoop 2 and Yarn. Our previous
configuration (still not a production cluster) was Spark on Mesos.
We're running a java application (which runs from tomcat server). The
application builds a singleton java spark context when it is first lunch and
then all users' requests are executed using this same spark context.
With Mesos - creating the context included few simple operation and was
possible via the java application.
I successfully executed Spark and Yarn example and even my own example
(although I was unable to find the output logs)
I noticed that it is being done using org.apache.spark.deploy.yarn.Client but
have no example regarding how it can be done.
Successful command:
SPARK_JAR=/app/spark-0.8.0-incubating/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.4-Intel.jar
./spark-class org.apache.spark.deploy.yarn.Client --jar
/app/iot/test/test3-0.0.1-SNAPSHOT.jar --class test3.yarntest
--args yarn-standalone --num-workers 3 --master-memory 4g
--worker-memory 2g --worker-cores
When I try to emulate the previous method we used and simple execute my test
jar - the execution hangs.
Our main goal is to be able to execute spark context on yarn from java code
(and not shell script) and create a singleton spark context.
In addition the application should be executed on a remote YARN server and not
on a local one.
Can you please advice?
Thanks,
Ido
Problematic Command:
/usr/java/latest/bin/java -cp
/usr/lib/hbase/hbase-0.94.7-Intel.jar:/usr/lib/hadoop/hadoop-auth-2.0.4-Intel.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/app/spark-0.8.0-incubating/conf:/app/spark-0.8.0-incubating/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.4-Intel.jar:/etc/hadoop/conf:/etc/hbase/conf:/etc/hadoop/conf:/app/iot/test/test3-0.0.1-SNAPSHOT.jar
-Djava.library.path=/usr/lib/hadoop/lib/native -Xms512m -Xmx512m test3.yarntest
Spark Context code piece:
JavaSparkContext sc = new JavaSparkContext(
"yarn-standalone",
"SPARK YARN TEST"
);
Log:
13/12/12 17:30:36 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/12 17:30:36 INFO spark.SparkEnv: Registering BlockManagerMaster
13/12/12 17:30:36 INFO storage.MemoryStore: MemoryStore started with capacity
323.9 MB.
13/12/12 17:30:36 INFO storage.DiskStore: Created local directory at
/tmp/spark-local-20131212173036-09c0
13/12/12 17:30:36 INFO network.ConnectionManager: Bound socket to port 39426
with id =
ConnectionManagerId(ip-172-31-43-121.eu<http://ip-172-31-43-121.eu/>-west-1.compute.internal,39426)
13/12/12 17:30:36 INFO storage.BlockManagerMaster: Trying to register
BlockManager
13/12/12 17:30:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/12/12 17:30:37 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/12 17:30:37 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:43438<mailto:SocketConnector@0.0.0.0:43438>
13/12/12 17:30:37 INFO broadcast.HttpBroadcast: Broadcast server started at
http://172.31.43.121:43438<http://172.31.43.121:43438/>
13/12/12 17:30:37 INFO spark.SparkEnv: Registering MapOutputTracker
13/12/12 17:30:37 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-b48abc5a-53c6-4af1-9c3c-725e1cd7fbb9
13/12/12 17:30:37 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/12 17:30:37 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:60476<mailto:SocketConnector@0.0.0.0:60476>
13/12/12 17:30:37 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/stage,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/pool,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/environment,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/executors,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/json,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/static,null}
13/12/12 17:30:37 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/,null}
13/12/12 17:30:37 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040<mailto:SelectChannelConnector@0.0.0.0:4040>
13/12/12 17:30:37 INFO ui.SparkUI: Started Spark Web UI at
http://ip-172-31-43-121.eu-west-1.compute.internal:4040<http://ip-172-31-43-121.eu-west-1.compute.internal:4040/>
13/12/12 17:30:37 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
13/12/12 17:30:37 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook for
context
org.apache.spark.SparkContext@475a07bf<mailto:org.apache.spark.SparkContext@475a07bf>
---------------------------------------------------------------------
Intel Electronics Ltd.
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
