Hi there,
I tried the SimpleApp WordCount example and it works perfect on local
environment. My code:
object SimpleApp {
def main(args: Array[String]) {
val logFile = "README.md"
val conf = new SparkConf()
.setMaster("zk://172.31.0.11:2181/mesos")
.setAppName("Simple App")
.setSparkHome("/opt/spark")
.setJars(List("app/target/scala-2.10/app-assembly-0.1-SNAPSHOT.jar"))
.set("spark.executor.memory", "5g")
.set("spark.cores.max", "10")
.set("spark.executor.uri",
"http://domain.com/spark/spark-1.0.0-wr-bin-cdh4.tgz";)
val sc = new SparkContext(conf)
System.out.println("[info] Spark Context created: " + sc)
System.out.println("[info] Spark user: " + sc.sparkUser)
val logData = sc.textFile(logFile, 2).cache()
System.out.println("[info] Log Data: " + logData)
val numAs = logData.filter(line => line.contains("a")).count()
System.out.println("[info] numAs: " + numAs)
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}
However, if I run the spark code in cluster mode (Mesos), the framework will
encounter errors and it doesn't work.
Log sbt run:
----------------------------------------------
vagrant@master1:/vagrant/workspace/spark-quick-start$ sbt run
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
[info] Loading project definition from
/vagrant/workspace/spark-quick-start/project
[info] Set current project to Simple Project (in build
file:/vagrant/workspace/spark-quick-start/)
[info] Running com.domain.spark.SimpleApp
14/03/05 09:53:36 WARN util.Utils: Your hostname, master1 resolves to a
loopback address: 127.0.1.1; using 172.31.1.11 instead (on interface eth1)
14/03/05 09:53:36 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
14/03/05 09:53:41 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/03/05 09:53:42 INFO Remoting: Starting remoting
14/03/05 09:53:45 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://spark@172.31.1.11:43210]
14/03/05 09:53:45 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://spark@172.31.1.11:43210]
14/03/05 09:53:45 INFO spark.SparkEnv: Registering BlockManagerMaster
14/03/05 09:53:45 INFO storage.DiskBlockManager: Created local directory at
/tmp/spark-local-20140305095345-a5ea
14/03/05 09:53:46 INFO storage.MemoryStore: MemoryStore started with
capacity 593.9 MB.
14/03/05 09:53:46 INFO network.ConnectionManager: Bound socket to port 50585
with id = ConnectionManagerId(172.31.1.11,50585)
14/03/05 09:53:46 INFO storage.BlockManagerMaster: Trying to register
BlockManager
14/03/05 09:53:46 INFO storage.BlockManagerMasterActor$BlockManagerInfo:
Registering block manager 172.31.1.11:50585 with 593.9 MB RAM
14/03/05 09:53:46 INFO storage.BlockManagerMaster: Registered BlockManager
14/03/05 09:53:46 INFO spark.HttpServer: Starting HTTP Server
14/03/05 09:53:47 INFO server.Server: jetty-7.6.8.v20121106
14/03/05 09:53:47 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:41819
14/03/05 09:53:47 INFO broadcast.HttpBroadcast: Broadcast server started at
http://172.31.1.11:41819
14/03/05 09:53:48 INFO spark.SparkEnv: Registering MapOutputTracker
14/03/05 09:53:48 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-fcdabf4d-33bc-4505-a9ca-a2bb2ad43da4
14/03/05 09:53:48 INFO spark.HttpServer: Starting HTTP Server
14/03/05 09:53:48 INFO server.Server: jetty-7.6.8.v20121106
14/03/05 09:53:48 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:43856
14/03/05 09:53:53 INFO server.Server: jetty-7.6.8.v20121106
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/stage,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/pool,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/environment,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/executors,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/json,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/static,null}
14/03/05 09:53:53 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/,null}
14/03/05 09:53:53 INFO server.AbstractConnector: Started
SelectChannelConnector@172.31.1.11:4040
14/03/05 09:53:53 INFO ui.SparkUI: Started Spark Web UI at
http://172.31.1.11:4040
14/03/05 09:53:54 INFO spark.SparkContext: Added JAR
app/target/scala-2.10/app-assembly-0.1-SNAPSHOT.jar at
http://172.31.1.11:43856/jars/app-assembly
-0.1-SNAPSHOT.jar with timestamp 1394013234949
2014-03-05 09:53:55,877:4907(0x7f44240fb700):ZOO_INFO@log_env@658: Client
environment:zookeeper.version=zookeeper C client 3.3.4
2014-03-05 09:53:55,878:4907(0x7f44240fb700):ZOO_INFO@log_env@662: Client
environment:host.name=master1
2014-03-05 09:53:55,878:4907(0x7f44240fb700):ZOO_INFO@log_env@669: Client
environment:os.name=Linux
2014-03-05 09:53:55,879:4907(0x7f44240fb700):ZOO_INFO@log_env@670: Client
environment:os.arch=3.8.0-35-generic
2014-03-05 09:53:55,879:4907(0x7f44240fb700):ZOO_INFO@log_env@671: Client
environment:os.version=#50-Ubuntu SMP Tue Dec 3 01:24:59 UTC 2013
2014-03-05 09:53:55,879:4907(0x7f44240fb700):ZOO_INFO@log_env@679: Client
environment:user.name=vagrant
2014-03-05 09:53:55,880:4907(0x7f44240fb700):ZOO_INFO@log_env@687: Client
environment:user.home=/home/vagrant
2014-03-05 09:53:55,880:4907(0x7f44240fb700):ZOO_INFO@log_env@699: Client
environment:user.dir=/vagrant/workspace/spark-quick-start
2014-03-05 09:53:55,880:4907(0x7f44240fb700):ZOO_INFO@zookeeper_init@727:
Initiating client connection, host=172.31.0.11:2181 sessionTimeout=10000 wat
cher=0x7f442b945dc0 sessionId=0 sessionPasswd=<null> context=0x7f4444004d40
flags=0
2014-03-05 09:53:55,882:4907(0x7f44228f8700):ZOO_INFO@check_events@1585:
initiated connection to server [172.31.0.11:2181]
2014-03-05 09:53:55,884:4907(0x7f44228f8700):ZOO_INFO@check_events@1632:
session establishment complete on server [172.31.0.11:2181],
sessionId=0x1449
1506d6f000f, negotiated timeout=10000
I0305 09:53:55.885061 4985 group.cpp:310] Group process
((2)@127.0.1.1:59604) connected to ZooKeeper
I0305 09:53:55.885396 4985 group.cpp:752] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0305 09:53:55.885670 4985 group.cpp:367] Trying to create path '/mesos' in
ZooKeeper
I0305 09:53:55.967910 4985 detector.cpp:134] Detected a new leader:
(id='0')
I0305 09:53:55.969091 4985 group.cpp:629] Trying to get '/mesos/0000000000'
in ZooKeeper
I0305 09:53:56.017066 4985 detector.cpp:354] A new leading master
(UPID=master@172.31.1.11:5050) is detected
I0305 09:53:56.017913 4985 sched.cpp:218] No credentials provided.
Attempting to register without authentication
I0305 09:53:56.019975 4985 sched.cpp:230] Detecting new master
14/03/05 09:53:56 INFO mesos.CoarseMesosSchedulerBackend: Registered as
framework ID 201403050816-184623020-5050-1350-0006
[info] Spark Context created: org.apache.spark.SparkContext@30d4c046
[info] Spark user: vagrant
14/03/05 09:53:57 INFO storage.MemoryStore: ensureFreeSpace(32856) called
with curMem=0, maxMem=622775500
14/03/05 09:53:57 INFO storage.MemoryStore: Block broadcast_0 stored as
values to memory (estimated size 32.1 KB, free 593.9 MB)
[info] Log Data: MappedRDD[1] at textFile at SimpleApp.scala:29
14/03/05 09:54:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/03/05 09:54:00 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/05 09:54:00 INFO mapred.FileInputFormat: Total input paths to process
: 1
14/03/05 09:54:00 INFO spark.SparkContext: Starting job: count at
SimpleApp.scala:32
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Got job 0 (count at
SimpleApp.scala:32) with 2 output partitions (allowLocal=false)
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Final stage: Stage 0 (count
at SimpleApp.scala:32)
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Parents of final stage:
List()
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Missing parents: List()
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Submitting Stage 0
(FilteredRDD[2] at filter at SimpleApp.scala:32), which has no missing
parents
14/03/05 09:54:01 INFO scheduler.DAGScheduler: Submitting 2 missing tasks
from Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:32)
14/03/05 09:54:01 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with
2 tasks
14/03/05 09:54:16 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
regis
tered and have sufficient memory
14/03/05 09:54:31 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
regis
tered and have sufficient memory
----------------------------------------------
Master LOG:
----------------------------------------------
I0305 09:55:11.704283 1438 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-70 on slave
201403050816-184623020-5050-1350-1 (slave2) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:11.704797 1438 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:11.705227 1438 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-1 for 5secs
I0305 09:55:12.699676 1439 master.cpp:1787] Sending 1 offers to framework
201403050816-184623020-5050-1350-0006
I0305 09:55:12.701704 1444 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-71 on slave
201403050816-184623020-5050-1350-0 (slave1) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:12.701987 1444 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:12.702188 1444 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-0 for 5secs
I0305 09:55:13.314255 1445 http.cpp:373] HTTP request for
'/master/state.json'
I0305 09:55:16.324359 1441 http.cpp:373] HTTP request for
'/master/state.json'
I0305 09:55:16.706665 1443 master.cpp:1787] Sending 1 offers to framework
201403050816-184623020-5050-1350-0006
I0305 09:55:16.712649 1440 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-72 on slave
201403050816-184623020-5050-1350-1 (slave2) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:16.713141 1440 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:16.713718 1440 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-1 for 5secs
I0305 09:55:17.707922 1442 master.cpp:1787] Sending 1 offers to framework
201403050816-184623020-5050-1350-0006
I0305 09:55:17.709288 1444 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-73 on slave
201403050816-184623020-5050-1350-0 (slave1) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:17.709404 1444 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:17.709539 1444 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-0 for 5secs
I0305 09:55:19.333492 1442 http.cpp:373] HTTP request for
'/master/state.json'
I0305 09:55:22.350966 1438 http.cpp:373] HTTP request for
'/master/state.json'
I0305 09:55:22.714210 1442 master.cpp:1787] Sending 2 offers to framework
201403050816-184623020-5050-1350-0006
I0305 09:55:22.718286 1443 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-74 on slave
201403050816-184623020-5050-1350-1 (slave2) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:22.718905 1443 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:22.719311 1443 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-1 for 5secs
I0305 09:55:22.719779 1438 master.cpp:2124] Processing reply for offer
201403050816-184623020-5050-1350-75 on slave
201403050816-184623020-5050-1350-0 (slave1) for framework
201403050816-184623020-5050-1350-0006
W0305 09:55:22.719988 1438 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0305 09:55:22.720192 1438 hierarchical_allocator_process.hpp:590]
Framework 201403050816-184623020-5050-1350-0006 filtered slave
201403050816-184623020-5050-1350-0 for 5secs
I0305 09:55:25.362354 1441 http.cpp:373] HTTP request for
'/master/state.json'
----------------------------------------------
SLAVE LOG:
----------------------------------------------
I0305 09:56:15.905223 1292 slave.cpp:2602] Current usage 10.64%. Max
allowed age: 5.555137022165880days
I0305 09:57:15.907424 1293 slave.cpp:2602] Current usage 10.64%. Max
allowed age: 5.555137022165880days
I0305 09:57:18.543018 1292 slave.cpp:1134] Asked to shut down framework
201403050816-184623020-5050-1350-0006 by master@172.31.1.11:5050
W0305 09:57:18.543279 1292 slave.cpp:1149] Cannot shut down unknown
framework 201403050816-184623020-5050-1350-0006
I0305 09:58:15.676760 1290 http.cpp:302] HTTP request for
'/slave(1)/state.json'
I0305 09:58:15.909160 1295 slave.cpp:2602] Current usage 10.64%. Max
allowed age: 5.555137022165880days
----------------------------------------------
Any help is greatly appreciated!
Thanks,
Juan Pedro
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Word-Count-on-Mesos-Cluster-tp2299.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
