object SparkPi {
private val logger = Logger(this.getClass)
val sparkConf = new SparkConf()
.setAppName("Spark Pi")
.setMaster("spark://10.100.103.192:7077")
lazy val sc = new SparkContext(sparkConf)
sc.addJar("/Users/yfang/workspace/mcs/target/scala-2.11/root-assembly-0.1.0.jar")
def main(args: Array[String]) {
val x = (1 to 4)
val a = sc.parallelize(x)
val mean = a.mean()
print(mean)
}
}
spark://10.100.103.192:7077 is a remote standalone cluster I created on
AWS.
I ran it locally with IntelliJ.
I can see the job is submitted. But the calculation can never finish.
The log shows:
15:34:21.674 [Timer-0] WARN org.apache.spark.scheduler.TaskSchedulerImpl -
Initial job has not accepted any resources; check your cluster UI to ensure
that workers are registered and have sufficient resources
Any help will be highly appreciated!
Thanks!
Yuan
--
This message is intended exclusively for the individual or entity to which
it is addressed. This communication may contain information that is
proprietary, privileged or confidential or otherwise legally prohibited
from disclosure. If you are not the named addressee, you are not authorized
to read, print, retain, copy or disseminate this message or any part of it.
If you have received this message in error, please notify the sender
immediately by e-mail and delete all copies of the message.
