Hi, I am building a Spark-based service which requires initialization of a SparkContext in a main():
def main(args: Array[String]) { val conf = new SparkConf(false) .setMaster("spark://foo.example.com:7077") .setAppName("foobar") val sc = new SparkContext(conf) val rdd = sc.parallelize(0 until 255) val res = rdd.mapPartitions(it => it).take(1) println(s"res=$res") sc.stop() } This code works fine via REPL, but not as a standalone program; it causes a ClassNotFoundException. This has me confused about how code is shipped out to executors. When using via REPL, does the mapPartitions closure, it=>it, get sent out when the REPL statement is executed? When this code is run as a standalone program (not via spark-submit), is the compiled code expected to be present at the the executor? Thanks, Akshat