You can build a jar of your project and add it to the sparkContext
(sc.addJar("/path/to/your/project.jar")) then it will get shipped to the
worker and hence no classNotfoundException!
Thanks
Best Regards
On Thu, Dec 18, 2014 at 10:06 PM, Akshat Aranya <aara...@gmail.com> wrote:
>
> Hi,
>
> I am building a Spark-based service which requires initialization of a
> SparkContext in a main():
>
> def main(args: Array[String]) {
> val conf = new SparkConf(false)
> .setMaster("spark://foo.example.com:7077")
> .setAppName("foobar")
>
> val sc = new SparkContext(conf)
> val rdd = sc.parallelize(0 until 255)
> val res = rdd.mapPartitions(it => it).take(1)
> println(s"res=$res")
> sc.stop()
> }
>
> This code works fine via REPL, but not as a standalone program; it causes
> a ClassNotFoundException. This has me confused about how code is shipped
> out to executors. When using via REPL, does the mapPartitions closure,
> it=>it, get sent out when the REPL statement is executed? When this code
> is run as a standalone program (not via spark-submit), is the compiled code
> expected to be present at the the executor?
>
> Thanks,
> Akshat
>
>
