RE: How to add a custom jar file to the Spark driver?

Hi guys, Thanks for responding. Re SPARK_CLASSPATH (Daoyuan): I think you are right. We tried it, and that’s what the warning we got said. Re SparkConf (Daoyuan): We need the custom jar in the driver code, so I don’t know how that would work. Re EMR -u (Sonal): The documentation says that

Re: How to add a custom jar file to the Spark driver?

Hi Gerhard, I just stumbled upon some documentation on EMR - link below. Seems there is a -u option to add jars in S3 to your classpath, have you tried that ? http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-spark-configure.html Best Regards, Sonal Founder, Nube

RE: How to add a custom jar file to the Spark driver?

Hi Gerhard, How does EMR set its conf for spark? I think if you set SPARK_CLASSPATH and spark.dirver.extraClassPath, spark would ignore SPARK_CLASSPATH. I think you can do this by read the configuration from SparkConf, and then add your custom settings to the corresponding key, and use the