Re: Dynamic jar loading

2015-12-19 Thread Jeff Zhang
Actually I would say yes and no. Yes means the jar will be fetched by executor and added to classpath, No means it would not be added to classpath of driver. That means you can not invoke the class in the jar explicitly. But you can call them indirectly. like following (or if the jar is only

Re: Dynamic jar loading

2015-12-18 Thread Jim Lohse
I am going to say no, but have not actually tested this. Just going on this line in the docs: http://spark.apache.org/docs/latest/configuration.html |spark.driver.extraClassPath| (none) Extra classpath entries to prepend to the classpath of the driver. /Note:/ In client mode, this config

Dynamic jar loading

2015-12-17 Thread amarouni
Hello guys, Do you know if the method SparkContext.addJar("file:///...") can be used on a running context (an already started spark-shell) ? And if so, does it add the jar to the class-path of the Spark workers (Yarn containers in case of yarn-client) ? Thanks,