You can try to play with experimental flags [1] `spark.executor.userClassPathFirst` and `spark.driver.userClassPathFirst`. But this can also potentially break other things (like: dependencies that Spark master required initializing overridden by Spark app and so on) so, you will need to verify.
[1] https://spark.apache.org/docs/latest/configuration.html On Thu, Jul 7, 2016 at 4:05 PM, Chen Song <chen.song...@gmail.com> wrote: > Sorry to spam people who are not interested. Greatly appreciate it if > anyone who is familiar with this can share some insights. > > On Wed, Jul 6, 2016 at 2:28 PM Chen Song <chen.song...@gmail.com> wrote: > >> Hi >> >> I ran into problems to use class loader in Spark. In my code (run within >> executor), I explicitly load classes using the ContextClassLoader as below. >> >> Thread.currentThread().getContextClassLoader() >> >> The jar containing the classes to be loaded is added via the --jars >> option in spark-shell/spark-submit. >> >> I always get the class not found exception. However, it seems to work if >> I compile these classes in main jar for the job (the jar containing the >> main job class). >> >> I know Spark implements its own class loaders in a particular way. Is >> there a way to work around this? In other words, what is the proper way to >> programmatically load classes in other jars added via --jars in Spark? >> >> -- -- Cheers, Praj
