[
https://issues.apache.org/jira/browse/SPARK-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15011202#comment-15011202
]
Jungtaek Lim edited comment on SPARK-11818 at 11/18/15 3:30 PM:
----------------------------------------------------------------
One thing to note is,
if I remove "spark.repl.class.uri" config to SparkConf, and create new
SparkContext with new SparkConf, and use new sc to run codes, it runs normally.
Without removing "spark.repl.class.uri", creating new SparkConf / SparkContext
doesn't help.
Let me explain to the point.
ExecutorClassLoader extends ClassLoader with no parent. It just has parent
class loader as a field wrapped to ParentClassLoader.
ExecutorClassLoader overrides findClass() which interacts with parent class
loader so there's no problem loading classes. But ExecutorClassLoader doesn't
override methods related to resources, so we can't find any resources from
ExecutorClassLoader although parent class loader can find them.
May pull request makes it more clear.
was (Author: kabhwan):
One thing to note is,
if I remove "spark.repl.class.uri" config to SparkConf, and create new
SparkContext with new SparkConf, and use new sc to run codes, it runs normally.
Without removing "spark.repl.class.uri" new SparkConf / SparkContext doesn't
help.
Let me explain to the point.
ExecutorClassLoader extends ClassLoader with no parent. It just has parent
class loader as a field wrapped to ParentClassLoader.
ExecutorClassLoader overrides findClass() which interacts with parent class
loader so there's no problem loading classes. But ExecutorClassLoader doesn't
override methods related to resources, so we can't find any resources from
ExecutorClassLoader although parent class loader can find them.
May pull request makes it more clear.
> REPL ExecutorClassLoader cannot see any resources from parent class loader
> --------------------------------------------------------------------------
>
> Key: SPARK-11818
> URL: https://issues.apache.org/jira/browse/SPARK-11818
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Environment: CentOS 6, spark 1.4.1-hadoop2.4, mesos 0.22.1
> Reporter: Jungtaek Lim
>
> This issue starts from finding root reason from strange problem from
> spark-shell (and zeppelin) which is not a problem for spark-submit.
> https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAF5108jMXyOjiGmCgr%3Ds%2BNvTMcyKWMBVM1GsrH7Pz4xUj48LfA%40mail.gmail.com%3E
> After some hours (over days) digging into the detail, I found that
> ExecutorClassLoader cannot see any resource files which can be seen from
> parent class loader.
> ExecutorClassLoader itself doesn't need to lookup resource files cause REPL
> doesn't generate these, but it should delegate lookup to parent class loader.
> I'll provide the pull request which includes tests which fails on master soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
