Adam Lewandowski created SPARK-11093:
----------------------------------------
Summary: ChildFirstURLClassLoader#getResources should return all
found resources, not just those in the child classloader
Key: SPARK-11093
URL: https://issues.apache.org/jira/browse/SPARK-11093
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.5.1
Reporter: Adam Lewandowski
Currently when using a child-first classloader
(spark.{driver|executor}.userClassPathFirst = true), the getResources method
does not return any matching resources from the parent classloader if the child
classloader contains any. This is not child-first, it's child-only and is
inconsistent with how the default parent-first classloaders work in the JDK
(all found resources are returned from both classloaders). It is also
inconsistent with how child-first classloaders work in other environments
(Servlet containers, for example).
ChildFirstURLClassLoader#getResources() should return resources found from both
the child and the parent classloaders, placing any found from the child
classloader first.
For reference, the specific use case where I encountered this problem was
running Spark on AWS EMR in a child-first arrangement (due to guava version
conflicts), where Akka's configuration file (reference.conf) was made available
in the parent classloader, but was not visible to the Typesafe config library
which uses Classloader.getResources() on the Thread's context classloader to
find them. This resulted in a fatal error from the Config library:
"com.typesafe.config.ConfigException$Missing: No configuration setting found
for key 'akka.version'" .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
