[
https://issues.apache.org/jira/browse/SPARK-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Lewandowski updated SPARK-11093:
-------------------------------------
Description:
Currently when using a child-first classloader
(spark.driver|executor.userClassPathFirst = true), the getResources method does
not return any matching resources from the parent classloader if the child
classloader contains any. This is not child-first, it's child-only and is
inconsistent with how the default parent-first classloaders work in the JDK
(all found resources are returned from both classloaders). It is also
inconsistent with how child-first classloaders work in other environments
(Servlet containers, for example).
ChildFirstURLClassLoader#getResources() should return resources found from both
the child and the parent classloaders, placing any found from the child
classloader first.
For reference, the specific use case where I encountered this problem was
running Spark on AWS EMR in a child-first arrangement (due to guava version
conflicts), where Akka's configuration file (reference.conf) was made available
in the parent classloader, but was not visible to the Typesafe config library
which uses Classloader.getResources() on the Thread's context classloader to
find them. This resulted in a fatal error from the Config library:
"com.typesafe.config.ConfigException$Missing: No configuration setting found
for key 'akka.version'" .
was:
Currently when using a child-first classloader
(spark.{driver|executor}.userClassPathFirst = true), the getResources method
does not return any matching resources from the parent classloader if the child
classloader contains any. This is not child-first, it's child-only and is
inconsistent with how the default parent-first classloaders work in the JDK
(all found resources are returned from both classloaders). It is also
inconsistent with how child-first classloaders work in other environments
(Servlet containers, for example).
ChildFirstURLClassLoader#getResources() should return resources found from both
the child and the parent classloaders, placing any found from the child
classloader first.
For reference, the specific use case where I encountered this problem was
running Spark on AWS EMR in a child-first arrangement (due to guava version
conflicts), where Akka's configuration file (reference.conf) was made available
in the parent classloader, but was not visible to the Typesafe config library
which uses Classloader.getResources() on the Thread's context classloader to
find them. This resulted in a fatal error from the Config library:
"com.typesafe.config.ConfigException$Missing: No configuration setting found
for key 'akka.version'" .
> ChildFirstURLClassLoader#getResources should return all found resources, not
> just those in the child classloader
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-11093
> URL: https://issues.apache.org/jira/browse/SPARK-11093
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.5.1
> Reporter: Adam Lewandowski
>
> Currently when using a child-first classloader
> (spark.driver|executor.userClassPathFirst = true), the getResources method
> does not return any matching resources from the parent classloader if the
> child classloader contains any. This is not child-first, it's child-only and
> is inconsistent with how the default parent-first classloaders work in the
> JDK (all found resources are returned from both classloaders). It is also
> inconsistent with how child-first classloaders work in other environments
> (Servlet containers, for example).
> ChildFirstURLClassLoader#getResources() should return resources found from
> both the child and the parent classloaders, placing any found from the child
> classloader first.
> For reference, the specific use case where I encountered this problem was
> running Spark on AWS EMR in a child-first arrangement (due to guava version
> conflicts), where Akka's configuration file (reference.conf) was made
> available in the parent classloader, but was not visible to the Typesafe
> config library which uses Classloader.getResources() on the Thread's context
> classloader to find them. This resulted in a fatal error from the Config
> library: "com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'akka.version'" .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
