[
https://issues.apache.org/jira/browse/BEAM-8113?focusedWorklogId=307541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307541
]
ASF GitHub Bot logged work on BEAM-8113:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Sep/19 23:56
Start Date: 05/Sep/19 23:56
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9451: [BEAM-8113]
Stage files from context classloader
URL: https://github.com/apache/beam/pull/9451#discussion_r321530670
##########
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineResources.java
##########
@@ -40,34 +44,42 @@
public class PipelineResources {
/**
- * Attempts to detect all the resources the class loader has access to. This
does not recurse to
- * class loader parents stopping it from pulling in resources from the
system class loader.
+ * Detects all URLs that are present in all class loaders in between context
class loader of
+ * calling thread and class loader of class passed as parameter. It doesn't
follow parents above
+ * this class loader stopping it from pulling in resources from the system
class loader.
*
- * @param classLoader The URLClassLoader to use to detect resources to stage.
- * @throws IllegalArgumentException If either the class loader is not a
URLClassLoader or one of
- * the resources the class loader exposes is not a file resource.
+ * @param cls Class whose class loader stops recursion into parent loaders
+ * @throws IllegalArgumentException no classloader in context hierarchy is
URLClassloader or if
+ * one of the resources any class loader exposes is not a file resource.
* @return A list of absolute paths to the resources the class loader uses.
*/
- public static List<String> detectClassPathResourcesToStage(ClassLoader
classLoader) {
- if (!(classLoader instanceof URLClassLoader)) {
- String message =
- String.format(
- "Unable to use ClassLoader to detect classpath elements. "
- + "Current ClassLoader is %s, only URLClassLoaders are
supported.",
- classLoader);
- throw new IllegalArgumentException(message);
- }
+ public static List<String> detectClassPathResourcesToStage(Class<?> cls) {
+ return detectClassPathResourcesToStage(cls.getClassLoader());
+ }
- List<String> files = new ArrayList<>();
- for (URL url : ((URLClassLoader) classLoader).getURLs()) {
- try {
- files.add(new File(url.toURI()).getAbsolutePath());
- } catch (IllegalArgumentException | URISyntaxException e) {
- String message = String.format("Unable to convert url (%s) to file.",
url);
- throw new IllegalArgumentException(message, e);
- }
+ /**
+ * Detect resources to stage, by using hierarchy between current context
classloader and the given
+ * one. Always include the given class loader in the process.
+ *
+ * @param loader the loader to use as stopping condition when traversing
context class loaders
+ * @return A list of absolute paths to the resources the class loader uses.
+ */
+ @VisibleForTesting
+ static List<String> detectClassPathResourcesToStage(final ClassLoader
loader) {
+ Set<String> files = new HashSet<>();
Review comment:
Making this a set even when there is a single classloader means that we will
reorder that classloaders classpath which will cause lots of dependency
conflict issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 307541)
Time Spent: 7h 10m (was: 7h)
> FlinkRunner: Stage files from context classloader
> -------------------------------------------------
>
> Key: BEAM-8113
> URL: https://issues.apache.org/jira/browse/BEAM-8113
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Jan Lukavský
> Assignee: Jan Lukavský
> Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Currently, only files from {{FlinkRunner.class.getClassLoader()}} are staged
> by default. Add also files from
> {{Thread.currentThread().getContextClassLoader()}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
