[ https://issues.apache.org/jira/browse/FLINK-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14948502#comment-14948502 ]
ASF GitHub Bot commented on FLINK-1789: --------------------------------------- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/593#issuecomment-146509420 @twalthr Doesn't seem to compile :) > Allow adding of URLs to the usercode class loader > ------------------------------------------------- > > Key: FLINK-1789 > URL: https://issues.apache.org/jira/browse/FLINK-1789 > Project: Flink > Issue Type: Improvement > Components: Distributed Runtime > Reporter: Timo Walther > Assignee: Timo Walther > Priority: Minor > > Currently, there is no option to add customs classpath URLs to the > FlinkUserCodeClassLoader. JARs always need to be shipped to the cluster even > if they are already present on all nodes. > It would be great if RemoteEnvironment also accepts valid classpaths URLs and > forwards them to BlobLibraryCacheManager. > The problem with the current approach is that the code loaded by the regular > JVM class loader cannot refer to job specific types (which can be accessed > only at the UserCodeClassLoader level). Unfortunately, this is the case if we > use the classpath entry to generate the dataflows dynamically at runtime. > Currently this functionality needs to be done by "hacks" (hardcode a > filesystem path next to the list of jars when initializing the BlobManager > entry). It makes sense to open an issue which makes this list parameterizable > via an additional ExecutionEnvironment argument (this is basically the only > main feature which prohibits the use of Emma project with "off-the-shelf" > Flink). > This, of course, would require that the folders are shared (e.g. via NFS) > between client, master and workers. I think what made Stephan so excited is > the idea of using the same URL mechanism in order to ship the code to all > dependent parties (most probably by running a dedicated HTTP or FTP server on > the client). > We make the following assumptions for the use case where we need the global > class path: > - The URL is either a file path that points to a directory accessible to all > nodes (NFS or so) and the client runs in the cluster as well. > - The URL is an HTTP URL or so that points to a file server that serves the > classes to work in non-shared directory settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)