[
https://issues.apache.org/jira/browse/BEAM-8113?focusedWorklogId=304080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304080
]
ASF GitHub Bot logged work on BEAM-8113:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Aug/19 07:46
Start Date: 30/Aug/19 07:46
Worklog Time Spent: 10m
Work Description: je-ik commented on issue #9451: [BEAM-8113] Stage files
from context classloader
URL: https://github.com/apache/beam/pull/9451#issuecomment-526497002
The discussion in #8775 is interesting, but I think that it would need an
extension. What I actually need is a way of propagating classes that are
generated *at runtime*, and therefore cannot be part of any system variable
(classes are generated inside a REPL shell). The general case is that
application starts with some classloader, then creates another and sets it as
context classloader. Now we have two issues:
a) classes that are part of this new classloader have to be externalized
(stored to jar), and
b) this new jar have to be made available to *all possible runners*
The problem a) is not related to Beam and have to be handled by the
application, but problem b) is actually more complicated, than it seems,
because runners don't have any common way to ship dependencies (maybe this is
changed in portability, I'm not sure). The only way, that seems to work is
actually not to use the `filesToStage` pipeline option (because it might be
ignored, and for instance `FlinkRunner` (local) ignores it), but to propagate
this file to applications classloader (essentially by something like
`Thread.currentThread().setContextClassLoader(new URLClassLoader(new URL[] {
myNewJar }, Thread.currentThread().getContextClassLoader())`. This works well
for both local and remote submission of a job (tested on flink and spark), but
currently breaks for JDK >= 9.
This PR fixes that partly, flink local runner is unfortunately somehow
broken and doesn't use context classloader to deserialize UDFs. There is a
workaround (see [this
hack](https://github.com/datadrivencz/proxima-platform/blob/363dee45e9c1edff3b6383b3fe13c463a0169f01/beam/tools/src/main/java/cz/o2/proxima/beam/tools/groovy/BeamStreamProvider.java#L264)),
but it most probably needs fixing in flink directly. The hack is obviously not
working for JDK >= 9 either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 304080)
Time Spent: 1.5h (was: 1h 20m)
> FlinkRunner: Stage files from context classloader
> -------------------------------------------------
>
> Key: BEAM-8113
> URL: https://issues.apache.org/jira/browse/BEAM-8113
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Jan Lukavský
> Assignee: Jan Lukavský
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently, only files from {{FlinkRunner.class.getClassLoader()}} are staged
> by default. Add also files from
> {{Thread.currentThread().getContextClassLoader()}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
