[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991190#comment-16991190 ] Guowei Ma commented on FLINK-13993: --- [~tison] Sorry for the late reply. I think your method could work. But the method leads that the directory structure of TM is almost the same in every mode. [~liyu] yes I would do it. > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Original Estimate: 30h > Time Spent: 20m > Remaining Estimate: 29h 40m > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991163#comment-16991163 ] Yu Li commented on FLINK-13993: --- [~maguowei] Since all sub-tasks have been completed, I guess we could close this JIRA as resolved? [~tison] IMHO we could open discussion thread in ML and follow up JIRAs for further implementation (if any). What do you think? Thanks. > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Original Estimate: 30h > Time Spent: 20m > Remaining Estimate: 29h 40m > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[
https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977045#comment-16977045
]
Zili Chen commented on FLINK-13993:
---
DISCLAIMER: I don't want to challenge this implementation
Alternative shares: we possibly pass user classpath via YARN dynamic properties
and normally ship files without adding user classpath in system classpath. When
retrieved {{JobGraph}} from {{JobGraphRetriever}} we add back classpath from
that dynamic properties. The same for other case we pass user classpath and
exclude then from system classpath.
> Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
> -
>
> Key: FLINK-13993
> URL: https://issues.apache.org/jira/browse/FLINK-13993
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Original Estimate: 30h
> Time Spent: 20m
> Remaining Estimate: 29h 40m
>
> Currently, Flink has the FlinkUserCodeClassLoader, which is using to load
> user’s class. However, the user class and the system class are all loaded by
> the system classloader in the perjob mode. This introduces some conflicts.
> This document[1] gives a proposal that makes the FlinkUserClassLoader load
> the user class in perjob mode. (disscuss with Till[2])
>
> [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7]
> [2]
> [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933201#comment-16933201 ] Guowei Ma commented on FLINK-13993: --- Sorry, I change the time. Actually, what I mean is one man week. > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Original Estimate: 30h > Remaining Estimate: 30h > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933112#comment-16933112 ] Till Rohrmann commented on FLINK-13993: --- Thanks for updating the work estimate [~maguowei]. So we think that the feature will take a man month to be completed, right? > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930613#comment-16930613 ] Guowei Ma commented on FLINK-13993: --- I will do it. thanks. [~till.rohrmann] > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930457#comment-16930457 ] Till Rohrmann commented on FLINK-13993: --- [~maguowei] did you start working on this issue? If yes, then we should assign you or the person working on this problem to this issue. > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927484#comment-16927484 ] Guowei Ma commented on FLINK-13993: --- Thanks for reminding me. I'll close it. [~till.rohrmann] > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13993) Using FlinkUserCodeClassLoaders to load the user class in the perjob mode
[ https://issues.apache.org/jira/browse/FLINK-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925734#comment-16925734 ] Till Rohrmann commented on FLINK-13993: --- If this issue is duplicated by/duplicates FLINK-13840, can we close one of them as duplicate [~maguowei]? > Using FlinkUserCodeClassLoaders to load the user class in the perjob mode > - > > Key: FLINK-13993 > URL: https://issues.apache.org/jira/browse/FLINK-13993 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Priority: Major > Fix For: 1.10.0 > > > Currently, Flink has the FlinkUserCodeClassLoader, which is using to load > user’s class. However, the user class and the system class are all loaded by > the system classloader in the perjob mode. This introduces some conflicts. > This document[1] gives a proposal that makes the FlinkUserClassLoader load > the user class in perjob mode. (disscuss with Till[2]) > > [1][https://docs.google.com/document/d/1fH2Cwrrmps5RxxvVuUdeprruvDNabEaIHPyYps28WM8/edit#heading=h.815t5dodlxh7] > [2] > [https://docs.google.com/document/d/1SUhFt1BmsGMLUYVa72SWLbNrrWzunvcjAlEm8iusvq0/edit] -- This message was sent by Atlassian Jira (v8.3.2#803003)
