[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[ https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281200#comment-17281200 ] Maciej Prochniak commented on FLINK-21138: -- I only had time this weekend to look at this: [https://github.com/apache/flink/pull/14902] - here is the change, I added integration test to verify the fix works > KvStateServerHandler is not invoked with user code classloader > -- > > Key: FLINK-21138 > URL: https://issues.apache.org/jira/browse/FLINK-21138 > Project: Flink > Issue Type: Bug > Components: Runtime / Queryable State >Affects Versions: 1.11.2 >Reporter: Maciej Prochniak >Assignee: Maciej Prochniak >Priority: Major > Labels: pull-request-available > Attachments: TestJob.java, stacktrace > > > When using e.g. custom Kryo serializers user code classloader has to be set > as context classloader during invocation of methods such as > TypeSerializer.duplicat() > KvStateServerHandler does not do this, which leads to exceptions like > ClassNotFound etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[ https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273766#comment-17273766 ] Till Rohrmann commented on FLINK-21138: --- I think you are right [~mproch]. The problem with context classloader is that it is not very explicit and can easily break (e.g. if a method further downstream sets a different context classloader). The clean solution would probably be to pass in an explicit classloader. But this would be super involved. Hence, I am ok with your proposal. I've assigned you to the ticket. > KvStateServerHandler is not invoked with user code classloader > -- > > Key: FLINK-21138 > URL: https://issues.apache.org/jira/browse/FLINK-21138 > Project: Flink > Issue Type: Bug > Components: Runtime / Queryable State >Affects Versions: 1.11.2 >Reporter: Maciej Prochniak >Priority: Major > Attachments: TestJob.java, stacktrace > > > When using e.g. custom Kryo serializers user code classloader has to be set > as context classloader during invocation of methods such as > TypeSerializer.duplicat() > KvStateServerHandler does not do this, which leads to exceptions like > ClassNotFound etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[ https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273371#comment-17273371 ] Maciej Prochniak commented on FLINK-21138: -- I think using original.getClass().getClassLoader() won't work here - "original" is SerializableSerializer wrapper, which is probably loaded by framework classloader. I think with such custom serializers the part loaded by user classloader can hidden via quite a few wrappers/decorators and using contextClassloader is safer, it's done like that also here: [here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/JavaSerializer.java#L59] I'll try to come up with PR with test > KvStateServerHandler is not invoked with user code classloader > -- > > Key: FLINK-21138 > URL: https://issues.apache.org/jira/browse/FLINK-21138 > Project: Flink > Issue Type: Bug > Components: Runtime / Queryable State >Affects Versions: 1.11.2 >Reporter: Maciej Prochniak >Priority: Major > Attachments: TestJob.java, stacktrace > > > When using e.g. custom Kryo serializers user code classloader has to be set > as context classloader during invocation of methods such as > TypeSerializer.duplicat() > KvStateServerHandler does not do this, which leads to exceptions like > ClassNotFound etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[
https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273026#comment-17273026
]
Till Rohrmann commented on FLINK-21138:
---
Thanks for posting this example. I am wondering whether the problem isn't that
we are using the {{Thread.currentThread().getContextClassLoader()}}
[here|https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L626]
instead of {{original.getClass().getClassLoader()}}. I think this should fix
the problem.
Test-wise you could take a look at the
{{PojoSerializerUpgradeTest.testPojoSerializerUpgrade}} which creates a new
class and instantiates an {{URLClassLoader}} to load it. If you then use the
test thread, it should not know about this new class.
> KvStateServerHandler is not invoked with user code classloader
> --
>
> Key: FLINK-21138
> URL: https://issues.apache.org/jira/browse/FLINK-21138
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.11.2
>Reporter: Maciej Prochniak
>Priority: Major
> Attachments: TestJob.java, stacktrace
>
>
> When using e.g. custom Kryo serializers user code classloader has to be set
> as context classloader during invocation of methods such as
> TypeSerializer.duplicat()
> KvStateServerHandler does not do this, which leads to exceptions like
> ClassNotFound etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[ https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272393#comment-17272393 ] Maciej Prochniak commented on FLINK-21138: -- Hi [~trohrmann], thanks for quick response. I attach smallest case I could come up with, and stack trace that is thrown. I think the problem is relatively straightforward and user classloader just needs to be passed down to KvStateEntry. I think I can prepare a PR, but I'm not quite sure what kind of test would be suitable - as the problem involves custom classloaders etc. > KvStateServerHandler is not invoked with user code classloader > -- > > Key: FLINK-21138 > URL: https://issues.apache.org/jira/browse/FLINK-21138 > Project: Flink > Issue Type: Bug > Components: Runtime / Queryable State >Affects Versions: 1.11.2 >Reporter: Maciej Prochniak >Priority: Major > Attachments: TestJob.java, stacktrace > > > When using e.g. custom Kryo serializers user code classloader has to be set > as context classloader during invocation of methods such as > TypeSerializer.duplicat() > KvStateServerHandler does not do this, which leads to exceptions like > ClassNotFound etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21138) KvStateServerHandler is not invoked with user code classloader
[ https://issues.apache.org/jira/browse/FLINK-21138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272095#comment-17272095 ] Till Rohrmann commented on FLINK-21138: --- Thanks for reporting this issue [~mproch]. Could you give a short example illustrating the problem? Maybe you have the problematic job already in some repository. > KvStateServerHandler is not invoked with user code classloader > -- > > Key: FLINK-21138 > URL: https://issues.apache.org/jira/browse/FLINK-21138 > Project: Flink > Issue Type: Bug > Components: Runtime / Queryable State >Affects Versions: 1.11.2 >Reporter: Maciej Prochniak >Priority: Major > > When using e.g. custom Kryo serializers user code classloader has to be set > as context classloader during invocation of methods such as > TypeSerializer.duplicat() > KvStateServerHandler does not do this, which leads to exceptions like > ClassNotFound etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
