[
https://issues.apache.org/jira/browse/PIG-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16838353#comment-16838353
]
Nandor Kollar commented on PIG-5387:
------------------------------------
[~knoguchi], [~rohini] the class loading related changes is an ugly hack which
I'm not proud of, but I didn't find any easy fix which is compatible with Java
11 as well as Java 8, and this one seems to work. Fortunately it is only a test
problem, not a production code issue. Until Java 8 the system class loader
([AppClassLoader|https://github.com/openjdk/jdk/blob/jdk8-b120/jdk/src/share/classes/sun/misc/Launcher.java#L259])
extends URLClassLoader, thus casting and registering the jar at runtime works
fine. This has changes in Java 11 (I think since Java 9, but didn't verify
this): system class loader no longer extends URLClassLoader (see
[here|https://github.com/openjdk/jdk11u/blob/master/src/java.base/share/classes/jdk/internal/loader/ClassLoaders.java#L151]).
I tried to create a new URLClassLoader and set the Thread's context class
loader to it, but Pig uses
[getSystemResources|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/PigServer.java#L575]
to locate the jars, which will ask system classloader, for the locations, and
won't find resources registered below it. That's why I came up with this hack:
on Java 8 it will work as before, on Java 11, it will register the new
URLClassLoader as the parent of system class loader, and will find the jars. I
know that this is an ugly hack, I guess the correct answer to this problem
would be Java 11 module loading, but to remain compatible with Java 8 we should
either introduce several reflective invocations, or we should introduce
something like Java 11 shim. I don't think either option pays off for a single
test fix like this one. The only incompatibility in this case is the casting of
system class loader to URLClassLoader, and since we don't do it in any other
place apart from these two test classes, I don't think it causes any issue with
registering commands in Pig. Refactoring registerNewResource to Utils class is
a good idea, as well as adding a comment which briefly describes the
aforementioned situation.
> Test failures on JRE 11
> -----------------------
>
> Key: PIG-5387
> URL: https://issues.apache.org/jira/browse/PIG-5387
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Nandor Kollar
> Priority: Major
> Attachments: PIG-5387_1.patch
>
>
> I tried to compile Pig with JDK 8 and execute the test with Java 11, and
> faced with several test failures. For example TestCommit#testCheckin2 failed
> with the following exception:
> {code}
> 2019-05-08 16:06:09,712 WARN [Thread-108] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1000317333_0003
> java.lang.Exception: java.io.IOException: Deserialization error: null
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> Caused by: java.io.IOException: Deserialization error: null
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:62)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:183)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.plan.Operator.hashCode(Operator.java:106)
> at java.base/java.util.HashMap.hash(HashMap.java:339)
> at java.base/java.util.HashMap.readObject(HashMap.java:1461)
> at
> java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160)
> {code}
> It deserialization of one of the map plan failed, it appears we ran into
> [JDK-8201131|https://bugs.openjdk.java.net/browse/JDK-8201131]. I seems that
> the workaround in the issue report works, adding a readObject method to
> org.apache.pig.impl.plan.Operator:
> {code}
> private void readObject(ObjectInputStream in) throws
> ClassNotFoundException, IOException {
> in.defaultReadObject();
> }
> {code}
> solves the problem, however I'm not sure that this is the optimal solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
