register statement fail to change classloader in JobSubmitter thread
--------------------------------------------------------------------
Key: PIG-2449
URL: https://issues.apache.org/jira/browse/PIG-2449
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Daniel Dai
When I run TestPigRunner.classLoaderTest() in hadoop 23, I hit the following
error:
org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.pig.test.PigTestLoader$TestPathFilter
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:281)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.pig.test.PigTestLoader$TestPathFilter
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1211)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getInputPathFilter(FileInputFormat.java:182)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:214)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
... 12 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.pig.test.PigTestLoader$TestPathFilter
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1203)
... 17 more
Caused by: java.lang.ClassNotFoundException:
org.apache.pig.test.PigTestLoader$TestPathFilter
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
... 18 more
The reason is when we register a jar, we do put the jar in the classloader of
the current thread (PIG-1489). However, JobSubmitter is in a different thread
and still use the default classloader. I am not sure if there is a way to
change that classloader.
TestPigRunner.classLoaderTest() is actually wrong. The same test case will fail
in mapreduce mode and 23 test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
