[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869647#comment-17869647
]
Koji Noguchi commented on PIG-5459:
-----------------------------------
It confused me on why regular run (e2e) is requiring junit jar.
It turns out
"from org.apache.hadoop.conf import *"
line matches classes from test jars that Hadoop3 added as part of regular lib.
For example
{noformat}
/tmp/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6-tests.jar ===========
0 Sun Jun 18 08:22:40 UTC 2023 org/apache/hadoop/conf/
2151 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestConfigurationDeprecation$1.class
522 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances$SampleClass.class
2291 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestConfigurationDeprecation$2.class
333 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances$ChildInterface.class
2203 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances.class
2358 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestConfigurationSubclass.class
3335 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestDeprecatedKeys.class
71538 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestConfiguration.class
...
/tmp/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6-tests.jar
===========
0 Sun Jun 18 08:42:34 UTC 2023 org/apache/hadoop/conf/
4469 Sun Jun 18 08:42:34 UTC 2023
org/apache/hadoop/conf/TestNoDefaultsJobConf.class
...
{noformat}
Now, these classes requires junit.
One option is to skip these test jars but that requires changes on the hadoop
side (since pig is calling hadoop commandline to start up pig.)
Second option is to give it up and add the required junit jars to lib dir.
Third option is to skip this test and let users add junit jars if they really
need to call
"from org.apache.hadoop.conf import *". but it's pretty tough to understand
what's happening when users hit this.
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
> ----------------------------------------------------------------
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
