[ https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869647#comment-17869647 ]
Koji Noguchi commented on PIG-5459: ----------------------------------- It confused me on why regular run (e2e) is requiring junit jar. It turns out "from org.apache.hadoop.conf import *" line matches classes from test jars that Hadoop3 added as part of regular lib. For example {noformat} /tmp/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6-tests.jar =========== 0 Sun Jun 18 08:22:40 UTC 2023 org/apache/hadoop/conf/ 2151 Sun Jun 18 08:22:38 UTC 2023 org/apache/hadoop/conf/TestConfigurationDeprecation$1.class 522 Sun Jun 18 08:22:38 UTC 2023 org/apache/hadoop/conf/TestGetInstances$SampleClass.class 2291 Sun Jun 18 08:22:38 UTC 2023 org/apache/hadoop/conf/TestConfigurationDeprecation$2.class 333 Sun Jun 18 08:22:38 UTC 2023 org/apache/hadoop/conf/TestGetInstances$ChildInterface.class 2203 Sun Jun 18 08:22:38 UTC 2023 org/apache/hadoop/conf/TestGetInstances.class 2358 Sun Jun 18 08:22:36 UTC 2023 org/apache/hadoop/conf/TestConfigurationSubclass.class 3335 Sun Jun 18 08:22:36 UTC 2023 org/apache/hadoop/conf/TestDeprecatedKeys.class 71538 Sun Jun 18 08:22:36 UTC 2023 org/apache/hadoop/conf/TestConfiguration.class ... /tmp/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6-tests.jar =========== 0 Sun Jun 18 08:42:34 UTC 2023 org/apache/hadoop/conf/ 4469 Sun Jun 18 08:42:34 UTC 2023 org/apache/hadoop/conf/TestNoDefaultsJobConf.class ... {noformat} Now, these classes requires junit. One option is to skip these test jars but that requires changes on the hadoop side (since pig is calling hadoop commandline to start up pig.) Second option is to give it up and add the required junit jars to lib dir. Third option is to skip this test and let users add junit jars if they really need to call "from org.apache.hadoop.conf import *". but it's pretty tough to understand what's happening when users hit this. > Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3) > ---------------------------------------------------------------- > > Key: PIG-5459 > URL: https://issues.apache.org/jira/browse/PIG-5459 > Project: Pig > Issue Type: Bug > Reporter: Koji Noguchi > Assignee: Koji Noguchi > Priority: Minor > > {noformat} > turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_ > from org.apache.hadoop.conf import * > java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException; > at java.lang.Class.getDeclaredFields0(Native Method) > at java.lang.Class.privateGetDeclaredFields(Class.java:2583) > at java.lang.Class.privateGetPublicFields(Class.java:2614) > at java.lang.Class.getFields(Class.java:1557) > at org.python.core.PyJavaType.init(PyJavaType.java:419) > at org.python.core.PyType.createType(PyType.java:1523) > at org.python.core.PyType.addFromClass(PyType.java:1462) > at org.python.core.PyType.fromClass(PyType.java:1551) > at > org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77) > at > org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44) > at > org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131) > at org.python.core.Py.java2py(Py.java:2017) > at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86) > at > org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113) > at > org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148) > at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120) > at org.python.core.imp.importAll(imp.java:1189) > at org.python.core.imp.importAll(imp.java:1177) > at > org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8) > at > org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig) > at org.python.core.PyTableCode.call(PyTableCode.java:171) > at org.python.core.PyCode.call(PyCode.java:18) > at org.python.core.Py.runCode(Py.java:1614) > at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296) > at > org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217) > at > org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440) > at > org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424) > at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310) > at org.apache.pig.Main.runEmbeddedScript(Main.java:1096) > at org.apache.pig.Main.run(Main.java:584) > at org.apache.pig.Main.main(Main.java:175) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:328) > at org.apache.hadoop.util.RunJar.main(RunJar.java:241) > Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 37 more > java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: > Lorg/junit/rules/ExpectedException; > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)