[
https://issues.apache.org/jira/browse/PIG-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-2979:
-------------------------------
Description:
To reproduce, please do the following:
1) ensure that no hadoop is installed and therefore no hadoop classes are
present in classpath.
2) ant clean jar -Dhadoopversion=23
3) ./bin/pig -x local
4) fail with the following error: (you may not see this if slf4j is available
in classpath.)
{code}
cheolsoo@localhost:~/workspace/pig-trunk $cat
/home/cheolsoo/workspace/pig-trunk/pig_1350687456711.log
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/slf4j/LoggerFactory
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
...
{code}
5) cp ./build/ivy/lib/Pig/slf4j-api-1.6.1.jar lib
6) ./bin/pig -x local
7) fail with the following error:
{code}
cheolsoo@localhost:~/workspace/pig-trunk $cat
/home/cheolsoo/workspace/pig-trunk/pig_1350687052995.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:204)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:117)
at org.apache.pig.impl.PigContext.connect(PigContext.java:229)
at org.apache.pig.PigServer.<init>(PigServer.java:213)
at org.apache.pig.PigServer.<init>(PigServer.java:198)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
at org.apache.pig.Main.run(Main.java:535)
at org.apache.pig.Main.main(Main.java:154)
Caused by: java.io.IOException: No FileSystem for scheme: file
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
================================================================================
{code}
8) cp ./build/ivy/lib/Pig/hadoop-common-2.0.0-alpha.jar lib
9) ./bin/pig -x local
10) works fine!
In fact, this is also an issue with running e2e test in local mode:
{code}
ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
-Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
-Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
{code}
The ant test-e2e-local fails with the following error:
{code}
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at
org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
{code}
was:
To reproduce, please run on machine where no Hadoop is installed:
{code}
ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
-Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
-Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
{code}
The ant test-e2e-local fails with the following error:
{code}
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at
org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
{code}
In fact, this is also an issue with running Pig in local mode with the fat jar
where no hadoop dependencies are available in classpath. For example, the
following command also fails with the same error:
{code}
ant clean jar -Dhadoopversion=23
./bin/pig -x local
{code}
Summary: Pig.jar doesn't work with hadoop-2.0.x (was: ant
test-e2e-local fails due to missing run-time dependencies in classpath with
hadoop-2.0.x)
> Pig.jar doesn't work with hadoop-2.0.x
> --------------------------------------
>
> Key: PIG-2979
> URL: https://issues.apache.org/jira/browse/PIG-2979
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2979.patch
>
>
> To reproduce, please do the following:
> 1) ensure that no hadoop is installed and therefore no hadoop classes are
> present in classpath.
> 2) ant clean jar -Dhadoopversion=23
> 3) ./bin/pig -x local
> 4) fail with the following error: (you may not see this if slf4j is available
> in classpath.)
> {code}
> cheolsoo@localhost:~/workspace/pig-trunk $cat
> /home/cheolsoo/workspace/pig-trunk/pig_1350687456711.log
> Error before Pig is launched
> ----------------------------
> ERROR 2998: Unhandled internal error. org/slf4j/LoggerFactory
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> ...
> {code}
> 5) cp ./build/ivy/lib/Pig/slf4j-api-1.6.1.jar lib
> 6) ./bin/pig -x local
> 7) fail with the following error:
> {code}
> cheolsoo@localhost:~/workspace/pig-trunk $cat
> /home/cheolsoo/workspace/pig-trunk/pig_1350687052995.log
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> java.lang.RuntimeException: Failed to create DataStorage
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:204)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:117)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:229)
> at org.apache.pig.PigServer.<init>(PigServer.java:213)
> at org.apache.pig.PigServer.<init>(PigServer.java:198)
> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
> at org.apache.pig.Main.run(Main.java:535)
> at org.apache.pig.Main.main(Main.java:154)
> Caused by: java.io.IOException: No FileSystem for scheme: file
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
> ... 9 more
> ================================================================================
> {code}
> 8) cp ./build/ivy/lib/Pig/hadoop-common-2.0.0-alpha.jar lib
> 9) ./bin/pig -x local
> 10) works fine!
> In fact, this is also an issue with running e2e test in local mode:
> {code}
> ant clean
> ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
> -Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
> ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir
> -Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
> {code}
> The ant test-e2e-local fails with the following error:
> {code}
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> at
> org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
