dmitry d created PIG-3450:
-----------------------------
Summary: error when pigrunner.run embedded python stored in hdfs
Key: PIG-3450
URL: https://issues.apache.org/jira/browse/PIG-3450
Project: Pig
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: dmitry d
I try to run embedded python from java program using PigRunner. Script file
store in hdfs. And I get the following error:
{quote}
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. null
java.lang.NullPointerException
at
org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
at
org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
at org.apache.pig.PigServer.registerJar(PigServer.java:458)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
at org.apache.pig.Main.run(Main.java:427)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
================================================================================
{quote}
No error when I run python script from local path.
My Java code:
{quote}
PigStats stats = PigRunner.run(new String[] {"-f",
"hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
{quote}
My Python code:
{quote}
#!/usr/bin/python
import sys
print('hello from python');
{quote}
I saw source code where error is happened - FileLocalizer.java:
{quote}
FileSystem localFs = FileSystem.getLocal(conf);
FileSystem srcFs;
if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
&& uri.getScheme() == null )||
uri.getScheme().equals("local") ) {
srcFs = localFs;
} else {
srcFs = path.getFileSystem(conf);
}
{quote}
uri is local path to jython-standalone.jar. And uri.getScheme() return null.
And uri.getScheme().equals("local") is throw exception
Maybe I must redefine some property to get jython-standalone.jar from hdfs?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
