Hi all:
I try to run simple transform script in hive and my script is written in
python. But when I try to import other file in the script the task fails.
There may be some basic classes which are used by many transform scripts,
so I would like to know how to import other file in my transform script.
Below is my test data:
Hive CLI Query:
add file /home/hive_test/empty.py;
add file /home/hive_test/normal.py;
insert overwrite table test
select
transform(d.*)
using './normal.py'
as (`name`)
from
(
select
name
from test_1
) d;
normal.py:
#!/usr/bin/python
import sys
import empty
if __name__ == "__main__":
for l in sys.stdin:
print l.strip('\n')
empty.py:
#!/usr/bin/python
class Empty:
pass
error msg:
java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hit error
while closing ..
at
org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:452)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
... 8 more
Thanks.