I'm getting errors while running contributed UDFs in Hive.
The environment is Hadoop 0.20.2 with Hive from the trunk on 9/30/10
I've tried a few UDFs, and get the same results:
1) The reduce oscillates between 0 and 30% before failing
2) I get the errors below
The "create temporary function" succeeds for a properly named class, so it
seems the jar is being found properly.
Here are a few ways I've invoked the UDFs
drop temporary function row_sequence;
create temporary function row_sequence as
'org.apache.hadoop.hive.contrib.udf.UDFRowSequence';
set mapred.reduce.tasks=1;
select row_sequence(), t.* from userid_new t;
set mapred.reduce.tasks=1;
select user_id, row_sequence() as r
from (
select user_id
from userid_new_lim10
order by user_id
) o
order by user_id;
drop temporary function udftest;
create temporary function udftest as
'org.apache.hadoop.hive.contrib.udf.example.UDFExampleAdd';
select t.n, udftest(t.n) from userid_new_lim10 t group by t.n;
select t.n, udftest(t.n) from userid_new_lim10 t order by t.n;
Here's an example of the errors I see:
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 10 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 13 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:157)
... 18 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:80)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:821)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:833)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:427)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:383)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:113)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:415)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:96)
... 18 more
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
... 31 more
