Hi,
I looked through the syslogs and found the following exceptions.
Can anyone help me to figure out the point of error?
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 22 more
Caused by: java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
... 37 more
2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
------------------------------
Thanks,
Abhishek
---------- Forwarded message ----------
From: Abhishek Bhattacharya <[email protected]>
Date: Tue, Feb 12, 2013 at 12:48 PM
Subject: Re: Help to solve UDAF errors!
To: [email protected]
Hi Mark,
Thanks for the response!
The UDAFPercentile.java have two terminate() methods since it is handling
two different input types by the two inner classes: PercentileLongEvaluator
and PercentileLongArrayEvaluator.
I am handling only a single input type of double from one table column to
the iterate() method and wish to return an ArrayList<DoubleWritable> from
the terminate() method.
What is wrong in my class?
Moreover, is there any way for UDF/UDAF/UDTF which can process all the rows
of the table and output only a subset of the total rows based on some
aggregation function of one column attribute i.e., similar to my case of
computing the top-n-percent of a column attribute and output the entire set
of filtered rows with all other columns from the table?
Thanks,
Abhishek
On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover
<[email protected]>wrote:
> Hi Abhishek,
> The code looks incomplete.
>
> See the comment at
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
> Those are all the methods your UDAF class needs to implement but you seem
> to be missing them.
>
> Mark
>
> On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya
> <[email protected]>wrote:
>
>> Thanks for the response.
>> The link to the code is:
>> https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
>> Please let me know to fix it!
>>
>> Thanks,
>> Abhishek
>>
>>
>>
>> On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover
>> <[email protected]>wrote:
>>
>>> Abhishek,
>>> The code doesn't seem to be complete.
>>>
>>> Look at
>>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor
>>> reference. It has two terminate()'s - one for UDAF and one for the
>>> Evaluator.
>>>
>>> Do you mind posting your complete code on github somewhere so it's
>>> easier to analyze?
>>>
>>> Mark
>>>
>>> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya
>>> <[email protected]>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have implemented a simple UDAF for top-n-percent as follows:
>>>> import java.util.ArrayList;
>>>> import java.util.Collections;
>>>>
>>>> import org.apache.hadoop.hive.ql.exec.UDAF;
>>>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>>>
>>>> public class UDAFTopNPercent extends UDAF{
>>>>
>>>> public static class Result {
>>>> ArrayList<Double> list;
>>>> double min;
>>>> }
>>>>
>>>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>>>
>>>> private Result res;
>>>> private int rowIndex;
>>>> private int percent;
>>>>
>>>> public TopNPercentEvaluator() {
>>>> super();
>>>> res = new Result();
>>>> init();
>>>> rowIndex = 0;
>>>> }
>>>> @Override
>>>> public void init() {
>>>> res.list = new ArrayList<Double>();
>>>> res.min = Double.MAX_VALUE;
>>>> }
>>>>
>>>> public boolean iterate(Double rowVal, int pct) {
>>>> ArrayList<Double> resList = res.list;
>>>> rowIndex++;
>>>> resList.add(rowVal);
>>>> percent = pct;
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminatePartial() {
>>>> ArrayList<Double> resList = res.list;
>>>> Collections.sort(resList);
>>>> return resList;
>>>> }
>>>>
>>>> public boolean merge(ArrayList<Double> otherList) {
>>>> ArrayList<Double> resList = res.list;
>>>> resList.addAll(otherList);
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminate() {
>>>> ArrayList<Double> resList = res.list;
>>>> double num_rows = (double)percent/100.0*rowIndex;
>>>> Collections.sort(resList);
>>>> int lastIdx = resList.size()- (int) num_rows;
>>>> if(lastIdx <= 0) {
>>>> return resList;
>>>> }
>>>> for(int i=0; i<lastIdx; i++) {
>>>> resList.remove(i);
>>>> }
>>>> return resList;
>>>> }
>>>> }
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> public static void main(String[] args) {
>>>> // TODO Auto-generated method stub
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> But throws some error such as first few lines are:
>>>> FAILED: Hive Internal Error:
>>>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>>> cannot be cast to
>>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
>>>> java.lang.ClassCastException:
>>>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>>> cannot be cast to
>>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
>>>> at
>>>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
>>>> at
>>>> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
>>>> at
>>>> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
>>>>
>>>> Please help me to debug this!
>>>> Is it throwing from returning ArrayList<Double> in terminate()?
>>>> How should I return a List from UDAF?
>>>>
>>>> Thanks,
>>>> Abhishek
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>>
>> Abhishek Bhattacharya
>> PhD Computer Science
>> School of Computing and Information Sciences
>> Florida International University
>>
>
>
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
