Hi,
You might well have found the error yourself by now, but if not, the problem is
that you missed the "static" keyword off the declaration of
TopNPercentEvaluator. Line 18 of your code should read
public static class TopNPercentEvaluator implements UDAFEvaluator {
then this error goes away.
A new error appears, however, because there's a bug in your terminate()
function that reads off the end of the array.
Robin
From: Abhishek Bhattacharya <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Tuesday, February 12, 2013 10:29 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Fwd: Help to solve UDAF errors!
Hi,
I looked through the syslogs and found the following exceptions.
Can anyone help me to figure out the point of error?
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 22 more
Caused by: java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
... 37 more
2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
________________________________
Thanks,
Abhishek
---------- Forwarded message ----------
From: Abhishek Bhattacharya <[email protected]<mailto:[email protected]>>
Date: Tue, Feb 12, 2013 at 12:48 PM
Subject: Re: Help to solve UDAF errors!
To: [email protected]<mailto:[email protected]>
Hi Mark,
Thanks for the response!
The UDAFPercentile.java have two terminate() methods since it is handling two
different input types by the two inner classes: PercentileLongEvaluator and
PercentileLongArrayEvaluator.
I am handling only a single input type of double from one table column to the
iterate() method and wish to return an ArrayList<DoubleWritable> from the
terminate() method.
What is wrong in my class?
Moreover, is there any way for UDF/UDAF/UDTF which can process all the rows of
the table and output only a subset of the total rows based on some aggregation
function of one column attribute i.e., similar to my case of computing the
top-n-percent of a column attribute and output the entire set of filtered rows
with all other columns from the table?
Thanks,
Abhishek
On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover
<[email protected]<mailto:[email protected]>> wrote:
Hi Abhishek,
The code looks incomplete.
See the comment at
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
Those are all the methods your UDAF class needs to implement but you seem to be
missing them.
Mark
On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya
<[email protected]<mailto:[email protected]>> wrote:
Thanks for the response.
The link to the code is:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Please let me know to fix it!
Thanks,
Abhishek
On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover
<[email protected]<mailto:[email protected]>> wrote:
Abhishek,
The code doesn't seem to be complete.
Look at
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java
for reference. It has two terminate()'s - one for UDAF and one for the
Evaluator.
Do you mind posting your complete code on github somewhere so it's easier to
analyze?
Mark
On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I have implemented a simple UDAF for top-n-percent as follows:
import java.util.ArrayList;
import java.util.Collections;
import org.apache.hadoop.hive.ql.exec.UDAF;
import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
public class UDAFTopNPercent extends UDAF{
public static class Result {
ArrayList<Double> list;
double min;
}
public class TopNPercentEvaluator implements UDAFEvaluator {
private Result res;
private int rowIndex;
private int percent;
public TopNPercentEvaluator() {
super();
res = new Result();
init();
rowIndex = 0;
}
@Override
public void init() {
res.list = new ArrayList<Double>();
res.min = Double.MAX_VALUE;
}
public boolean iterate(Double rowVal, int pct) {
ArrayList<Double> resList = res.list;
rowIndex++;
resList.add(rowVal);
percent = pct;
return true;
}
public ArrayList<Double> terminatePartial() {
ArrayList<Double> resList = res.list;
Collections.sort(resList);
return resList;
}
public boolean merge(ArrayList<Double> otherList) {
ArrayList<Double> resList = res.list;
resList.addAll(otherList);
return true;
}
public ArrayList<Double> terminate() {
ArrayList<Double> resList = res.list;
double num_rows = (double)percent/100.0*rowIndex;
Collections.sort(resList);
int lastIdx = resList.size()- (int) num_rows;
if(lastIdx <= 0) {
return resList;
}
for(int i=0; i<lastIdx; i++) {
resList.remove(i);
}
return resList;
}
}
/**
* @param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
}
}
But throws some error such as first few lines are:
FAILED: Hive Internal Error:
java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
Please help me to debug this!
Is it throwing from returning ArrayList<Double> in terminate()?
How should I return a List from UDAF?
Thanks,
Abhishek
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
--
Thanks and Regards,
Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University
