Hi, You might well have found the error yourself by now, but if not, the problem is that you missed the "static" keyword off the declaration of TopNPercentEvaluator. Line 18 of your code should read
public static class TopNPercentEvaluator implements UDAFEvaluator { then this error goes away. A new error appears, however, because there's a bug in your terminate() function that reads off the end of the array. Robin From: Abhishek Bhattacharya <abhat...@fiu.edu<mailto:abhat...@fiu.edu>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Date: Tuesday, February 12, 2013 10:29 AM To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Subject: Fwd: Help to solve UDAF errors! Hi, I looked through the syslogs and found the following exceptions. Can anyone help me to figure out the point of error? java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 22 more Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160) at org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536) at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98) ... 22 more Caused by: java.lang.NoSuchMethodException: com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>() at java.lang.Class.getConstructor0(Class.java:2706) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109) ... 37 more 2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task ________________________________ Thanks, Abhishek ---------- Forwarded message ---------- From: Abhishek Bhattacharya <abhat...@fiu.edu<mailto:abhat...@fiu.edu>> Date: Tue, Feb 12, 2013 at 12:48 PM Subject: Re: Help to solve UDAF errors! To: user@hive.apache.org<mailto:user@hive.apache.org> Hi Mark, Thanks for the response! The UDAFPercentile.java have two terminate() methods since it is handling two different input types by the two inner classes: PercentileLongEvaluator and PercentileLongArrayEvaluator. I am handling only a single input type of double from one table column to the iterate() method and wish to return an ArrayList<DoubleWritable> from the terminate() method. What is wrong in my class? Moreover, is there any way for UDF/UDAF/UDTF which can process all the rows of the table and output only a subset of the total rows based on some aggregation function of one column attribute i.e., similar to my case of computing the top-n-percent of a column attribute and output the entire set of filtered rows with all other columns from the table? Thanks, Abhishek On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover <grover.markgro...@gmail.com<mailto:grover.markgro...@gmail.com>> wrote: Hi Abhishek, The code looks incomplete. See the comment at https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22 Those are all the methods your UDAF class needs to implement but you seem to be missing them. Mark On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya <abhat...@fiu.edu<mailto:abhat...@fiu.edu>> wrote: Thanks for the response. The link to the code is: https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java Please let me know to fix it! Thanks, Abhishek On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <grover.markgro...@gmail.com<mailto:grover.markgro...@gmail.com>> wrote: Abhishek, The code doesn't seem to be complete. Look at https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java for reference. It has two terminate()'s - one for UDAF and one for the Evaluator. Do you mind posting your complete code on github somewhere so it's easier to analyze? Mark On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <abhat...@fiu.edu<mailto:abhat...@fiu.edu>> wrote: Hi, I have implemented a simple UDAF for top-n-percent as follows: import java.util.ArrayList; import java.util.Collections; import org.apache.hadoop.hive.ql.exec.UDAF; import org.apache.hadoop.hive.ql.exec.UDAFEvaluator; public class UDAFTopNPercent extends UDAF{ public static class Result { ArrayList<Double> list; double min; } public class TopNPercentEvaluator implements UDAFEvaluator { private Result res; private int rowIndex; private int percent; public TopNPercentEvaluator() { super(); res = new Result(); init(); rowIndex = 0; } @Override public void init() { res.list = new ArrayList<Double>(); res.min = Double.MAX_VALUE; } public boolean iterate(Double rowVal, int pct) { ArrayList<Double> resList = res.list; rowIndex++; resList.add(rowVal); percent = pct; return true; } public ArrayList<Double> terminatePartial() { ArrayList<Double> resList = res.list; Collections.sort(resList); return resList; } public boolean merge(ArrayList<Double> otherList) { ArrayList<Double> resList = res.list; resList.addAll(otherList); return true; } public ArrayList<Double> terminate() { ArrayList<Double> resList = res.list; double num_rows = (double)percent/100.0*rowIndex; Collections.sort(resList); int lastIdx = resList.size()- (int) num_rows; if(lastIdx <= 0) { return resList; } for(int i=0; i<lastIdx; i++) { resList.remove(i); } return resList; } } /** * @param args */ public static void main(String[] args) { // TODO Auto-generated method stub } } But throws some error such as first few lines are: FAILED: Hive Internal Error: java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector) java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300) at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129) Please help me to debug this! Is it throwing from returning ArrayList<Double> in terminate()? How should I return a List from UDAF? Thanks, Abhishek -- Thanks and Regards, Abhishek Bhattacharya PhD Computer Science School of Computing and Information Sciences Florida International University -- Thanks and Regards, Abhishek Bhattacharya PhD Computer Science School of Computing and Information Sciences Florida International University -- Thanks and Regards, Abhishek Bhattacharya PhD Computer Science School of Computing and Information Sciences Florida International University