Re: Hive UDAF percentile_approx says This UDAF does not support the deprecated getEvaluator() method.
Looking at the source code for AbstractGenericUDAFResolver, the following (non-deprecated) method should be called: public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info) It is called by hiveUdfs.scala (master branch): val parameterInfo = new SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false) resolver.getEvaluator(parameterInfo) FYI On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Hi, The following SQL query select percentile_approx(variables.var1, 0.95) p95 from model will throw ERROR SparkSqlInterpreter: Error org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not support the deprecated getEvaluator() method. at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195) at org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203) at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) I'm using latest branch-1.2 I found in PR that percentile and percentile_approx are supported. A bug? Thanks, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
Hive UDAF percentile_approx says This UDAF does not support the deprecated getEvaluator() method.
Hi, The following SQL query select percentile_approx(variables.var1, 0.95) p95 from model will throw ERROR SparkSqlInterpreter: Error org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not support the deprecated getEvaluator() method. at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195) at org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203) at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) I'm using latest branch-1.2 I found in PR that percentile and percentile_approx are supported. A bug? Thanks, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
Re: Hive UDAF percentile_approx says This UDAF does not support the deprecated getEvaluator() method.
Yeah, it's a bug. It has been fixed by https://issues.apache.org/jira/browse/SPARK-3891 in master. On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu yuzhih...@gmail.com wrote: Looking at the source code for AbstractGenericUDAFResolver, the following (non-deprecated) method should be called: public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info) It is called by hiveUdfs.scala (master branch): val parameterInfo = new SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false) resolver.getEvaluator(parameterInfo) FYI On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Hi, The following SQL query select percentile_approx(variables.var1, 0.95) p95 from model will throw ERROR SparkSqlInterpreter: Error org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not support the deprecated getEvaluator() method. at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195) at org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203) at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) I'm using latest branch-1.2 I found in PR that percentile and percentile_approx are supported. A bug? Thanks, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
Re: Hive UDAF percentile_approx says This UDAF does not support the deprecated getEvaluator() method.
Ah, thx Ted and Yin! I'll build a new version. :) Jianshi On Wed, Jan 14, 2015 at 7:24 AM, Yin Huai yh...@databricks.com wrote: Yeah, it's a bug. It has been fixed by https://issues.apache.org/jira/browse/SPARK-3891 in master. On Tue, Jan 13, 2015 at 2:41 PM, Ted Yu yuzhih...@gmail.com wrote: Looking at the source code for AbstractGenericUDAFResolver, the following (non-deprecated) method should be called: public GenericUDAFEvaluator getEvaluator(GenericUDAFParameterInfo info) It is called by hiveUdfs.scala (master branch): val parameterInfo = new SimpleGenericUDAFParameterInfo(inspectors.toArray, false, false) resolver.getEvaluator(parameterInfo) FYI On Tue, Jan 13, 2015 at 1:51 PM, Jianshi Huang jianshi.hu...@gmail.com wrote: Hi, The following SQL query select percentile_approx(variables.var1, 0.95) p95 from model will throw ERROR SparkSqlInterpreter: Error org.apache.hadoop.hive.ql.parse.SemanticException: This UDAF does not support the deprecated getEvaluator() method. at org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:53) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector$lzycompute(hiveUdfs.scala:196) at org.apache.spark.sql.hive.HiveGenericUdaf.objectInspector(hiveUdfs.scala:195) at org.apache.spark.sql.hive.HiveGenericUdaf.dataType(hiveUdfs.scala:203) at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:105) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) at org.apache.spark.sql.catalyst.plans.logical.Aggregate$$anonfun$output$6.apply(basicOperators.scala:143) I'm using latest branch-1.2 I found in PR that percentile and percentile_approx are supported. A bug? Thanks, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/ -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github Blog: http://huangjs.github.com/
