[jira] [Commented] (SPARK-16228) "Percentile" needs explicit cast to double

2016-11-17 Thread Fabian Boehnlein (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673869#comment-15673869
 ] 

Fabian Boehnlein commented on SPARK-16228:
--

Shouldn't this also have resolved the below error we see when porting from 1.6 
to 2.0.1?

See reproducible snippet below.

{code}/usr/local/tl/spark-2.0.1-bin-hadoop2.7/bin/spark-shell{code}

{code}
scala> case class Record(key: Long, value: String)
defined class Record

scala> val recordsDF = spark.createDataFrame((1 to 100).map(i => 
Record(i.toLong, s"val_$i")))
recordsDF: org.apache.spark.sql.DataFrame = [key: int, value: string]

scala> recordsDF.createOrReplaceTempView("records")
scala> sql("SELECT percentile(key, Array(0.95, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 
0.3, 0.2, 0.1)) AS test FROM records")
org.apache.spark.sql.AnalysisException: No handler for Hive UDF 
'org.apache.hadoop.hive.ql.udf.UDAFPercentile': 
org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method 
for class org.apache.had
oop.hive.ql.udf.UDAFPercentile with (bigint, array). Possible 
choices: _FUNC_(bigint, array)  _FUNC_(bigint, double)  ; line 1 pos 7
  at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getMethodInternal(FunctionRegistry.java:1164)
  at 
org.apache.hadoop.hive.ql.exec.DefaultUDAFEvaluatorResolver.getEvaluatorClass(DefaultUDAFEvaluatorResolver.java:83)
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge.getEvaluator(GenericUDAFBridge.java:56)
  at 
org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector$lzycompute(hiveUDFs.scala:286)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector(hiveUDFs.scala:284)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector$lzycompute(hiveUDFs.scala:294)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector(hiveUDFs.scala:294)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.dataType$lzycompute(hiveUDFs.scala:342)
  at org.apache.spark.sql.hive.HiveUDAFFunction.dataType(hiveUDFs.scala:342)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:142)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:122)
  at 
org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:87)
  at 
org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:848)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:186)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$4.apply(HiveSessionCatalog.scala:186)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$4.apply(HiveSessionCatalog.scala:186)
  at scala.util.Try$.apply(Try.scala:192)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:186)
  at 
org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:172)
  at 
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:886)
  at 
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:886)
  at 
org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)

{code}

> "Percentile" needs explicit cast to double
> --
>
> Key: SPARK-16228
> URL: https://issues.apache.org/jira/browse/SPARK-16228
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Egor Pahomov
>Assignee: Dongjoon Hyun
> Fix For: 2.0.0
>
>
> {quote}
>  select percentile(cast(id as bigint), cast(0.5 as double)) from temp.bla
> {quote}
> Works.
> {quote}
>  select percentile(cast(id as bigint), 0.5 ) from temp.bla
> {quote}
> Throws
> {quote}
> Error in query: No handler for Hive UDF 
> 'org.apache.hadoop.hive.ql.udf.UDAFPercentile': 
> org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method 
> for class org.apache.hadoop.hive.ql.udf.UDAFPercentile with (bigint, 
> decimal(38,18)). Possible choices: _FUNC_(bigint, array)  
> _FUNC_(bigint, double)  ; line 1 pos 7
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-16228) "Percentile" needs explicit cast to double

2016-06-27 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351727#comment-15351727
 ] 

Apache Spark commented on SPARK-16228:
--

User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/13930

> "Percentile" needs explicit cast to double
> --
>
> Key: SPARK-16228
> URL: https://issues.apache.org/jira/browse/SPARK-16228
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Egor Pahomov
>
> {quote}
>  select percentile(cast(id as bigint), cast(0.5 as double)) from temp.bla
> {quote}
> Works.
> {quote}
>  select percentile(cast(id as bigint), 0.5 ) from temp.bla
> {quote}
> Throws
> {quote}
> Error in query: No handler for Hive UDF 
> 'org.apache.hadoop.hive.ql.udf.UDAFPercentile': 
> org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method 
> for class org.apache.hadoop.hive.ql.udf.UDAFPercentile with (bigint, 
> decimal(38,18)). Possible choices: _FUNC_(bigint, array)  
> _FUNC_(bigint, double)  ; line 1 pos 7
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-16228) "Percentile" needs explicit cast to double

2016-06-27 Thread Dongjoon Hyun (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351708#comment-15351708
 ] 

Dongjoon Hyun commented on SPARK-16228:
---

Hi, [~epahomov] and [~srowen].
The root cause is that Spark 2.0 uses `Decimal` as a default type for literal 
'0.5'.
This happens for `percentile_approx`, too. I guess it will happen for all 
double-type-only external functions.
I'll make a PR for this soon.

> "Percentile" needs explicit cast to double
> --
>
> Key: SPARK-16228
> URL: https://issues.apache.org/jira/browse/SPARK-16228
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Egor Pahomov
>
> {quote}
>  select percentile(cast(id as bigint), cast(0.5 as double)) from temp.bla
> {quote}
> Works.
> {quote}
>  select percentile(cast(id as bigint), 0.5 ) from temp.bla
> {quote}
> Throws
> {quote}
> Error in query: No handler for Hive UDF 
> 'org.apache.hadoop.hive.ql.udf.UDAFPercentile': 
> org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method 
> for class org.apache.hadoop.hive.ql.udf.UDAFPercentile with (bigint, 
> decimal(38,18)). Possible choices: _FUNC_(bigint, array)  
> _FUNC_(bigint, double)  ; line 1 pos 7
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-16228) "Percentile" needs explicit cast to double

2016-06-27 Thread Egor Pahomov (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351697#comment-15351697
 ] 

Egor Pahomov commented on SPARK-16228:
--

[~srowen] "blocker" is questionable, I agree. I just believe, that everything 
which prevent you from moving from 1.6.1 to 2.0 without major code changes is 
"blocker". It's just if I move and such bug would be there a lot of my analysts 
notebook would be invalid. 

> "Percentile" needs explicit cast to double
> --
>
> Key: SPARK-16228
> URL: https://issues.apache.org/jira/browse/SPARK-16228
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Egor Pahomov
>
> {quote}
>  select percentile(cast(id as bigint), cast(0.5 as double)) from temp.bla
> {quote}
> Works.
> {quote}
>  select percentile(cast(id as bigint), 0.5 ) from temp.bla
> {quote}
> Throws
> {quote}
> Error in query: No handler for Hive UDF 
> 'org.apache.hadoop.hive.ql.udf.UDAFPercentile': 
> org.apache.hadoop.hive.ql.exec.NoMatchingMethodException: No matching method 
> for class org.apache.hadoop.hive.ql.udf.UDAFPercentile with (bigint, 
> decimal(38,18)). Possible choices: _FUNC_(bigint, array)  
> _FUNC_(bigint, double)  ; line 1 pos 7
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org