While reading some book on Java 8, I saw a reference to the following w.r.t. declaration-site variance :
https://bugs.openjdk.java.net/browse/JDK-8043488 The above reportedly targets Java 9. FYI On Thu, Jan 14, 2016 at 12:33 PM, Michael Armbrust <mich...@databricks.com> wrote: > I don't believe that Java 8 got rid of erasure. In fact I think its > actually worse when you use Java 8 lambdas. > > On Thu, Jan 14, 2016 at 10:54 AM, Raghu Ganti <raghuki...@gmail.com> > wrote: > >> Would this go away if the Spark source was compiled against Java 1.8 >> (since the problem of type erasure is solved through proper generics >> implementation in Java 1.8). >> >> On Thu, Jan 14, 2016 at 1:42 PM, Michael Armbrust <mich...@databricks.com >> > wrote: >> >>> We automatically convert types for UDFs defined in Scala, but we can't >>> do it in Java because the types are erased by the compiler. If you want to >>> use double you should cast before calling the UDF. >>> >>> On Wed, Jan 13, 2016 at 8:10 PM, Raghu Ganti <raghuki...@gmail.com> >>> wrote: >>> >>>> So, when I try BigDecimal, it works. But, should it not parse based on >>>> what the UDF defines? Am I missing something here? >>>> >>>> On Wed, Jan 13, 2016 at 4:57 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Please take a look >>>>> at >>>>> sql/hive/src/test/java/org/apache/spark/sql/hive/aggregate/MyDoubleSum.java >>>>> which shows a UserDefinedAggregateFunction that works on DoubleType >>>>> column. >>>>> >>>>> sql/hive/src/test/java/org/apache/spark/sql/hive/JavaDataFrameSuite.java >>>>> shows how it is registered. >>>>> >>>>> Cheers >>>>> >>>>> On Wed, Jan 13, 2016 at 11:58 AM, raghukiran <raghuki...@gmail.com> >>>>> wrote: >>>>> >>>>>> While registering and using SQL UDFs, I am running into the following >>>>>> problem: >>>>>> >>>>>> UDF registered: >>>>>> >>>>>> ctx.udf().register("Test", new UDF1<Double, String>() { >>>>>> /** >>>>>> * >>>>>> */ >>>>>> private static final long serialVersionUID = >>>>>> -8231917155671435931L; >>>>>> >>>>>> public String call(Double x) throws Exception >>>>>> { >>>>>> return "testing"; >>>>>> } >>>>>> }, DataTypes.StringType); >>>>>> >>>>>> Usage: >>>>>> query = "SELECT Test(82.4)"; >>>>>> result = sqlCtx.sql(query).first(); >>>>>> System.out.println(result.toString()); >>>>>> >>>>>> Problem: Class Cast exception thrown >>>>>> Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot >>>>>> be cast >>>>>> to java.lang.Double >>>>>> >>>>>> This problem occurs with Spark v1.5.2 and 1.6.0. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/SQL-UDF-problem-with-re-to-types-tp25968.html >>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>>> >>>> >>> >> >
