[
https://issues.apache.org/jira/browse/SPARK-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221656#comment-15221656
]
Joan Goyeau edited comment on SPARK-13929 at 4/1/16 1:02 PM:
-------------------------------------------------------------
I investigated a way to implement Scala reflection for the part you show which
involves Annotations.
This is a bit tricky because reflection of annotations in Scala is for now only
at the AST level.
What we can do is mix both Scala and Java reflection which does something like
that:
{code}
case t if getClassFromType(t).isAnnotationPresent(classOf[SQLUserDefinedType])
=>
val udt =
getClassFromType(t).getAnnotation(classOf[SQLUserDefinedType]).udt().newInstance()
{code}
But then `getClassFromType` is using `runtimeClass` and therefore we need to
move `schemaFor` from the `ScalaReflection` trait (compile time and runtime
universe) to his companion object (runtime universe) as it works only in the
runtime universe.
Let me know your thoughts.
was (Author: j...@goyeau.com):
I investigated a way to implement Scala reflection for the part you show which
involves Annotations.
This is a bit tricky because reflection of annotations in Scala is for now only
to the AST level.
What we can do is mix both Scala and Java reflection which does something like
that:
{code}
case t if getClassFromType(t).isAnnotationPresent(classOf[SQLUserDefinedType])
=>
val udt =
getClassFromType(t).getAnnotation(classOf[SQLUserDefinedType]).udt().newInstance()
{code}
But then `getClassFromType` is using `runtimeClass` and therefore we need to
move `schemaFor` from the `ScalaReflection` trait (compile time and runtime
universe) to his companion object (runtime universe) as it works only in the
runtime universe.
Let me know your thoughts.
> Use Scala reflection for UDFs
> -----------------------------
>
> Key: SPARK-13929
> URL: https://issues.apache.org/jira/browse/SPARK-13929
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Jakob Odersky
> Priority: Minor
>
> {{ScalaReflection}} uses native Java reflection for User Defined Types which
> would fail if such types are not plain Scala classes that map 1:1 to Java.
> Consider the following extract (from here
> https://github.com/apache/spark/blob/92024797a4fad594b5314f3f3be5c6be2434de8a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L376
> ):
> {code}
> case t if Utils.classIsLoadable(className) &&
> Utils.classForName(className).isAnnotationPresent(classOf[SQLUserDefinedType])
> =>
> val udt =
> Utils.classForName(className).getAnnotation(classOf[SQLUserDefinedType]).udt().newInstance()
> //...
> {code}
> If {{t}}'s runtime class is actually synthetic (something that doesn't exist
> in Java and hence uses a dollar sign internally), such as nested classes or
> package objects, the above code will fail.
> Currently there are no known use-cases of synthetic user-defined types (hence
> the minor priority), however it would be best practice to remove plain Java
> reflection and rely on Scala reflection instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
