[
https://issues.apache.org/jira/browse/SPARK-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274401#comment-15274401
]
Shixiong Zhu commented on SPARK-14642:
--------------------------------------
The implicit TypeTag[String] is inserted into the codes by the compiler via
macro expansion. However, the generated codes contain
"scala.collection.immutable.List.apply()" when the type paramter is a `type`
(such as, String, List in Predef.scala. For non types, such as
"java.lang.String", it will work). So if the user imports some packages ending
with ".scala", the root scala is hidden, then
`scala.collection.immutable.List.apply()` won't be compiled. Hence, TypeTag
macro expansion fails and the implicit TypeTag won't be created. Since macro
expansion is optional, the complier will continue its work and finally report
no implicit TypeTag for String.
For Spark, I think we can avoid to use the `scala` package name.
> import org.apache.spark.sql.expressions._ breaks udf under functions
> --------------------------------------------------------------------
>
> Key: SPARK-14642
> URL: https://issues.apache.org/jira/browse/SPARK-14642
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Yin Huai
> Priority: Blocker
>
> The following code works
> {code}
> scala> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.functions._
> scala> udf((v: String) => v.stripSuffix("-abc"))
> res0: org.apache.spark.sql.expressions.UserDefinedFunction =
> UserDefinedFunction(<function1>,StringType,Some(List(StringType)))
> {code}
> But, the following does not
> {code}
> scala> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.functions._
> scala> import org.apache.spark.sql.expressions._
> import org.apache.spark.sql.expressions._
> scala> udf((v: String) => v.stripSuffix("-abc"))
> <console>:30: error: No TypeTag available for String
> udf((v: String) => v.stripSuffix("-abc"))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
