[ https://issues.apache.org/jira/browse/SPARK-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171627#comment-16171627 ]
Simeon H.K. Fitch edited comment on SPARK-12823 at 9/19/17 12:53 PM: --------------------------------------------------------------------- When you say "Row", if you meant `org.apache.spark.sql.Row`? then yes, this works: {code:java} val udf2 = udf((row: Row) => row.getString(1)) {code} But that's requiring me to keep track of the schema when Catalyst (through `Encoder`) has all the type information it needs to be able to reify that for me. There shouldn't be any technical reason why `udf((row: KV) ⇒ row.value)` shouldn't be allowed. And even if there is (i.e. I'm wrong), it should be a compile time error, not a runtime one. But the example defined its own `Row` class making this all more confusing. I tried `udf((row: NotTheSparkRow) => row.kv.value)` but that doesn't work either. was (Author: metasim): When you say "Row", if you meant `org.apache.spark.sql.Row`? then yes, this works: {code:java} val udf2 = udf((row: Row) => row.getString(1)) {code} But that's requiring me to keep track of the schema when Catalyst (through `Encoder`) has all the type information it needs to be able to reify that for me. There shouldn't be any technical reason why `udf((row: KV) ⇒ row.value)` shouldn't be allowed. And even if there is (i.e. I'm wrong), it should be a compile time error, not a runtime one. But the example defined its own `Row` class making this all more confusing. I tried `udf((row: Row) => row.kv.value)` but that doesn't work either. > Cannot create UDF with StructType input > --------------------------------------- > > Key: SPARK-12823 > URL: https://issues.apache.org/jira/browse/SPARK-12823 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.2 > Reporter: Frank Rosner > > h5. Problem > It is not possible to apply a UDF to a column that has a struct data type. > Two previous requests to the mailing list remained unanswered. > h5. How-To-Reproduce > {code} > val sql = new org.apache.spark.sql.SQLContext(sc) > import sql.implicits._ > case class KV(key: Long, value: String) > case class Row(kv: KV) > val df = sc.parallelize(List(Row(KV(1L, "a")), Row(KV(5L, "b")))).toDF > val udf1 = org.apache.spark.sql.functions.udf((kv: KV) => kv.value) > df.select(udf1(df("kv"))).show > // java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast > to $line78.$read$$iwC$$iwC$KV > val udf2 = org.apache.spark.sql.functions.udf((kv: (Long, String)) => kv._2) > df.select(udf2(df("kv"))).show > // org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(kv)' due to > data type mismatch: argument 1 requires struct<_1:bigint,_2:string> type, > however, 'kv' is of struct<key:bigint,value:string> type.; > {code} > h5. Mailing List Entries > - > https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCACUahd8M=ipCbFCYDyein_=vqyoantn-tpxe6sq395nh10g...@mail.gmail.com%3E > - https://www.mail-archive.com/user@spark.apache.org/msg43092.html > h5. Possible Workaround > If you create a {{UserDefinedFunction}} manually, not using the {{udf}} > helper functions, it works. See https://github.com/FRosner/struct-udf, which > exposes the {{UserDefinedFunction}} constructor (public from package > private). However, then you have to work with a {{Row}}, because it does not > automatically convert the row to a case class / tuple. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org