Ohad Raviv created SPARK-18861: ---------------------------------- Summary: Spark-SQL unconsistent behavior with "struct" expressions Key: SPARK-18861 URL: https://issues.apache.org/jira/browse/SPARK-18861 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.0 Reporter: Ohad Raviv
We are getting strangly inconsistent behavior with expressions involving "struct". Let's start with this simple table: {quote} Seq((1, 2, 3), (2, 3, 4)).toDF("a", "b", "c").createOrReplaceTempView("t1") sql("desc t1").show() {quote} Then we get this DF: {quote} |col_name|data_type|comment| | a| int| | | b| int| | | c| int| | {quote} Now, although we can clearly see that all the fields are of type int, we we run: {quote} sql("SELECT case when a>b then struct(a,b) else struct(c,c) end from t1") {quote} we get this error: {quote} org.apache.spark.sql.AnalysisException: cannot resolve 'CASE WHEN (t1.`a` > t1.`b`) THEN struct(t1.`a`, t1.`b`) ELSE struct(t1.`c`, t1.`c`) END' due to data type mismatch: THEN and ELSE expressions should all be same type or coercible to a common type; line 1 pos 7 {quote} if we try this: {quote} sql("SELECT case when a>b then struct(cast(a as int), cast(b as int)) else struct(cast(c as int), cast(c as int)) end from t1") {quote} we get another exception: {quote} requirement failed: Unresolved attributes found when constructing LocalRelation. java.lang.IllegalArgumentException: requirement failed: Unresolved attributes found when constructing LocalRelation. at scala.Predef$.require(Predef.scala:224) at org.apache.spark.sql.catalyst.plans.logical.LocalRelation.<init>(LocalRelation.scala:49) {quote} However, these do work: {quote} sql("SELECT case when a>b then struct(cast(a as double), cast(b as double)) else struct(cast(c as double), cast(c as double)) end from t1") sql("SELECT case when a>b then struct(cast(a as string), cast(b as string)) else struct(cast(c as string), cast(c as string)) end from t1") {quote} any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org