Tomasz Bartczak created SPARK-23477: ---------------------------------------
Summary: Misleading exception message when union fails due to metadata Key: SPARK-23477 URL: https://issues.apache.org/jira/browse/SPARK-23477 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.2.1 Reporter: Tomasz Bartczak When I have two DF's that are different only in terms of metadata in fields inside a struct - I cannot union them but the error message shows that they are the same: {code:java} df = spark.createDataFrame([{'a':1}]) a = df.select(struct('a').alias('x')) b = df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x')) a.union(b).printSchema(){code} gives: {code:java} An error occurred while calling o1076.union. : org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> at the first column of the second table{code} and this part: {code:java} struct<a:bigint> <> struct<a:bigint>{code} does not make any sense because those are the same. Since metadata must be the same for union -> it should be incuded in the error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org