GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/18460
[SPARK-21247][SQL] Allow case-insensitive type equality in Set operation
## What changes were proposed in this pull request?
Spark supports case-sensitivity in columns. Especially, for Struct types,
with `spark.sql.caseSensitive=true`, the following is supported.
```scala
scala> sql("select named_struct('a', 1, 'A', 2).a").show
+--------------------------+
|named_struct(a, 1, A, 2).a|
+--------------------------+
| 1|
+--------------------------+
scala> sql("select named_struct('a', 1, 'A', 2).A").show
+--------------------------+
|named_struct(a, 1, A, 2).A|
+--------------------------+
| 2|
+--------------------------+
```
And vice versa, with `spark.sql.caseSensitive=false`, the following is
supported.
```scala
scala> sql("select named_struct('a', 1).A, named_struct('A', 1).a").show
+--------------------+--------------------+
|named_struct(a, 1).A|named_struct(A, 1).a|
+--------------------+--------------------+
| 1| 1|
+--------------------+--------------------+
```
However, types are considered different. For example, SET operations fail.
```scala
scala> sql("SELECT struct(1 a) UNION ALL (SELECT struct(2 A))").show
org.apache.spark.sql.AnalysisException: Union can only be performed on
tables with the compatible column types. struct<A:int> <> struct<a:int> at the
first column of the second table;;
'Union
:- Project [named_struct(a, 1) AS named_struct(a, 1 AS `a`)#2]
: +- OneRowRelation$
+- Project [named_struct(A, 2) AS named_struct(A, 2 AS `A`)#3]
+- OneRowRelation$
```
This PR aims to support case-insensitive type equality in Set operation.
Please note that this issue does not aim to change all type equality semantics.
## How was this patch tested?
Pass the Jenkins with a newly add test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-21247
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18460
----
commit f600448f710312cc1795abff7771daab8cefcb4a
Author: Dongjoon Hyun <[email protected]>
Date: 2017-06-28T22:30:09Z
[SPARK-21247][SQL] Allow case-insensitive type equality in Set operation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
