[ https://issues.apache.org/jira/browse/SPARK-34543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Gekk updated SPARK-34543: ------------------------------- Description: SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config *spark.sql.caseSensitive* which is false by default, for instance: {code:sql} spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part); spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0; spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0); Location: file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 spark-sql> ALTER TABLE tbl ADD PARTITION (part=1); spark-sql> SELECT * FROM tbl; 0 0 {code} Create new partition folder in the file system: {code} $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa {code} Set new location for the partition part=1: {code:sql} spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa'; spark-sql> SELECT * FROM tbl; 0 0 0 1 spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2); spark-sql> SELECT * FROM tbl; 0 0 0 1 {code} Set location for a partition in the upper case: {code} $ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb {code} {code:sql} spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb'; Error in query: Partition spec is invalid. The spec (PART) must match the partition spec (part) defined in table '`default`.`tbl`' {code} was: SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config *spark.sql.caseSensitive* which is false by default, for instance: {code:sql} spark-sql> CREATE TABLE tbl1 (price int, qty int, year int, month int) > USING parquet > PARTITIONED BY (year, month); spark-sql> INSERT INTO tbl1 PARTITION(year = 2015, month = 1) SELECT 1, 1; spark-sql> SHOW PARTITIONS tbl1 PARTITION(YEAR = 2015, Month = 1); Error in query: Non-partitioning column(s) [YEAR, Month] are specified for SHOW PARTITIONS; {code} > Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION > ---------------------------------------------------------- > > Key: SPARK-34543 > URL: https://issues.apache.org/jira/browse/SPARK-34543 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 2.4.7, 3.0.1, 3.1.0 > Reporter: Maxim Gekk > Assignee: Maxim Gekk > Priority: Major > Fix For: 2.4.8, 3.0.2, 3.1.0 > > > SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config > *spark.sql.caseSensitive* which is false by default, for instance: > {code:sql} > spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part); > spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0; > spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0); > Location: > file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 > spark-sql> ALTER TABLE tbl ADD PARTITION (part=1); > spark-sql> SELECT * FROM tbl; > 0 0 > {code} > Create new partition folder in the file system: > {code} > $ cp -r > /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 > /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa > {code} > Set new location for the partition part=1: > {code:sql} > spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION > '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa'; > spark-sql> SELECT * FROM tbl; > 0 0 > 0 1 > spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2); > spark-sql> SELECT * FROM tbl; > 0 0 > 0 1 > {code} > Set location for a partition in the upper case: > {code} > $ cp -r > /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 > /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb > {code} > {code:sql} > spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION > '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb'; > Error in query: Partition spec is invalid. The spec (PART) must match the > partition spec (part) defined in table '`default`.`tbl`' > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org