Michael Allman created SPARK-17990:
--------------------------------------

             Summary: ALTER TABLE ... ADD PARTITION does not play nice with 
mixed-case partition column names
                 Key: SPARK-17990
                 URL: https://issues.apache.org/jira/browse/SPARK-17990
             Project: Spark
          Issue Type: Bug
          Components: SQL
         Environment: Linux
Mac OS with a case-sensitive filesystem
            Reporter: Michael Allman


Writing partition data to an external table's file location and then adding 
those as table partition metadata is a common use case. However, for tables 
with partition column names with upper case letters, the SQL command {{ALTER 
TABLE ... ADD PARTITION}} does not work, as illustrated in the following 
example:

{code}
scala> sql("create external table mixed_case_partitioning (a bigint) 
PARTITIONED BY (partCol bigint) STORED AS parquet LOCATION 
'/tmp/mixed_case_partitioning'")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sqlContext.range(10).selectExpr("id as a", "id as 
partCol").write.partitionBy("partCol").mode("overwrite").parquet("/tmp/mixed_case_partitioning")
{code}

At this point, doing a {{hadoop fs -ls /tmp/mixed_case_partitioning}} produces 
the following:

{code}
[msa@jupyter ~]$ hadoop fs -ls /tmp/mixed_case_partitioning
Found 11 items
-rw-r--r--   3 msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/_SUCCESS
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=0
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=1
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=2
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=3
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=4
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=5
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=6
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=7
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=8
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=9
{code}

Returning to the Spark shell, we execute the following to add the partition 
metadata:

{code}
scala> (0 to 9).foreach { p => sql(s"alter table mixed_case_partitioning add 
partition(partCol=$p)") }
{code}

Examining the HDFS file listing again, we see:

{code}
[msa@jupyter ~]$ hadoop fs -ls /tmp/mixed_case_partitioning
Found 21 items
-rw-r--r--   3 msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/_SUCCESS
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=0
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=1
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=2
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=3
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=4
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=5
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=6
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=7
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=8
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:52 
/tmp/mixed_case_partitioning/partCol=9
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=0
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=1
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=2
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=3
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=4
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=5
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=6
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=7
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=8
drwxr-xr-x   - msa supergroup          0 2016-10-18 17:53 
/tmp/mixed_case_partitioning/partcol=9
{code}

Note that {{msck repair table mixed_case_partitioning}} does not exhibit this 
behavior—it handles this use case correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to