[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

2021-07-19 Thread Apache Spark (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383322#comment-17383322
 ] 

Apache Spark commented on SPARK-36093:
--

User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/33418

> The result incorrect if the partition path case is inconsistent
> ---
>
> Key: SPARK-36093
> URL: https://issues.apache.org/jira/browse/SPARK-36093
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Apache Spark
>Priority: Major
>  Labels: correctness
>
> Please reproduce this issue using HDFS. Local HDFS can not reproduce this 
> issue.
> {code:scala}
> sql("create table t1(cal_dt date) using parquet")
> sql("insert into t1 values 
> (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
> sql("create view t1_v as select * from t1")
> sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS 
> FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
> sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'")
> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> sql("SELECT * FROM t2 ").show
> {code}
> {noformat}
> // It should not empty.
> scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> ++--+
> scala> sql("SELECT * FROM t2 ").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   1|2021-06-27|
> |   1|2021-06-28|
> ++--+
> scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   2|2021-06-29|
> |   2|2021-06-30|
> ++--+
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

2021-07-19 Thread Apache Spark (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383321#comment-17383321
 ] 

Apache Spark commented on SPARK-36093:
--

User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/33417

> The result incorrect if the partition path case is inconsistent
> ---
>
> Key: SPARK-36093
> URL: https://issues.apache.org/jira/browse/SPARK-36093
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Apache Spark
>Priority: Major
>  Labels: correctness
>
> Please reproduce this issue using HDFS. Local HDFS can not reproduce this 
> issue.
> {code:scala}
> sql("create table t1(cal_dt date) using parquet")
> sql("insert into t1 values 
> (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
> sql("create view t1_v as select * from t1")
> sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS 
> FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
> sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'")
> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> sql("SELECT * FROM t2 ").show
> {code}
> {noformat}
> // It should not empty.
> scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> ++--+
> scala> sql("SELECT * FROM t2 ").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   1|2021-06-27|
> |   1|2021-06-28|
> ++--+
> scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   2|2021-06-29|
> |   2|2021-06-30|
> ++--+
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

2021-07-13 Thread Apache Spark (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379905#comment-17379905
 ] 

Apache Spark commented on SPARK-36093:
--

User 'AngersZh' has created a pull request for this issue:
https://github.com/apache/spark/pull/33324

> The result incorrect if the partition path case is inconsistent
> ---
>
> Key: SPARK-36093
> URL: https://issues.apache.org/jira/browse/SPARK-36093
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Priority: Major
>  Labels: correctness
>
> Please reproduce this issue using HDFS. Local HDFS can not reproduce this 
> issue.
> {code:scala}
> sql("create table t1(cal_dt date) using parquet")
> sql("insert into t1 values 
> (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
> sql("create view t1_v as select * from t1")
> sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS 
> FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
> sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'")
> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> sql("SELECT * FROM t2 ").show
> {code}
> {noformat}
> // It should not empty.
> scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> ++--+
> scala> sql("SELECT * FROM t2 ").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   1|2021-06-27|
> |   1|2021-06-28|
> ++--+
> scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   2|2021-06-29|
> |   2|2021-06-30|
> ++--+
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36093) The result incorrect if the partition path case is inconsistent

2021-07-12 Thread angerszhu (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379224#comment-17379224
 ] 

angerszhu commented on SPARK-36093:
---

Working on this

> The result incorrect if the partition path case is inconsistent
> ---
>
> Key: SPARK-36093
> URL: https://issues.apache.org/jira/browse/SPARK-36093
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Priority: Major
>  Labels: correctness
>
> Please reproduce this issue using HDFS. Local HDFS can not reproduce this 
> issue.
> {code:scala}
> sql("create table t1(cal_dt date) using parquet")
> sql("insert into t1 values 
> (date'2021-06-27'),(date'2021-06-28'),(date'2021-06-29'),(date'2021-06-30')")
> sql("create view t1_v as select * from t1")
> sql("CREATE TABLE t2 USING PARQUET PARTITIONED BY (CAL_DT) AS SELECT 1 AS 
> FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN '2021-06-27' AND '2021-06-28'")
> sql("INSERT INTO t2 SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'")
> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> sql("SELECT * FROM t2 ").show
> {code}
> {noformat}
> // It should not empty.
> scala> sql("SELECT * FROM t2 WHERE CAL_DT BETWEEN '2021-06-29' AND 
> '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> ++--+
> scala> sql("SELECT * FROM t2 ").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   1|2021-06-27|
> |   1|2021-06-28|
> ++--+
> scala> sql("SELECT 2 AS FLAG,CAL_DT FROM t1_v WHERE CAL_DT BETWEEN 
> '2021-06-29' AND '2021-06-30'").show
> ++--+
> |FLAG|CAL_DT|
> ++--+
> |   2|2021-06-29|
> |   2|2021-06-30|
> ++--+
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org