[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26812: -- Affects Version/s: 2.0.2 > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 2.4.4, 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26812: -- Affects Version/s: 2.1.3 > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.3, 2.2.3, 2.3.4, 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 2.4.4, 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26812: -- Affects Version/s: 2.2.3 > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.2.3, 2.3.4, 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 2.4.4, 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26812: -- Affects Version/s: 2.3.4 > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4, 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 2.4.4, 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26812: -- Fix Version/s: 2.4.4 > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 2.4.4, 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26812) PushProjectionThroughUnion nullability issue
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-26812: --- Labels: correctness (was: ) > PushProjectionThroughUnion nullability issue > > > Key: SPARK-26812 > URL: https://issues.apache.org/jira/browse/SPARK-26812 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bogdan Raducanu >Assignee: Marco Gaido >Priority: Major > Labels: correctness > Fix For: 3.0.0 > > > Union output data types are the output data types of the first child. > However the other union children may have different values nullability. > This means that we can't always push down a project on the children. > To reproduce > {code} > Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1") > sql("SELECT 1 AS b").write.saveAsTable("table2") > sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL > SELECT a FROM table1") > sql("select * from test1").show > {code} > This fails becaus the plan is no longer resolved. > The plan is broken by the PushProjectionThroughUnion rule which pushed down a > cast to map with values nullability=true on a child with type > map with values nullability=false. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org