[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[
https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242274#comment-17242274
]
Apache Spark commented on SPARK-33544:
--
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/30570
> explode should not filter when used with CreateArray
>
>
> Key: SPARK-33544
> URL: https://issues.apache.org/jira/browse/SPARK-33544
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>Priority: Major
> Fix For: 3.1.0
>
>
> https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
> insert a filter for not null and size > 0 when using inner explode/inline.
> This is fine in most cases but the extra filter is not needed if the explode
> is with a create array and not using Literals (it already handles LIterals).
> When this happens you know that the values aren't null and it has a size. It
> already handles the empty array.
> for instance:
> val df = someDF.selectExpr("number", "explode(array(word, col3))")
> So in this case we shouldn't be inserting the extra Filter and that filter
> can get pushed down into like a parquet reader as well. This is just causing
> extra overhead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[
https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238892#comment-17238892
]
Apache Spark commented on SPARK-33544:
--
User 'tgravescs' has created a pull request for this issue:
https://github.com/apache/spark/pull/30504
> explode should not filter when used with CreateArray
>
>
> Key: SPARK-33544
> URL: https://issues.apache.org/jira/browse/SPARK-33544
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Thomas Graves
>Priority: Major
>
> https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
> insert a filter for not null and size > 0 when using inner explode/inline.
> This is fine in most cases but the extra filter is not needed if the explode
> is with a create array and not using Literals (it already handles LIterals).
> When this happens you know that the values aren't null and it has a size. It
> already handles the empty array.
> for instance:
> val df = someDF.selectExpr("number", "explode(array(word, col3))")
> So in this case we shouldn't be inserting the extra Filter and that filter
> can get pushed down into like a parquet reader as well. This is just causing
> extra overhead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[
https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238473#comment-17238473
]
L. C. Hsieh commented on SPARK-33544:
-
Thanks [~hyukjin.kwon]. Will help review if [~tgraves] create a patch.
> explode should not filter when used with CreateArray
>
>
> Key: SPARK-33544
> URL: https://issues.apache.org/jira/browse/SPARK-33544
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Thomas Graves
>Priority: Major
>
> https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
> insert a filter for not null and size > 0 when using inner explode/inline.
> This is fine in most cases but the extra filter is not needed if the explode
> is with a create array and not using Literals (it already handles LIterals).
> When this happens you know that the values aren't null and it has a size. It
> already handles the empty array.
> for instance:
> val df = someDF.selectExpr("number", "explode(array(word, col3))")
> So in this case we shouldn't be inserting the extra Filter and that filter
> can get pushed down into like a parquet reader as well. This is just causing
> extra overhead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[
https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238469#comment-17238469
]
Hyukjin Kwon commented on SPARK-33544:
--
cc [~viirya] FYI
> explode should not filter when used with CreateArray
>
>
> Key: SPARK-33544
> URL: https://issues.apache.org/jira/browse/SPARK-33544
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Thomas Graves
>Priority: Major
>
> https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
> insert a filter for not null and size > 0 when using inner explode/inline.
> This is fine in most cases but the extra filter is not needed if the explode
> is with a create array and not using Literals (it already handles LIterals).
> When this happens you know that the values aren't null and it has a size. It
> already handles the empty array.
> for instance:
> val df = someDF.selectExpr("number", "explode(array(word, col3))")
> So in this case we shouldn't be inserting the extra Filter and that filter
> can get pushed down into like a parquet reader as well. This is just causing
> extra overhead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[
https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238371#comment-17238371
]
Thomas Graves commented on SPARK-33544:
---
I'm working on a patch for this.
> explode should not filter when used with CreateArray
>
>
> Key: SPARK-33544
> URL: https://issues.apache.org/jira/browse/SPARK-33544
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Thomas Graves
>Priority: Major
>
> https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
> insert a filter for not null and size > 0 when using inner explode/inline.
> This is fine in most cases but the extra filter is not needed if the explode
> is with a create array and not using Literals (it already handles LIterals).
> When this happens you know that the values aren't null and it has a size. It
> already handles the empty array.
> for instance:
> val df = someDF.selectExpr("number", "explode(array(word, col3))")
> So in this case we shouldn't be inserting the extra Filter and that filter
> can get pushed down into like a parquet reader as well. This is just causing
> extra overhead.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
