[ https://issues.apache.org/jira/browse/SPARK-29164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-29164. ---------------------------------- Resolution: Won't Fix > Rewrite coalesce(boolean, booleanLit) as boolean expression > ----------------------------------------------------------- > > Key: SPARK-29164 > URL: https://issues.apache.org/jira/browse/SPARK-29164 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Josh Rosen > Priority: Major > > I propose the following expression rewrite optimizations: > {code:java} > coalesce(x: Boolean, true) -> x or isnull(x) > coalesce(x: Boolean, false) -> x and isnotnull(x){code} > This pattern appears when translating Dataset filters on {{Option[Boolean]}} > columns: we might have a typed Dataset filter which looks like > {code:java} > .filter(_.boolCol.getOrElse(DEFAULT_VALUE)){code} > and the most idiomatic, user-friendly translation of this in Catalyst is to > use {{coalesce()}}. However, the {{coalesce()}} form of this expression is > not eligible for Parquet / data source filter pushdown. > (We should write out truth-tables to double-check this rewrite's correctness) -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org