[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(p, null) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(not(p), null) {code} iff predicate {{p}} is deterministic, i.e., can be evaluated to either true or false, but not null. {{and}} and {{or}} clauses are more optimization friendly. For instance, by converting {{if(col > 42, null, false)}} to {{and(col > 42, null)}}, we can potentially push the filter {{col > 42}} down to data sources to avoid unnecessary IO. was: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(p, null) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(not(p), null) {code} iff predicate {{p}} is deterministic, i.e., can be evaluated to either true or false, but not null. > Simplify if clauses with null and boolean > - > > Key: SPARK-32721 > URL: https://issues.apache.org/jira/browse/SPARK-32721 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Chao Sun >Priority: Major > > The following if clause: > {code:sql} > if(p, null, false) > {code} > can be simplified to: > {code:sql} > and(p, null) > {code} > And similarly, the following clause: > {code:sql} > if(p, null, true) > {code} > can be simplified to: > {code:sql} > or(not(p), null) > {code} > iff predicate {{p}} is deterministic, i.e., can be evaluated to either true > or false, but not null. > {{and}} and {{or}} clauses are more optimization friendly. For instance, by > converting {{if(col > 42, null, false)}} to {{and(col > 42, null)}}, we can > potentially push the filter {{col > 42}} down to data sources to avoid > unnecessary IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(p, null) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(not(p), null) {code} iff predicate {{p}} is deterministic, i.e., can be evaluated to either true or false, but not null. was: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(p, null) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(p, null) {code} > Simplify if clauses with null and boolean > - > > Key: SPARK-32721 > URL: https://issues.apache.org/jira/browse/SPARK-32721 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Chao Sun >Priority: Major > > The following if clause: > {code:sql} > if(p, null, false) > {code} > can be simplified to: > {code:sql} > and(p, null) > {code} > And similarly, the following clause: > {code:sql} > if(p, null, true) > {code} > can be simplified to: > {code:sql} > or(not(p), null) > {code} > iff predicate {{p}} is deterministic, i.e., can be evaluated to either true > or false, but not null. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(p, null) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(p, null) {code} was: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(null, false) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(null, true) {code} > Simplify if clauses with null and boolean > - > > Key: SPARK-32721 > URL: https://issues.apache.org/jira/browse/SPARK-32721 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Chao Sun >Priority: Major > > The following if clause: > {code:sql} > if(p, null, false) > {code} > can be simplified to: > {code:sql} > and(p, null) > {code} > And similarly, the following clause: > {code:sql} > if(p, null, true) > {code} > can be simplified to: > {code:sql} > or(p, null) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(null, false) {code} And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(null, true) {code} was: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:sql} and(null, false). And similarly, the following clause: {code:sql} if(p, null, true) {code} can be simplified to: {code:sql} or(null, true) {code} > Simplify if clauses with null and boolean > - > > Key: SPARK-32721 > URL: https://issues.apache.org/jira/browse/SPARK-32721 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Chao Sun >Priority: Major > > The following if clause: > {code:sql} > if(p, null, false) > {code} > can be simplified to: > {code:sql} > and(null, false) > {code} > And similarly, the following clause: > {code:sql} > if(p, null, true) > {code} > can be simplified to: > {code:sql} > or(null, true) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org