[ https://issues.apache.org/jira/browse/SPARK-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liang-Chi Hsieh closed SPARK-18130. ----------------------------------- Resolution: Won't Fix > Don't add inferred redundant isnotnull condition from constraints > ----------------------------------------------------------------- > > Key: SPARK-18130 > URL: https://issues.apache.org/jira/browse/SPARK-18130 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Liang-Chi Hsieh > Priority: Minor > > In Optimizer, we will infer additional conditions from Filter's constraints > which include IsNotNull. Some inferred IsNotNulls are useful because they can > be short circuit for other predicates which refer the same attributes. But > for an IsNotNull condition, we also infer other IsNotNull from it. Those > IsNotNulls are redundant. We don't need to include them. > E.g., for a condition like IsNotNull(a + Rand()), an additional IsNotNull(a) > will be added into Filter's condition. So we need to evaluate IsNotNull(a + > Rand()) && IsNotNull(a). Actually we don't need IsNotNull(a) because it is > guaranteed by the original IsNotNull(a + Rand()). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org