Jordan Halterman created SPARK-17897:
----------------------------------------
Summary: not isnotnull is converted to the always false condition
isnotnull && not isnotnull
Key: SPARK-17897
URL: https://issues.apache.org/jira/browse/SPARK-17897
Project: Spark
Issue Type: Bug
Components: Optimizer
Affects Versions: 2.0.1, 2.0.0
Reporter: Jordan Halterman
When a logical plan is built containing the following somewhat nonsensical
filter:
{{Filter (NOT isnotnull($f0#212))}}
During optimization the filter is converted into a condition that will always
fail:
{{Filter (isnotnull($f0#212) && NOT isnotnull($f0#212))}}
This appears to be caused by the following check for {{NullIntolerant}}:
https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R63
Which recurses through the expression and extracts nested {{IsNotNull}} calls,
converting them to {{IsNotNull}} calls on the attribute at the root level:
https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R49
This results in the nonsensical condition above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
