Asif created SPARK-54881:
----------------------------
Summary: BooleanSimplification rule using transformExpressionsUp
instead of transformExpressionsDown, is inefficient in some cases resulting in
delayed idempotency
Key: SPARK-54881
URL: https://issues.apache.org/jira/browse/SPARK-54881
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0, 4.1.1
Reporter: Asif
In case of simplification of an expression of type
Not(A <= B OR A >= B)
to
(A > B AND A < B)
it does not happen in a single pass.
This is due to use of "transformUp" instead of transformDown.
Because of transformUp logic, Not (A <= B OR A >=B) is converted to
Not(A <= B) AND Not(A >=B) in the first pass.
Only in second pass would it result in final valid form of
(A > B AND A < B)
Will be opening a PR and bugtest in some time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
