[
https://issues.apache.org/jira/browse/SPARK-54881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif updated SPARK-54881:
-------------------------
Description:
In case of simplification of an expression of type
Not(A <= B OR A >= B)
to
(A > B AND A < B)
it does not happen in a single pass.
This is due to use of "transformUp" instead of transformDown.
Because of transformUp logic, Not (A <= B OR A >=B) is converted to
Not(A <= B) AND Not(A >=B) in the first pass.
Only in second pass would it result in final valid form of
(A > B AND A < B) .
However, just replacing transformUp with transformDown though fixes this issue,
but it breaks idempotency in single pass for some cases as seen by failure in
ConstantFoldingTest.
so the fix should be such that it satisfies both of the cases.
Will be opening a PR and bugtest in some time.
was:
In case of simplification of an expression of type
Not(A <= B OR A >= B)
to
(A > B AND A < B)
it does not happen in a single pass.
This is due to use of "transformUp" instead of transformDown.
Because of transformUp logic, Not (A <= B OR A >=B) is converted to
Not(A <= B) AND Not(A >=B) in the first pass.
Only in second pass would it result in final valid form of
(A > B AND A < B)
Will be opening a PR and bugtest in some time.
> BooleanSimplification rule using transformExpressionsUp instead of
> transformExpressionsDown, is inefficient in some cases resulting in delayed
> idempotency
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-54881
> URL: https://issues.apache.org/jira/browse/SPARK-54881
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0, 4.1.1
> Reporter: Asif
> Priority: Major
> Labels: pull-request-available
>
> In case of simplification of an expression of type
> Not(A <= B OR A >= B)
> to
> (A > B AND A < B)
> it does not happen in a single pass.
> This is due to use of "transformUp" instead of transformDown.
> Because of transformUp logic, Not (A <= B OR A >=B) is converted to
> Not(A <= B) AND Not(A >=B) in the first pass.
> Only in second pass would it result in final valid form of
> (A > B AND A < B) .
> However, just replacing transformUp with transformDown though fixes this
> issue, but it breaks idempotency in single pass for some cases as seen by
> failure in ConstantFoldingTest.
> so the fix should be such that it satisfies both of the cases.
>
> Will be opening a PR and bugtest in some time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
