Github user viirya commented on the issue: https://github.com/apache/spark/pull/16998 > Not really. Constraint propagation will still be enabled by default in Spark. The flag would just be a hammer to quickly get around issues like this and SPARK-17733. Yeah, of course. I meant that when you disable the flag, you would enjoy the optimization relying on constraint propagation. I will create another PR for this option. > I'll take a closer look at this patch but given that this PR is primarily introducing a data structure that keeps track of aliased constraints, is there a fundamental reason for changing the underlying behavior (and restricting the optimization space)? Can there be a simpler alternative where we can still keep the old semantics? I don't find an alternative fixing to keep the old semantics and not change the propagation structure, and also can largely improve performance at the same time. #16785 keeps the old semantics and not change the propagation structure, but it just can cut half of the running time regarding the benchmark. Adding the flag is one simpler option.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org