Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19295 @gatorsmile thanks for your comments. Here are my thoughts, thanks for correcting me if i'm wrong. (sorry for the big comment though :)) 1. This PR don't change any existing API, it adds a new one. 2. In the usual cases, for the people who don't use `ExperimentalMethods`, it don't affect or break anything. 3. For the people who use `ExperimentalMethods`, irrespective of whether it is pre-optimizer or post-optimizer rule, it will break anyway if they do it wrong. 4. One of the advantages of this PR `sparkSession.experimental.extraPreOptimizations` is that the user provided rule can get further optimizer by the native rules of spark, which is not possible with `sparkSession.experimental.extraOptimizations`. I'm writing a blog post regarding this with an example, i will post the link soon. 5. Last but not least, one of the main intention of the spark catalyst optimizer, as mentioned in its sigmod paper, is it's simplicity in defining new optimization rules and plug it into the query optimizer during runtime, so we should consider not to limit it even if it only concerns a rare case.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org