[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19295 Sorry, we do not expect users to add rules before our internal optimizer rules finish, as I explained above. To avoid the potential issues, I suggest to close it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19295 I pushed a new commit that addresses @wzhfy review comments.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19295 @gatorsmile thanks for your comments. Here are my thoughts, thanks for correcting me if i'm wrong. (sorry for the big comment though :)) 1. This PR don't change any existing API, it adds a new one. 2. In the usual cases, for the people who don't use `ExperimentalMethods`, it don't affect or break anything. 3. For the people who use `ExperimentalMethods`, irrespective of whether it is pre-optimizer or post-optimizer rule, it will break anyway if they do it wrong. 4. One of the advantages of this PR `sparkSession.experimental.extraPreOptimizations` is that the user provided rule can get further optimizer by the native rules of spark, which is not possible with `sparkSession.experimental.extraOptimizations`. I'm writing a blog post regarding this with an example, i will post the link soon. 5. Last but not least, one of the main intention of the spark catalyst optimizer, as mentioned in its sigmod paper, is it's simplicity in defining new optimization rules and plug it into the query optimizer during runtime, so we should consider not to limit it even if it only concerns a rare case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19295 I do not think we should do it. The extra pre-optimizer rules can easily break our existing optimizer rules. Adding post optimizer rules should be enough for 99% cases. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19295 ping @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19295 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19295 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19295 why can't user just use `preOptimizationBatches`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19295 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org