[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620961 Right. Thank you so much for enriching ideas! I'll update this PR with `FoldablePropagation`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620606 the scope of `NullPropagation` is one operator, but we need a `FoldablePropagation` whose scope is the whole plan tree. Think about `Sort(a, Filter(true, Project(1 AS a)))`, we should be able to propagate the foldable information up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620522 Oh, I got. Thanks. I will try to generalize. * Sort(_, Project(_)) * Project(_, Project(...)) And so on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620369 If that's just about how to handle `Sort(_, Project(_,_))` expressions in `EliminateSorts`, I can easily modify this PR according to your advice. After moving up the foldables, and the existing `case` statement removes them eventually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215619413 Actually, `Sort` is dead end, we can not propagate up anymore. So, in that case, removing looks more efficient. Do you mean more generalized `FoldablePropagation` like `NullPropagation` by 'not only Sort'? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215618718 `select 1 as a from tbl order by a` is equal to `select 1 as a from tbl order by 1`. When the child operator is `Project` and has foldable output, if the parent operator references the foldable output, we should replace the attribute with the real foldable expression in `Project`. (and keep the alias to preserve the naming info) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215616875 Thank you for review, @cloud-fan ! Do you mean removing aliases by replacing the base expression(?) by using `transformUp`? Maybe, except the top most aliases? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215614987 instead of doing this, can we propagate foldable alias bottom up? So that it's not only `Sort`, but all operators can benefit from it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215564542 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57271/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215564535 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215564274 **[Test build #57271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57271/consoleFull)** for PR 12719 at commit [`b7deb89`](https://github.com/apache/spark/commit/b7deb89ddd124e243d5691a541ceaec0399bd835). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215547799 Thank you so much, @gatorsmile ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215546800 Sure, will do it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215545987 Hi, @gatorsmile and @cloud-fan . Could you take a look at this PR when you have some time? I think you are the best persons to review this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215541446 **[Test build #57271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57271/consoleFull)** for PR 12719 at commit [`b7deb89`](https://github.com/apache/spark/commit/b7deb89ddd124e243d5691a541ceaec0399bd835). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
