[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43959321 Ok, close this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin closed the pull request at: https://github.com/apache/spark/pull/825 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43951637 Probably makes sense to close this one now. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43842591 @rxin Thanks a lot! Well, should I close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43824842 Done & merged #836. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43742049 @rxin BTW, speaking of performance, could you please review the code #836? I think this is a kind of blocker issue of join strategy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43741634 @rxin Ah, you mean that we should add right/full outer join support in addition to #734? I agree with the unfavorable performance of the nested loop join, so we should wait for being merged and then add the right/full outer join support at another issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43727197 We can easily add right outer join support to the hash join though. In general, the nested loop join performs very unfavorably compared with a hash join implementation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43599450 @rxin Thank you for your comment. I checked the code #734, not deeply yet, though. It seems like broadcast hash join is used only for `Inner` join so broadcast nested loop join will not be replaced. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43585497 Would be great if you can review that code, @ueshin. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43585484 Thanks for submitting this. I actually think this thing is going away to be replaced by broadcast hash join. https://github.com/apache/spark/pull/734 Maybe we should wait for that to be merged? Otherwise a lot of unnecessary conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43482897 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15079/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43482896 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43476605 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/825#issuecomment-43476612 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/825 [SPARK-1880] [SQL] Eliminate unnecessary job executions. There are unnecessary job executions in `BroadcastNestedLoopJoin`. When `Innner` or `LeftOuter` join, preparation of `rightOuterMatches` for `RightOuter` or `FullOuter` join is not neccessary. And when `RightOuter` or `FullOuter`, it should use not `count` and then `reduce` but `fold`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-1880 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/825.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #825 commit 7bff4d978fa0cb64ef6d4e935aaa8c394a6fa4ae Author: Takuya UESHIN Date: 2014-05-19T07:47:39Z Move preparing of rightOuterMatches to after checking joinType not to execute unnecessary job when joinType is Inner or LeftOuter. commit 63cacbfd41090185de64c27586cacc5265c3c0d7 Author: Takuya UESHIN Date: 2014-05-19T07:52:33Z Use fold instead of count and then reduce to eliminate unnecessary job executions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
