GitHub user zhonghaihua opened a pull request: https://github.com/apache/spark/pull/10128
pull out nondeterministic expressions from Join Currently,`nondeterministic expressions` are only allowed in `Project` or `Filter`,And only when we use nondeterministic expressions in `UnaryNode` can be pulled out. But,Sometime in many case,we will use nondeterministic expressions to process `join keys` avoiding data skew.for example: ``` select * from tableA a join (select * from tableB) b on upper((case when (a.brand_code is null or a.brand_code = '' ) then cast( (-rand() * 10000000 ) as string ) else a.brand_code end )) = b.brand_code ``` This PR introduce a mechanism to pull out nondeterministic expressions from `Join`,so we can use nondeterministic expression in `Join` appropriately. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhonghaihua/spark pulloutJoinNondeterministic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10128.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10128 ---- commit 6e166578a5c1a1faf260389509663ac8c71ec015 Author: zhonghaihua <793507...@qq.com> Date: 2015-11-30T07:44:49Z pull out nondeterministic expressions from Join pull out nondeterministic expressions from Join pull out nondeterministic expressions from Join ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org