GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/18299
Spark 21092
## What changes were proposed in this pull request?
It is really painful to not have configs in logical plan and expressions.
We had to add all sorts of hacks (e.g. pass SQLConf explicitly in functions).
This patch exposes SQLConf in logical plan, using a thread local variable and a
getter closure that's set once there is an active SparkSession.
The implementation is a bit of a hack, since we didn't anticipate this need
in the beginning (config was only exposed in physical plan).
In terms of future work, we should follow up to clean up CBO (remove the
need for passing in config).
## How was this patch tested?
Updated relevant tests for constraint propagation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-21092
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18299.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18299
----
commit bbcd26f0cdfaba602eacbcac79a563e06cb33736
Author: Reynold Xin <[email protected]>
Date: 2017-06-14T06:41:29Z
[SPARK-21091][SQL] Move constraint code into QueryPlanConstraints
commit ced6e9b754fee8df23b0b554a621c185c2a2b211
Author: Reynold Xin <[email protected]>
Date: 2017-06-14T07:19:13Z
[SPARK-21092][SQL] Wire SQLConf in logical plan and expressions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
