GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/21138
[SPARK-24062][Thrift Server] Fix SASL encryption cannot enabled issue in
thrift server
## What changes were proposed in this pull request?
For the details of the exception please see
[SPARK-24062](https://issues.apache.org/jira/browse/SPARK-24062).
The issue is:
Spark on Yarn stores SASL secret in current UGI's credentials, this
credentials will be distributed to AM and executors, so that executors and
drive share the same secret to communicate. But STS/Hive library code will
refresh the current UGI by UGI's loginFromKeytab() after Spark application is
started, this will create a new UGI in the current driver's context with empty
tokens and secret keys, so secret key is lost in the current context's UGI,
that's why Spark driver throws secret key not found exception.
In Spark 2.2 code, Spark also stores this secret key in SecurityManager's
class variable, so even UGI is refreshed, the secret is still existed in the
object, so STS with SASL can still be worked in Spark 2.2. But in Spark 2.3, we
always search key from current UGI, which makes it fail to work in Spark 2.3.
To fix this issue, there're two possible solutions:
1. Fix in STS/Hive library, when a new UGI is refreshed, copy the secret
key from original UGI to the new one. The difficulty is that some codes to
refresh the UGI is existed in Hive library, which makes us hard to change the
code.
2. Roll back the logics in SecurityManager to match Spark 2.2, so that this
issue can be fixed.
2nd solution seems a simple one. So I will propose a PR with 2nd solution.
## How was this patch tested?
Verified in local cluster.
CC @vanzin @tgravescs please help to review. Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-24062
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21138.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21138
----
commit 00776858c2e776f46dbe542effe52a19283e752f
Author: jerryshao <sshao@...>
Date: 2018-04-24T08:14:27Z
Fix secret key not found issue in STS
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
