[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21138 Merging to master and branch 2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89864/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21138 **[Test build #89864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89864/testReport)** for PR 21138 at commit [`0077685`](https://github.com/apache/spark/commit/00776858c2e776f46dbe542effe52a19283e752f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2683/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21138 **[Test build #89864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89864/testReport)** for PR 21138 at commit [`0077685`](https://github.com/apache/spark/commit/00776858c2e776f46dbe542effe52a19283e752f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21138 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21138 Thanks for the review @mridulm @vanzin . Let me test again. I will merge the code when test is passed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21138 I'm fine with the fix. Not familiar with the internals of the STS / Hive to suggest anything different. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/21138 Sounds good to me; any thoughts @vanzin ? (since you changed this last) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21138 @mridulm I would treat the current fix as a workaround for SASL issue, since it is a regression in 2.3. For UGI refreshing issue (mainly cause STS long running failure, also lead to SASL failure here), I think we can create a separate JIRA to fix the issue. Since this is not a regression. What do you think? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/21138 @jerryshao As we discussed IRL `HiveClientImpl` was one place this is happening (now fixed - git pull delays, my bad). The other is in `HadoopThriftAuthBridge` we rely on - `createServer` results in always doing a `UGI.loginUserFromKeytab` : resulting in changing the static `loginUser` in UGI (even though principal and keytab are the same). This results in : * The current bug - where secret token's are now gone. * Long running STS in secure cluster failing (due to hadoop IPC failure) ** This is the reason why I found/had worked on this - though for 1.6 release. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21138 Hi @mridulm , thanks a lot for your comments. UGI.loginUserFromKeytab is not existed any more in Spark 2.3+ (https://github.com/apache/spark/commit/dc2714da50ecba1bf1fdf555a82a4314f763a76e#diff-6fd847124f8eae45ba2de1cf7d6296fe). Actually it is the code here (https://github.com/apache/spark/blob/e77d62a722941ce1cf235861d21b1f73089be134/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIService.scala#L53) in ThriftServer and somewhere else in Hive Library which calls UGI.loginUserFromKeytab, note here the principal and keytab is hive one, not sure if we can remove this. PS. I saw two "Login successful" from UGI in the thrift server log, but I can only find out one login in the thrift server code. So I assume another one is in the Hive library. Yes, I agree with you, ideally we should not login from keytab unnecessarily, but thinking of thrift server as a Spark application, it doesn't know the context of Spark's UGI and do login to refresh the UGI in its context, seems we cannot defend user to do that in the user layer. So I think my fix could workaround such issue, though may not be the elegant fix. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/21138 IMO the fix would be to not do UGI.loginUserFromKeytab in HiveClientImpl; or rather, do it only if it is absolutely necessary. I will share the snippet with you @jerryshao - this is something I have fixed in the past (invoking UGI.login multiple times; not this specific bug - though it is a manifestation of the same). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89773/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21138 **[Test build #89773 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89773/testReport)** for PR 21138 at commit [`0077685`](https://github.com/apache/spark/commit/00776858c2e776f46dbe542effe52a19283e752f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2622/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21138: [SPARK-24062][Thrift Server] Fix SASL encryption cannot ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21138 **[Test build #89773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89773/testReport)** for PR 21138 at commit [`0077685`](https://github.com/apache/spark/commit/00776858c2e776f46dbe542effe52a19283e752f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
