[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-161064237 The change doesn't affect pyspark; so merging (master and 1.6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9859 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-160819668 **[Test build #46923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)** for PR 9859 at commit [`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-160841511 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-160841354 **[Test build #46923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)** for PR 9859 at commit [`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-160841515 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46923/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-160819008 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159915514 Thank you @vanzin for your help. I commited the nits in the documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159921198 **[Test build #46766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)** for PR 9859 at commit [`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159970308 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159970290 **[Test build #46766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)** for PR 9859 at commit [`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159970309 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46766/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159734089 Also, just for posterity, if you're running local mode, you should be able to `kinit` before running your Spark app, and then you don't need to provide principal / keytab at all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9859#discussion_r45920128 --- Diff: docs/running-on-yarn.md --- @@ -349,14 +349,14 @@ If you need a reference to the proper location to put log files in the YARN so t The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the Secure Distributed Cache, - for renewing the login tickets and the delegation tokens periodically. + for renewing the login tickets and the delegation tokens periodically. (Works also in `LOCAL` mode) spark.yarn.principal (none) - Principal to be used to login to KDC, while running on secure HDFS. + Principal to be used to login to KDC, while running on secure HDFS. (Works also in `LOCAL` mode) --- End diff -- Same. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159733870 Just minor nits, otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9859#discussion_r45920123 --- Diff: docs/running-on-yarn.md --- @@ -349,14 +349,14 @@ If you need a reference to the proper location to put log files in the YARN so t The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the Secure Distributed Cache, - for renewing the login tickets and the delegation tokens periodically. + for renewing the login tickets and the delegation tokens periodically. (Works also in `LOCAL` mode) --- End diff -- nit: Works also with the "local" master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9859#discussion_r45920176 --- Diff: docs/sql-programming-guide.md --- @@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes Hive. Note that this Hive a on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries (SerDes) in order to access data stored in Hive. -Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running +Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` (in order to security parameters, --- End diff -- nit: "for security configuration" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9859#discussion_r45920195 --- Diff: docs/sql-programming-guide.md --- @@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes Hive. Note that this Hive a on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries (SerDes) in order to access data stored in Hive. -Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running +Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` (in order to security parameters, + like Kerberos), `hdfs-site.xml` (parameters of HDFS HA, etc.) file in `conf/`. Please note when running --- End diff -- nit: "for HDFS configuration" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159530732 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159530547 **[Test build #46672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)** for PR 9859 at commit [`9365a7f`](https://github.com/apache/spark/commit/9365a7f8766b081625ed8fe9066cef10df25b198). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159530735 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46672/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159447241 **[Test build #46625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)** for PR 9859 at commit [`7802c0c`](https://github.com/apache/spark/commit/7802c0ca3cf4fb6d19180dbda48291d5a9f50bf5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159447342 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159447346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46625/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159412787 **[Test build #46625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)** for PR 9859 at commit [`7802c0c`](https://github.com/apache/spark/commit/7802c0ca3cf4fb6d19180dbda48291d5a9f50bf5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9859#discussion_r45796647 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala --- @@ -167,6 +169,8 @@ private[hive] class ClientWrapper( } else { logInfo("Attempting to login to Kerberos" + s" using principal: ${principalName} and keytab: ${keytabFileName}") +// make sure the core-site.xml is provided in your classpath --- End diff -- Better than this would be to update `docs/sql-programming-guide.md`. It currently only mentions `hive-site.xml`, when really you need `core-site.xml` and `hdfs-site.xml` too (otherwise kerberos, in your case, or things like HDFS HA won't work). The other changes in this file can be reverted, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159408859 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159351434 I had little time today to see this issue, but even if I provide a core-site.xml in my classpath the authentication method in the UGI config is updated to "simple" a one moment (haven't spotted when so far). I need to check it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159367661 I see my problem. Defining the environment variable HADOOP_CONF_DIR is not enough and doesn't work (not read config files there). Copy of configs to the classpath works. Thanks @vanzin . You are right about the hive-site. All in all- I don't need to set additional config, just have keytab properties in YARN mode as well as LOCAL mode. I've made a commit. I plan to read data from tables by `sql` method on a `HiveContext` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159346275 That would be more acceptable; although you already have to provide the Hive configuration if you're accessing the metastore, so for correctness you should be providing everything (otherwise how do you plan to read the data stored in the tables?). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159499824 @vanzin thanks for your support. I've made a commit with documentation update and clean the code as you asked. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159503152 **[Test build #46672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)** for PR 9859 at commit [`9365a7f`](https://github.com/apache/spark/commit/9365a7f8766b081625ed8fe9066cef10df25b198). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159214470 Hi! I checked the code, you are right, by default xmls are loaded. However, in local mode it's cumbersome to attach hadoop config from the cluster. What do you think to set this parameter in local mode only? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159023025 So, is there any code in any of the other backends to actually use those config options? Looking at the bug, isn't the fix actually achieved by just doing the login (`UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159052976 No It's not enough. Please look at the code of the method UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab). There is a check of authentication type. If it's simple (what is default in the classloader context) then parameters are not used. So that I think the recent patch need this fix. I faced the situation I described. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159053620 I know that, but that's not what I'm asking about. I'm asking about all the other code; why do you need to set `spark.yarn.keytab` for other backends? I also agree with Tom that the authentication config value should come from the hadoop configuration, and not be set explicitly like this. If you need to set that, you're using the wrong configuration. Also, I'm a little skeptic that this would work; anybody who has kerberos auth for the Hive metastore probably has the same for HDFS, and while this patch would make it work for local mode, it would not work for Standalone nor Mesos. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159057200 Let's see the situation when I wrap the context of SparkContext with my UGI and even wrap the HiveContext with my UGI then most of Spark functions work with my provided credentials. However, the ClientWrapper object is created in another (the classloader) context, so the UGI provided in application doesn't work anymore. That's why the property has to be set. Is it clear or should I describe it in details? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159010452 Hi! Can you test it on all environments? If not, then I can shrink the scope to the YARN and the local mode. In case of HADOOP_SECURITY_AUTHENTICATION there isn't this parameter in SparkConf, neither in HiveContext. What can you suggest? I think if the keytab is defined then authentication method should be Kerberos, shouldn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-158964727 so Spark has never officially supported this outside of YARN - local mode, standalone, etc. So this isn't a bug, but would be an improvement. --keytab and --principal options are under the YARN only section, documentation is for yarn only, etc.. I'm fine if you want to take this on but I think if you make it more then YARN you need to make sure it works in all the environments or at least the one we intend to support and its all documented, tested, etc. There have been attempts at adding this to standalone in the past but have always had various issues. cc @pwendell as he might remember more about that mode. I also don't like forcing the HADOOP_SECURITY_AUTHENTICATION setting for all environments to kerberos. For YARN this should be picked up from the hadoop configuration if other modes require it thats fine but at the same time if other modes are using hadoop and kerberos perhaps they should pick up the proper hadoop configuration instead of hardcoding it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159059097 Ah, I see. `ClientWrapper` is also reading those properties (which was my original question). Still, I don't think changing the Hadoop configuration is the right thing (as has been pointed out, you're just using the wrong configuration from the get go if you need to do that). Also, in non-local, non-yarn mode, this will just lead to different issues down the line, so those other modes should probably be excluded from this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159060652 Ok, I can prepare a commit with: 1. Shrinking the scope of this feature to YARN or LOCAL modes 2. I'll try to load the "authentication method" property from Hadoop config file specified under a HADOOP_CONF_DIR environment variable or the classpath (file core-site.xml). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159063343 No. I need to load the property, because it's not loaded automatically by `UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`, so it's required to load a keytab properly for objects created in another context than mine (e.g. classloader context for ClientWrapper). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159064233 `UserGroupInformation.loginUserFromKeytab` calls `isSecurityEnabled` which calls `ensureInitialized` which calls `initialize(new Configuration(), false);`. That `new Configuration()` call should read `core-site.xml` from your classpath, which happens when you do either of the things I mentioned above. So perhaps there's some issue with how `ClientWrapper` is reading the configuration, but your solution and your analysis of the underlying problem are not entirely correct. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159061342 > I'll try to load the "authentication method" property from Hadoop config file It's still unclear to me why you need to do that at all. If you set `HADOOP_CONF_DIR` to the right config directory, or place both `core-site.xml` and `hive-site.xml` in `$SPARK_HOME/conf`, wouldn't either of those be enough for the code to pick up the value of the authentication config? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-159064971 I'll check it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on a diff in the pull request: https://github.com/apache/spark/pull/9837#discussion_r45450002 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala --- @@ -166,7 +168,11 @@ private[hive] class ClientWrapper( " specified in spark.yarn.keytab does not exist") } else { logInfo("Attempting to login to Kerberos" + - s" using principal: ${principalName} and keytab: ${keytabFileName}") +s" using principal: ${principalName} and keytab: ${keytabFileName}") +val hadoopConfiguration = new Configuration() + hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, --- End diff -- Hi! I think it is required because of another context of execution in this part of code. Here is a context of the classloader, which cause an empty environment, with default value of authentication, which is "simple". In case of "simple" a keytab is not used and credentials cannot be granted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
GitHub user woj-i opened a pull request: https://github.com/apache/spark/pull/9859 [SPARK-11821] Propagate Kerberos keytab for all environments @andrewor14 the same PR as in branch 1.5 @harishreedharan You can merge this pull request into a Git repository by running: $ git pull https://github.com/woj-i/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9859.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9859 commit 0b55df5c433cd1052793e03eea4022b3f3b5f94b Author: woj-iDate: 2015-11-18T18:14:31Z [SPARK-11821] location of Kerberos keytab in a property for all environments, bugfix of renewing credentials in the Classloader context commit b3b0d59bfb92b86434d673280e43ce1ba3c6c080 Author: woj-i Date: 2015-11-20T09:51:05Z [SPARK-11821] bugfix of renewing Kerberos credentials in context of the classloader --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-158344013 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on a diff in the pull request: https://github.com/apache/spark/pull/9837#discussion_r45495036 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala --- @@ -166,7 +168,11 @@ private[hive] class ClientWrapper( " specified in spark.yarn.keytab does not exist") } else { logInfo("Attempting to login to Kerberos" + - s" using principal: ${principalName} and keytab: ${keytabFileName}") +s" using principal: ${principalName} and keytab: ${keytabFileName}") +val hadoopConfiguration = new Configuration() + hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, --- End diff -- Yes, I did. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/9837#discussion_r45494051 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala --- @@ -166,7 +168,11 @@ private[hive] class ClientWrapper( " specified in spark.yarn.keytab does not exist") } else { logInfo("Attempting to login to Kerberos" + - s" using principal: ${principalName} and keytab: ${keytabFileName}") +s" using principal: ${principalName} and keytab: ${keytabFileName}") +val hadoopConfiguration = new Configuration() + hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, --- End diff -- Ah, ok. Did you hit this issue when running an app? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158491049 @woj-i would you mind tagging this one as `[branch-1.5]` in the title of the PR so it's clearer? @harishreedharan please review #9859 instead, which is the same patch opened against master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-158503831 This is the one parameter needed to update kerberos credentials based on a keytab. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/9859#issuecomment-158498642 LGTM. Are there any other configs required? I remember Hadoop security had a bunch of configs. /cc @tgravescs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user woj-i commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158020139 I also see, that a change of authentication method from a simple to a kerberos was required while renewing credentials. I've made a commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-157995224 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
GitHub user woj-i opened a pull request: https://github.com/apache/spark/pull/9837 [SPARK-11821] Propagate Kerberos keytab for all environments I prepared a patch for recent bugfix. The scope of the previous bugfix is too narrow- it works only on YARN. I need it on local mode and I think the other modes also need the information (because reflection works the same for each environment having JVM). You can merge this pull request into a Git repository by running: $ git pull https://github.com/woj-i/spark branch-1.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9837.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9837 commit 7803fd30be4c14b23611a61216a4560da5604194 Author: woj-iDate: 2015-11-19T09:03:55Z [SPARK-11821] Propagate Kerberos keytab for all environments (not only YARN) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158154158 @woj-i please open the patch against the master branch instead. Committers will backport it into branch-1.5 when they merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9837#issuecomment-158154058 @tgravescs @harishreedharan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/9837#discussion_r45425265 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala --- @@ -166,7 +168,11 @@ private[hive] class ClientWrapper( " specified in spark.yarn.keytab does not exist") } else { logInfo("Attempting to login to Kerberos" + - s" using principal: ${principalName} and keytab: ${keytabFileName}") +s" using principal: ${principalName} and keytab: ${keytabFileName}") +val hadoopConfiguration = new Configuration() + hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, --- End diff -- This seems wrong. If you are talking to hadoop cluster that is secure, your client configuration should already have this. Setting this here alone is not going to help, since all other instances used all over Spark will not have this information (where required). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org