[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-161064237
  
The change doesn't affect pyspark; so merging (master and 1.6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9859


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-160819668
  
**[Test build #46923 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)**
 for PR 9859 at commit 
[`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-160841511
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-160841354
  
**[Test build #46923 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)**
 for PR 9859 at commit 
[`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-160841515
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46923/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-160819008
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159915514
  
Thank you @vanzin for your help. I commited the nits in the documentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159921198
  
**[Test build #46766 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)**
 for PR 9859 at commit 
[`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159970308
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159970290
  
**[Test build #46766 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)**
 for PR 9859 at commit 
[`449cbbb`](https://github.com/apache/spark/commit/449cbbb61bf25781c5b4bb4f1b2ca62da71307fd).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159970309
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46766/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159734089
  
Also, just for posterity, if you're running local mode, you should be able 
to `kinit` before running your Spark app, and then you don't need to provide 
principal / keytab at all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9859#discussion_r45920128
  
--- Diff: docs/running-on-yarn.md ---
@@ -349,14 +349,14 @@ If you need a reference to the proper location to put 
log files in the YARN so t
   
   The full path to the file that contains the keytab for the principal 
specified above.
   This keytab will be copied to the node running the YARN Application 
Master via the Secure Distributed Cache,
-  for renewing the login tickets and the delegation tokens periodically.
+  for renewing the login tickets and the delegation tokens periodically. 
(Works also in `LOCAL` mode)
   
 
 
   spark.yarn.principal
   (none)
   
-  Principal to be used to login to KDC, while running on secure HDFS.
+  Principal to be used to login to KDC, while running on secure HDFS. 
(Works also in `LOCAL` mode)
--- End diff --

Same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159733870
  
Just minor nits, otherwise LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9859#discussion_r45920123
  
--- Diff: docs/running-on-yarn.md ---
@@ -349,14 +349,14 @@ If you need a reference to the proper location to put 
log files in the YARN so t
   
   The full path to the file that contains the keytab for the principal 
specified above.
   This keytab will be copied to the node running the YARN Application 
Master via the Secure Distributed Cache,
-  for renewing the login tickets and the delegation tokens periodically.
+  for renewing the login tickets and the delegation tokens periodically. 
(Works also in `LOCAL` mode)
--- End diff --

nit: Works also with the "local" master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9859#discussion_r45920176
  
--- Diff: docs/sql-programming-guide.md ---
@@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes 
Hive. Note that this Hive a
 on all of the worker nodes, as they will need access to the Hive 
serialization and deserialization libraries
 (SerDes) in order to access data stored in Hive.
 
-Configuration of Hive is done by placing your `hive-site.xml` file in 
`conf/`. Please note when running
+Configuration of Hive is done by placing your `hive-site.xml`, 
`core-site.xml` (in order to security parameters,
--- End diff --

nit: "for security configuration"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9859#discussion_r45920195
  
--- Diff: docs/sql-programming-guide.md ---
@@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes 
Hive. Note that this Hive a
 on all of the worker nodes, as they will need access to the Hive 
serialization and deserialization libraries
 (SerDes) in order to access data stored in Hive.
 
-Configuration of Hive is done by placing your `hive-site.xml` file in 
`conf/`. Please note when running
+Configuration of Hive is done by placing your `hive-site.xml`, 
`core-site.xml` (in order to security parameters,
+ like Kerberos), `hdfs-site.xml` (parameters of HDFS HA, etc.) file in 
`conf/`. Please note when running
--- End diff --

nit: "for HDFS configuration"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159530732
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159530547
  
**[Test build #46672 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)**
 for PR 9859 at commit 
[`9365a7f`](https://github.com/apache/spark/commit/9365a7f8766b081625ed8fe9066cef10df25b198).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159530735
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46672/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159447241
  
**[Test build #46625 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)**
 for PR 9859 at commit 
[`7802c0c`](https://github.com/apache/spark/commit/7802c0ca3cf4fb6d19180dbda48291d5a9f50bf5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159447342
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159447346
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46625/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159412787
  
**[Test build #46625 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)**
 for PR 9859 at commit 
[`7802c0c`](https://github.com/apache/spark/commit/7802c0ca3cf4fb6d19180dbda48291d5a9f50bf5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9859#discussion_r45796647
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -167,6 +169,8 @@ private[hive] class ClientWrapper(
   } else {
 logInfo("Attempting to login to Kerberos" +
   s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+// make sure the core-site.xml is provided in your classpath
--- End diff --

Better than this would be to update `docs/sql-programming-guide.md`. It 
currently only mentions `hive-site.xml`, when really you need `core-site.xml` 
and `hdfs-site.xml` too (otherwise kerberos, in your case, or things like HDFS 
HA won't work).

The other changes in this file can be reverted, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159408859
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159351434
  
I had little time today to see this issue, but even if I provide a 
core-site.xml in my classpath the authentication method in the UGI config is 
updated to "simple" a one moment (haven't spotted when so far). I need to check 
it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159367661
  
I see my problem. Defining the environment variable HADOOP_CONF_DIR is not 
enough and doesn't work (not read config files there). Copy of configs to the 
classpath works. Thanks @vanzin . You are right about the hive-site. All in 
all- I don't need to set additional config, just have keytab properties in YARN 
mode as well as LOCAL mode. I've made a commit.

I plan to read data from tables by `sql` method on a `HiveContext`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159346275
  
That would be more acceptable; although you already have to provide the 
Hive configuration if you're accessing the metastore, so for correctness you 
should be providing everything (otherwise how do you plan to read the data 
stored in the tables?).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159499824
  
@vanzin thanks for your support. I've made a commit with documentation 
update and clean the code as you asked.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159503152
  
**[Test build #46672 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)**
 for PR 9859 at commit 
[`9365a7f`](https://github.com/apache/spark/commit/9365a7f8766b081625ed8fe9066cef10df25b198).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159214470
  
Hi!
I checked the code, you are right, by default xmls are loaded. However, in 
local mode it's cumbersome to attach hadoop config from the cluster. What do 
you think to set this parameter in local mode only?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159023025
  
So, is there any code in any of the other backends to actually use those 
config options?

Looking at the bug, isn't the fix actually achieved by just doing the login 
(`UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159052976
  
No It's not enough. Please look at the code of the method 
UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab). There is 
a check of authentication type. If it's simple (what is default in the 
classloader context) then parameters are not used. So that I think the recent 
patch need this fix. I faced the situation I described. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159053620
  
I know that, but that's not what I'm asking about. I'm asking about all the 
other code; why do you need to set `spark.yarn.keytab` for other backends?

I also agree with Tom that the authentication config value should come from 
the hadoop configuration, and not be set explicitly like this. If you need to 
set that, you're using the wrong configuration.

Also, I'm a little skeptic that this would work; anybody who has kerberos 
auth for the Hive metastore probably has the same for HDFS, and while this 
patch would make it work for local mode, it would not work for Standalone nor 
Mesos.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159057200
  
Let's see the situation when I wrap the context of SparkContext with my UGI 
and even wrap the HiveContext with my UGI then most of Spark functions work 
with my provided credentials. However, the ClientWrapper object is created in 
another (the classloader) context, so the UGI provided in application doesn't 
work anymore. That's why the property has to be set. Is it clear or should I 
describe it in details?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159010452
  
Hi!
Can you test it on all environments? If not, then I can shrink the scope to 
the YARN and the local mode.

In case of HADOOP_SECURITY_AUTHENTICATION there isn't this parameter in 
SparkConf, neither in HiveContext. What can you suggest? I think if the keytab 
is defined then authentication method should be Kerberos, shouldn't it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-158964727
  
so Spark has never officially supported this outside of YARN - local mode, 
standalone, etc. So this isn't a bug, but would be an improvement.

--keytab and --principal options are under the YARN only section, 
documentation is for yarn only, etc..

I'm fine if you want to take this on but I think if you make it more then 
YARN you need to make sure it works in all the environments or at least the one 
we intend to support and its all documented, tested, etc. 

There have been attempts at adding this to standalone in the past but have 
always had various issues.  cc @pwendell  as he might remember more about that 
mode.

I also don't like forcing the HADOOP_SECURITY_AUTHENTICATION setting for 
all environments to kerberos.  For YARN this should be picked up from the 
hadoop configuration if other modes require it thats fine but at the same time 
if other modes are using hadoop and kerberos perhaps they should pick up the 
proper hadoop configuration instead of hardcoding it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159059097
  
Ah, I see. `ClientWrapper` is also reading those properties (which was my 
original question). Still, I don't think changing the Hadoop configuration is 
the right thing (as has been pointed out, you're just using the wrong 
configuration from the get go if you need to do that).

Also, in non-local, non-yarn mode, this will just lead to different issues 
down the line, so those other modes should probably be excluded from this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159060652
  
Ok, I can prepare a commit with:
1. Shrinking the scope of this feature to YARN or LOCAL modes
2. I'll try to load the "authentication method" property from Hadoop config 
file specified under a HADOOP_CONF_DIR environment variable or the classpath 
(file core-site.xml).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159063343
  
No. I need to load the property, because it's not loaded automatically by 
`UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`, so 
it's required to load a keytab properly for objects created in another context 
than mine (e.g. classloader context for ClientWrapper).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159064233
  
`UserGroupInformation.loginUserFromKeytab` calls `isSecurityEnabled` which 
calls `ensureInitialized` which calls `initialize(new Configuration(), false);`.

That `new Configuration()` call should read `core-site.xml` from your 
classpath, which happens when you do either of the things I mentioned above. So 
perhaps there's some issue with how `ClientWrapper` is reading the 
configuration, but your solution and your analysis of the underlying problem 
are not entirely correct.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159061342
  
> I'll try to load the "authentication method" property from Hadoop config 
file

It's still unclear to me why you need to do that at all. If you set 
`HADOOP_CONF_DIR` to the right config directory, or place both `core-site.xml` 
and `hive-site.xml` in `$SPARK_HOME/conf`, wouldn't either of those be enough 
for the code to pick up the value of the authentication config?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-159064971
  
I'll check it tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45450002
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

Hi!
I think it is required because of another context of execution in this part 
of code. Here is a context of the classloader, which cause an empty 
environment, with default value of authentication, which is "simple". In case 
of "simple" a keytab is not used and credentials cannot be granted. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

GitHub user woj-i opened a pull request:

https://github.com/apache/spark/pull/9859

[SPARK-11821] Propagate Kerberos keytab for all environments

@andrewor14 the same PR as in branch 1.5
@harishreedharan 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/woj-i/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9859


commit 0b55df5c433cd1052793e03eea4022b3f3b5f94b
Author: woj-i 
Date:   2015-11-18T18:14:31Z

[SPARK-11821] location of Kerberos keytab in a property for all 
environments, bugfix of renewing credentials in the Classloader context

commit b3b0d59bfb92b86434d673280e43ce1ba3c6c080
Author: woj-i 
Date:   2015-11-20T09:51:05Z

[SPARK-11821] bugfix of renewing Kerberos credentials in context of the 
classloader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-158344013
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45495036
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

Yes, I did.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[harishreedharan]

Github user harishreedharan commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45494051
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

Ah, ok. Did you hit this issue when running an app?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/9837#issuecomment-158491049
  
@woj-i would you mind tagging this one as `[branch-1.5]` in the title of 
the PR so it's clearer?
@harishreedharan please review #9859 instead, which is the same patch 
opened against master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-158503831
  
This is the one parameter needed to update kerberos credentials based on a 
keytab.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/9859#issuecomment-158498642
  
LGTM. Are there any other configs required? I remember Hadoop security had 
a bunch of configs.

/cc @tgravescs 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

Github user woj-i commented on the pull request:

https://github.com/apache/spark/pull/9837#issuecomment-158020139
  
I also see, that a change of authentication method from a simple to a 
kerberos was required while renewing credentials. I've made a commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9837#issuecomment-157995224
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[woj-i]

GitHub user woj-i opened a pull request:

https://github.com/apache/spark/pull/9837

[SPARK-11821] Propagate Kerberos keytab for all environments

I prepared a patch for recent bugfix. The scope of the previous bugfix is 
too narrow- it works only on YARN. I need it on local mode and I think the 
other modes also need the information (because reflection works the same for 
each environment having JVM).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/woj-i/spark branch-1.5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9837.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9837


commit 7803fd30be4c14b23611a61216a4560da5604194
Author: woj-i 
Date:   2015-11-19T09:03:55Z

[SPARK-11821] Propagate Kerberos keytab for all environments (not only YARN)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/9837#issuecomment-158154158
  
@woj-i please open the patch against the master branch instead. Committers 
will backport it into branch-1.5 when they merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/9837#issuecomment-158154058
  
@tgravescs @harishreedharan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11821] Propagate Kerberos keytab for al...

[harishreedharan]

Github user harishreedharan commented on a diff in the pull request:

https://github.com/apache/spark/pull/9837#discussion_r45425265
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
   " specified in spark.yarn.keytab does not exist")
   } else {
 logInfo("Attempting to login to Kerberos" +
-  s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+s" using principal: ${principalName} and keytab: 
${keytabFileName}")
+val hadoopConfiguration = new Configuration()
+
hadoopConfiguration.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
--- End diff --

This seems wrong. If you are talking to hadoop cluster that is secure, your 
client configuration should already have this. Setting this here alone is not 
going to help, since all other instances used all over Spark will not have this 
information (where required).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org