[GitHub] spark pull request: SPARK-11265 hive tokens
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150726454 @steveloughran could you use PR titles following the convention described in the following document? https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-PullRequest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150704233 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150704237 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44264/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150704157 **[Test build #44264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44264/consoleFull)** for PR 9232 at commit [`b417ca3`](https://github.com/apache/spark/commit/b417ca32d16edbb7c305d1c3922278999e302137). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701247 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701248 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44260/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701221 **[Test build #44264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44264/consoleFull)** for PR 9232 at commit [`b417ca3`](https://github.com/apache/spark/commit/b417ca32d16edbb7c305d1c3922278999e302137). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701155 **[Test build #44260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44260/consoleFull)** for PR 9232 at commit [`bc6ea26`](https://github.com/apache/spark/commit/bc6ea26a139d377dcb8fbb27a02284b0b27454cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701020 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150701035 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150700891 I've tightened this up and am happy with the code now, except I'm not sure that NoSuchMethodExceptions should be downgraded. It will hide the situation of incompatible Hive version on the CP yet principal required to talk to the hive metastore. Provided the spark app doesn't want to talk to hive, it's not going to matter. But if the app does want't to talk to hive, there'll be no cue except for a log message at launch time, and failures in executors whenever they try to set up an RPC connection. Leaving it as is retains backwards compatibility though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150697465 **[Test build #44260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44260/consoleFull)** for PR 9232 at commit [`bc6ea26`](https://github.com/apache/spark/commit/bc6ea26a139d377dcb8fbb27a02284b0b27454cf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150695732 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150695706 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150693686 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150693689 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44257/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150693568 **[Test build #44257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44257/consoleFull)** for PR 9232 at commit [`9713cf5`](https://github.com/apache/spark/commit/9713cf5c2f69a07f8ed03d64a0cd3956350ed232). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150688788 **[Test build #44257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44257/consoleFull)** for PR 9232 at commit [`9713cf5`](https://github.com/apache/spark/commit/9713cf5c2f69a07f8ed03d64a0cd3956350ed232). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150688417 Updated patch 1. Moves all arg validation up before any hive class reflection 1. gets all hive class methods before attempting to call any 1. calls `Hive.closeCurrent()` in a finally block (if it could be loaded); catches and logs exceptions @ warn With this load-methods-first strategy, the `closeCurrent` variable will always contain a value prior to `Hive.get()` being invoked. As a result, `Hive.closeCurrent()` will always be invoked after `Hive.get()` method has been invoked (irrespective of outcome), so is guaranteed to trigger a cleanup once a Hive instance was created. Previously, any failure of `Hive.getDelegationToken()` would skip the cleanup. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150684298 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150684229 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150662642 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44244/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150662640 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150662508 **[Test build #44244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44244/consoleFull)** for PR 9232 at commit [`5158afe`](https://github.com/apache/spark/commit/5158afe1e15966f0eb94b67bd097709d7b333606). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150658089 Patch with revert posted For reviewers, here some things to consider 1. Which exceptions should trigger log & skip for token, vs which are critical enough to fail the process. The patch ignores no classpath found (`NoClassDefFoundError` and `ClassNotFoundException`) (i.e. hive not on the CP), but also `NoSuchMethodException`. Which means that even if the security settings require a token, if there's some method incompatibility, it will be ignored. 1. the Hadoop config now passed down is used to help define the hive configuration used in the binding. This is needed for testing. Previously, an empty `HiveConf` instance was created: enough to force in the hive-default.xml and hive-site.xml, but not able to pick on any custom options. This is a change: does it constitute a regression, or a feature? Also, I've just realised the final`closeCurrent()` call must be in a finally(); will add --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150657934 **[Test build #44244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44244/consoleFull)** for PR 9232 at commit [`5158afe`](https://github.com/apache/spark/commit/5158afe1e15966f0eb94b67bd097709d7b333606). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r42898469
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1337,55 +1337,8 @@ object Client extends Logging {
conf: Configuration,
credentials: Credentials) {
if (shouldGetTokens(sparkConf, "hive") &&
UserGroupInformation.isSecurityEnabled) {
--- End diff --
Actually I don't really think the delegation tokens are needed in client
mode (for Hive only). Only the driver talks to the metastore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150655380 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9232#issuecomment-150655438 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r42897746
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1337,55 +1337,8 @@ object Client extends Logging {
conf: Configuration,
credentials: Credentials) {
if (shouldGetTokens(sparkConf, "hive") &&
UserGroupInformation.isSecurityEnabled) {
--- End diff --
As @tgravescs points out, the tokens are needed throughout the cluster, and
yes, must be obtained irrespective of deployment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r42897540
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -322,8 +322,10 @@ private[spark] class Client(
// multiple times, YARN will fail to launch containers for the app
with an internal
// error.
val distributedUris = new HashSet[String]
-obtainTokenForHiveMetastore(sparkConf, hadoopConf, credentials)
-obtainTokenForHBase(sparkConf, hadoopConf, credentials)
+if (isClusterMode) {
--- End diff --
suggestion from @dougb , but yes, I didn't think that through. It's not for
the driver, its for the nodes. Will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11265 hive tokens
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r42881320
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -322,8 +322,10 @@ private[spark] class Client(
// multiple times, YARN will fail to launch containers for the app
with an internal
// error.
val distributedUris = new HashSet[String]
-obtainTokenForHiveMetastore(sparkConf, hadoopConf, credentials)
-obtainTokenForHBase(sparkConf, hadoopConf, credentials)
+if (isClusterMode) {
--- End diff --
why is this cluster mode only? I can run spark shell to access hive or
hbase and this won't get tokens for those to ship to executors?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
