Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9859
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-161064237
The change doesn't affect pyspark; so merging (master and 1.6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-160841515
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-160841511
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-160841354
**[Test build #46923 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)**
for PR 9859 at commit
[`449cbbb`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-160819668
**[Test build #46923 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46923/consoleFull)**
for PR 9859 at commit
[`449cbbb`](https://gith
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-160819008
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159970309
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159970290
**[Test build #46766 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)**
for PR 9859 at commit
[`449cbbb`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159970308
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159921198
**[Test build #46766 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46766/consoleFull)**
for PR 9859 at commit
[`449cbbb`](https://gith
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159915514
Thank you @vanzin for your help. I commited the nits in the documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159734089
Also, just for posterity, if you're running local mode, you should be able
to `kinit` before running your Spark app, and then you don't need to provide
principal / keytab
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159733870
Just minor nits, otherwise LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9859#discussion_r45920195
--- Diff: docs/sql-programming-guide.md ---
@@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes
Hive. Note that this Hive a
on all
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9859#discussion_r45920176
--- Diff: docs/sql-programming-guide.md ---
@@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes
Hive. Note that this Hive a
on all
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9859#discussion_r45920123
--- Diff: docs/running-on-yarn.md ---
@@ -349,14 +349,14 @@ If you need a reference to the proper location to put
log files in the YARN so t
The
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9859#discussion_r45920128
--- Diff: docs/running-on-yarn.md ---
@@ -349,14 +349,14 @@ If you need a reference to the proper location to put
log files in the YARN so t
The
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159530732
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159530547
**[Test build #46672 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)**
for PR 9859 at commit
[`9365a7f`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159530735
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159503152
**[Test build #46672 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46672/consoleFull)**
for PR 9859 at commit
[`9365a7f`](https://gith
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159499824
@vanzin thanks for your support. I've made a commit with documentation
update and clean the code as you asked.
---
If your project is set up for it, you can reply to this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159447346
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159447342
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159447241
**[Test build #46625 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)**
for PR 9859 at commit
[`7802c0c`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159412787
**[Test build #46625 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46625/consoleFull)**
for PR 9859 at commit
[`7802c0c`](https://gith
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159408859
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enab
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9859#discussion_r45796647
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -167,6 +169,8 @@ private[hive] class ClientWrapper(
}
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159367661
I see my problem. Defining the environment variable HADOOP_CONF_DIR is not
enough and doesn't work (not read config files there). Copy of configs to the
classpath works. T
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159351434
I had little time today to see this issue, but even if I provide a
core-site.xml in my classpath the authentication method in the UGI config is
updated to "simple" a one m
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159346275
That would be more acceptable; although you already have to provide the
Hive configuration if you're accessing the metastore, so for correctness you
should be providing e
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159214470
Hi!
I checked the code, you are right, by default xmls are loaded. However, in
local mode it's cumbersome to attach hadoop config from the cluster. What do
you think t
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159064971
I'll check it tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159064233
`UserGroupInformation.loginUserFromKeytab` calls `isSecurityEnabled` which
calls `ensureInitialized` which calls `initialize(new Configuration(), false);`.
That `
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159063343
No. I need to load the property, because it's not loaded automatically by
`UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)`, so
it's required to load
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159061342
> I'll try to load the "authentication method" property from Hadoop config
file
It's still unclear to me why you need to do that at all. If you set
`HADOOP_CONF_
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159060652
Ok, I can prepare a commit with:
1. Shrinking the scope of this feature to YARN or LOCAL modes
2. I'll try to load the "authentication method" property from Hadoop c
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159059097
Ah, I see. `ClientWrapper` is also reading those properties (which was my
original question). Still, I don't think changing the Hadoop configuration is
the right thing (a
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159057200
Let's see the situation when I wrap the context of SparkContext with my UGI
and even wrap the HiveContext with my UGI then most of Spark functions work
with my provided cr
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159053620
I know that, but that's not what I'm asking about. I'm asking about all the
other code; why do you need to set `spark.yarn.keytab` for other backends?
I also agre
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159052976
No It's not enough. Please look at the code of the method
UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab). There is
a check of authentication type. I
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159023025
So, is there any code in any of the other backends to actually use those
config options?
Looking at the bug, isn't the fix actually achieved by just doing the log
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-159010452
Hi!
Can you test it on all environments? If not, then I can shrink the scope to
the YARN and the local mode.
In case of HADOOP_SECURITY_AUTHENTICATION there is
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-158964727
so Spark has never officially supported this outside of YARN - local mode,
standalone, etc. So this isn't a bug, but would be an improvement.
--keytab and --pr
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-158503831
This is the one parameter needed to update kerberos credentials based on a
keytab.
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-158498642
LGTM. Are there any other configs required? I remember Hadoop security had
a bunch of configs.
/cc @tgravescs
---
If your project is set up for it, yo
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9837#issuecomment-158491049
@woj-i would you mind tagging this one as `[branch-1.5]` in the title of
the PR so it's clearer?
@harishreedharan please review #9859 instead, which is the same pa
Github user woj-i commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45495036
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45494051
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9859#issuecomment-158344013
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
GitHub user woj-i opened a pull request:
https://github.com/apache/spark/pull/9859
[SPARK-11821] Propagate Kerberos keytab for all environments
@andrewor14 the same PR as in branch 1.5
@harishreedharan
You can merge this pull request into a Git repository by running:
$ git
Github user woj-i commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45450002
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9837#discussion_r45425265
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -166,7 +168,11 @@ private[hive] class ClientWrapper(
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9837#issuecomment-158154158
@woj-i please open the patch against the master branch instead. Committers
will backport it into branch-1.5 when they merge it.
---
If your project is set up for it,
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9837#issuecomment-158154058
@tgravescs @harishreedharan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user woj-i commented on the pull request:
https://github.com/apache/spark/pull/9837#issuecomment-158020139
I also see, that a change of authentication method from a simple to a
kerberos was required while renewing credentials. I've made a commit.
---
If your project is set up
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9837#issuecomment-157995224
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
GitHub user woj-i opened a pull request:
https://github.com/apache/spark/pull/9837
[SPARK-11821] Propagate Kerberos keytab for all environments
I prepared a patch for recent bugfix. The scope of the previous bugfix is
too narrow- it works only on YARN. I need it on local mode and I
59 matches
Mail list logo
