Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77941 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77941/testReport)**
for PR 17723 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17723
Thanks! Will review it tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77822/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77822 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77822/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77820/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77820 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77820/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77822 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77822/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77820 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77820/testReport)**
for PR 17723 at commit
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
Thanks. I updated the description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
You still haven't updated the PR description. It still references things
that you have removed from the PR.
I tested the latest version with our kerberos tests and so far it looks
good.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/9/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #9 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #9 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/testReport)**
for PR 17723 at commit
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin tests are passing. ready for re-review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77761/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77761 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77761/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77761 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77761/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77688/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77688 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77688/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77688 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77688/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77686 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77686/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77686/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77686 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77686/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77683/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77683 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77683/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77683 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77683/testReport)**
for PR 17723 at commit
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin All comments addressed. Ready for re-review (after tests pass).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
Thanks @mridulm. We'll move forward without you on this one. Have a great
vacation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
Hi @mgummelt, I am quite tied up and will be back from vacation July mid -
which is when I will have bandwidth for reviews (which is, IMO, too long to
wait for this PR !).
I see that
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
Perfect, thanks.
@mridulm Any thoughts? He might still be out on vacation...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
I'm super busy with things as usual. But I hope to be able to take a look
at this sometime this week.
I'm also sort of waiting for Mridul to chime in since he's the one that's
super
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
Hey guys, this has been open for > 1 month now and I'd love to get it over
the line. Are there any other concerns or questions I can answer?
---
If your project is set up for it, you can reply
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@gatorsmile @vanzin @mridulm Any thoughts here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@gatorsmile All comments addressed. Ready for re-review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77259/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77259/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77259/testReport)**
for PR 17723 at commit
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77232/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77232 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77232/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77232 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77232/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77212 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77212/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77212/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77212 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77212/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77210 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77210/testReport)**
for PR 17723 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77210/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17723
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77210 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77210/testReport)**
for PR 17723 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77065 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77065/testReport)**
for PR 17723 at commit
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
I have:
* Made ServiceCredentialProvider private
* Made ConfigurableCredentialManager load a hardcoded list of (HDFS, Hive,
HBase) providers, rather than service loading
* Renamed
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17723
**[Test build #77064 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77064/testReport)**
for PR 17723 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
I'm fine with that if you're ok with Mesos being restricted to the built-in
providers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@jerryshao Are those providers different than the Hive and HBase providers
already in the Spark codebase?
Regardless, with what I'm proposing, the `yarn.ServiceCredentialProvider`
would
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17723
@mgummelt We have in house delegation provider for HiveServer2, multi HBase
cluster. I think this is useful in Hadoop world. So better to keep it.
---
If your project is set up for it, you can
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
I'm working on this now, and am definitely willing to execute the plan
we've agreed on, but the more I think about it, the more I think it makes sense
to make `ServiceCredentialProvider` private
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
BTW, if @mgummelt is ok with spending time working on this super generic
interface, then it's up to him. I find it unnecessary and a waste of resources
at this point in time, but it's his time. :-)
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> On the contrary, this is handled if you read the proposal -
ServiceTokenManager.acquireTokens will do the necessary setup
Yes, that's approach (b) which I said still assumes that the
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin
> But it's just a small piece of the puzzle. For example, for Hadoop token
providers, obtainCredentials has to be called with a very specific environment
set up depending on when it's
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> b) Eliminate ConfigurableCredentialManager entirely, and have each
ServiceTokenProvider load their specific sub-interface of
BaseServiceTokenProvider; which gives better type safety.
This
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin:
>> // Modify ConfigurableCredentialManager to segregate by serviceType of
ServiceTokenProvider
> See, you're already making assumptions that these other systems use
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
OK, I'll submit an updated version once I get back from vacation around May
15th.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> I'm saying avoid exposing Hadoop APIs. Wrap them around something if
possible.
If that's all you'd like to see, it's not hard. But at the same time, it
doesn't solve a whole lot of
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin I agree with you, but it looks like the best way to make progress
is to generalize the interface. Are you OK with this?
---
If your project is set up for it, you can reply to this email
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17723
I'm saying avoid exposing Hadoop APIs. Wrap them around something if
possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
> I didn't read through the super long debate here, but I have a strong
preference to not expose Hadoop APIs directly. I'm seeing more and more
deployments out there that do not use Hadoop (e.g.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17723
I didn't read through the super long debate here, but I have a strong
preference to not expose Hadoop APIs directly. I'm seeing more and more
deployments out there that do not use Hadoop (e.g. connect
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
Technically anyone can -1 a change, AFAIK. In practice I see that done very
rarely.
To clarify my position, I see it as counter-productive to try to create an
abstraction now. We don't know
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin @mridulm
I think we've reached the crux of the disagreement. @mridulm thinks we
should generalize ServiceCredentialProvider. @vanzin and I think this would do
more harm than
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17723
I see. Will try to catch the whole history. Thanks! @mgummelt @vanzin
Also cc @jerryshao I just read his [design
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
@gatorsmile Yes. This PR originally contained the Mesos propagation code
as well, but I split it out to simplify the review process (can't you see how
simple it's become!) :)
The old PR
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
@gatorsmile
> Does that mean Mesos plans to use a different solution for delegation
token management?
There's still Mesos code to be added for this; if you trace back through
the JIRA
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> As I detailed above, the exposed spi contract is not just the interface,
but also how it is to be used by spark.
There's very little lee way on that front. Those semantics are mostly
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17723
Based on the PR title `Move kerberos delegation token code from yarn to
core`, it sounds like it moves the whole delegation token logics into core. I
found it does not move the logics for
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
> The only public interface being added in this change is
ServiceCredentialProvider. It's an interface that service-specific libraries
(e.g. a Solr connector, or a Kudu connector) would
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
To ask a more direct question:
The only public interface being added in this change is
`ServiceCredentialProvider`. It's an interface that service-specific libraries
(e.g. a Solr connector,
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
> a) we have explicitly based our support on it
What does this mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> This was my point - we should not introduce system specific api's into
spark core infrastructure api's/spi's
Sorry, I still have no idea what your point is. How do you suggest we
support
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzip wrote:
> So, this is purely about handling Hadoop authentication for Hadoop
services.
This was my point - we should not introduce system specific api's into
spark core
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> Do we need to generalize ServiceCredentialProvider to support non-hadoop
delegation tokens?
I don't see a need for that.
---
If your project is set up for it, you can reply to this email
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> but I am unsure about kubernetes (perusing their documentation left me
unsure) or any others
That's kinda orthogonal. This PR isn't Mesos-specific. Mesos, AFAICT, does
not support
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
I also have to admit that even after all this discussion, I'm still unclear
what the concern is. Hadoop security classes, such as `UGI` are already
exposed to users. For example, through
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
> If I understand from @mgummelt, mesos does not require it
Mesos has its own authn/authz system, but it has nothing to do with this
PR:
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin Thanks for the details and examples, that definitely clarifies
things better for me !
I agree with you that opening it up, even if as unstable api, would be
better than keeping it
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> Support for long running applications (which require token renewal, etc)
was added much later in spark
That's different and not what this change is about. Support for Hadoop
security
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin:
> @mridulm the main argument for just dealing with Hadoop security is that
it's been sufficient since the inception of Spark. I have never seen anyone ask
for integration with any
Github user mgummelt commented on the issue:
https://github.com/apache/spark/pull/17723
> My point is that spark core should not depend on hadoop-security;
@mridulm You keep mentioning `hadoop-security` as if it's a library. It's
not. `UserGroupInformation` and
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
@mridulm the main argument for just dealing with Hadoop security is that
it's been sufficient since the inception of Spark. I have never seen anyone ask
for integration with any other type of
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
Hi @mgummelt, hopefully I have clarified some of my thinking above.
Responding to specific points below.
> It seems the first point of contention is the distinction between Hadoop
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin Looking at it from view of specific API usage, while relevant for
analyzing implementation, is not the whole picture. Is hadoop-security model
sufficient for spark ? Will spark need to
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
It would be easier if instead you said what particular API that is being
proposed here you have issues with. Is it the storing of credentials in UGI? Or
what?
If that's what you're saying.
1 - 100 of 113 matches
Mail list logo
