[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8218 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38580853
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
--- End diff --
That would be true, but I don't believe you can actually set things up like
that. Authentication is either enabled on the server or its not, for all
clients.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137236868 @vanzin sorry I couldn't review this in time. While looking through the code I was wondering about two things: (1) See my inline comment (2) Looks like the app ID is relatively easy to spoof? Aren't they listed in the standalone Master / RM UI? Should we use something that's more like a secret, like what we do in `YarnShuffleService`? The only problem there is that in standalone mode I'm not sure if we can securely pass secrets from drivers to executors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38579943
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
--- End diff --
I'm confused about this: since `getClientId` returns null if the client did
not enable `spark.authenticate`, does that mean any application that did not
enable SASL can read my shuffle files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137225646 These pyspark tests fail on every other PR. I'll merge this since I haven't seen any more feedback. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137238198 Hi @andrewor14, This patch is not very useful outside YARN. On Standalone, all apps run as the same user, and authenticate using the same user and secret. So there's no way to prevent one app from reading another's shuffle files (either through the shuffle service or reading them directly from disk). On YARN, each app authenticates itself using the app's ID as the user name, and a secure, per-app secret (see `SecurityManager::generateSecretKey`). Authentication is not based on the app simply saying who it is - the app needs to know that secret. After the SASL auth occurs, then we just do simple matching of what app the shuffle file belongs to and what app the connection was authenticated as. You can't spoof it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38587368
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
--- End diff --
Correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137257022 > it seems then that there are two kinds of authentication No, there's one kind of authentication. I don't know what's this "handshake" you talk about. Whenever you open a connection to read shuffle blocks and `spark.authenticate` is enabled (no matter whether you are connecting to a block manager directly, or to an external shuffle service), you will perform SASL authentication and provide the user name (= app id) and secret for your application. The secret is already distributed securely on YARN; it's stashed in the credentials held by the `UserGroupInformation` object used to start the application. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38595554
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
+ StreamState state = streams.get(streamId);
+ Preconditions.checkArgument(state != null, "Unknown stream ID.");
+ if (!client.getClientId().equals(state.appId)) {
+throw new SecurityException(String.format(
+ "Client %s not authorized to read stream %d (app %s).",
--- End diff --
Should we not disclose the actual appId in the exception message ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8218#issuecomment-137267685
Yes, and if you look at that code, that secret is the return value of
`SecurityManager.getSecretKey()`, which on YARN is stored in the
`UserGroupInformation` object. Here's the code to make it clear:
val secretKey =
SparkHadoopUtil.get.getSecretKeyFromUserCredentials(sparkSecretLookupKey)
if (secretKey != null) {
logDebug("in yarn mode, getting secret from credentials")
return new Text(secretKey).toString
} else {
logDebug("getSecretKey: yarn mode, secret key from credentials is
null")
}
val cookie = akka.util.Crypt.generateSecureCookie
// if we generated the secret then we must be the first so lets set
it so t
// gets used by everyone else
SparkHadoopUtil.get.addSecretKeyToUserCredentials(sparkSecretLookupKey, cookie)
Is the secret doesn't yet exist (i.e. before the app is submitted), then a
new one is created and stashed in the user's credentials. If it already exists
(e.g. for the AM and all executors), the it's used. Authentication works fine
just as it was originally designed. This patch is not about authentication.
It's about authorization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137252475 I see, so it seems then that there are two kinds of authentication, one during the handshake where we use some secure secret (i.e. the one used passed from `ExecutorRunnable` to `YarnShuffleService`), and another for reading each shuffle block where we use the app ID. For the latter, could we use the same shuffle secret instead of the app ID? That would require us to pass the secret to the executor JVM securely, which could be difficult. This patch is already a strict improvement as is, so I'm just wondering whether it's worth it to strengthen the security guarantees further. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38586992
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
--- End diff --
I see, do you mean if the server enabled authentication, then any client
that did not also enable it will fail the handshake in the first place?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-137266073 > No, there's one kind of authentication. I don't know what's this "handshake" you talk about. What? I'm talking about the shuffle secret the executor container needs when it initially registers with the shuffle service: https://github.com/apache/spark/blob/6cd98c1878a9c5c6475ed5974643021ab27862a7/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L116. The whole motivation for adding that in the first place is for app-level authentication during the registration. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38595606
--- Diff:
network/common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java
---
@@ -109,15 +111,34 @@ public void connectionTerminated(Channel channel) {
}
}
+ @Override
+ public void checkAuthorization(TransportClient client, long streamId) {
+if (client.getClientId() != null) {
+ StreamState state = streams.get(streamId);
+ Preconditions.checkArgument(state != null, "Unknown stream ID.");
+ if (!client.getClientId().equals(state.appId)) {
+throw new SecurityException(String.format(
+ "Client %s not authorized to read stream %d (app %s).",
--- End diff --
App IDs are not secret.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136434485 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136434540 @aarondav do you have any comments to add? Otherwise I really want to merge this soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136435282 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136435247 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136438158 [Test build #41835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41835/consoleFull) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136489638 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41835/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136489559 [Test build #41835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41835/console) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136489635 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136520566 I triggered the tests again. I think for this one, since it is so early in the release cycle for 1.6, we can also optimistically merge it for now and do post-hoc reviews, provided that @vanzin is not going to disappear :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-136520802 [Test build #1707 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1707/consoleFull) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-13657 [Test build #1707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1707/console) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135945729 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135945713 [Test build #41777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41777/console) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135945731 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41777/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135643130 Looks alright to me - would be good if @aarondav takes a look too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r38173030 --- Diff: network/common/src/main/java/org/apache/spark/network/client/TransportClient.java --- @@ -70,6 +70,7 @@ private final Channel channel; private final TransportResponseHandler handler; + private String clientId; --- End diff -- Can we use an Optional here, or annotate it as @Nullable? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38173055
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -85,6 +86,25 @@ public SocketAddress getSocketAddress() {
}
/**
+ * Returns the ID used by the client to authenticate itself when
authentication is enabled.
+ *
+ * @return The client ID, or null if authentication is disabled.
--- End diff --
this can also be null if clientId hasn't been set yet right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135933770 [Test build #41777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41777/consoleFull) for PR 8218 at commit [`b491ac7`](https://github.com/apache/spark/commit/b491ac7f44880208a5e22d3dea4194eff6c53590). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135933055 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135933058 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r38239163
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -85,6 +86,25 @@ public SocketAddress getSocketAddress() {
}
/**
+ * Returns the ID used by the client to authenticate itself when
authentication is enabled.
+ *
+ * @return The client ID, or null if authentication is disabled.
--- End diff --
Technically yes, but I'm pretty sure code cannot get a `TransportClient`
handle before the auth bootstraps have run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135637236 ping @aarondav --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-135215117 Ping? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134684632 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134684652 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134685164 [Test build #41540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41540/consoleFull) for PR 8218 at commit [`d25c6cf`](https://github.com/apache/spark/commit/d25c6cf93566d70a9d4c811044cfe41c98a8f273). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134695334 @pwendell here's an example of more timeouts; last timed out build took 154m just for Java / Scala tests; the failed build above took 124m for the same tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134731994 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134731996 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41540/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134731860 [Test build #41540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41540/console) for PR 8218 at commit [`d25c6cf`](https://github.com/apache/spark/commit/d25c6cf93566d70a9d4c811044cfe41c98a8f273). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134303181 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134304652 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134304695 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134306161 [Test build #41460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41460/consoleFull) for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134363520 **[Test build #41460 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41460/console)** for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134363628 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41460/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134363624 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134427069 [Test build #41481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41481/console) for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134427109 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41481/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134396675 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134427108 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134397827 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134397800 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-134398249 [Test build #41481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41481/consoleFull) for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133564829 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133564844 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133565347 [Test build #41386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41386/consoleFull) for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133593795 **[Test build #41386 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41386/console)** for PR 8218 at commit [`8153497`](https://github.com/apache/spark/commit/81534974f798e2d3b8bfbec922ddee12a07857d2) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133593857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41386/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133106018 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133106078 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133107601 [Test build #41329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41329/consoleFull) for PR 8218 at commit [`4d19ed5`](https://github.com/apache/spark/commit/4d19ed51616ff6019bcb3b762d82823ced49b675). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133169345 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133168977 [Test build #41329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41329/console) for PR 8218 at commit [`4d19ed5`](https://github.com/apache/spark/commit/4d19ed51616ff6019bcb3b762d82823ced49b675). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133169349 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41329/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132320704 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132320734 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132321977 [Test build #41156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41156/consoleFull) for PR 8218 at commit [`3cc9321`](https://github.com/apache/spark/commit/3cc9321bcd048242a985221a7a5656bd93a832ef). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132369820 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41156/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132369642 [Test build #41156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41156/console) for PR 8218 at commit [`3cc9321`](https://github.com/apache/spark/commit/3cc9321bcd048242a985221a7a5656bd93a832ef). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132369817 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-132418179 Let's try pinging a couple of people. @rxin @aarondav --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131969424 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131969443 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131970027 [Test build #41057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41057/consoleFull) for PR 8218 at commit [`fadff27`](https://github.com/apache/spark/commit/fadff2790d765fba458999fb3952d626fa15214b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131982761 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41049/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131982691 [Test build #41049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41049/console) for PR 8218 at commit [`292a299`](https://github.com/apache/spark/commit/292a2995a0702f92111e07d316a740574da7db4e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131982759 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131997859 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131997860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41057/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131997808 [Test build #41057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41057/console) for PR 8218 at commit [`fadff27`](https://github.com/apache/spark/commit/fadff2790d765fba458999fb3952d626fa15214b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r37239534
--- Diff:
network/shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java
---
@@ -39,44 +46,41 @@
import org.apache.spark.network.server.StreamManager;
import org.apache.spark.network.server.TransportServer;
import org.apache.spark.network.server.TransportServerBootstrap;
+import org.apache.spark.network.shuffle.BlockFetchingListener;
import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler;
+import org.apache.spark.network.shuffle.ExternalShuffleBlockResolver;
+import org.apache.spark.network.shuffle.OneForOneBlockFetcher;
+import org.apache.spark.network.shuffle.protocol.BlockTransferMessage;
+import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo;
+import org.apache.spark.network.shuffle.protocol.OpenBlocks;
+import org.apache.spark.network.shuffle.protocol.RegisterExecutor;
+import org.apache.spark.network.shuffle.protocol.StreamHandle;
import org.apache.spark.network.util.SystemPropertyConfigProvider;
import org.apache.spark.network.util.TransportConf;
public class SaslIntegrationSuite {
- static ExternalShuffleBlockHandler handler;
+ private final Logger logger =
LoggerFactory.getLogger(SaslIntegrationSuite.class);
--- End diff --
nit: unused, delete (and then the imports too)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/8218#discussion_r37239500 --- Diff: network/shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java --- @@ -19,17 +19,24 @@ import java.io.IOException; import java.util.Arrays; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; --- End diff -- nit: unused, delete --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131946775 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131946794 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131947298 [Test build #41049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41049/consoleFull) for PR 8218 at commit [`292a299`](https://github.com/apache/spark/commit/292a2995a0702f92111e07d316a740574da7db4e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r37155642
--- Diff:
network/shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java
---
@@ -160,6 +164,111 @@ public void testNoSaslServer() {
}
}
+ /**
+ * This test is not actually testing SASL behavior, but testing that the
shuffle service
+ * performs correct authorization checks based on the SASL
authentication data.
+ */
+ @Test
+ public void testAppIsolation() throws Exception {
+// Start a new server with the correct RPC handler to serve block data.
+ExternalShuffleBlockResolver blockResolver =
mock(ExternalShuffleBlockResolver.class);
+ExternalShuffleBlockHandler blockHandler = new
ExternalShuffleBlockHandler(
+ new OneForOneStreamManager(), blockResolver);
+TransportServerBootstrap bootstrap = new SaslServerBootstrap(conf,
secretKeyHolder);
+TransportContext blockServerContext = new TransportContext(conf,
blockHandler);
+TransportServer blockServer =
blockServerContext.createServer(Arrays.asList(bootstrap));
+
+TransportClient client1 = null;
+TransportClient client2 = null;
+TransportClientFactory clientFactory2 = null;
+try {
+ // Create a client, and make a request to fetch blocks from a
different app.
+ clientFactory = blockServerContext.createClientFactory(
+Lists.TransportClientBootstrapnewArrayList(
+ new SaslClientBootstrap(conf, app-1, secretKeyHolder)));
+ client1 = clientFactory.createClient(TestUtils.getLocalHost(),
+blockServer.getPort());
+
+ final AtomicBoolean result = new AtomicBoolean(false);
+
+ BlockFetchingListener listener = new BlockFetchingListener() {
+@Override
+public synchronized void onBlockFetchSuccess(String blockId,
ManagedBuffer data) {
+ notifyAll();
+}
+
+@Override
+public synchronized void onBlockFetchFailure(String blockId,
Throwable exception) {
+
result.set(exception.getMessage().contains(SecurityException.class.getName()));
+ notifyAll();
+}
+ };
+
+ String[] blockIds = new String[] { shuffle_2_3_4, shuffle_6_7_8
};
+ OneForOneBlockFetcher fetcher = new OneForOneBlockFetcher(client1,
app-2, 0,
+blockIds, listener);
+ synchronized (listener) {
+fetcher.start();
+listener.wait();
+ }
+ assertTrue(Should have failed to fetch blocks from non-authorized
app., result.get());
+
+ // Register an executor so that the next steps work.
+ ExecutorShuffleInfo executorInfo = new ExecutorShuffleInfo(
+new String[] { System.getProperty(java.io.tmpdir) }, 1,
+org.apache.spark.shuffle.sort.SortShuffleManager);
+ RegisterExecutor regmsg = new RegisterExecutor(app-1, 0,
executorInfo);
+ client1.sendRpcSync(regmsg.toByteArray(), 1);
+
+ // Make a successful request to fetch blocks, which creates a new
stream. But do not actually
+ // fetch any blocks, to keep the stream open.
+ result.set(false);
--- End diff --
I think you can delete this, its not checked between here and the next
`result.set(false)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r37155626
--- Diff:
network/shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java
---
@@ -160,6 +164,111 @@ public void testNoSaslServer() {
}
}
+ /**
+ * This test is not actually testing SASL behavior, but testing that the
shuffle service
+ * performs correct authorization checks based on the SASL
authentication data.
+ */
+ @Test
+ public void testAppIsolation() throws Exception {
+// Start a new server with the correct RPC handler to serve block data.
+ExternalShuffleBlockResolver blockResolver =
mock(ExternalShuffleBlockResolver.class);
+ExternalShuffleBlockHandler blockHandler = new
ExternalShuffleBlockHandler(
+ new OneForOneStreamManager(), blockResolver);
+TransportServerBootstrap bootstrap = new SaslServerBootstrap(conf,
secretKeyHolder);
+TransportContext blockServerContext = new TransportContext(conf,
blockHandler);
+TransportServer blockServer =
blockServerContext.createServer(Arrays.asList(bootstrap));
+
+TransportClient client1 = null;
+TransportClient client2 = null;
+TransportClientFactory clientFactory2 = null;
+try {
+ // Create a client, and make a request to fetch blocks from a
different app.
+ clientFactory = blockServerContext.createClientFactory(
+Lists.TransportClientBootstrapnewArrayList(
+ new SaslClientBootstrap(conf, app-1, secretKeyHolder)));
+ client1 = clientFactory.createClient(TestUtils.getLocalHost(),
+blockServer.getPort());
+
+ final AtomicBoolean result = new AtomicBoolean(false);
--- End diff --
how about renaming this to `gotException`? I was momentarily confused
thinking this was the result of a successful request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8218#discussion_r37155796
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -85,6 +86,24 @@ public SocketAddress getSocketAddress() {
}
/**
+ * Returns the ID used by the client to authenticate itself when
authentication is enabled.
+ *
+ * @return The client ID.
+ */
+ public String getClientId() {
+return clientId;
+ }
+
+ /**
+ * Sets the authenticated client ID. This is meant to be used by the
authentication layer;
+ * trying to set a different client ID after it's been set will result
in an exception.
+ */
+ public void setClientId(String id) {
+Preconditions.checkState(clientId == null, Client ID has already been
set.);
--- End diff --
If I understand correctly, this will never get called when you aren't using
authentication, right? Maybe drop in a comment here explaining that, and on
`getClientId` that it will return `null` without authentication?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131663356 I'm not an expert on this part of the code, but it looks sane. I just left a few minor comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131457210 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131459031 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131473374 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131473352 [Test build #40976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40976/console) for PR 8218 at commit [`c68deab`](https://github.com/apache/spark/commit/c68deabdf4f0a5070a928802c18e0330b38c3df0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131473375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40976/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-131459060 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
