[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12760 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216869015 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216679755 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57662/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216679747 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216679477 **[Test build #57662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57662/consoleFull)** for PR 12760 at commit [`5da94b8`](https://github.com/apache/spark/commit/5da94b884ddfc63da41ce3e670e60006ae5c4e2e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216649376 **[Test build #57662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57662/consoleFull)** for PR 12760 at commit [`5da94b8`](https://github.com/apache/spark/commit/5da94b884ddfc63da41ce3e670e60006ae5c4e2e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216625526 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57645/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216625523 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216625205 **[Test build #57645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57645/consoleFull)** for PR 12760 at commit [`03ac94e`](https://github.com/apache/spark/commit/03ac94e458b7ea80f328a7868a45ba361781d764). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user dhruve commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61929871
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -60,6 +110,43 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
assert(securityManager.checkUIViewPermissions(null) === true)
}
+ test("set security with api for groups") {
+val conf = new SparkConf
+conf.set("spark.user.groups.mapping",
"org.apache.spark.DummyGroupMappingServiceProvider")
+
+val securityManager = new SecurityManager(conf);
+securityManager.setAcls(true)
+securityManager.setViewAclsGroups("group1,group2")
+
+// group1,group2 match
+assert(securityManager.checkUIViewPermissions("user1") === true)
+assert(securityManager.checkUIViewPermissions("user2") === true)
+
+// change groups so they do not match
+securityManager.setViewAclsGroups("group4,group5")
+assert(securityManager.checkUIViewPermissions("user1") === false)
+assert(securityManager.checkUIViewPermissions("user2") === false)
+
+
+val conf2 = new SparkConf
+conf.set("spark.user.groups.mapping", "BogusServiceProvider")
--- End diff --
This one checks the BogusServiceProvider against an empty group set. The
one above checks against groups which are explicitly specified.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61925009
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -60,6 +110,43 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
assert(securityManager.checkUIViewPermissions(null) === true)
}
+ test("set security with api for groups") {
+val conf = new SparkConf
+conf.set("spark.user.groups.mapping",
"org.apache.spark.DummyGroupMappingServiceProvider")
+
+val securityManager = new SecurityManager(conf);
+securityManager.setAcls(true)
+securityManager.setViewAclsGroups("group1,group2")
+
+// group1,group2 match
+assert(securityManager.checkUIViewPermissions("user1") === true)
+assert(securityManager.checkUIViewPermissions("user2") === true)
+
+// change groups so they do not match
+securityManager.setViewAclsGroups("group4,group5")
+assert(securityManager.checkUIViewPermissions("user1") === false)
+assert(securityManager.checkUIViewPermissions("user2") === false)
+
+
+val conf2 = new SparkConf
+conf.set("spark.user.groups.mapping", "BogusServiceProvider")
--- End diff --
I don't think we need this section of tests, we already test
BogusServiceProvider in the test above. One is enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61924849
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -112,6 +126,25 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
securityManager.setViewAclsGroups("group4,group5")
assert(securityManager.checkUIViewPermissions("user1") === false)
assert(securityManager.checkUIViewPermissions("user2") === false)
+
+
+val conf2 = new SparkConf
--- End diff --
I don't think we need this section of tests, the one above that uses
BogusServiceProvider is enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-216592024 **[Test build #57645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57645/consoleFull)** for PR 12760 at commit [`03ac94e`](https://github.com/apache/spark/commit/03ac94e458b7ea80f328a7868a45ba361781d764). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61893139
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2179,6 +2179,24 @@ private[spark] object Utils extends Logging {
.getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
}
+ val EMPTY_USER_GROUPS = Set[String]()
+
+ // Returns the groups to which the current user belongs.
+ def getCurrentUserGroups(sparkConf: SparkConf, username: String):
Set[String] = {
+val groupProviderClassName =
sparkConf.get("spark.user.groups.mapping", "")
--- End diff --
lets leave ShellBasedGroupsMappingProvider as the default because it won't
hurt anything since we now catch and ignore exceptions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61893228
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2179,6 +2179,24 @@ private[spark] object Utils extends Logging {
.getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
}
+ val EMPTY_USER_GROUPS = Set[String]()
+
+ // Returns the groups to which the current user belongs.
+ def getCurrentUserGroups(sparkConf: SparkConf, username: String):
Set[String] = {
+val groupProviderClassName =
sparkConf.get("spark.user.groups.mapping", "")
+if (groupProviderClassName != "") {
+ try {
+val groupMappingServiceProvider =
classForName(groupProviderClassName).newInstance.
+
asInstanceOf[org.apache.spark.security.GroupMappingServiceProvider]
+val currentUserGroups =
groupMappingServiceProvider.getGroups(username)
+return currentUserGroups
+ } catch {
+case e: Exception => logError("Unable to get groups for user=" +
username + " msg" + e)
--- End diff --
logError has a syntax logError(msg, e) change to use that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61892471
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -166,6 +284,56 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
assert(securityManager.checkModifyPermissions("user8") === true)
}
+ test("set security with * in acls for groups") {
+val conf = new SparkConf
+conf.set("spark.ui.acls.enable", "true")
+conf.set("spark.admin.acls.groups", "group4,group5")
+conf.set("spark.ui.view.acls.groups", "*")
+conf.set("spark.modify.acls.groups", "group6")
+
+val securityManager = new SecurityManager(conf)
+assert(securityManager.aclsEnabled() === true)
+
+// check for viewAclsGroups with *
+assert(securityManager.checkUIViewPermissions("user1") === true)
+assert(securityManager.checkUIViewPermissions("user2") === true)
+assert(securityManager.checkModifyPermissions("user1") === false)
+assert(securityManager.checkModifyPermissions("user2") === false)
+
+// check for modifyAcls with *
+securityManager.setModifyAclsGroups("*")
+securityManager.setViewAclsGroups("group6")
+assert(securityManager.checkUIViewPermissions("user1") === false)
+assert(securityManager.checkUIViewPermissions("user2") === false)
+assert(securityManager.checkModifyPermissions("user1") === true)
+assert(securityManager.checkModifyPermissions("user2") === true)
+
+// check for adminAcls with *
+securityManager.setAdminAclsGroups("group9,*")
+securityManager.setModifyAclsGroups("group4,group5")
+securityManager.setViewAclsGroups("group6,group7")
+assert(securityManager.checkUIViewPermissions("user5") === true)
+assert(securityManager.checkUIViewPermissions("user6") === true)
+assert(securityManager.checkModifyPermissions("user7") === true)
+assert(securityManager.checkModifyPermissions("user8") === true)
+ }
+
+ test("security for groups default behavior") {
+// no groups or userToGroupsMapper is provided
--- End diff --
assume you mean is not provided. Also add a test for setting it to a bogus
value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215920023 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57379/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215920022 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215919905 **[Test build #57379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57379/consoleFull)** for PR 12760 at commit [`3ecd704`](https://github.com/apache/spark/commit/3ecd704c3306e25d159fb383c14c899bf8aa44e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215905387 **[Test build #57379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57379/consoleFull)** for PR 12760 at commit [`3ecd704`](https://github.com/apache/spark/commit/3ecd704c3306e25d159fb383c14c899bf8aa44e2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215549032 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215549034 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57266/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215548834 **[Test build #57266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57266/consoleFull)** for PR 12760 at commit [`9a96269`](https://github.com/apache/spark/commit/9a962697bf9ca93f8daf862970cf50eb5f8dec29). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/12760#discussion_r61494141 --- Diff: docs/configuration.md --- @@ -1249,8 +1249,32 @@ Apart from these, the following properties are also available, and may be useful Comma separated list of users/administrators that have view and modify access to all Spark jobs. This can be used if you run on a shared cluster and have a set of administrators or devs who -help debug when things work. Putting a "*" in the list means any user can have the privilege -of admin. +help debug when things do not work. Putting a "*" in the list means any user can have the +privilege of admin. --- End diff -- There is a Security.md document we should update with the new acls as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61492158
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2179,6 +2179,16 @@ private[spark] object Utils extends Logging {
.getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
}
+
+ // Returns the groups to which the current user belongs.
+ def getCurrentUserGroups(sparkConf: SparkConf, username: String):
Set[String] = {
+val groupMappingServiceProvider =
Utils.classForName(sparkConf.get("spark.user.groups.mapping",
--- End diff --
Also we want this conf to be basically optional so it should be able to set
to empty string and not error out in case people don't want it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61487225
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2179,6 +2179,16 @@ private[spark] object Utils extends Logging {
.getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
}
+
+ // Returns the groups to which the current user belongs.
+ def getCurrentUserGroups(sparkConf: SparkConf, username: String):
Set[String] = {
+val groupMappingServiceProvider =
Utils.classForName(sparkConf.get("spark.user.groups.mapping",
+
"org.apache.spark.security.ShellBasedGroupsMappingProvider")).newInstance.
+ asInstanceOf[org.apache.spark.security.GroupMappingServiceProvider]
--- End diff --
we should have a try catch around here incase users specifies a class that
isn't found and the newInstance fails. Just log an error if it fails.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61486445
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2179,6 +2179,16 @@ private[spark] object Utils extends Logging {
.getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
}
+
+ // Returns the groups to which the current user belongs.
+ def getCurrentUserGroups(sparkConf: SparkConf, username: String):
Set[String] = {
+val groupMappingServiceProvider =
Utils.classForName(sparkConf.get("spark.user.groups.mapping",
--- End diff --
dont' need the Utils. since in the Utils class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61485864
--- Diff:
core/src/main/scala/org/apache/spark/security/ShellBasedGroupsMappingProvider.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.security
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * This class is responsible for getting the groups for a particular user
in Unix based
+ * environments. This implementation uses the Unix Shell based id command
to fetch the user groups
+ * for the specified user. It does not cache the user groups as the
invocations are expected
+ * to be infrequent.
+*/
+
+private[spark] class ShellBasedGroupsMappingProvider extends
GroupMappingServiceProvider
+ with Logging {
+
+ override def getGroups(username: String): Set[String] = {
+val userGroups = getUnixGroups(username)
+logInfo("User: " + username + " Groups: " + userGroups.mkString(","))
+userGroups
+ }
+
+ private def getUnixGroups(username: String): Set[String] = {
+logDebug("getUnixGroupsFromSparkUtil got username=" + username)
+val cmdSeq = Seq("bash", "-c", "id -Gn " + username)
+var result: String = null
+try {
+ result = Utils.executeAndGetOutput(cmdSeq)
+ // we need to get rid of the trailing "\n" from the result of
command execution
+ result = result.stripLineEnd
+ logDebug("Usergroups from executeAndGetOutput= " + result)
+} catch {
+ case e: Exception => logError("Unable to get groups for user=" +
username + e.getMessage)
--- End diff --
with logError, instead of calling e.getMessage, use logError("msg", e)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12760#discussion_r61485159
--- Diff:
core/src/main/scala/org/apache/spark/security/ShellBasedGroupsMappingProvider.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.security
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.Utils
+
+/**
+ * This class is responsible for getting the groups for a particular user
in Unix based
+ * environments. This implementation uses the Unix Shell based id command
to fetch the user groups
+ * for the specified user. It does not cache the user groups as the
invocations are expected
+ * to be infrequent.
+*/
+
+private[spark] class ShellBasedGroupsMappingProvider extends
GroupMappingServiceProvider
+ with Logging {
+
+ override def getGroups(username: String): Set[String] = {
+val userGroups = getUnixGroups(username)
+logInfo("User: " + username + " Groups: " + userGroups.mkString(","))
--- End diff --
make this a debug message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core][YARN] - Support group acls...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/12760#discussion_r61482000 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -303,16 +321,33 @@ private[spark] class SecurityManager(sparkConf: SparkConf) } /** + * Admin acls groups should be set before the view or modify acls groups. If you modify the admin --- End diff -- fixing spacing here to line up the first line of *. Throughout the file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core, YARN] - Support group acls...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215519529 **[Test build #57266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57266/consoleFull)** for PR 12760 at commit [`9a96269`](https://github.com/apache/spark/commit/9a962697bf9ca93f8daf862970cf50eb5f8dec29). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core, YARN] - Support group acls...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215518103 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4224] [Core, YARN] - Support group acls...
Github user dhruve commented on the pull request: https://github.com/apache/spark/pull/12760#issuecomment-215517725 @vanzin - My bad. Have added the description now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
