[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 merged to master and 2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 If no other comments, I'll commit this? I'll leave it open for a bit longer --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98269/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98269/testReport)** for PR 22852 at commit [`ebf4789`](https://github.com/apache/spark/commit/ebf47895c07b33684d5a206ba37d1ac2aaed36a5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4628/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 added sections to the resource manager sections. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98269/testReport)** for PR 22852 at commit [`ebf4789`](https://github.com/apache/spark/commit/ebf47895c07b33684d5a206ba37d1ac2aaed36a5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I can add a note here for deployments here and then we can do version specific ones after --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22852 I think these are good changes. In a separate PR for the versions-specific docs, we could add a similar note to https://spark.apache.org/docs/latest/spark-standalone.html as much of the security concern is around the standalone master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98267/testReport)** for PR 22852 at commit [`a4616bf`](https://github.com/apache/spark/commit/a4616bf8cd5adfb94ad9146d1f1d620f213ab041). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98267/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4626/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 Updated to have a section on security in the quickstart and overview, let me know what you think and if wording needs updated. If this ok I can followup with something on the website --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98267/testReport)** for PR 22852 at commit [`a4616bf`](https://github.com/apache/spark/commit/a4616bf8cd5adfb94ad9146d1f1d620f213ab041). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22852 A quick pointer to security issues in other key places sounds good. As long as it is increasing the chance users understand the specific issue and isn't more general text to skip past, it is helping --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would be fine with adding it more places, including perhaps the overview page: http://spark.apache.org/docs/latest/ and quick start pages. Perhaps we should agree upon the wording here first though. I'm not exactly sure where this pr stands honestly. @srowen are you going to put up a different one with wording you prefer? > If someone lands on this page, do they pretty easily come away with the impression they need to set spark.authenticate and network security if they care about security? Everyone reads text slightly different and I'm by no means a doc expert, so I'm definitely open to reword if there is consensus on it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would rather see someone more familiar with K8s that uses it document it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22852 I don't feel strongly about it; go ahead. If someone lands on this page, do they pretty easily come away with the impression they need to set spark.authenticate and network security if they care about security? if so, great. If the text is just adding to the text they might skip over, maybe revise it. That's how I think about it. I think you can make edits for Mesos and K8S here too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 so I had filed a jira to update mesos docs more detail about security things (https://issues.apache.org/jira/browse/SPARK-25024) which I need to follow up on, but I didn't file one for k8s. It would be good to have one for k8s if its not clear as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22852 I looked at the existing docs after I wrote my comment and saw that paragraph. I think it's fine, maybe just in need of some update now that we have k8s, although I'm not sure yet how spark.authenticate behaves on k8s. It's probably ok as is. It could be clarified a little bit (e.g. mentioning explicitly the Master and Worker daemons) but not a big deal. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 we can add stronger wording for standalone if you want, I know the text was recently updated (I believe by you) to have the below: > For other resource managers, spark.authenticate.secret must be configured on each of the nodes. This secret will be shared by all the daemons and applications, so this deployment configuration is not as secure as the above, especially when considering multi-tenant clusters. In this configuration, a user with the secret can effectively impersonate any other user. Do you have specific suggestion on where you want to put that? The reason I didn't put stronger was because if you are running it in isolated one client environment then the authentication part via secret doesn't matter that much. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22852 I'm fine with this, although I wonder if having a stronger wording about Spark standalone just not being secure would be better. Even if you enable auth, everybody needs to know the same auth secret, which is not optimal. It gives you a little bit of security, but not much. No isolation or anything. I'm also generally skeptical that people read and follow these things, but at least it's there. :-) (And just to comment on a previous comment by Sean, just enabling auth does not enable encryption.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 the intention is not a we told you so, its meant to grab their attention and to get people to think about it because in the end it is their responsibility in my opinion. I'm fine if you want to take a crack at listing a few things or adding a table of the bullets we have in the below sections. I just want to make sure we also say that this is not a comprehensive list. Many users who don't read all the docs look at the cheat sheet and stop there and think its comprehensive. There is no way we could list all variances of peoples environments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98091/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98091/testReport)** for PR 22852 at commit [`1320795`](https://github.com/apache/spark/commit/13207950f5a187943cb201220b59698d653d9398). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22852 I get it, the "it's your responsibility" stance, and it is. For any risk there's a sentence in this doc we could point to and say, "see, told you". If we're going to make a change here, adding another para saying "below, we told you so" isn't additive. Especially if we're trying to use this change to actively mitigate security issues. More useful is a cheat-sheet, TL;DR, simply enumerating the top things you don't want to miss. I think it's more useful than redundant. I can take a crack at that too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98091/testReport)** for PR 22852 at commit [`1320795`](https://github.com/apache/spark/commit/13207950f5a187943cb201220b59698d653d9398). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4536/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98090/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98090/testReport)** for PR 22852 at commit [`8b4aaf5`](https://github.com/apache/spark/commit/8b4aaf5a85f7f925baf7365283e950b9d7676a4b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22852 **[Test build #98090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98090/testReport)** for PR 22852 at commit [`8b4aaf5`](https://github.com/apache/spark/commit/8b4aaf5a85f7f925baf7365283e950b9d7676a4b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4535/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 @vanzin @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
