[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575134#comment-16575134 ] Arthur Rand commented on SPARK-25024: - Just to chime in here. Unless a lot has changed, all of the Spark security features when running on Mesos are available in "vanilla Mesos", as long as you have the required plug-ins. The problem is that the users' suite of security plug-ins is impossible to predict so the spark docs only tell you how to _configure Spark_. Some of the questions you bring up [~tgraves], depend on the specific setup, for example auth when submitting jobs. However, I think it's safe to say that if you have a _secure Mesos_ cluster (meaning you have some form of plug-ins) then it'll work with Spark. > Update mesos documentation to be clear about security supported > --- > > Key: SPARK-25024 > URL: https://issues.apache.org/jira/browse/SPARK-25024 > Project: Spark > Issue Type: Bug > Components: Documentation >Affects Versions: 2.2.2 >Reporter: Thomas Graves >Priority: Major > > I was reading through our mesos deployment docs and security docs and its not > clear at all what type of security and how to set it up for mesos. I think > we should clarify this and have something about exactly what is supported and > what is not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22402) Allow fetcher URIs to be downloaded to specific locations relative to Mesos Sandbox
[
https://issues.apache.org/jira/browse/SPARK-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265732#comment-16265732
]
Arthur Rand commented on SPARK-22402:
-
Hello [~felixcheung], Yes I'll submit a patch very soon - should be relatively
simple. Thanks.
> Allow fetcher URIs to be downloaded to specific locations relative to Mesos
> Sandbox
> ---
>
> Key: SPARK-22402
> URL: https://issues.apache.org/jira/browse/SPARK-22402
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Arthur Rand
>Priority: Minor
>
> Currently {{spark.mesos.uris}} will only place files in the sandbox, but some
> configuration files and applications may need to be in specific locations.
> The Mesos proto allows for this with the optional {{output_file}} field
> (https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L671).
> We can expose this through the command line with {{--conf
> spark.mesos.uris=:}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22402) Allow fetcher URIs to be downloaded to specific locations relative to Mesos Sandbox
Arthur Rand created SPARK-22402:
---
Summary: Allow fetcher URIs to be downloaded to specific locations
relative to Mesos Sandbox
Key: SPARK-22402
URL: https://issues.apache.org/jira/browse/SPARK-22402
Project: Spark
Issue Type: Improvement
Components: Mesos
Affects Versions: 2.2.1
Reporter: Arthur Rand
Priority: Minor
Currently {{spark.mesos.uris}} will only place files in the sandbox, but some
configuration files and applications may need to be in specific locations. The
Mesos proto allows for this with the optional {{output_file}} field
(https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L671).
We can expose this through the command line with {{--conf
spark.mesos.uris=:}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22133) Document Mesos reject offer duration configutations
Arthur Rand created SPARK-22133:
---
Summary: Document Mesos reject offer duration configutations
Key: SPARK-22133
URL: https://issues.apache.org/jira/browse/SPARK-22133
Project: Spark
Issue Type: Improvement
Components: Mesos
Affects Versions: 2.3.0
Reporter: Arthur Rand
Mesos has multiple configurable timeouts {{spark.mesos.rejectOfferDuration}},
{{spark.mesos.rejectOfferDurationForUnmetConstraints}}, and
{{spark.mesos.rejectOfferDurationForReachedMaxCores}} that can have a large
effect on Spark performance when sharing a Mesos cluster with other frameworks
and users. These configurations aren't documented, add documentation and
information for non-Mesos experts on how these settings should be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22132) Document the Dispatcher REST API
Arthur Rand created SPARK-22132: --- Summary: Document the Dispatcher REST API Key: SPARK-22132 URL: https://issues.apache.org/jira/browse/SPARK-22132 Project: Spark Issue Type: Improvement Components: Mesos Affects Versions: 2.3.0 Reporter: Arthur Rand Priority: Minor The Dispatcher has a REST API for managing jobs in a Mesos cluster but it's currently undocumented meaning that users have to reference the source code for programmatic access. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22131) Add Mesos Secrets Support to the Mesos Driver
Arthur Rand created SPARK-22131: --- Summary: Add Mesos Secrets Support to the Mesos Driver Key: SPARK-22131 URL: https://issues.apache.org/jira/browse/SPARK-22131 Project: Spark Issue Type: New Feature Components: Mesos Affects Versions: 2.3.0 Reporter: Arthur Rand We recently added Secrets support to the Dispatcher (SPARK-20812). In order to have Driver-to-Executor TLS we need the same support in the Mesos Driver so a secret can be disseminated to the executors. This JIRA is to move the current secrets implementation to be used by both frameworks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21842) Support Kerberos ticket renewal and creation in Mesos
[ https://issues.apache.org/jira/browse/SPARK-21842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168278#comment-16168278 ] Arthur Rand edited comment on SPARK-21842 at 9/18/17 12:42 AM: --- Hey [~kalvinnchau] I'm currently of the mind that using the RPC/broadcast approach is better (for Mesos) for a couple reasons. 1. We recently added Secret support to Spark on Mesos, this uses a temporary file system to put the keytab or TGT in the sandbox of the Spark driver. They are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236) which is broadcast to the executors, so using RPC/broadcast is consistent with this. 2. Keeps all transfers of secure information within Spark. 3. Doesn't _require_ HDFS. However I understand that there is a potential risk with executors falsely registering with the Driver and getting tokens. I know in the case of DC/OS this is less of a concern (we have some protections around this). But this could still happen today due to the code mentioned above. We could prevent this by keeping track of the executor IDs and only allowing executors to register when they have an expected ID..? was (Author: arand): Hey [~kalvinnchau] I'm currently of the mind that using the RPC/broadcast approach is better (for Mesos) for a couple reasons. 1. We recently added Secret support to Spark on Mesos, this uses a temporary file system to put the keytab or TGT in the sandbox of the Spark driver. They are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236) which is broadcast to the executors, so using RPC/broadcast is consistent with this. 2. Keeps all transfers of secure information within Spark. 3. Doesn't require HDFS. There is a little bit of a chicken-and-egg situation here w.r.t. YARN, but I'm obviously not familiar enough with how Spark-YARN-HDFS work together. However I understand that there is a potential risk with executors falsely registering with the Driver and getting tokens. I know in the case of DC/OS this is less of a concern (we have some protections around this). But this could still happen today due to the code mentioned above. We could prevent this by keeping track of the executor IDs and only allowing executors to register when they have an expected ID..? > Support Kerberos ticket renewal and creation in Mesos > -- > > Key: SPARK-21842 > URL: https://issues.apache.org/jira/browse/SPARK-21842 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Arthur Rand > > We at Mesosphere have written Kerberos support for Spark on Mesos. The code > to use Kerberos on a Mesos cluster has been added to Apache Spark > (SPARK-16742). This ticket is to complete the implementation and allow for > ticket renewal and creation. Specifically for long running and streaming jobs. > Mesosphere design doc (needs revision, wip): > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21842) Support Kerberos ticket renewal and creation in Mesos
[ https://issues.apache.org/jira/browse/SPARK-21842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168278#comment-16168278 ] Arthur Rand commented on SPARK-21842: - Hey [~kalvinnchau] I'm currently of the mind that using the RPC/broadcast approach is better (for Mesos) for a couple reasons. 1. We recently added Secret support to Spark on Mesos, this uses a temporary file system to put the keytab or TGT in the sandbox of the Spark driver. They are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236) which is broadcast to the executors, so using RPC/broadcast is consistent with this. 2. Keeps all transfers of secure information within Spark. 3. Doesn't require HDFS. There is a little bit of a chicken-and-egg situation here w.r.t. YARN, but I'm obviously not familiar enough with how Spark-YARN-HDFS work together. However I understand that there is a potential risk with executors falsely registering with the Driver and getting tokens. I know in the case of DC/OS this is less of a concern (we have some protections around this). But this could still happen today due to the code mentioned above. We could prevent this by keeping track of the executor IDs and only allowing executors to register when they have an expected ID..? > Support Kerberos ticket renewal and creation in Mesos > -- > > Key: SPARK-21842 > URL: https://issues.apache.org/jira/browse/SPARK-21842 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Arthur Rand > > We at Mesosphere have written Kerberos support for Spark on Mesos. The code > to use Kerberos on a Mesos cluster has been added to Apache Spark > (SPARK-16742). This ticket is to complete the implementation and allow for > ticket renewal and creation. Specifically for long running and streaming jobs. > Mesosphere design doc (needs revision, wip): > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21842) Support Kerberos ticket renewal and creation in Mesos
[ https://issues.apache.org/jira/browse/SPARK-21842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154785#comment-16154785 ] Arthur Rand commented on SPARK-21842: - Hello [~kalvinnchau], I apologize for the slow response. Thanks for your interest. We're very keen on getting this working asap so we're working on it actively. Of course we'd like to work together if possible. Do you have a timeline you're looking at? > Support Kerberos ticket renewal and creation in Mesos > -- > > Key: SPARK-21842 > URL: https://issues.apache.org/jira/browse/SPARK-21842 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Arthur Rand > > We at Mesosphere have written Kerberos support for Spark on Mesos. The code > to use Kerberos on a Mesos cluster has been added to Apache Spark > (SPARK-16742). This ticket is to complete the implementation and allow for > ticket renewal and creation. Specifically for long running and streaming jobs. > Mesosphere design doc (needs revision, wip): > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142337#comment-16142337 ] Arthur Rand commented on SPARK-16742: - Gotcha, https://issues.apache.org/jira/browse/SPARK-21842 is to track work. > Kerberos support for Spark on Mesos > --- > > Key: SPARK-16742 > URL: https://issues.apache.org/jira/browse/SPARK-16742 > Project: Spark > Issue Type: New Feature > Components: Mesos >Reporter: Michael Gummelt >Assignee: Arthur Rand > Fix For: 2.3.0 > > > We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be > contributing it to Apache Spark soon. > Mesosphere design doc: > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 > Mesosphere code: > https://github.com/mesosphere/spark/commit/73ba2ab8d97510d5475ef9a48c673ce34f7173fa -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21842) Support Kerberos ticket renewal and creation in Mesos
Arthur Rand created SPARK-21842: --- Summary: Support Kerberos ticket renewal and creation in Mesos Key: SPARK-21842 URL: https://issues.apache.org/jira/browse/SPARK-21842 Project: Spark Issue Type: New Feature Components: Mesos Affects Versions: 2.3.0 Reporter: Arthur Rand Fix For: 2.3.0 We at Mesosphere have written Kerberos support for Spark on Mesos. The code to use Kerberos on a Mesos cluster has been added to Apache Spark (SPARK-16742). This ticket is to complete the implementation and allow for ticket renewal and creation. Specifically for long running and streaming jobs. Mesosphere design doc (needs revision, wip): https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140111#comment-16140111 ] Arthur Rand commented on SPARK-16742: - Hello [~vanzin], I'm assuming you're talking about automatic ticket renewal, correct? I was just starting to look into that w.r.t. Mesos, I'll create a ticket. > Kerberos support for Spark on Mesos > --- > > Key: SPARK-16742 > URL: https://issues.apache.org/jira/browse/SPARK-16742 > Project: Spark > Issue Type: New Feature > Components: Mesos >Reporter: Michael Gummelt >Assignee: Arthur Rand > Fix For: 2.3.0 > > > We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be > contributing it to Apache Spark soon. > Mesosphere design doc: > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 > Mesosphere code: > https://github.com/mesosphere/spark/commit/73ba2ab8d97510d5475ef9a48c673ce34f7173fa -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20812) Add Mesos Secrets support to the spark dispatcher
[ https://issues.apache.org/jira/browse/SPARK-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113731#comment-16113731 ] Arthur Rand commented on SPARK-20812: - https://github.com/apache/spark/pull/18837 > Add Mesos Secrets support to the spark dispatcher > - > > Key: SPARK-20812 > URL: https://issues.apache.org/jira/browse/SPARK-20812 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 2.3.0 >Reporter: Michael Gummelt > > Mesos 1.4 will support secrets. In order to support sending keytabs through > the Spark Dispatcher, or any other secret, we need to integrate this with the > Spark Dispatcher. > The integration should include support for both file-based and env-based > secrets. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106501#comment-16106501 ] Arthur Rand commented on SPARK-16742: - Hello [~vanzin], I addressed the comments for the second PR (https://github.com/apache/spark/pull/18519). It is ready for final review. > Kerberos support for Spark on Mesos > --- > > Key: SPARK-16742 > URL: https://issues.apache.org/jira/browse/SPARK-16742 > Project: Spark > Issue Type: New Feature > Components: Mesos >Reporter: Michael Gummelt > > We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be > contributing it to Apache Spark soon. > Mesosphere design doc: > https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6 > Mesosphere code: > https://github.com/mesosphere/spark/commit/73ba2ab8d97510d5475ef9a48c673ce34f7173fa -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark
[
https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100016#comment-16100016
]
Arthur Rand commented on SPARK-18935:
-
Are people still interested in this being fixed? I'm going to try and reproduce
the problem and scope out the effort to fix it.
> Use Mesos "Dynamic Reservation" resource for Spark
> --
>
> Key: SPARK-18935
> URL: https://issues.apache.org/jira/browse/SPARK-18935
> Project: Spark
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.0.1, 2.0.2
>Reporter: jackyoh
>
> I'm running spark on Apache Mesos
> Please follow these steps to reproduce the issue:
> 1. First, run Mesos resource reserve:
> curl -i -d slaveId=c24d1cfb-79f3-4b07-9f8b-c7b19543a333-S0 -d
> resources='[{"name":"cpus","type":"SCALAR","scalar":{"value":20},"role":"spark","reservation":{"principal":""}},{"name":"mem","type":"SCALAR","scalar":{"value":4096},"role":"spark","reservation":{"principal":""}}]'
> -X POST http://192.168.1.118:5050/master/reserve
> 2. Then run spark-submit command:
> ./spark-submit --class org.apache.spark.examples.SparkPi --master
> mesos://192.168.1.118:5050 --conf spark.mesos.role=spark
> ../examples/jars/spark-examples_2.11-2.0.2.jar 1
> And the console will keep loging same warning message as shown below:
> 16/12/19 22:33:28 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
