[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[
https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980038#comment-14980038
]
Lei Xu commented on SPARK-6284:
---
join this thread.
We have the same problems.
Mesos 0.25.0 - 3 masters & 38 slaves, with roles:
* cpus(spark):16;
* port\(\*\):\[8000-32000\];mem\(\*\):126976
* other framework roles..
Spark 1.5.0 was deployed in "cluster" mode, with 3 scheduler properties:
* spark.mesos.secret=4bbd7991de80b50bbbc5e272e7f67aff
* spark.mesos.role=spark
* spark.mesos.principal=spark
and task failed with message:
state: TASK_ERROR message: "Task uses more resources cpus\(\*\):1;
mem\(\*\):2048 than available mem\(\*\):126976; ports\(\*\):\[8000-32000\];
disk\(\*\):1.04152e+06; cpus(spark):16" slave_id { value:
"69d525a7-3a63-4b4d-a269-5aa478f8c15f-S36" } timestamp: 1.446104736756061E9
source: SOURCE_MASTER reason: REASON_TASK_INVALID
Could you please show some examples to describe how to specify a job run with
"spark" role on Mesos ?
> Support framework authentication and role in Mesos framework
>
>
> Key: SPARK-6284
> URL: https://issues.apache.org/jira/browse/SPARK-6284
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Fix For: 1.5.0
>
>
> Support framework authentication and role in both Coarse grain and fine grain
> mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357181#comment-14357181 ] Apache Spark commented on SPARK-6284: - User 'tnachen' has created a pull request for this issue: https://github.com/apache/spark/pull/4960 > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357180#comment-14357180 ] Timothy Chen commented on SPARK-6284: - https://github.com/apache/spark/pull/4960 > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575628#comment-14575628 ] Kapil Malik commented on SPARK-6284: ^ +1. It's a must have for our usecase as well (multi-tenant mesos production setup). Can you kindly update on expected timeline for this change ? > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580780#comment-14580780 ] DarinJ commented on SPARK-6284: --- +1 we're currently patching spark with similar code to make it play nice on our cluster. Would be great to have the ability out of the box. > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902336#comment-14902336 ] Oleh Halenok commented on SPARK-6284: - Have a cluster, Mesos 0.22.1, hadoop 2.5.0, 3 masters, 8 slaves, which are 3 - role "fast", 3 - role "*", 2 other roles. Spark 1.5.0 is deployed to use cluster mode with dispatcher, it looks working, however I can not get it working on the particular nodes with role "fast" configured. Job submitted as # bin/spark-submit --class org.apache.spark.examples.SparkPi --master master.novalocal:7077 --deploy-mode cluster --conf spark.mesos.role=fast http://www/spark.jar 10 Also, I've tried to configure Spark defaults for submitted jobs and added to /opt/spark/conf/spark-defaults.conf the line "spark.mesos.role fast" In any case after submitting job is failing with the next error throwing, it looks driver starts with some defaults - cpu, mem, from spark ui: State: TASK_ERROR, Message: Task uses more resources cpus(*):1; mem(*):1024 than available cpus(fast):8; mem(fast):14863; disk(fast):45243; ports(fast):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-65535], Source: SOURCE_MASTER, Reason: REASON_TASK_INVALID, Time: 1.442868857232848E9 Can you please describe how to configure/start it properly? > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Fix For: 1.5.0 > > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530117#comment-14530117 ] Bharath Ravi Kumar commented on SPARK-6284: --- [~tnachen] This issue makes Spark on Mesos a non-starter in a multi-tenant mesos production setup. Any possibility that this will be back-ported into a minor release (1.3.x or 1.2.x) ? On a related note, is any further assistance required for testing this change? I could help with that. Thanks. > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630687#comment-14630687 ] Andrew Or commented on SPARK-6284: -- This is now merged and will appear in Spark 1.5! Thanks [~Timothy Chen] for the changes. > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Fix For: 1.5.0 > > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
