[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[
https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980038#comment-14980038
]
Lei Xu commented on SPARK-6284:
---
join this thread.
We have the same problems.
Mesos 0.25.0 - 3 masters & 38 slaves, with roles:
* cpus(spark):16;
* port\(\*\):\[8000-32000\];mem\(\*\):126976
* other framework roles..
Spark 1.5.0 was deployed in "cluster" mode, with 3 scheduler properties:
* spark.mesos.secret=4bbd7991de80b50bbbc5e272e7f67aff
* spark.mesos.role=spark
* spark.mesos.principal=spark
and task failed with message:
state: TASK_ERROR message: "Task uses more resources cpus\(\*\):1;
mem\(\*\):2048 than available mem\(\*\):126976; ports\(\*\):\[8000-32000\];
disk\(\*\):1.04152e+06; cpus(spark):16" slave_id { value:
"69d525a7-3a63-4b4d-a269-5aa478f8c15f-S36" } timestamp: 1.446104736756061E9
source: SOURCE_MASTER reason: REASON_TASK_INVALID
Could you please show some examples to describe how to specify a job run with
"spark" role on Mesos ?
> Support framework authentication and role in Mesos framework
>
>
> Key: SPARK-6284
> URL: https://issues.apache.org/jira/browse/SPARK-6284
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Fix For: 1.5.0
>
>
> Support framework authentication and role in both Coarse grain and fine grain
> mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902336#comment-14902336 ] Oleh Halenok commented on SPARK-6284: - Have a cluster, Mesos 0.22.1, hadoop 2.5.0, 3 masters, 8 slaves, which are 3 - role "fast", 3 - role "*", 2 other roles. Spark 1.5.0 is deployed to use cluster mode with dispatcher, it looks working, however I can not get it working on the particular nodes with role "fast" configured. Job submitted as # bin/spark-submit --class org.apache.spark.examples.SparkPi --master master.novalocal:7077 --deploy-mode cluster --conf spark.mesos.role=fast http://www/spark.jar 10 Also, I've tried to configure Spark defaults for submitted jobs and added to /opt/spark/conf/spark-defaults.conf the line "spark.mesos.role fast" In any case after submitting job is failing with the next error throwing, it looks driver starts with some defaults - cpu, mem, from spark ui: State: TASK_ERROR, Message: Task uses more resources cpus(*):1; mem(*):1024 than available cpus(fast):8; mem(fast):14863; disk(fast):45243; ports(fast):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-65535], Source: SOURCE_MASTER, Reason: REASON_TASK_INVALID, Time: 1.442868857232848E9 Can you please describe how to configure/start it properly? > Support framework authentication and role in Mesos framework > > > Key: SPARK-6284 > URL: https://issues.apache.org/jira/browse/SPARK-6284 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Fix For: 1.5.0 > > > Support framework authentication and role in both Coarse grain and fine grain > mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630687#comment-14630687 ] Andrew Or commented on SPARK-6284: -- This is now merged and will appear in Spark 1.5! Thanks [~Timothy Chen] for the changes. Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Assignee: Timothy Chen Fix For: 1.5.0 Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14580780#comment-14580780 ] DarinJ commented on SPARK-6284: --- +1 we're currently patching spark with similar code to make it play nice on our cluster. Would be great to have the ability out of the box. Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575628#comment-14575628 ] Kapil Malik commented on SPARK-6284: ^ +1. It's a must have for our usecase as well (multi-tenant mesos production setup). Can you kindly update on expected timeline for this change ? Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530117#comment-14530117 ] Bharath Ravi Kumar commented on SPARK-6284: --- [~tnachen] This issue makes Spark on Mesos a non-starter in a multi-tenant mesos production setup. Any possibility that this will be back-ported into a minor release (1.3.x or 1.2.x) ? On a related note, is any further assistance required for testing this change? I could help with that. Thanks. Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357180#comment-14357180 ] Timothy Chen commented on SPARK-6284: - https://github.com/apache/spark/pull/4960 Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6284) Support framework authentication and role in Mesos framework
[ https://issues.apache.org/jira/browse/SPARK-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357181#comment-14357181 ] Apache Spark commented on SPARK-6284: - User 'tnachen' has created a pull request for this issue: https://github.com/apache/spark/pull/4960 Support framework authentication and role in Mesos framework Key: SPARK-6284 URL: https://issues.apache.org/jira/browse/SPARK-6284 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Timothy Chen Support framework authentication and role in both Coarse grain and fine grain mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
