Yeah this is so Mesos can use multiple executors from different machines and allow dynamic allocations to scale up to 20 CORES but limit the maximum amount to 20 with spark.cores.max=20?
Am I missing something? \-- **Rodrick Brown** / Systems Engineer +1 917 445 6839 / [rodr...@orchardplatform.com](mailto:char...@orchardplatform.com) **Orchard Platform** 101 5th Avenue, 4th Floor, New York, NY 10003 [http://www.orchardplatform.com](http://www.orchardplatform.com/) [Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/) On Apr 29 2016, at 12:39 pm, June Taylor <j...@umn.edu> wrote: > You have specified total executor cores of 500. > > > > > > Thanks, > > June Taylor > > System Administrator, Minnesota Population Center > > University of Minnesota > > > > On Fri, Apr 29, 2016 at 11:35 AM, Rodrick Brown <[rodrick@orchard- app.com](mailto:rodr...@orchard-app.com)> wrote: > >> I'm running a spark app on Mesos in coarse grain mode with dynmaic allocation after my task launches I see it still takes more CPU's than what I specify in my config. Does the following parameter spark.cores.max=20 work in Mesos or is this a YARN only parameter? is their something equivalent for Mesos? >> >> >> >> When I go to the framework tab in Mesos I see this task consuming 30 CPU's instead of 20 I limited it to. >> >> >> >> /usr/bin/timeout 0 /opt/spark-1.6.1/bin/spark-submit \ >> >> \--master "mesos://zk://[prod-zookeeper-1.aws.xx.com:2181](http://prod- zookeeper-1.aws.xx.com:2181),[prod-zookeeper-2.aws.xx.com:2181](http://prod- zookeeper-2.aws.xx.com:2181),[prod-zookeeper-3.aws.xx.com:2181/mesos](http ://prod-zookeeper-3.aws.xx.com:2181/mesos)" \ >> >> \--conf spark.ui.port=31354 \ >> >> \--conf spark.mesos.coarse=true \ >> >> \--conf spark.cores.max=20 \ >> >> \--conf spark.mesos.constraints="rack:spark" \ >> >> \--conf spark.sql.tungsten.enabled=true \ >> >> \--conf spark.shuffle.service.enabled=true \ >> >> \--conf spark.dynamicAllocation.enabled=true \ >> >> \--conf spark.mesos.executor.memoryOverhead=1606 \ >> >> \--class com.orchard.sparkjob.library.kafka.KafkaDirectConsumer \ >> >> \--total-executor-cores 500 \ >> >> \--driver-memory 2867M \ >> >> \--executor-memory 4014M \ >> >> \--jars /data/orchard/etc/config/es-services.jar \ >> >> /data/orchard/jars/spark-job-library- 3e30539922ff540f4632d1d0745501c48300b89b-assembled.jar >> >> >> >> \-- >> >> **Rodrick Brown** / Systems Engineer >> >> +1 917 445 6839 / [rodr...@orchardplatform.com](mailto:char...@orchardplatform.com) >> >> **Orchard Platform** >> >> 101 5th Avenue, 4th Floor, New York, NY 10003 >> >> [http://www.orchardplatform.com](http://www.orchardplatform.com/) >> >> [Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/) >> >> >> >> **NOTICE TO RECIPIENTS**: This communication is confidential and intended for the use of the addressee only. If you are not an intended recipient of this communication, please delete it immediately and notify the sender by return email. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. This communication does not constitute an offer to sell or a solicitation of an indication of interest to purchase any loan, security or any other financial product or instrument, nor is it an offer to sell or a solicitation of an indication of interest to purchase any products or services to any persons who are prohibited from receiving such information under applicable law. The contents of this communication may not be accurate or complete and are subject to change without notice. As such, Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding the accuracy or completeness of the information contained herein. The intended recipient is advised to consult its own professional advisors, including those specializing in legal, tax and accounting matters. Orchard does not provide legal, tax or accounting advice. > > -- *NOTICE TO RECIPIENTS*: This communication is confidential and intended for the use of the addressee only. If you are not an intended recipient of this communication, please delete it immediately and notify the sender by return email. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. This communication does not constitute an offer to sell or a solicitation of an indication of interest to purchase any loan, security or any other financial product or instrument, nor is it an offer to sell or a solicitation of an indication of interest to purchase any products or services to any persons who are prohibited from receiving such information under applicable law. The contents of this communication may not be accurate or complete and are subject to change without notice. As such, Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding the accuracy or completeness of the information contained herein. The intended recipient is advised to consult its own professional advisors, including those specializing in legal, tax and accounting matters. Orchard does not provide legal, tax or accounting advice.
