Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

: Artemis User Sent: Thursday, November 3, 2022 8:35 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. Now I see what you want to do. If you have access to the cl

Re: [EXTERNAL] Re: Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

November 6, 2022 4:19 PM To: Shay Elbaz Cc: Artemis User ; Tom Graves ; Tom Graves ; user@spark.apache.org Subject: [EXTERNAL] Re: Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. May I ask why the ETL

Re: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

0 GPUs. > > I hope that was more clear. > Thank you very much for helping. > > Shay > > -- > *From:* Tom Graves > *Sent:* Friday, November 4, 2022 4:19 PM > *To:* Tom Graves ; Artemis User < > arte...@dtechspace.com>; user@spark.apache.org ; >

Re: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

ves ; Artemis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. So I'm not sure I completely follow. Are you asking for a way to change

Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

utes, if not much more. Any failure during this long time is pretty expensive. ShayFrom: Tom Graves Sent: Thursday, November 3, 2022 7:56 PM To: Artemis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using

Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

rsday, November 3, 2022 7:56 PM To: Artemis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. Stage level scheduling does not allow you to change co

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

ERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs *ATTENTION:*This email originated from outside of GM. Shay,  You may find this video helpful (with some API code samples that you are looking for). https://www.youtube.com/watch?v=JNQu-226wUc&t=171s <ht

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

he profile with 1 GPU per executor. So, the question is how do I limit the stage resources to 20 GPUs total?  Thanks again,Shay From: Artemis User Sent: Thursday, November 3, 2022 5:23 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

Er, wait, this is what stage-level scheduling is right? this has existed since 3.1 https://issues.apache.org/jira/browse/SPARK-27495 On Thu, Nov 3, 2022 at 12:10 PM bo yang wrote: > Interesting discussion here, looks like Spark does not support configuring > different number of execut

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

Interesting discussion here, looks like Spark does not support configuring different number of executors in different stages. Would love to see the community come out such a feature. On Thu, Nov 3, 2022 at 9:10 AM Shay Elbaz wrote: > Thanks again Artemis, I really appreciate it. I have watc

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

to 20 GPUs total? Thanks again, Shay From: Artemis User Sent: Thursday, November 3, 2022 5:23 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from out

Re: [EXTERNAL] Re: Stage level scheduling - lower the number of executors when using GPUs

Shay,  You may find this video helpful (with some API code samples that you are looking for). https://www.youtube.com/watch?v=JNQu-226wUc&t=171s.  The issue here isn't how to limit the number of executors but to request for the right GPU-enabled executors dynamically.  Those executor

Re: [EXTERNAL] Re: Stage level scheduling - lower the number of executors when using GPUs

Thanks Artemis. We are not using Rapids, but rather using GPUs through the Stage Level Scheduling feature with ResourceProfile. In Kubernetes you have to turn on shuffle tracking for dynamic allocation, anyhow. The question is how we can limit the number of executors when building a new

Re: Stage level scheduling - lower the number of executors when using GPUs

for a GPU stage than for a CPU stage. We are using dynamic allocation with stage level scheduling, and Spark tries to maximize the number of executors also during the GPU stage, causing a bit of resources chaos in the cluster. This forces us to use a lower value for 'maxExecutors' in

Stage level scheduling - lower the number of executors when using GPUs

Hi, Our typical applications need less executors for a GPU stage than for a CPU stage. We are using dynamic allocation with stage level scheduling, and Spark tries to maximize the number of executors also during the GPU stage, causing a bit of resources chaos in the cluster. This forces us to

Spark streaming job not able to launch more number of executors

, after which we fork out the DAG into two. Each of the forks has an action (forEach) at the end. In this case, we are observing that the number of executors is not exceeding the number of input kafka partitions. Job is not spawning more than 60 executors (2*10*3). And we see that the tasks from the

Re: Databricks - number of executors, shuffle.partitions etc

Thanks Ayan, I wasn't aware of such user group specifically for databricks. Thanks for the input, much appreciated! On Wed, May 15, 2019 at 10:07 PM ayan guha wrote: > Well its a databricks question so better be asked in their forum. > > You can set up cluster level params when you create new cl

Re: Databricks - number of executors, shuffle.partitions etc

Well its a databricks question so better be asked in their forum. You can set up cluster level params when you create new cluster or add them later. Go to cluster page, ipen one cluster, expand additional config section and add your param there as key value pair separated by space. On Thu, 16 May

Re: Databricks - number of executors, shuffle.partitions etc

Hi All, Any idea? Thanks, -Rishi On Tue, May 14, 2019 at 11:52 PM Rishi Shah wrote: > Hi All, > > How can we set spark conf parameter in databricks notebook? My cluster > doesn't take into account any spark.conf.set properties... it creates 8 > worker nodes (dat executors) but doesn't honor th

Databricks - number of executors, shuffle.partitions etc

Hi All, How can we set spark conf parameter in databricks notebook? My cluster doesn't take into account any spark.conf.set properties... it creates 8 worker nodes (dat executors) but doesn't honor the supplied conf parameters. Any idea? -- Regards, Rishi Shah

Re: Spark Streaming - Increasing number of executors slows down processing rate

Hi Edwin, I have faced a similar issue as well and this behaviour is very abrupt. I even created a question on StackOverflow but no solution yet. https://stackoverflow.com/questions/43496205/spark-job-processing-time-increases-to-4s-without-explanation For us, we sometimes had this constant delay

Spark Streaming - Increasing number of executors slows down processing rate

Hi All, I am struggling with an odd issue and would like your help in addressing it. Environment AWS Cluster (40 Spark Nodes & 4 node Kafka cluster) Spark Kafka Streaming submitted in Yarn cluster mode Kafka - Single topic, 400 partitions Spark 2.1 on Cloudera Kafka 10.0 on Cloudera We have zero

Spark streaming uses lesser number of executors

get 2 executors. My batch interval is 1 second. While running what I observe from event timeline is that only 3 of the executors are being used. The other 3 are not being used. As far as I know, there is no parameter in spark standalone mode to specify the number of executors. How do I make spark to

What factors decide the number of executors when doing a Spark SQL insert in Mesos?

Hi, What factors decide the number of executors when doing a Spark SQL insert? Right now when I submit my job in Mesos I see only 2 executors getting allocated all the time. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/What-factors-decide-the

Re: Number of executors change during job running

will give deeper understanding of the problem. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-change-during-job-running-tp9243p26866.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Number of executors in spark-1.6 and spark-1.5

increase no of worker instances on each node then >> jobs >> are running in parallel as no of workers but within same executor. >> >> Can anyone suggest, why spark 1.6 can not use multiple executors across >> different node at a time for parallel processing. >> Your

Re: Number of executors in spark-1.6 and spark-1.5

> Your suggestion will be highly appreciated. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-spark-1-5-tp26733.html > Sent from the Apache Spark User List mailing list archive

Number of executors in spark-1.6 and spark-1.5

parallel processing. Your suggestion will be highly appreciated. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-spark-1-5-tp26733.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: reasonable number of executors

Hi Igor, That's a great talk and an exact answer to my question. Thank you. Cheers, Alex. On Tue, Feb 23, 2016 at 8:27 PM, Igor Berman wrote: > > http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications > > there is a section that is connected to your q

Re: reasonable number of executors

http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications there is a section that is connected to your question On 23 February 2016 at 16:49, Alex Dzhagriev wrote: > Hello all, > > Can someone please advise me on the pros and cons on how to allocate the >

Re: reasonable number of executors

Hi Alex, take a look here : https://blogs.aws.amazon.com/bigdata/post/Tx3RD6EISZGHQ1C/The-Impact-of-Using-Latest-Generation-Instances-for-Your-Amazon-EMR-Job Bas

reasonable number of executors

Hello all, Can someone please advise me on the pros and cons on how to allocate the resources: many small heap machines with 1 core or few machines with big heaps and many cores? I'm sure that depends on the data flow and there is no best practise solution. E.g. with bigger heap I can perform map-

Re: Specify number of executors in standalone cluster mode

Max number of cores per executor can be controlled using spark.executor.cores. And maximum number of executors on a single worker can be determined by environment variable: SPARK_WORKER_INSTANCES. However, to ensure that all available cores are used, you will have to take care of how the stream

Specify number of executors in standalone cluster mode

Hi, I'm running a spark streaming application onto a spark cluster that spans 6 machines/workers. I'm using spark cluster standalone mode. Each machine has 8 cores. Is there any way to specify that I want to run my application on all 6 machines and just use 2 cores on each machine? Thanks

Re: Number of executors in Spark - Kafka

-line flag or spark.executor.instances > configuration > property control the number of executors requested. Starting in CDH > 5.4/Spark 1.3, you will be able to avoid setting this property by turning > on dynamic allocation > <https://spark.apache.org/docs/latest/job-scheduling.html#

Number of executors in Spark - Kafka

c the number executors to six. If I don't specific anything it just create one executor. Looking for information I have read: "The --num-executors command-line flag or spark.executor.instances configuration property control the number of executors requested. Starting in CDH 5.4/Spark 1.3, you

Re: number of executors in sparkR.init()

t with sparkR.init()? > > > _ > From: Franc Carter > Sent: Friday, December 25, 2015 9:23 PM > Subject: number of executors in sparkR.init() > To: > > > > Hi, > > I'm having trouble working out how to get the number of execut

Re: number of executors in sparkR.init()

9:23 PM Subject: number of executors in sparkR.init() To: Hi, I'm having trouble working out how to get the number of executors set when using sparkR.init(). If I start sparkR with   sparkR  --master yarn --num-execut

number of executors in sparkR.init()

Hi, I'm having trouble working out how to get the number of executors set when using sparkR.init(). If I start sparkR with sparkR --master yarn --num-executors 6 then I get 6 executors However, if start sparkR with sparkR followed by sc <- sparkR.init(master=&quo

RE: ideal number of executors per machine

spark jobs unstable. Thx tri From: Veljko Skarich [mailto:veljko.skar...@gmail.com] Sent: Tuesday, December 15, 2015 3:08 PM To: user@spark.apache.org Subject: ideal number of executors per machine Hi, I'm looking for suggestions on the ideal number of executors per machine. I run my jobs o

Re: ideal number of executors per machine

9:07 PM, Veljko Skarich >> wrote: >>> Hi, >>> >>> I'm looking for suggestions on the ideal number of executors per machine. I >>> run my jobs on 64G 32 core machines, and at the moment I have one executor >>> running per machine, on the s

Re: ideal number of executors per machine

> Hi, > > I'm looking for suggestions on the ideal number of executors per machine. I > run my jobs on 64G 32 core machines, and at the moment I have one executor > running per machine, on the spark standalone cluster. > > I could not find many guidelines for figuring o

Re: ideal number of executors per machine

Best Regards, Jerry > On Dec 15, 2015, at 5:18 PM, Jakob Odersky wrote: > > Hi Veljko, > I would assume keeping the number of executors per machine to a minimum is > best for performance (as long as you consider memory requirements as well). > Each executor is a proces

Re: ideal number of executors per machine

Hi Veljko, I would assume keeping the number of executors per machine to a minimum is best for performance (as long as you consider memory requirements as well). Each executor is a process that can run tasks in multiple threads. On a kernel/hardware level, thread switches are much cheaper than

ideal number of executors per machine

Hi, I'm looking for suggestions on the ideal number of executors per machine. I run my jobs on 64G 32 core machines, and at the moment I have one executor running per machine, on the spark standalone cluster. I could not find many guidelines for figuring out the ideal number of executors

Re: How to set the number of executors and tasks in a Spark Streaming job in Mesos

many partitions are you seeing for that job? You can try doing a dstream.repartition to see if it increase from 11 to a higher number. Thanks Best Regards On Thu, Aug 20, 2015 at 2:28 AM, swetha wrote: > Hi, > > How to set the number of executors and tasks in a Spark Streaming job in

Re: Finding the number of executors.

As I was writing a long-ish message to explain how it doesn't work, it dawned on me that maybe driver connects to executors only after there's some work to do (while I was trying to find the number of executors BEFORE starting the actual work). So the solution was to simply execute a

Re: Finding the number of executors.

= sc.getConf.get("spark.driver.host")    allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList  } On Friday, August 21, 2015 1:53 PM, Virgil Palanciuc wrote: Hi Akhil, I'm using spark 1.4.1. Number of executors is not in the command line, not in the g

Re: Finding the number of executors.

Hi Akhil, I'm using spark 1.4.1. Number of executors is not in the command line, not in the getExecutorMemoryStatus (I already mentioned that I tried that, works in spark-shell but not when executed via spark-submit). I tried looking at "defaultParallelism" too, it's 112 (7

Re: Finding the number of executors.

-laqgpdanvhm...@mail.gmail.com%3E On Aug 21, 2015 7:42 AM, "Virgil Palanciuc" wrote: > Is there any reliable way to find out the number of executors > programatically - regardless of how the job is run? A method that > preferably works for spark-standalone, yarn, mesos, regardless whet

Finding the number of executors.

Is there any reliable way to find out the number of executors programatically - regardless of how the job is run? A method that preferably works for spark-standalone, yarn, mesos, regardless whether the code runs from the shell or not? Things that I tried and don't

How to set the number of executors and tasks in a Spark Streaming job in Mesos

Hi, How to set the number of executors and tasks in a Spark Streaming job in Mesos? I have the following settings but my job still shows me 11 active tasks and 11 executors. Any idea as to why this is happening ? sparkConf.set("spark.mesos.coarse", "true") sparkConf.

Re: Controlling number of executors on Mesos vs YARN

s can specify desirable number of executors. If not available, Mesos (in a simple implementation) can provide/offer whatever is available. In a slightly complex implementation, we can build a simple protocol to negotiate. Regards, Ajay On Wed, Aug 12, 2015 at 5:51 PM, Tim Chen wrote: > You're

Re: Controlling number of executors on Mesos vs YARN

Hi Tim, An option like spark.mesos.executor.max to cap the number of executors per node/application would be very useful. However, having an option like spark.mesos.executor.num to specify desirable number of executors per node would provide even/much better control. Thanks, Ajay On Wed, Aug

Re: Controlling number of executors on Mesos vs YARN

You're referring to both fine grain and coarse grain? Desirable number of executors per node could be interesting but it can't be guaranteed (or we could try to and when failed abort the job). How would you imagine this new option to actually work? Tim On Wed, Aug 12, 2015 at 11:4

Re: Controlling number of executors on Mesos vs YARN

my iPad > > On 2015-08-11, at 2:38, Haripriya Ayyalasomayajula > wrote: > >> Hi Tim, >> >> Spark on Yarn allows us to do it using --num-executors and --executor_cores >> commandline arguments. I just got a chance to look at a similar spark user >> lis

Re: Controlling number of executors on Mesos vs YARN

. I just got a chance to look at a >> similar spark user list mail, but no answer yet. So does mesos allow >> setting the number of executors and cores? Is there a default number it >> assumes? >> >> On Mon, Jan 5, 2015 at 5:07 PM, Tim Chen wrote: >> >>> Forg

Re: Controlling number of executors on Mesos vs YARN

priya Ayyalasomayajula > wrote: > > Hi Tim, > > Spark on Yarn allows us to do it using --num-executors and > --executor_cores commandline arguments. I just got a chance to look at a > similar spark user list mail, but no answer yet. So does mesos allow > setting the number of exec

Re: Controlling number of executors on Mesos vs YARN

; Hi Tim, > > Spark on Yarn allows us to do it using --num-executors and --executor_cores > commandline arguments. I just got a chance to look at a similar spark user > list mail, but no answer yet. So does mesos allow setting the number of > executors and cores? Is there a default number

Re: Controlling number of executors on Mesos vs YARN

Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores? Is there a default number it assumes? On Mon

Re: Determining number of executors within RDD

re than 1 workers per machine > > > Sent from Samsung Mobile > > > Original message > From: Sandy Ryza > Date:2015/06/10 21:31 (GMT+00:00) > To: Evo Eftimov > Cc: maxdml ,user@spark.apache.org > Subject: Re: Determining number of executors within

Re: Determining number of executors within RDD

Sent from Samsung Mobile Original message From: maxdml Date:2015/06/10 19:56 (GMT+00:00) To: user@spark.apache.org Subject: Re: Determining number of executors within RDD Actually this is somehow confusing for two reasons: - First, the option 'spark.executor.instances',

Re: Determining number of executors within RDD

> > > Sent from Samsung Mobile > > > Original message > From: maxdml > Date:2015/06/10 19:56 (GMT+00:00) > To: user@spark.apache.org > Subject: Re: Determining number of executors within RDD > > Actually this is somehow confusing for two reasons

Re: Determining number of executors within RDD

/executot Sent from Samsung Mobile Original message From: maxdml Date:2015/06/10 19:56 (GMT+00:00) To: user@spark.apache.org Subject: Re: Determining number of executors within RDD Actually this is somehow confusing for two reasons: - First, the option

Re: Determining number of executors within RDD

icate that it is also available for this mode - Second, a post from Andrew Or states that this properties define the number of workers in the cluster, not the number of executors on a given worker. (http://apache-spark-user-list.1001560.n3.nabble.com/clarification-for-some-spark-on-yarn-configurati

Re: Determining number of executors within RDD

Note that this property is only available for YARN -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23256.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Determining number of executors within RDD

partitions. And of course if you want you can determine the number of executor as well by setting 'spark.executor.instances' property in 'sparkConf' object. Thank you. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of

Re: Determining number of executors within RDD

spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23234.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: number of executors

Yeah, I read that page before, but it does not mention the options should come before the application jar. Actually, if I put the --class option before the application jar, I will get ClassNotFoundException. Anyway, thanks again Sandy. On Tue, May 19, 2015 at 11:06 AM, Sandy Ryza wrote: > Awes

Re: number of executors

Awesome! It's documented here: https://spark.apache.org/docs/latest/submitting-applications.html -Sandy On Mon, May 18, 2015 at 8:03 PM, xiaohe lan wrote: > Hi Sandy, > > Thanks for your information. Yes, spark-submit --master yarn > --num-executors 5 --executor-cores 4 > target/scala-2.10/sim

Re: number of executors

Hi Sandy, Thanks for your information. Yes, spark-submit --master yarn --num-executors 5 --executor-cores 4 target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp is working awesomely. Is there any documentations pointing to this ? Thanks, Xiaohe On Tue, May 19, 2015 at 12:07 AM,

Re: number of executors

Oh BTW, it's spark 1.3.1 on hadoop 2.4. AIM 3.6. Sorry for lefting out this information. Appreciate for any help! Ed 2015-05-18 12:53 GMT-04:00 edward cui : > I actually have the same problem, but I am not sure whether it is a spark > problem or a Yarn problem. > > I set up a five nodes cluste

Re: number of executors

I actually have the same problem, but I am not sure whether it is a spark problem or a Yarn problem. I set up a five nodes cluster on aws emr, start yarn daemon on the master (The node manager will not be started on default on the master, I don't want to waste any resource since I have to pay). An

Re: number of executors

*All On Mon, May 18, 2015 at 9:07 AM, Sandy Ryza wrote: > Hi Xiaohe, > > The all Spark options must go before the jar or they won't take effect. > > -Sandy > > On Sun, May 17, 2015 at 8:59 AM, xiaohe lan > wrote: > >> Sorry, them both are assigned task actually. >> >> Aggregated Metrics by Exec

Re: number of executors

Hi Xiaohe, The all Spark options must go before the jar or they won't take effect. -Sandy On Sun, May 17, 2015 at 8:59 AM, xiaohe lan wrote: > Sorry, them both are assigned task actually. > > Aggregated Metrics by Executor > Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded TasksInpu

Re: number of executors

Sorry, them both are assigned task actually. Aggregated Metrics by Executor Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded TasksInput Size / RecordsShuffle Write Size / RecordsShuffle Spill (Memory)Shuffle Spill (Disk)1host1:61841.7 min505640.0 MB / 12318400382.3 MB / 121007701630.4 M

Re: number of executors

bash-4.1$ ps aux | grep SparkSubmit xilan 1704 13.2 1.2 5275520 380244 pts/0 Sl+ 08:39 0:13 /scratch/xilan/jdk1.8.0_45/bin/java -cp /scratch/xilan/spark/conf:/scratch/xilan/spark/lib/spark-assembly-1.3.1-hadoop2.4.0.jar:/scratch/xilan/spark/lib/datanucleus-core-3.2.10.jar:/scratch/xilan/sp

Re: number of executors

Did you try --executor-cores param? While you submit the job, do a ps aux | grep spark-submit and see the exact command parameters. Thanks Best Regards On Sat, May 16, 2015 at 12:31 PM, xiaohe lan wrote: > Hi, > > I have a 5 nodes yarn cluster, I used spark-submit to submit a simple app. > > s

Re: number of executors

What Spark release are you using ? Can you check driver log to see if there is some clue there ? Thanks On Sat, May 16, 2015 at 12:01 AM, xiaohe lan wrote: > Hi, > > I have a 5 nodes yarn cluster, I used spark-submit to submit a simple app. > > spark-submit --master yarn target/scala-2.10/sim

number of executors

Hi, I have a 5 nodes yarn cluster, I used spark-submit to submit a simple app. spark-submit --master yarn target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp --num-executors 5 I have set the number of executor to 5, but from sparkui I could see only two executors and it ran ve

Re: Number of Executors per worker process

>> SPARK_WORKER_CORES)? >> 3. If the worker can have many executors, how this is handled by the >> Spark? Or can I handle by myself to set up the number of executors per JVM? >> >> I look forward for your answers. >> Regards, >> Florin >> > >

Re: Number of Executors per worker process

he pool size > is equal with the number of the given cores (set up by the > SPARK_WORKER_CORES)? > 3. If the worker can have many executors, how this is handled by the > Spark? Or can I handle by myself to set up the number of executors per JVM? > > I look forward for your answers. > Regards, > Florin >

Number of Executors per worker process

y the SPARK_WORKER_CORES)? 3. If the worker can have many executors, how this is handled by the Spark? Or can I handle by myself to set up the number of executors per JVM? I look forward for your answers. Regards, Florin

Re: Setting the number of executors in standalone mode

though. > > > > Mohammed > > > > *From:* Yiannis Gkoufas [mailto:johngou...@gmail.com] > *Sent:* Friday, February 20, 2015 9:48 AM > *To:* Mohammed Guller > *Cc:* user@spark.apache.org > *Subject:* Re: Setting the number of executors in standalone mode > > > > Hi Mo

RE: Setting the number of executors in standalone mode

: Re: Setting the number of executors in standalone mode Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller mailto:moham

RE: Setting the number of executors in standalone mode

SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed From: Yiannis Gkoufas [mailto:johngou...@gmail.com] Sent: Friday, February 20, 2015 4:55 AM To: user@spark.apache.org Subject: Setting the number of executors in standalone

Re: Setting the number of executors in standalone mode

Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller wrote: > SPARK_WORKER_MEMORY=8g > > Will allocate 8GB memory to

Setting the number of executors in standalone mode

Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g

Re: Problems with GC and time to execute with different number of executors.

gt;> > System.currentTimeMillis()+ >>> >> > "/"); >>> >> > >>> >> > The parse function just takes an array of bytes and applies some >>> >> > transformations like,,, >>> >> > [0..3]

Re: Problems with GC and time to execute with different number of executors.

t a test code, I'd like to understand what it's happeing. >> >> > >> >> > 2015-02-04 18:57 GMT+01:00 Sandy Ryza : >> >> >> Hi Guillermo, >> >> >> >> >> >> What exactly do you mean by "each iteration&qu

Re: Problems with GC and time to execute with different number of executors.

yza : > >> >> Hi Guillermo, > >> >> > >> >> What exactly do you mean by "each iteration"? Are you caching data > in > >> >> memory? > >> >> > >> >> -Sandy > >> >> > >> >> On Wed, Feb 4, 2015 at 5

Re: Problems with GC and time to execute with different number of executors.

t; Hi Guillermo, >> >> >> >> What exactly do you mean by "each iteration"? Are you caching data in >> >> memory? >> >> >> >> -Sandy >> >> >> >> On Wed, Feb 4, 2015 at 5:02 AM, Guillermo Ortiz >> &

Re: Problems with GC and time to execute with different number of executors.

ry? > >> > >> -Sandy > >> > >> On Wed, Feb 4, 2015 at 5:02 AM, Guillermo Ortiz > >> wrote: > >>> > >>> I execute a job in Spark where I'm processing a file of 80Gb in HDFS. > >>> I have 5 slaves: > >>>

Re: Problems with GC and time to execute with different number of executors.

; >>> I execute a job in Spark where I'm processing a file of 80Gb in HDFS. >>> I have 5 slaves: >>> (32cores /256Gb / 7physical disks) x 5 >>> >>> I have been trying many different configurations with YARN. >>> yarn.nodemanager.resource.mem

Re: Problems with GC and time to execute with different number of executors.

ote: >> >> I execute a job in Spark where I'm processing a file of 80Gb in HDFS. >> I have 5 slaves: >> (32cores /256Gb / 7physical disks) x 5 >> >> I have been trying many different configurations with YARN. >> yarn.nodemanager.resource.memory-mb

Re: Problems with GC and time to execute with different number of executors.

hysical disks) x 5 > > I have been trying many different configurations with YARN. > yarn.nodemanager.resource.memory-mb 196Gb > yarn.nodemanager.resource.cpu-vcores 24 > > I have tried to execute the job with different number of executors a > memory (1-4g) > With 20 executo

Problems with GC and time to execute with different number of executors.

e the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting because GC. When I execute around 60 executors the process time it's about 45s and some tasks take until one minute because GC. I have n

Fwd: Controlling number of executors on Mesos vs YARN

Forgot to hit reply-all. -- Forwarded message -- From: Tim Chen Date: Sun, Jan 4, 2015 at 10:46 PM Subject: Re: Controlling number of executors on Mesos vs YARN To: mvle Hi Mike, You're correct there is no such setting in for Mesos coarse grain mode, since the assumpti

Controlling number of executors on Mesos vs YARN

I'm trying to compare the performance of Spark running on Mesos vs YARN. However, I am having problems being able to configure the Spark workload to run in a similar way on Mesos and YARN. When running Spark on YARN, you can specify the number of executors per node. So if I have a node w

Re: Specifying number of executors in Mesos

Gerard, Are you familiar with spark.deploy.spreadOut in Standalone mode? It sounds like you want the same thing in Mesos mode. On Thu, Dec 11, 2014 at 6:48 AM, Tathagata Das wrote: > Not that I am aware of. Spark will try to spread th

  1   2   >