Re: Understanding executor memory behavior

All else equal it is better to have the same resources in fewer executors. More tasks are local to other tasks which helps perf. There is more possibility of 'borrowing' extra mem and CPU in a task. On Thu, Mar 16, 2023, 2:14 PM Nikhil Goyal wrote: > Hi folks, > I am trying to understand what

Understanding executor memory behavior

Hi folks, I am trying to understand what would be the difference in running 8G 1 core executor vs 40G 5 core executors. I see that on yarn it can cause bin fitting issues but other than that are there any pros and cons on using either? Thanks Nikhil

Re: [EXTERNAL] [SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ?

Yosr Kchaou Sent: Wednesday, December 7, 2022 10:19 AM To: user@spark.apache.org Subject: [EXTERNAL] [SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ? ATTENTION: This email originated from outside of GM. Hello, We are running Spark on Kubernetes

[SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ?

Hello, We are running Spark on Kubernetes and noticed that driver/executors use the same value for memory request and memory limit. We see that limits/requests can be set only for cpu using the following options: spark.kubernetes.{driver/executor}.limit.cores and

Re: How to estimate the executor memory size according by the data

wrote: > Dear experts, > > Recently there’s some OOM issue in my demo jobs which consuming data from > the hive database, and I know I can increase the executor memory size to > eliminate the OOM error. While I don’t know how to do the executor memory > assessment and how to a

RE: How to estimate the executor memory size according by the data

23, 2021 15:11 To: user@spark.apache.org Subject: How to estimate the executor memory size according by the data Dear experts, Recently there’s some OOM issue in my demo jobs which consuming data from the hive database, and I know I can increase the executor memory size to eliminate the OOM

How to estimate the executor memory size according by the data

Dear experts, Recently there’s some OOM issue in my demo jobs which consuming data from the hive database, and I know I can increase the executor memory size to eliminate the OOM error. While I don’t know how to do the executor memory assessment and how to automatically adopt the executor

RE: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

. Regards Ranju From: Attila Zsolt Piros Sent: Monday, March 22, 2021 11:07 AM To: Ranju Jain Cc: Mich Talebzadeh ; user@spark.apache.org Subject: Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU Hi Ranju! I am quite sure for your requirement "monitor

Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

see all the > metrics can be viewed , but I have less idea for steps to install on > virtual server and integrate. I need to go through in detail the Grafana. > > > > Kindly suggest your views. > > > > Regards > > Ranju > > > > *From:*

RE: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

ebzadeh Cc: Ranju Jain ; user@spark.apache.org Subject: Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU Hi Ranju! You can configure Spark's metric system. Check the memoryMetrics.* of executor-metrics<https://spark.apache.org/docs/3.0.0-preview/monitoring.

Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

Hi Ranju! You can configure Spark's metric system. Check the *memoryMetrics.** of executor-metrics and in the component-instance-executor

Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

Hi, Have you considered spark GUI first? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on

Can JVisual VM monitoring tool be used to Monitor Spark Executor Memory and CPU

Hi All, Virtual Machine running an application, this application is having various other 3PPs components running such as spark, database etc . My requirement is to monitor every component and isolate the resources consuming individually by every component. I am thinking of using a common tool

Re: How shall I configure the Spark executor memory size and the Alluxio worker memory size on a machine?

oops, sorry for the confusion. I mean "20% of the size of your input data set" allocated to Alluxio as memory resource as the starting point. after that, you can checkout the cache hit ratio into Alluxio space based on the metrics collected in Alluxio web UI

Re: How shall I configure the Spark executor memory size and the Alluxio worker memory size on a machine?

Hi Andy, It really depends on your workloads. I would suggest to allocate 20% of the size of your input data set as the starting point and see how it works. Also depending on your data source as the under store of Alluxio, if it is remote (e.g., cloud storage like S3 or GCS), you can perhaps use

How shall I configure the Spark executor memory size and the Alluxio worker memory size on a machine?

Hey, We have a cluster of 10 nodes each of which consists 128GB memory. We are about to running Spark and Alluxio on the cluster. We wonder how shall allocate the memory to the Spark executor and the Alluxio worker on a machine? Are there some recommendations? Thanks! Best, Andy Li

Re: [E] Re: Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

This typically works ok for standalone mode with moderate resources ${SPARK_HOME}/bin/spark-submit \ --driver-memory 6G \ --executor-memory 2G \ --num-executors 2 \ --executor-cores 2 \ --master spark

Re: [E] Re: Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

7 at 5:02 PM To: Abdulfattah Safa <fattah.s...@gmail.com<mailto:fattah.s...@gmail.com>> Cc: User <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: [E] Re: Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory You can add memory in your command m

Re: Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

You can add memory in your command make sure given memory available on your executor ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --executor-memory 20G \ --total-executor-cores 100 \ /path/to/examples.jar \ 1000 https

Re: Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

setting > the Driver Memory to > Executor Memory, the submitted job is stuck at > Submitted in the driver and the application never starts.

Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

I'm working on Spark with Standalone Cluster mode. I need to increase the Driver Memory as I got OOM in t he driver thread. If found that when setting the Driver Memory to > Executor Memory, the submitted job is stuck at Submitted in the driver and the application never starts.

Spark Job is stuck at SUBMITTED when set Driver Memory > Executor Memory

I'm working on Spark with Standalone Cluster mode. I need to increase the Driver Memory as I got OOM in t he driver thread. If found that when setting the Driver Memory to > Executor Memory, the submitted job is stuck at Submitted in the driver and the application never starts.

Re: spark executor memory, jvm config

OK, I found the problem. There is a typo in my configuration. As a result, the executor dynamic allocation is not disabled. So, the executors get killed and requested from time to time. All good now. On Wed, Mar 8, 2017 at 2:45 PM, TheGeorge1918 . wrote: > Hello all, >

spark executor memory, jvm config

Hello all, I was running some spark job and some executors failed without error info. The executors were dead and new executors were requested but on the spark web UI, no failure found. Normally, if it's memory issue, I could find OOM ther, but not this time. Configuration: 1. each executor has

Spark executor memory and jvm heap memory usage metric

that if the max heap usage found by this method is less than the executor memory allotted, then we can safely tune down the executor memory to max heap usage (approx). Related to this - when we specify executor-memory as 'X'gb, is all of this X'gb being allotted from jvm heap memory? are there any other

Driver/Executor Memory values during Unit Testing

Hi there, I've been trying to increase the spark.driver.memory and spark.executor.memory during some unit tests. Most of the information I can find about increasing memory for Spark is based on either flags to spark-submit, or settings in the spark-defaults.conf file. Running unit tests with

Re: Question about executor memory setting

gt; wrote: > > Hi all, > > May I ask a question about executor memory setting? I was running PageRank > with input size 2.8GB on one workstation for testing. I gave PageRank one > executor. > > In case 1, I set --executor-cores to 4, and --executor-memory to 1GB, the

Question about executor memory setting

Hi all, May I ask a question about executor memory setting? I was running PageRank with input size 2.8GB on one workstation for testing. I gave PageRank one executor. In case 1, I set --executor-cores to 4, and --executor-memory to 1GB, the stage (stage 2) completion time is 14 min

Re: Standalone executor memory is fixed while executor cores are load balanced between workers

.executor.memory to some lower amount, than I can end > up with less executors, even a single one (if other nodes are full) which > wouldn't have enough memory. I don't know how to configure executor memory > in a predictable way. > > The only predictable way we found is to set

Standalone executor memory is fixed while executor cores are load balanced between workers

, each with 8GB => 64GB RAM, instead of about 8GB I need for app. If I would plan spark.executor.memory to some lower amount, than I can end up with less executors, even a single one (if other nodes are full) which wouldn't have enough memory. I don't know how to configure executor mem

Re: num-executors, executor-memory and executor-cores parameters

. There is no resource policing num-executors --> 1 executor-memory --> You can give as much as you can afford. executor-cores --> will go and grab what you have specified in --master local[n] Standalone mode Resources are managed by Spark resource manager itself. You start your master and slav

num-executors, executor-memory and executor-cores parameters

Hi I would like to know the exact definition for these three  parameters  num-executors executor-memory executor-cores for local, standalone and yarn modes I have looked at on-line doc but not convinced if I understand them correct. Thanking you 

Re: Executor memory requirement for reduceByKey

Even though it does not sound intuitive, reduce by key expects all values for a particular key for a partition to be loaded into memory. So once you increase the partitions you can run the jobs.

Re: Executor memory requirement for reduceByKey

Ok, so that worked flawlessly after I upped the number of partitions to 400 from 40. Thanks! On Fri, May 13, 2016 at 7:28 PM, Sung Hwan Chung wrote: > I'll try that, as of now I have a small number of partitions in the order > of 20~40. > > It would be great if

Re: Executor memory requirement for reduceByKey

I'll try that, as of now I have a small number of partitions in the order of 20~40. It would be great if there's some documentation on the memory requirement wrt the number of keys and the number of partitions per executor (i.e., the Spark's internal memory requirement outside of the user space).

Re: Executor memory requirement for reduceByKey

Have you taken a look at SPARK-11293 ? Consider using repartition to increase the number of partitions. FYI On Fri, May 13, 2016 at 12:14 PM, Sung Hwan Chung wrote: > Hello, > > I'm using Spark version 1.6.0 and have trouble with memory when trying to > do

Executor memory requirement for reduceByKey

Hello, I'm using Spark version 1.6.0 and have trouble with memory when trying to do reducebykey on a dataset with as many as 75 million keys. I.e. I get the following exception when I run the task. There are 20 workers in the cluster. It is running under the standalone mode with 12 GB assigned

How to take executor memory dump

o tried taking dump of running executor using jmap -dump. but it fails with exception in middle of it. It still generate some dump if I used -F option. However that file seem corrupted and not getting load into eclipse MAT or VisualVM. So what is the correct way to set this executor opts and ulti

coalesce executor memory explosion

an action. Without coalesce there is no memory issue. The size of the data makes no difference: hadoopFile (creates 14,844 partitions) -> sample (fraction 0.00075) -> count() Per executor memory usage: 0.4 GB Adding coalesce increases the memory usage substantially and it is still usin

Re: coalesce and executor memory

s issue. > > > - Original Message - > From: silvio.fior...@granturing.com > To: christopher.br...@oracle.com, ko...@tresata.com > Cc: user@spark.apache.org > Sent: Sunday, February 14, 2016 8:27:09 AM GMT -05:00 US/Canada Eastern > Subject: RE: coalesce and executor m

Re: coalesce and executor memory

Eastern Subject: RE: coalesce and executor memory Actually, rereading your email I see you're caching. But ‘cache’ uses MEMORY_ONLY. Do you see errors about losing partitions as your job is running? Are you sure you need to cache if you're just saving to disk? Can you try the coalesce

RE: coalesce and executor memory

<mailto:christopher.br...@oracle.com> Sent: Friday, February 12, 2016 8:34 PM To: Koert Kuipers<mailto:ko...@tresata.com>; Silvio Fiorito<mailto:silvio.fior...@granturing.com> Cc: user<mailto:user@spark.apache.org> Subject: Re: coalesce and executor memory Thank you for the responses. The m

Re: coalesce and executor memory

On Fri, Feb 12, 2016 at 11:10 PM, Koert Kuipers wrote: > in spark, every partition needs to fit in the memory available to the core > processing it. > That does not agree with my understanding of how it works. I think you could do

Re: coalesce and executor memory

On Fri, Feb 12, 2016 at 11:10 PM, Koert Kuipers wrote: > in spark, every partition needs to fit in the memory available to the core > processing it. > That does not agree with my understanding of how it works. I think you could do

Re: coalesce and executor memory

sorry i meant to say: and my way to deal with OOMs is almost always simply to increase number of partitions. maybe there is a better way that i am not aware of. On Sat, Feb 13, 2016 at 11:38 PM, Koert Kuipers wrote: > thats right, its the reduce operation that makes the

Re: coalesce and executor memory

thats right, its the reduce operation that makes the in-memory assumption, not the map (although i am still suspicious that the map actually streams from disk to disk record by record). in reality though my experience is that is spark can not fit partitions in memory it doesnt work well. i get

coalesce and executor memory

Can anyone help me understand why using coalesce causes my executors to crash with out of memory? What happens during coalesce that increases memory usage so much? If I do: hadoopFile -> sample -> cache -> map -> saveAsNewAPIHadoopFile everything works fine, but if I do: hadoopFile -> sample

Re: coalesce and executor memory

Coalesce essentially reduces parallelism, so fewer cores are getting more records. Be aware that it could also lead to loss of data locality, depending on how far you reduce. Depending on what you’re doing in the map operation, it could lead to OOM errors. Can you give more details as to what

Re: coalesce and executor memory

in spark, every partition needs to fit in the memory available to the core processing it. as you coalesce you reduce number of partitions, increasing partition size. at some point the partition no longer fits in memory. On Fri, Feb 12, 2016 at 4:50 PM, Silvio Fiorito <

Re: coalesce and executor memory

Thank you for the responses. The map function just changes the format of the record slightly, so I don't think that would be the cause of the memory problem. So if I have 3 cores per executor, I need to be able to fit 3 partitions per executor within whatever I specify for the executor memory

how to calculate -- executor-memory,num-executors,total-executor-cores

Hi, I would like to know how to calculate how much -executor-memory should we allocate , how many num-executors,total-executor-cores we should give while submitting spark jobs . Is there any formula for it ? Thanks, Divya

Re: how to calculate -- executor-memory,num-executors,total-executor-cores

Divya, According to my recent Spark tuning experiences, optimal executor-memory size not only depends on your workload characteristics (e.g. working set size at each job stage) and input data size, but also depends on your total available memory and memory requirements of other components like

Spark Yarn executor memory overhead content

Hello, In some of our spark applications, when writing outputs to hdfs we encountered an error about the spark yarn executor memory overhead : WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 3.0 GB of 3 GB physical memory used. Consider boosting

Re: Spark Yarn executor memory overhead content

On Thu, Jan 21, 2016 at 5:42 AM, Olivier Devoisin wrote: > The documentation states that it contains VM overheads, interned strings and > other native overheads. However it's really vague. It's intentionally vague, because it's "everything that is not Java

Different executor memory for different nodes

How to set different executor memory limits for different worker nodes? I'm using spark 1.5.2 in standalone deployment mode and launching using scripts. The executor memory is set via 'spark.executor.memory' in conf/spark-defaults.conf. This sets the same memory limit for all the worker nodes. I

Re: Spark on Mesos / Executor Memory

>>> https://github.com/apache/spark/pull/4027) >>>> >>>> It also might be interesting to include a cores to memory multiplier so >>>> that with a larger amount of cores we try to scale the memory with some >>>> factor, but I'm not

Re: Spark on Mesos / Executor Memory

fixed number. >>>>> >>>>> I have a patch out to configure how much max cpus should coarse grain >>>>> executor use, and it also allows multiple executors in coarse grain mode. >>>>> So you could say try to launch multiples of max 4 cores

Re: Spark on Mesos / Executor Memory

t;> factor, but I'm not entirely sure that's intuitive to use and what people >>> know what to set it to, as that can likely change with different workload. >>> >>> Tim >>> >>> >>> >>> >>> >>> >>>

Re: Spark on Mesos / Executor Memory

ad. >> >> Tim >> >> >> >> >> >> >> >> On Sat, Apr 11, 2015 at 9:51 AM, Tom Arnfeld <t...@duedil.com> wrote: >> >>> We're running Spark 1.3.0 (with a couple of patches over the top for >>> docker related bits). >&g

Re: Spark on Mesos / Executor Memory

at 9:51 AM, Tom Arnfeld <t...@duedil.com> wrote: > >> We're running Spark 1.3.0 (with a couple of patches over the top for >> docker related bits). >> >> I don't think SPARK-4158 is related to what we're seeing, things do run >> fine on the cluster, given a ridic

Setting Executor memory

Hello, I was looking for guidelines on what value to set executor memory to (via spark.executor.memory for example). This seems to be important to avoid OOM during tasks, especially in no swap environments (like AWS EMR clusters). This setting is really about the executor JVM heap. Hence

Re: Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

xamples is more than what is > necessary in usual cases and executor memory is quite less. > > I found this devops talk[1] at spark-summit here to be super useful in > understanding few of this configuration details. > > [1] https://.youtube.com/watch?v=l4ZYUfZuRbU > > Cheer

Re: Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

Hi Timothy, I think the driver memory in all your examples is more than what is necessary in usual cases and executor memory is quite less. I found this devops talk[1] at spark-summit here to be super useful in understanding few of this configuration details. [1] https://.youtube.com/watch?v

Re: Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

your message. */bin/spark-submit --class --master yarn-cluster --driver-memory 11g --executor-memory 1g --num-executors 3 --executor-cores 1 --jars * If I do not mess with the default memory overhead settings as above, I have to use driver memory greater than 10g for my job to run

Re: Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

Added log files and diagnostics to first and second cases and removed the images. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Effects-of-Driver-Memory-Executor-Memory-Driver-Memory-Overhead-and-Executor-Memory-Overhead-os-tp24507p24528.html Sent

Re: Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

st. > > First Case > > /`/bin/spark-submit --class --master yarn-cluster > --driver-memory 7g --executor-memory 1g --num-executors 3 --executor-cores > 1 > --jars ` > / > If I run my program with any driver memory less than 11g, I will get the > error below which is t

Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs

filters an RDD to make it smaller (removing examples as part of an algorithm), then does mapToPair and collect to gather the results and save them within a list. First Case /`/bin/spark-submit --class class name --master yarn-cluster --driver-memory 7g --executor-memory 1g --num-executors 3

Re: worker and executor memory

Additional Comment: I checked the disk usage on the 3 nodes (using iostat) and it seems that reading from HDFS partitions happen in a node-by-node basis. Only one of the nodes shows active IO (as read) at any given time while the other two nodes are idle IO-wise. I am not sure why the tasks are

worker and executor memory

Hi, I am using Spark 1.4 on a cluster (stand-alone mode), across 3 machines, for a workload similar to TPCH (analytical queries with multiple/multi-way large joins and aggregations). Each machine has 12GB of Memory and 4 cores. My total data size is 150GB, stored in HDFS (stored as Hive tables),

Re: Spark executor memory information

1. Yes open up the webui running on 8080 to see the memory/cores allocated to your workers, and open up the ui running on 4040 and click on the Executor tab to see the memory allocated for the executor. 2. mllib codes can be found over here https://github.com/apache/spark/tree/master/mllib and

Spark executor memory information

Hi, I am new to spark and need some guidance on below mentioned points: 1)I am using spark 1.2,is it possible to see how much memory is being allocated to an executor for web UI. If not how can we figure that out.2)    I am interested in source code of mlib,it is possible to get access to

Re: Executor memory allocations

default settings): 60% for cache 40% for tasks to process data Reading about how Spark implements shuffling, I've also seen it say 20% of executor memory is utilized for shuffles Does this 20% cut into the 40% for tasks to process data or the 60% for the data cache?

Executor memory allocations

So I've seen in the documentation that (after the overhead memory is subtracted), the memory allocations of each executor are as follows (assume default settings): 60% for cache 40% for tasks to process data Reading about how Spark implements shuffling, I've also seen it say 20% of executor

RE: Spark Executor Memory Usage

Subject: Re: Spark Executor Memory Usage From: yuzhih...@gmail.com To: valeramoisee...@gmail.com CC: user@spark.apache.org For #2, see http://unix.stackexchange.com/questions/65835/htop-reporting-much-higher-memory-usage-than-free-or-top Cheers On Fri, May 29, 2015 at 6:56 AM, Valerii Moisieienko

Spark Executor Memory Usage

in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor-Memory-Usage-tp23083.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: Spark Executor Memory Usage

questions regarding Spark's executors memory usage: 1. Why does Max Heap Size change during executor work? 2. Why does Memory usage via htop greater than executor's heap size? Thank you! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor

Re: Executor memory in web UI

This is the fraction available for caching, which is 60% * 90% * total by default. On Fri, Apr 17, 2015 at 11:30 AM, podioss grega...@hotmail.com wrote: Hi, i am a bit confused with the executor-memory option. I am running applications with Standalone cluster manager with 8 workers with 4gb

Executor memory in web UI

Hi, i am a bit confused with the executor-memory option. I am running applications with Standalone cluster manager with 8 workers with 4gb memory and 2 cores each and when i submit my application with spark-submit i use --executor-memory 1g. In the web ui in the completed applications table i see

Re: Spark on Mesos / Executor Memory

over the top for docker related bits). I don't think SPARK-4158 is related to what we're seeing, things do run fine on the cluster, given a ridiculously large executor memory configuration. As for SPARK-3535 although that looks useful I think we'e seeing something else. Put a different way

RE: EC2 spark-submit --executor-memory

If you are using Spark Standalone deployment, make sure you set the WORKER_MEMROY over 20G, and you do have 20G physical memory. Yong Date: Tue, 7 Apr 2015 20:58:42 -0700 From: li...@adobe.com To: user@spark.apache.org Subject: EC2 spark-submit --executor-memory Dear Spark team, I'm

Re: How to set Spark executor memory?

OK, I actually got the answer days ago from StackOverflow, but I did not check it :( When running in local mode, to set the executor memory - when using spark-submit, use --driver-memory - when running as a Java application, like executing from IDE, set the -Xmx vm option Thanks, David

Re: How to set Spark executor memory?

Hi Sean, It's getting strange now. If I ran from IDE, my executor memory is always set to 6.7G, no matter what value I set in code. I have check my environment variable, and there's no value of 6.7, or 12.5 Any idea? Thanks, David On Tue, 17 Mar 2015 00:35 null jishnu.prat...@wipro.com wrote

Re: How to set Spark executor memory?

If you are running from your IDE, then I don't know what you are running or in what mode. The discussion here concerns using standard mechanisms like spark-submit to configure executor memory. Please try these first instead of trying to directly invoke Spark, which will require more understanding

Re: How to set Spark executor memory?

In the log, I saw MemoryStorage: MemoryStore started with capacity 6.7GB But I still can not find where to set this storage capacity. On Sat, 21 Mar 2015 20:30 Xi Shen davidshe...@gmail.com wrote: Hi Sean, It's getting strange now. If I ran from IDE, my executor memory is always set

Re: How to set Spark executor memory?

wrote: In the log, I saw MemoryStorage: MemoryStore started with capacity 6.7GB But I still can not find where to set this storage capacity. On Sat, 21 Mar 2015 20:30 Xi Shen davidshe...@gmail.com wrote: Hi Sean, It's getting strange now. If I ran from IDE, my executor memory is always

Re: How to set Spark executor memory?

to set this storage capacity. On Sat, 21 Mar 2015 20:30 Xi Shen davidshe...@gmail.com wrote: Hi Sean, It's getting strange now. If I ran from IDE, my executor memory is always set to 6.7G, no matter what value I set in code. I have check my environment variable, and there's no value of 6.7

RE: How to set Spark executor memory?

, 2015 2:06 PM To: Xi Shen Cc: user@spark.apache.org Subject: Re: How to set Spark executor memory? By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the job using spark-submit and it is not able to override the value since you are running in local mode. Can you

Re: How to set Spark executor memory?

There are a number of small misunderstandings here. In the first instance, the executor memory is not actually being set to 2g and the default of 512m is being used. If you are writing code to launch an app, then you are trying to duplicate what spark-submit does, and you don't use spark-submit

How to set Spark executor memory?

Hi, I have set spark.executor.memory to 2048m, and in the UI Environment page, I can see this value has been set correctly. But in the Executors page, I saw there's only 1 executor and its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set? What am I missing here?

Re: How to set Spark executor memory?

How are you setting it? and how are you submitting the job? Thanks Best Regards On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen davidshe...@gmail.com wrote: Hi, I have set spark.executor.memory to 2048m, and in the UI Environment page, I can see this value has been set correctly. But in the

Re: How to set Spark executor memory?

I set it in code, not by configuration. I submit my jar file to local. I am working in my developer environment. On Mon, 16 Mar 2015 18:28 Akhil Das ak...@sigmoidanalytics.com wrote: How are you setting it? and how are you submitting the job? Thanks Best Regards On Mon, Mar 16, 2015 at

Re: How to set Spark executor memory?

By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the job using spark-submit and it is not able to override the value since you are running in local mode. Can you try it without using spark-submit as a standalone project? Thanks Best Regards On Mon, Mar 16,

Re: How to set Spark executor memory?

Hi Akhil, Yes, you are right. If I ran the program from IDE as a normal java program, the executor's memory is increased...but not to 2048m, it is set to 6.7GB...Looks like there's some formula to calculate this value. Thanks, David On Mon, Mar 16, 2015 at 7:36 PM Akhil Das

Re: How to set Spark executor memory?

How much memory are you having on your machine? I think default value is 0.6 of the spark.executor.memory as you can see from here http://spark.apache.org/docs/1.2.1/configuration.html#execution-behavior. Thanks Best Regards On Mon, Mar 16, 2015 at 2:26 PM, Xi Shen davidshe...@gmail.com wrote:

Re: How to set Spark executor memory?

I set spark.executor.memory to 2048m. If the executor storage memory is 0.6 of executor memory, it should be 2g * 0.6 = 1.2g. My machine has 56GB memory, and 0.6 of that should be 33.6G...I hate math xD On Mon, Mar 16, 2015 at 7:59 PM Akhil Das ak...@sigmoidanalytics.com wrote: How much

Re: How to set Spark executor memory?

of executor memory, it should be 2g * 0.6 = 1.2g. My machine has 56GB memory, and 0.6 of that should be 33.6G...I hate math xD On Mon, Mar 16, 2015 at 7:59 PM Akhil Das ak...@sigmoidanalytics.com wrote: How much memory are you having on your machine? I think default value is 0.6

How does unmanaged memory work with the executor memory limits?

Greetings! My executors apparently are being terminated because they are running beyond physical memory limits according to the yarn-hadoop-nodemanager logs on the worker nodes (/mnt/var/log/hadoop on AWS EMR).  I'm setting the driver-memory to 8G.However, looking at stdout in userlogs, I can

Re: How does unmanaged memory work with the executor memory limits?

Short answer: yes. Take a look at: http://spark.apache.org/docs/latest/running-on-yarn.html Look for memoryOverhead. On Mon, Jan 12, 2015 at 2:06 PM, Michael Albert m_albert...@yahoo.com.invalid wrote: Greetings! My executors apparently are being terminated because they are running beyond

Re: Executor memory

1.0.1 on Yarn 2.5 When i specify --executor-memory 4g, the spark UI shows each executor as having only 2.3 GB, and similarly for 8g, only 4.6 GB. I am guessing that the executor memory corresponds to the container memory, and that the task JVM gets only a percentage of the container

Re: Executor memory

on for the container memory. -Sandy On Dec 15, 2014, at 7:53 PM, Pala M Muthaia mchett...@rocketfuelinc.com wrote: Hi, Running Spark 1.0.1 on Yarn 2.5 When i specify --executor-memory 4g, the spark UI shows each executor as having only 2.3 GB, and similarly for 8g, only 4.6 GB. I

  1   2   >