Re: Idiomatic way to rate-limit streaming sources to avoid OutOfMemoryError?

at the source can generate > data faster than Spark can process it, eventually leading to an > OutOfMemoryError when Spark runs out of memory trying to queue up all > the pending data. > > I'm looking for advice on the most idiomatic/recommended way in Spark to > rate-limit data

Idiomatic way to rate-limit streaming sources to avoid OutOfMemoryError?

Hi Spark community, I have a Spark Structured Streaming application that reads data from a socket source (implemented very similarly to the TextSocketMicroBatchStream). The issue is that the source can generate data faster than Spark can process it, eventually leading to an OutOfMemoryError

Re: [Beginner Debug]: Executor OutOfMemoryError

Seems like you are having memory issues. Examine your settings. 1. It appears that your driver memory setting is too high. It should be a fraction of total memy provided by YARN 2. Use the Spark UI to monitor the job's memory consumption. Check the Storage tab to see how memory is

[Beginner Debug]: Executor OutOfMemoryError

Hi I'm new to Spark and I'm running into a lot of OOM issues while trying to scale up my first Spark application. I am running into these issues with only 1% of the final expected data size. Can anyone help me understand how to properly configure Spark to use limited memory or how to debug which

Re: OutOfMemoryError

Personally rather than Parameters here: val spark = SparkSession .builder .master("local[*]") .appName("OOM") .config("spark.driver.host", "localhost") .config("spark.driver.maxResultSize", "0") .config("spark.sql.caseSensitive", "false") .config("spark.sql.adaptive.enabled",

Re: OutOfMemoryError

Hi Sean, thx for the tip. I'm just running my app via spark-submit on CLI ie >spark-submit --class X --master local[*] assembly.jar so I'll now add to CLI args ie: spark-submit --class X --master local[*] --driver-memory 8g assembly.jar etc. Unless I have this wrong? Thx On Thu, Jul 1, 2021

Re: OutOfMemoryError

You need to set driver memory before the driver starts, on the CLI or however you run your app, not in the app itself. By the time the driver starts to run your app, its heap is already set. On Thu, Jul 1, 2021 at 12:10 AM javaguy Java wrote: > Hi, > > I'm getting Java OOM errors even though

OutOfMemoryError

Hi, I'm getting Java OOM errors even though I'm setting my driver memory to 24g and I'm executing against local[*] I was wondering if anyone can give me any insight. The server this job is running on has more than enough memory as does the spark driver. The final result does write 3 csv files

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

Thanks for the pointer, Vadim. However, I just tried it with Spark 2.4 and get the same failure. (I was previously testing with 2.2 and/or 2.3.) And I don't see this particular issue referred to there. The ticket that Harel commented on indeed appears to be the most similar one to this issue:

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

There's an umbrella ticket for various 2GB limitations https://issues.apache.org/jira/browse/SPARK-6235 On Fri, Nov 8, 2019 at 4:11 PM Jacob Lynn wrote: > > Sorry for the noise, folks! I understand that reducing the number of > partitions works around the issue (at the scale I'm working at,

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

Sorry for the noise, folks! I understand that reducing the number of partitions works around the issue (at the scale I'm working at, anyway) -- as I mentioned in my initial email -- and I understand the root cause. I'm not looking for advice on how to resolve my issue. I'm just pointing out that

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

Basically, the driver tracks partitions and sends it over to executors, so what it's trying to do is to serialize and compress the map but because it's so big, it goes over 2GiB and that's Java's limit on the max size of byte arrays, so the whole thing drops. The size of data doesn't matter here

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

File system is HDFS. Executors are 2 cores, 14GB RAM. But I don't think either of these relate to the problem -- this is a memory allocation issue on the driver side, and happens in an intermediate stage that has no HDFS read/write. On Fri, Nov 8, 2019 at 10:01 AM Spico Florin wrote: > Hi! >

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

Hi! What file system are you using: EMRFS or HDFS? Also what memory are you using for the reducer ? On Thu, Nov 7, 2019 at 8:37 PM abeboparebop wrote: > I ran into the same issue processing 20TB of data, with 200k tasks on both > the map and reduce sides. Reducing to 100k tasks each resolved

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

I ran into the same issue processing 20TB of data, with 200k tasks on both the map and reduce sides. Reducing to 100k tasks each resolved the issue. But this could/would be a major problem in cases where the data is bigger or the computation is heavier, since reducing the number of partitions may

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

I understand the error is because the number of partitions is very high, yet when processing 40 TB (and this number is expected to grow) this number seems reasonable: 40TB / 300,000 will result in partitions size of ~ 130MB (data should be evenly distributed). On Fri, Sep 7, 2018 at 6:28 PM Vadim

Re: Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

You have too many partitions, so when the driver is trying to gather the status of all map outputs and send back to executors it chokes on the size of the structure that needs to be GZipped, and since it's bigger than 2GiB, it produces OOM. On Fri, Sep 7, 2018 at 10:35 AM Harel Gliksman wrote: >

Driver OutOfMemoryError in MapOutputTracker$.serializeMapStatuses for 40 TB shuffle.

Hi, We are running a Spark (2.3.1) job on an EMR cluster with 500 r3.2xlarge (60 GB, 8 vcores, 160 GB SSD ). Driver memory is set to 25GB. It processes ~40 TB of data using aggregateByKey in which we specify numPartitions = 300,000. Map side tasks succeed, but reduce side tasks all fail. We

[Structured Streaming] HDFSBackedStateStoreProvider OutOfMemoryError

Hi All, I'm working on simple structured streaming query that uses flatMapGroupsWithState to maintain relatively a large size state. After running the application for few minutes on my local machine, it starts to slow down and then crashes with OutOfMemoryError. Tracking the code led me

Java heap space OutOfMemoryError in pyspark spark-submit (spark version:2.2)

Hi, I have a data set size of 10GB(example Test.txt). I wrote my pyspark script like below(Test.py): *from pyspark import SparkConf from pyspark.sql import SparkSession from pyspark.sql import SQLContext spark = SparkSession.builder.appName("FilterProduct").getOrCreate() sc = spark.sparkContext

OutOfMemoryError

y answers until now. Thanks! [0] https://gist.github.com/TwUxTLi51Nus/4accdb291494be9201abfad72541ce74 [1] http://stackoverflow.com/questions/43637913/apache-spark-outofmemoryerror-heapspace PS: As a workaround, I have been using "checkpoint" after every few iterations. --

OutOfMemoryError

t receive any answers until now. Thanks! [0] https://gist.github.com/TwUxTLi51Nus/4accdb291494be9201abfad72541ce74 [1] http://stackoverflow.com/questions/43637913/apache-spark-outofmemoryerror-heapspace PS: As a workaround, I have been writing and reading temporary parquet files on

Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

: user@spark.apache.org Subject: Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded Yea we also didn't find anything related to this online. Are you aware of any memory leaks in worker in 1.6.2 spark which might be causing this ? Do you know of any documentation which explai

Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

t; *From:* Behroz Sikander <behro...@gmail.com> > *Sent:* Friday, March 24, 2017 9:15 AM > *To:* Yong Zhang > *Cc:* user@spark.apache.org > *Subject:* Re: [Worker Crashing] OutOfMemoryError: GC overhead limit > execeeded > > Thank you for the response. > > Yes, I am su

Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

he.org Subject: Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded Thank you for the response. Yes, I am sure because the driver was working fine. Only 2 workers went down with OOM. Regards, Behroz On Fri, Mar 24, 2017 at 2:12 PM, Yong Zhang <java8...@hotmail.com<

Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

ns the > driver OOM. Are you sure your workers OOM? > > > Yong > > > -- > *From:* bsikander <behro...@gmail.com> > *Sent:* Friday, March 24, 2017 5:48 AM > *To:* user@spark.apache.org > *Subject:* [Worker Crashing] OutOfMemoryError: GC

Re: [Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

[Worker Crashing] OutOfMemoryError: GC overhead limit execeeded Spark version: 1.6.2 Hadoop: 2.6.0 Cluster: All VMS are deployed on AWS. 1 Master (t2.large) 1 Secondary Master (t2.large) 5 Workers (m4.xlarge) Zookeeper (t2.large) Recently, 2 of our workers went down with out of memory exception. java.lang.OutO

[Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

avoid that in future. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Worker-Crashing-OutOfMemoryError-GC-overhead-limit-execeeded-tp28535.html Sent from the Apache Spark User List mailing list archive at Nabble.com

[Worker Crashing] OutOfMemoryError: GC overhead limit execeeded

Hello, Spark version: 1.6.2 Hadoop: 2.6.0 Cluster: All VMS are deployed on AWS. 1 Master (t2.large) 1 Secondary Master (t2.large) 5 Workers (m4.xlarge) Zookeeper (t2.large) Recently, 2 of our workers went down with out of memory exception. > java.lang.OutOfMemoryError: GC overhead limit

OutOfMemoryError while running job...

I am trying to run a Spark job which reads from ElasticSearch and should write it's output back to a separate ElasticSearch index. Unfortunately I keep getting `java.lang.OutOfMemoryError: Java heap space` exceptions. I've tried running it with: --conf spark.memory.offHeap.enabled=true --conf

Re: Spark-xml - OutOfMemoryError: Requested array size exceeds VM limit

It seems a bit weird. Could we open an issue and talk in the repository link I sent? Let me try to reproduce your case with your data if possible. On 17 Nov 2016 2:26 a.m., "Arun Patel" wrote: > I tried below options. > > 1) Increase executor memory. Increased up to

Re: Spark-xml - OutOfMemoryError: Requested array size exceeds VM limit

I tried below options. 1) Increase executor memory. Increased up to maximum possibility 14GB. Same error. 2) Tried new version - spark-xml_2.10:0.4.1. Same error. 3) Tried with low level rowTags. It worked for lower level rowTag and returned 16000 rows. Are there any workarounds for this

Re: Spark-xml - OutOfMemoryError: Requested array size exceeds VM limit

Thanks for the quick response. Its a single XML file and I am using a top level rowTag. So, it creates only one row in a Dataframe with 5 columns. One of these columns will contain most of the data as StructType. Is there a limitation to store data in a cell of a Dataframe? I will check with

Re: Spark-xml - OutOfMemoryError: Requested array size exceeds VM limit

Hi Arun, I have few questions. Dose your XML file have like few huge documents? In this case of a row having a huge size like (like 500MB), it would consume a lot of memory becuase at least it should hold a row to iterate if I remember correctly. I remember this happened to me before while

Spark-xml - OutOfMemoryError: Requested array size exceeds VM limit

I am trying to read an XML file which is 1GB is size. I am getting an error 'java.lang.OutOfMemoryError: Requested array size exceeds VM limit' after reading 7 partitions in local mode. In Yarn mode, it throws 'java.lang.OutOfMemoryError: Java heap space' error after reading 3 partitions. Any

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

<yuzhih...@gmail.com> > *Sent:* Wednesday, August 3, 2016 6:10 PM > *To:* Rychnovsky, Dusan > *Cc:* user@spark.apache.org > *Subject:* Re: Managed memory leak detected + OutOfMemoryError: Unable to > acquire X bytes of memory, got 0 > > The latest QA run

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

OK, thank you. What do you suggest I do to get rid of the error? From: Ted Yu <yuzhih...@gmail.com> Sent: Wednesday, August 3, 2016 6:10 PM To: Rychnovsky, Dusan Cc: user@spark.apache.org Subject: Re: Managed memory leak detected + OutOfMemoryError:

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

> > > -- > *From:* Rychnovsky, Dusan > *Sent:* Wednesday, August 3, 2016 3:58 PM > *To:* Ted Yu > > *Cc:* user@spark.apache.org > *Subject:* Re: Managed memory leak detected + OutOfMemoryError: Unable to

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

2016 3:58 PM To: Ted Yu Cc: user@spark.apache.org Subject: Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0 Yes, I believe I'm using Spark 1.6.0. > spark-submit --

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

I have 1.6.0 and therefore should have it fixed, right? Or what do I do to fix it? Thanks, Dusan From: Ted Yu <yuzhih...@gmail.com> Sent: Wednesday, August 3, 2016 3:52 PM To: Rychnovsky, Dusan Cc: user@spark.apache.org Subject: Re: Managed memory leak detected

Re: Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

Are you using Spark 1.6+ ? See SPARK-11293 On Wed, Aug 3, 2016 at 5:03 AM, Rychnovsky, Dusan < dusan.rychnov...@firma.seznam.cz> wrote: > Hi, > > > I have a Spark workflow that when run on a relatively small portion of > data works fine, but when run on big data fails with strange errors. In

Managed memory leak detected + OutOfMemoryError: Unable to acquire X bytes of memory, got 0

Hi, I have a Spark workflow that when run on a relatively small portion of data works fine, but when run on big data fails with strange errors. In the log files of failed executors I found the following errors: Firstly > Managed memory leak detected; size = 263403077 bytes, TID = 6524 And

Re: OutOfMemoryError - When saving Word2Vec

any response. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemoryError-When-saving-Word2Vec-tp27142p27151.html > Sent from the Apache Spark User List m

Re: OutOfMemoryError - When saving Word2Vec

Is this the right forum to post Spark related issues ? I have tried this forum along with StackOverflow but not seeing any response. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemoryError-When-saving-Word2Vec-tp27142p27151.html Sent from

Re: OutOfMemoryError - When saving Word2Vec

Hi Sharad. The array size you (or the serializer) tries to allocate is just too big for the JVM. You can also split your input further by increasing parallelism. Following is good explanintion https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit regards, Vaquar khan

OutOfMemoryError - When saving Word2Vec

) at org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:131) at org.apache.spark.ml.feature.Word2VecModel.save(Word2Vec.scala:172) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemoryError-When-saving-Word2Vec-tp27142.html Sent from the Apache Spark User List mailing

Re: SqlContext parquet read OutOfMemoryError: Requested array size exceeds VM limit error

Thanks for the suggestions and links. The problem arises when I used DataFrame api to write but it works fine when doing insert overwrite in hive table. # Works good hive_context.sql("insert overwrite table {0} partiton (e_dt, c_dt) select * from temp_table".format(table_name)) # Doesn't work,

Re: SqlContext parquet read OutOfMemoryError: Requested array size exceeds VM limit error

If you are running on 64-bit JVM with less than 32G heap, you might want to enable -XX:+UseCompressedOops[1]. And if your dataframe is somehow generating more than 2^31-1 number of arrays, you might have to rethink your options. [1] https://spark.apache.org/docs/latest/tuning.html On Wed, May 4,

Re: SqlContext parquet read OutOfMemoryError: Requested array size exceeds VM limit error

Have you seen this thread ? http://search-hadoop.com/m/q3RTtyXr2N13hf9O=java+lang+OutOfMemoryError+Requested+array+size+exceeds+VM+limit On Wed, May 4, 2016 at 2:44 PM, Bijay Kumar Pathak <bkpat...@mtu.edu> wrote: > Hi, > > I am reading the parquet file around 50+ G which has

SqlContext parquet read OutOfMemoryError: Requested array size exceeds VM limit error

Hi, I am reading the parquet file around 50+ G which has 4013 partitions with 240 columns. Below is my configuration driver : 20G memory with 4 cores executors: 45 executors with 15G memory and 4 cores. I tried to read the data using both Dataframe read and using hive context to read the data

Re: PCA OutOfMemoryError

Hello Alex, Thanks for the response. There isn't much other data on the driver, so the issue is probably inherent to this particular PCA implementation. I'll try the alternative approach that you suggested instead. Thanks again. -Bharath On Wed, Jan 13, 2016 at 11:24 PM, Alex Gittens

Re: PCA OutOfMemoryError

The PCA.fit function calls the RowMatrix PCA routine, which attempts to construct the covariance matrix locally on the driver, and then computes the SVD of that to get the PCs. I'm not sure what's causing the memory error: RowMatrix.scala:124 is only using 3.5 GB of memory (n*(n+1)/2 with n=29604

Re: PCA OutOfMemoryError

Any suggestion/opinion? On 12-Jan-2016 2:06 pm, "Bharath Ravi Kumar" wrote: > We're running PCA (selecting 100 principal components) on a dataset that > has ~29K columns and is 70G in size stored in ~600 parts on HDFS. The > matrix in question is mostly sparse with tens of

PCA OutOfMemoryError

We're running PCA (selecting 100 principal components) on a dataset that has ~29K columns and is 70G in size stored in ~600 parts on HDFS. The matrix in question is mostly sparse with tens of columns populate in most rows, but a few rows with thousands of columns populated. We're running spark on

Lost tasks due to OutOfMemoryError (GC overhead limit exceeded)

ionRequired","true"); sparkConf.set("spark.kryoserializer.buffer.max.mb","512"); sparkConf.set("spark.default.parallelism","300"); sparkConf.set("spark.rpc.askTimeout","500"); I'm trying to load data from hdfs and running some sqls on it (m

Re: Lost tasks due to OutOfMemoryError (GC overhead limit exceeded)

k.rpc.askTimeout","500"); > > I'm trying to load data from hdfs and running some sqls on it (mostly > groupby) using DataFrames. The logs keep saying that tasks are lost due to > OutOfMemoryError (GC overhead limit exceeded). > > Can you advice what is the recommended settings (memory, cores, > partitions, etc.) for the given hardware? > > Thanks! >

Re: OutOfMemoryError When Reading Many json Files

Hi Why dont you check if you can just process the large file standalone and then do the outer loop next. sqlContext.read.json(jsonFile) .select($"some", $"fields") .withColumn( "new_col", some_transformations($"col")) .rdd.map( x: Row => (k, v) ) .combineByKey() Deenar On 14 October 2015 at

Re: OutOfMemoryError When Reading Many json Files

Yes it went wrong when processing a large file only. I removed transformations on DF, and it worked just fine. But doing a simple filter operation on the DF became the last straw that breaks the camel’s back. That’s confusing. ​ On Wed, Oct 14, 2015 at 2:11 PM Deenar Toraskar

OutOfMemoryError When Reading Many json Files

Hey Spark Users, I kept getting java.lang.OutOfMemoryError: Java heap space as I read a massive amount of json files, iteratively via read.json(). Even the result RDD is rather small, I still get the OOM Error. The brief structure of my program reads as following, in psuedo-code:

OutOfMemoryError OOM ByteArrayOutputStream.hugeCapacity

I have one job which fails if I enable KryoSerializer I use spark 1.5.0 on emr-4.1.0 Settings: spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.max 1024m spark.executor.memory47924M spark.yarn.executor.memoryOverhead 5324 The

Re: OutOfMemoryError

How to increase the Xmx of the workers ? *Thanks*, On Mon, Oct 5, 2015 at 3:48 PM, Ramkumar V wrote: > No. I didn't try to increase xmx. > > *Thanks*, > > > > On Mon, Oct 5, 2015 at

Re: OutOfMemoryError

You can add it in in conf/spark-defaults.conf # spark.executor.extraJavaOptions -XX:+PrintGCDetails FYI On Fri, Oct 9, 2015 at 3:07 AM, Ramkumar V wrote: > How to increase the Xmx of the workers ? > > *Thanks*, > > > > On

OutOfMemoryError

Hi, When i submit java spark job in cluster mode, i'm getting following exception. *LOG TRACE :* INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, -Djava.io.tmpdir={{PWD}}/tmp,

Re: OutOfMemoryError

Hi Ramkumar, did you try to increase Xmx of the workers ? Regards JB On 10/05/2015 08:56 AM, Ramkumar V wrote: Hi, When i submit java spark job in cluster mode, i'm getting following exception. *LOG TRACE :* INFO yarn.ExecutorRunnable: Setting up executor with commands:

Re: OutOfMemoryError

No. I didn't try to increase xmx. *Thanks*, On Mon, Oct 5, 2015 at 1:36 PM, Jean-Baptiste Onofré wrote: > Hi Ramkumar, > > did you try to increase Xmx of the workers ? > > Regards > JB > > On 10/05/2015 08:56 AM, Ramkumar V wrote: >

What happens to this RDD? OutOfMemoryError

Hi All, I'm using PySpark to create a corpus of labeled data points. I create an RDD called corpus, and then join to this RDD each newly-created feature RDD as I go. My code repeats something like this for each feature: feature = raw_data_rdd.map(...).reduceByKey(...).map(...) # create feature

Re: How to unit test HiveContext without OutOfMemoryError (using sbt)

Thanks for your response Yana, I can increase the MaxPermSize parameter and it will allow me to run the unit test a few more times before I run out of memory. However, the primary issue is that running the same unit test in the same JVM (multiple times) results in increased memory (each run of

Re: How to unit test HiveContext without OutOfMemoryError (using sbt)

I'd suggest setting sbt to fork when running tests. On Wed, Aug 26, 2015 at 10:51 AM, Mike Trienis mike.trie...@orcsol.com wrote: Thanks for your response Yana, I can increase the MaxPermSize parameter and it will allow me to run the unit test a few more times before I run out of memory.

How to unit test HiveContext without OutOfMemoryError (using sbt)

Hello, I am using sbt and created a unit test where I create a `HiveContext` and execute some query and then return. Each time I run the unit test the JVM will increase it's memory usage until I get the error: Internal error when running tests: java.lang.OutOfMemoryError: PermGen space Exception

Re: How to unit test HiveContext without OutOfMemoryError (using sbt)

The PermGen space error is controlled with MaxPermSize parameter. I run with this in my pom, I think copied pretty literally from Spark's own tests... I don't know what the sbt equivalent is but you should be able to pass it...possibly via SBT_OPTS? plugin

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

That looks like it's during recovery from a checkpoint, so it'd be driver memory not executor memory. How big is the checkpoint directory that you're trying to restore from? On Mon, Aug 10, 2015 at 10:57 AM, Dmitry Goldenberg dgoldenberg...@gmail.com wrote: We're getting the below error.

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

I wonder during recovery from a checkpoint whether we can estimate the size of the checkpoint and compare with Runtime.getRuntime().freeMemory(). If the size of checkpoint is much bigger than free memory, log warning, etc Cheers On Mon, Aug 10, 2015 at 9:34 AM, Dmitry Goldenberg

How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

We're getting the below error. Tried increasing spark.executor.memory e.g. from 1g to 2g but the below error still happens. Any recommendations? Something to do with specifying -Xmx in the submit job scripts? Thanks. Exception in thread main java.lang.OutOfMemoryError: GC overhead limit

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

Would there be a way to chunk up/batch up the contents of the checkpointing directories as they're being processed by Spark Streaming? Is it mandatory to load the whole thing in one go? On Mon, Aug 10, 2015 at 12:42 PM, Ted Yu yuzhih...@gmail.com wrote: I wonder during recovery from a

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

You need to keep a certain number of rdds around for checkpointing, based on e.g. the window size. Those would all need to be loaded at once. On Mon, Aug 10, 2015 at 11:49 AM, Dmitry Goldenberg dgoldenberg...@gmail.com wrote: Would there be a way to chunk up/batch up the contents of the

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

Looks like workaround is to reduce *window length.* *Cheers* On Mon, Aug 10, 2015 at 10:07 AM, Cody Koeninger c...@koeninger.org wrote: You need to keep a certain number of rdds around for checkpointing, based on e.g. the window size. Those would all need to be loaded at once. On Mon, Aug

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

You need to keep a certain number of rdds around for checkpointing -- that seems like a hefty expense to pay in order to achieve fault tolerance. Why does Spark persist whole RDD's of data? Shouldn't it be sufficient to just persist the offsets, to know where to resume from? Thanks. On Mon,

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

Well, RDDs also contain data, don't they? The question is, what can be so hefty in the checkpointing directory to cause Spark driver to run out of memory? It seems that it makes checkpointing expensive, in terms of I/O and memory consumption. Two network hops -- to driver, then to workers.

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

The rdd is indeed defined by mostly just the offsets / topic partitions. On Mon, Aug 10, 2015 at 3:24 PM, Dmitry Goldenberg dgoldenberg...@gmail.com wrote: You need to keep a certain number of rdds around for checkpointing -- that seems like a hefty expense to pay in order to achieve fault

Re: How to fix OutOfMemoryError: GC overhead limit exceeded when using Spark Streaming checkpointing

No, it's not like a given KafkaRDD object contains an array of messages that gets serialized with the object. Its compute method generates an iterator of messages as needed, by connecting to kafka. I don't know what was so hefty in your checkpoint directory, because you deleted it. My

Re: Spark - Timeout Issues - OutOfMemoryError

Hello Dean Others, Thanks for the response. I tried with 100,200, 400, 600 and 1200 repartitions with 100,200,400 and 800 executors. Each time all the tasks of join complete in less than a minute except one and that one tasks runs forever. I have a huge cluster at my disposal. The data for each

Re: Spark - Timeout Issues - OutOfMemoryError

IMHO If your data or your algorithm is prone to data skew, I think you have to fix this from application level, Spark itself cannot overcome this problem (if one key has large amount of values), you may change your algorithm to choose another shuffle key, somethings like this to avoid shuffle on

Re: Spark - Timeout Issues - OutOfMemoryError

Hello Shao, Can you talk more about shuffle key or point me to APIs that allow me to change shuffle key. I will try with different keys and see the performance. What is the shuffle key by default ? On Mon, May 4, 2015 at 2:37 PM, Saisai Shao sai.sai.s...@gmail.com wrote: IMHO If your data or

Re: Spark - Timeout Issues - OutOfMemoryError

Shuffle key is depending on your implementation, I'm not sure if you are familiar with MapReduce, the mapper output is a key-value pair, where the key is the shuffle key for shuffling, Spark is also the same. 2015-05-04 17:31 GMT+08:00 ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com: Hello Shao, Can you

Re: Spark - Timeout Issues - OutOfMemoryError

One dataset (RDD Pair) val lstgItem = listings.map { lstg = (lstg.getItemId().toLong, lstg) } Second Dataset (RDDPair) val viEvents = viEventsRaw.map { vi = (vi.get(14).asInstanceOf[Long], vi) } As i want to join based on item Id that is used as first element in the tuple in both cases and i

Re: Spark - Timeout Issues - OutOfMemoryError

Four tasks are now failing with IndexIDAttemptStatus ▾Locality LevelExecutor ID / HostLaunch TimeDurationGC TimeShuffle Read Size / RecordsShuffle Spill (Memory)Shuffle Spill (Disk) Errors 0 3771 0 FAILED PROCESS_LOCAL 114 / host1 2015/05/04 01:27:44 / ExecutorLostFailure (executor 114 lost)

Re: Spark - Timeout Issues - OutOfMemoryError

From the symptoms you mentioned that one task's shuffle write is much larger than all the other task, it is quite similar to normal data skew behavior, I just give some advice based on your descriptions, I think you need to detect whether data is actually skewed or not. The shuffle will put data

Re: Spark - Timeout Issues - OutOfMemoryError

I ran it against one file instead of 10 files and i see one task is still running after 33 mins its shuffle read size is 780MB/50 mil records. I did a count of records for each itemId from dataset-2 [One FILE] (Second Dataset (RDDPair) val viEvents = viEventsRaw.map { vi = (vi.get(14

Re: Spark - Timeout Issues - OutOfMemoryError

I tried this val viEventsWithListings: RDD[(Long, (DetailInputRecord, VISummary, Long))] = lstgItem.join(viEvents, new org.apache.spark.RangePartitioner(partitions = 1200, rdd = viEvents)).map { It fired two jobs and still i have 1 task that never completes. IndexIDAttemptStatusLocality

Re: Spark - Timeout Issues - OutOfMemoryError

Data Set 1 : viEvents : Is the event activity data of 1 day. I took 10 files out of it and 10 records *Item ID Count* 201335783004 3419 191568402102 1793 111654479898 1362 181503913062 1310 261798565828 1028 111654493548 994 231516683056 862

Re: Spark - Timeout Issues - OutOfMemoryError

I don't know the full context of what you're doing, but serialization errors usually mean you're attempting to serialize something that can't be serialized, like the SparkContext. Kryo won't help there. The arguments to spark-submit you posted previously look good: 2) --num-executors 96

Re: Spark - Timeout Issues - OutOfMemoryError

How big is the data you're returning to the driver with collectAsMap? You are probably running out of memory trying to copy too much data back to it. If you're trying to force a map-side join, Spark can do that for you in some cases within the regular DataFrame/RDD context. See

Re: Spark - Timeout Issues - OutOfMemoryError

IMHO, you are trying waaay to hard to optimize work on what is really a small data set. 25G, even 250G, is not that much data, especially if you've spent a month trying to get something to work that should be simple. All these errors are from optimization attempts. Kryo is great, but if it's not

Re: Spark - Timeout Issues - OutOfMemoryError

Hello Dean Others, Thanks for your suggestions. I have two data sets and all i want to do is a simple equi join. I have 10G limit and as my dataset_1 exceeded that it was throwing OOM error. Hence i switched back to use .join() API instead of map-side broadcast join. I am repartitioning the data

Re: Spark - Timeout Issues - OutOfMemoryError

Hello Deam, If I don;t use Kryo serializer i got Serialization error and hence am using it. If I don';t use partitionBy/reparition then the simply join never completed even after 7 hours and infact as next step i need to run it against 250G as that is my full dataset size. Someone here suggested

Re: Spark - Timeout Issues - OutOfMemoryError

You could try repartitioning your listings RDD, also doing a collectAsMap would basically bring all your data to driver, in that case you might want to set the storage level as Memory and disk not sure that will do any help on the driver though. Thanks Best Regards On Thu, Apr 30, 2015 at 11:10

Re: Spark - Timeout Issues - OutOfMemoryError

You could try increasing your heap space explicitly. like export _JAVA_OPTIONS=-Xmx10g, its not the correct approach but try. Thanks Best Regards On Tue, Apr 28, 2015 at 10:35 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote: I have a SparkApp that runs completes in 45 mins for 5 files (5*750MB

Re: Spark - Timeout Issues - OutOfMemoryError

Did not work. Same problem. On Thu, Apr 30, 2015 at 1:28 PM, Akhil Das ak...@sigmoidanalytics.com wrote: You could try increasing your heap space explicitly. like export _JAVA_OPTIONS=-Xmx10g, its not the correct approach but try. Thanks Best Regards On Tue, Apr 28, 2015 at 10:35 PM,

Re: Spark - Timeout Issues - OutOfMemoryError

Full Exception *15/04/30 09:59:49 INFO scheduler.DAGScheduler: Stage 1 (collectAsMap at VISummaryDataProvider.scala:37) failed in 884.087 s* *15/04/30 09:59:49 INFO scheduler.DAGScheduler: Job 0 failed: collectAsMap at VISummaryDataProvider.scala:37, took 1093.418249 s* 15/04/30 09:59:49 ERROR

Spark - Timeout Issues - OutOfMemoryError

I have a SparkApp that runs completes in 45 mins for 5 files (5*750MB size) and it takes 16 executors to do so. I wanted to run it against 10 files of each input type (10*3 files as there are three inputs that are transformed). [Input1 = 10*750 MB, Input2=10*2.5GB, Input3 = 10*1.5G], Hence i used

  1   2   >