Well if the system doesn't change, then the data must be different. The
exact exception probably won't be helpful since it only tells us the last
allocation that failed. My guess is that your ingestion changed and there
is either now slightly more data than previously or it's skewed
differently.
please help.
Thanks
Amit
On Mon, Nov 9, 2020 at 4:18 PM Amit Sharma wrote:
> Please find below the exact exception
>
> Exception in thread "streaming-job-executor-3" java.lang.OutOfMemoryError:
> Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
>
Please find below the exact exception
Exception in thread "streaming-job-executor-3" java.lang.OutOfMemoryError:
Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
Can you please help.
Thanks
Amit
On Sun, Nov 8, 2020 at 1:35 PM Amit Sharma wrote:
> Hi , I am using 16 nodes spark cluster with below config
> 1. Executor memory 8 GB
> 2. 5 cores per executor
> 3. Driver memory 12 GB.
>
>
> We have streaming job. We do not see problem but sometimes we get
Any idea about this?
From: Kürşat Kurt [mailto:kur...@kursatkurt.com]
Sent: Sunday, October 30, 2016 7:59 AM
To: 'Jörn Franke' <jornfra...@gmail.com>
Cc: 'user@spark.apache.org' <user@spark.apache.org>
Subject: RE: Out Of Memory issue
Hi Jörn;
I am reading 300.000 l
Hi Muthu, this could be related to a known issue in the release notes
http://spark.apache.org/releases/spark-release-1-6-0.html
Known issues
SPARK-12546 - Save DataFrame/table as Parquet with dynamic partitions may
cause OOM; this can be worked around by decreasing the memory used by both
Thanks Ewan Leith. This seems like a good start, as it seem to match up to
the symptoms I am seeing :).
But, how do I specify "parquet.memory.pool.ratio"?
Parquet code seem to take this parameter from
ParquetOutputFormat.getRecordWriter()
(ref code: float
