[
https://issues.apache.org/jira/browse/SPARK-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039953#comment-14039953
]
Patrick Wendell edited comment on SPARK-1392 at 6/21/14 9:15 PM:
-----------------------------------------------------------------
Okay great, I confirmed this is fixed by SPARK-1777. I tested as follows:
{code}
SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean
assembly/assembly
sc.textFile("/tmp/wiki_links").cache.count
{code}
The wiki_links file was download and extracted from here:
https://drive.google.com/file/d/0BwrkCxCycBCyTmlWYXp0MmdEakk/edit?usp=sharing
This worked with the proposed patch but failed with the default build.
was (Author: pwendell):
Okay great, I confirmed this is fixed by SPARK-1777. I tested as follows:
{code}
SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean
assembly/assembly
sc.textFile("/tmp/wiki_links").cache.count
{code}
The wiki_links file was download and extracted from here:
This worked with the proposed patch but failed with the default build.
> Local spark-shell Runs Out of Memory With Default Settings
> ----------------------------------------------------------
>
> Key: SPARK-1392
> URL: https://issues.apache.org/jira/browse/SPARK-1392
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 0.9.0
> Environment: OS X 10.9.2, Java 1.7.0_51, Scala 2.10.3
> Reporter: Pat McDonough
>
> Using the spark-0.9.0 Hadoop2 binary from the project download page, running
> the spark-shell locally in out of the box configuration, and attempting to
> cache all the attached data, spark OOMs with: java.lang.OutOfMemoryError: GC
> overhead limit exceeded
> You can work around the issue by either decreasing
> spark.storage.memoryFraction or increasing SPARK_MEM
--
This message was sent by Atlassian JIRA
(v6.2#6252)
