[ https://issues.apache.org/jira/browse/SPARK-36966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sam updated SPARK-36966: ------------------------ Affects Version/s: 1.6.0 > Spark evicts RDD partitions instead of allowing OOM > --------------------------------------------------- > > Key: SPARK-36966 > URL: https://issues.apache.org/jira/browse/SPARK-36966 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.0, 2.4.4 > Reporter: sam > Priority: Major > > In the past Spark (pre 1.6) jobs would give OOM if an RDD could not fit into > memory (when trying to cache with MEMORY_ONLY). These days it seems Spark > jobs will evict partitions from the cache and recompute them from scratch. > We have some jobs that cache an RDD then traverse it 300 times. In order to > know when we need to increase the memory on our cluster, we need to know when > it has run out of memory. > The new behaviour of Spark makes this difficult ... rather than the job > OOMing (like in the past), the job instead just takes forever (our > surrounding logic eventually times out the job). Diagnosing why the job > failed becomes difficult because it's not immediately obvious from the logs > that the job has run out of memory (since no OOM is thrown). One can find > "evicted" log lines. > As a bit of a hack, we are using accumulators with mapPartitionsWithIndex and > updating a count of the number of times each partition is traversed, then we > forcibly blow up the job when this count is 2 or more (indicating an > eviction). This hack doesn't work very well as it seems to give false > positives (RCA not yet understood, it doesn't seem to be speculative > execution, nor are tasks failing (`egrep "task .* in stage [0-9]+\.1" | wc > -l` gives 0). > *Question 1*: Is there a way to disable this new behaviour of Spark and make > it behave like it used to (i.e. just blow up with OOM) - I've looked in the > Spark configuration and cannot find anything like "disable-eviction". > *Question 2*: Is there any method on the SparkSession or SparkContext that we > can call to easily detect when eviction is happening? > If not, then for our use case this is an effective regression - we need a way > to make Spark behave predictably, or at least a way to determine > automatically when Spark is running slowly due to lack of memory. > FOR CONTEXT > > Borrowed storage memory may be evicted when memory pressure arises > From > https://issues.apache.org/jira/secure/attachment/12765646/unified-memory-management-spark-10000.pdf, > which is attached to https://issues.apache.org/jira/browse/SPARK-10000 which > was implemented in 1.6 > https://spark.apache.org/releases/spark-release-1-6-0.html -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org