[
https://issues.apache.org/jira/browse/SPARK-36966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam updated SPARK-36966:
------------------------
Description:
In the past Spark (pre 1.6) jobs would give OOM if an RDD could not fit into
memory (when trying to cache with MEMORY_ONLY). These days it seems Spark jobs
will evict partitions from the cache and recompute them from scratch.
We have some jobs that cache an RDD then traverse it 300 times. In order to
know when we need to increase the memory on our cluster, we need to know when
it has run out of memory.
The new behaviour of Spark makes this difficult ... rather than the job OOMing
(like in the past), the job instead just takes forever (our surrounding logic
eventually times out the job). Diagnosing why the job failed becomes difficult
because it's not immediately obvious from the logs that the job has run out of
memory (since no OOM is thrown). One can find "evicted" log lines.
As a bit of a hack, we are using accumulators with mapPartitionsWithIndex and
updating a count of the number of times each partition is traversed, then we
forcibly blow up the job when this count is 2 or more (indicating an eviction).
This hack doesn't work very well as it seems to give false positives (RCA not
yet understood, it doesn't seem to be speculative execution, nor are tasks
failing (`egrep "task .* in stage [0-9]+\.1" | wc -l` gives 0).
*Question 1*: Is there a way to disable this new behaviour of Spark and make it
behave like it used to (i.e. just blow up with OOM) - I've looked in the Spark
configuration and cannot find anything like "disable-eviction".
*Question 2*: Is there any method on the SparkSession or SparkContext that we
can call to easily detect when eviction is happening?
If not, then for our use case this is an effective regression - we need a way
to make Spark behave predictably, or at least a way to determine automatically
when Spark is running slowly due to lack of memory.
FOR CONTEXT
> Borrowed storage memory may be evicted when memory pressure arises
>From
>https://issues.apache.org/jira/secure/attachment/12765646/unified-memory-management-spark-10000.pdf,
> which is attached to https://issues.apache.org/jira/browse/SPARK-10000 which
>was implemented in 1.6
>https://spark.apache.org/releases/spark-release-1-6-0.html
was:
In the past Spark jobs would give OOM if an RDD could not fit into memory (when
trying to cache with MEMORY_ONLY). These days it seems Spark jobs will evict
partitions from the cache and recompute them from scratch.
We have some jobs that cache an RDD then traverse it 300 times. In order to
know when we need to increase the memory on our cluster, we need to know when
it has run out of memory.
The new behaviour of Spark makes this difficult ... rather than the job OOMing
(like in the past), the job instead just takes forever (our surrounding logic
eventually times out the job). Diagnosing why the job failed becomes difficult
because it's not immediately obvious from the logs that the job has run out of
memory (since no OOM is thrown). One can find "evicted" log lines.
As a bit of a hack, we are using accumulators with mapPartitionsWithIndex and
updating a count of the number of times each partition is traversed, then we
forcably blow up the job when this count is 2 or more (indicating an
evicition). This hack doesn't work very well as it seems to give false
positives (RCA not yet understood, it doesn't seem to be speculative execution,
nor are tasks failing (`egrep "task .* in stage [0-9]+\.1" | wc -l` gives 0).
*Question 1*: Is there a way to disable this new behaviour of Spark and make it
behave like it used to (i.e. just blow up with OOM) - I've looked in the Spark
configuration and cannot find anything like "disable-eviction".
*Question 2*: Is there any method on the SparkSession or SparkContext that we
can call to easily detect when eviction is happening?
If not, then for our use case this is an effective regression - we need a way
to make Spark behave predictably, or at least a way to determine automatically
when Spark is running slowly due to lack of memory.
FOR CONTEXT
> Borrowed storage memory may be evicted when memory pressure arises
>From
>https://issues.apache.org/jira/secure/attachment/12765646/unified-memory-management-spark-10000.pdf,
> which is attached to https://issues.apache.org/jira/browse/SPARK-10000 which
>was implemented in 1.6
>https://spark.apache.org/releases/spark-release-1-6-0.html
> Spark evicts RDD partitions instead of allowing OOM
> ---------------------------------------------------
>
> Key: SPARK-36966
> URL: https://issues.apache.org/jira/browse/SPARK-36966
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Reporter: sam
> Priority: Major
>
> In the past Spark (pre 1.6) jobs would give OOM if an RDD could not fit into
> memory (when trying to cache with MEMORY_ONLY). These days it seems Spark
> jobs will evict partitions from the cache and recompute them from scratch.
> We have some jobs that cache an RDD then traverse it 300 times. In order to
> know when we need to increase the memory on our cluster, we need to know when
> it has run out of memory.
> The new behaviour of Spark makes this difficult ... rather than the job
> OOMing (like in the past), the job instead just takes forever (our
> surrounding logic eventually times out the job). Diagnosing why the job
> failed becomes difficult because it's not immediately obvious from the logs
> that the job has run out of memory (since no OOM is thrown). One can find
> "evicted" log lines.
> As a bit of a hack, we are using accumulators with mapPartitionsWithIndex and
> updating a count of the number of times each partition is traversed, then we
> forcibly blow up the job when this count is 2 or more (indicating an
> eviction). This hack doesn't work very well as it seems to give false
> positives (RCA not yet understood, it doesn't seem to be speculative
> execution, nor are tasks failing (`egrep "task .* in stage [0-9]+\.1" | wc
> -l` gives 0).
> *Question 1*: Is there a way to disable this new behaviour of Spark and make
> it behave like it used to (i.e. just blow up with OOM) - I've looked in the
> Spark configuration and cannot find anything like "disable-eviction".
> *Question 2*: Is there any method on the SparkSession or SparkContext that we
> can call to easily detect when eviction is happening?
> If not, then for our use case this is an effective regression - we need a way
> to make Spark behave predictably, or at least a way to determine
> automatically when Spark is running slowly due to lack of memory.
> FOR CONTEXT
> > Borrowed storage memory may be evicted when memory pressure arises
> From
> https://issues.apache.org/jira/secure/attachment/12765646/unified-memory-management-spark-10000.pdf,
> which is attached to https://issues.apache.org/jira/browse/SPARK-10000 which
> was implemented in 1.6
> https://spark.apache.org/releases/spark-release-1-6-0.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
