[
https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965408#comment-15965408
]
Miguel Pérez commented on SPARK-20286:
--------------------------------------
Yes. I suppose this is only checked when an executors passes from active to
idle status (maybe when the last action has finished), and it's not changed
when unpersist is called. But this is problematic, for example, if the job is
interactive. Imagine you have a Zeppelin notebook or a Spark shell. The same
driver could have different purposes and the session could be opened a long
time. Once you have called a single persist, all these executors will have to
wait {{dynamicAllocation.cachedExecutorIdleTimeout}} (which is Infinity by
default) or the user executing another action (executors will become active
again and then idle, but this time without cached data).
I don't know the solution. Maybe the best approach is to change from
cachedExecutorIdleTimeout to executorIdleTimeout on all cached executors when
the last RDD has been unpersisted, and then restart the time counter (unpersist
will then count as an action).
> dynamicAllocation.executorIdleTimeout is ignored after unpersist
> ----------------------------------------------------------------
>
> Key: SPARK-20286
> URL: https://issues.apache.org/jira/browse/SPARK-20286
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.1
> Reporter: Miguel Pérez
>
> With dynamic allocation enabled, it seems that executors with cached data
> which are unpersisted are still being killed using the
> {{dynamicAllocation.cachedExecutorIdleTimeout}} configuration, instead of
> {{dynamicAllocation.executorIdleTimeout}}. Assuming the default configuration
> ({{dynamicAllocation.cachedExecutorIdleTimeout = Infinity}}), an executor
> with unpersisted data won't be released until the job ends.
> *How to reproduce*
> - Set different values for {{dynamicAllocation.executorIdleTimeout}} and
> {{dynamicAllocation.cachedExecutorIdleTimeout}}
> - Load a file into a RDD and persist it
> - Execute an action on the RDD (like a count) so some executors are activated.
> - When the action has finished, unpersist the RDD
> - The application UI removes correctly the persisted data from the *Storage*
> tab, but if you look in the *Executors* tab, you will find that the executors
> remain *active* until ({{dynamicAllocation.cachedExecutorIdleTimeout}} is
> reached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
