Maxim Gekk created SPARK-25286:
----------------------------------
Summary: Remove dangerous parmap
Key: SPARK-25286
URL: https://issues.apache.org/jira/browse/SPARK-25286
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Maxim Gekk
One of parmap methods accepts an execution context created outside of parmap.
If the parmap method is called recursively on a thread pool limited by size, it
could lead to deadlocks. See the JIRA tickets: SPARK-25240 and SPARK-25283 . To
eliminate the problems in the future, need to remove parmap() with the
signature:
{code:scala}
def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]]
(in: Col[I])
(f: I => O)
(implicit
cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For in.map
cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence
ec: ExecutionContext
): Col[O]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
