Maxim Gekk created SPARK-25286: ---------------------------------- Summary: Remove dangerous parmap Key: SPARK-25286 URL: https://issues.apache.org/jira/browse/SPARK-25286 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.0 Reporter: Maxim Gekk
One of parmap methods accepts an execution context created outside of parmap. If the parmap method is called recursively on a thread pool limited by size, it could lead to deadlocks. See the JIRA tickets: SPARK-25240 and SPARK-25283 . To eliminate the problems in the future, need to remove parmap() with the signature: {code:scala} def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]] (in: Col[I]) (f: I => O) (implicit cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For in.map cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence ec: ExecutionContext ): Col[O] {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org