Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21322 Introducing a new concept which is similar to broadcast seems like an overkill. We can just update broadcast, to allow it to be memory-only. However, there might be simpler solutions to fit your case, without broadcast. e.g. ``` val myObj = ... rdd.mapPartitions { it => try { // process data } finally { myObj.close() } } ``` I think we need to clear define the use case and think about whether we need a new API or not.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org