[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...
Github user 10110346 closed the pull request at: https://github.com/apache/spark/pull/21079 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/21079#discussion_r181942230 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -113,3 +118,24 @@ private[spark] class ShuffleMapTask( override def toString: String = "ShuffleMapTask(%d, %d)".format(stageId, partitionId) } + +object ShuffleMapTask extends Logging { + private val cache = CacheBuilder.newBuilder() --- End diff -- we don't need to clear, this cache can recovery itself --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/21079#discussion_r181938225 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -113,3 +118,24 @@ private[spark] class ShuffleMapTask( override def toString: String = "ShuffleMapTask(%d, %d)".format(stageId, partitionId) } + +object ShuffleMapTask extends Logging { + private val cache = CacheBuilder.newBuilder() --- End diff -- Do we need to clear this `cache` at the end of a app ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/21079 [SPARK-23992][CORE] ShuffleDependency does not need to be deserialized every time ## What changes were proposed in this pull request? In the same stage, 'ShuffleDependency' is not necessary to be deserialized each time. ## How was this patch tested? Existing unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/10110346/spark notdeserializedep Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21079.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21079 commit 2239e786cf0b631457ce6c6acfa173e21e314e61 Author: liuxianDate: 2018-04-16T11:48:09Z fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org