[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456115333 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -55,6 +58,24 @@ private[spark] class IndexShuffleBlockRe

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456115190 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -55,6 +58,24 @@ private[spark] class IndexShuffleBlockRe

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456063110 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456062915 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +662,23 @@ private[spark] class BlockManager( blockId: B

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456062494 ## File path: core/src/main/scala/org/apache/spark/storage/BlockId.scala ## @@ -38,7 +38,10 @@ sealed abstract class BlockId { // convenience methods

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456061843 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r456062326 ## File path: core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala ## @@ -168,7 +168,10 @@ private[spark] class NettyBlockT

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455369270 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -55,6 +58,24 @@ private[spark] class IndexShuffleBlockRe

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455367802 ## File path: core/src/main/scala/org/apache/spark/shuffle/MigratableResolver.scala ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455368622 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -44,9 +47,9 @@ import org.apache.spark.util.Utils // o

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455337945 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,29 @@ package object config { .booleanConf

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455188725 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ## @@ -1866,13 +1903,57 @@ class BlockManagerSuite extends SparkFunSuit

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455184586 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Ap

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455184084 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala ## @@ -0,0 +1,230 @@ +/* + * Licensed to the Ap

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r455181019 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1285,6 +1314,9 @@ private[spark] class BlockManager( require(b

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r454058089 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -242,8 +244,7 @@ private[spark] class BlockManager( private var b

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r447249716 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -242,8 +244,7 @@ private[spark] class BlockManager( private var b

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r447247802 ## File path: core/src/main/scala/org/apache/spark/storage/BlockId.scala ## @@ -40,6 +40,9 @@ sealed abstract class BlockId { def isRDD: Boolean = isIn

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r447247346 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -148,6 +170,82 @@ private[spark] class IndexShuffleBlock

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r447247000 ## File path: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ## @@ -55,6 +58,25 @@ private[spark] class IndexShuffleBlockRe

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r447190973 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r446475360 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -57,7 +57,7 @@ import org.apache.spark.resource._ import org.apache.spark.res

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r445108940 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -157,7 +158,8 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r445108498 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -489,6 +491,24 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r445073751 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r53446 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r442604350 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -775,7 +802,12 @@ private[spark] class MapOutputTrackerMaster( override

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r441027429 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440443890 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440442024 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440407328 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,21 @@ package object config { .booleanConf

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440405708 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -775,7 +802,12 @@ private[spark] class MapOutputTrackerMaster( override

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440378850 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -121,12 +121,28 @@ private class ShuffleStatus(numPartitions: Int) {

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440378160 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -334,13 +336,14 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440377849 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ## @@ -14,7 +14,6 @@ * See the License for the specific language gov

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440377043 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440377415 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440376585 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440375923 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +658,23 @@ private[spark] class BlockManager( blockId: B

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440375111 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440374116 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -1725,6 +1725,17 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440373566 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440372693 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440372013 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440372154 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440370903 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,21 @@ package object config { .booleanConf

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r440370509 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -775,7 +802,12 @@ private[spark] class MapOutputTrackerMaster( override

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439700057 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439386843 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439384531 ## File path: core/src/main/scala/org/apache/spark/shuffle/MigratableResolver.scala ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r438436447 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionUnitSuite.scala ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Sof

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r438269803 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissionManager.scala ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r438269652 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionUnitSuite.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Sof

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r436071464 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1813,107 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575853 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575539 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435575094 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r43809 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -1790,6 +1822,108 @@ private[spark] class BlockManager( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435447040 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -1725,6 +1725,16 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435446889 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -479,6 +497,16 @@ private[spark] class MapOutputTrackerMaster( } }

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435425845 ## File path: core/src/main/scala/org/apache/spark/storage/RDDBlockSavedOnDecommissionedBlockManagerException.scala ## @@ -0,0 +1,21 @@ +/* + * Licensed

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r435425022 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +657,19 @@ private[spark] class BlockManager( blockId: B

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r434896155 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +657,19 @@ private[spark] class BlockManager( blockId: B

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r434895820 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +657,19 @@ private[spark] class BlockManager( blockId: B