agrawaldevesh commented on a change in pull request #29211: URL: https://github.com/apache/spark/pull/29211#discussion_r465266535
########## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala ########## @@ -266,18 +266,17 @@ class BlockManagerDecommissionIntegrationSuite extends SparkFunSuite with LocalS val execIdToBlocksMapping = storageStatus.map( status => (status.blockManagerId.executorId, status.blocks)).toMap // No cached blocks should be present on executor which was decommissioned - assert(execIdToBlocksMapping(execToDecommission).keys.filter(_.isRDD).toSeq === Seq(), + assert( + !execIdToBlocksMapping.contains(execToDecommission) || + execIdToBlocksMapping(execToDecommission).keys.filter(_.isRDD).toSeq === Seq(), "Cache blocks should be migrated") if (persist) { // There should still be all the RDD blocks cached assert(execIdToBlocksMapping.values.flatMap(_.keys).count(_.isRDD) === numParts) } - // Make the executor we decommissioned exit - sched.client.killExecutors(List(execToDecommission)) - - // Wait for the executor to be removed - executorRemovedSem.acquire(1) + // Wait for the executor to be removed automatically after migration. + assert(executorRemovedSem.tryAcquire(1, 5L, TimeUnit.MINUTES)) Review comment: I get that it can take up to 5 minutes, but I would like to understand better whether it should take 5 minutes ? The executor is supposed to exit as soon as the migration is over, right ? Or if there is some polling there -- that polling should ideally be configurable. So the only thing we are really waiting for here is the migration : That migration shouldn't take 5 minutes -- we don't have that much data to migrate. Perhaps you can greatly reduce the executor heartbeat interval such that this can be sped up ? (I use this trick in the other block manager decommissioning test) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org