Usage of Explicit Future in Spark program

[Chetan Khatri]

Hello Spark Users,

Someone has suggested by breaking 5-5 unpredictable transformation blocks
to Future[ONE STRING ARGUMENT] and claim this can tune the performance. I
am wondering this is a use of explicit Future! in Spark?

Sample code is below:

 def writeData( tableName: String): Future[String] =  Future {

// some heavy lifting Spark transformations, 5-6 read->transform->load.

}

writeDataFutures += writeData("dynamicFieldData")

 writeDataFutures foreach { writeDataFuture =>
      Await.ready(writeDataFuture, Duration.Inf).onComplete {
        case Success(table) => logger.info(s"All Success")
        case Failure(e) => e.printStackTrace()
      }
    }


Please suggest technical doubt.

Thanks