Hello all,
I am writing a program which calls from a database. A run a couple
computations, but in the end I have a while loop, in which I make a
modification to the persisted thata. eg:
val data = PairRDD... persist()
var i = 0
while (i < 10) {
val data_mod = data.map(_._1 + 1, _._2)
val data_joined = data.join(data_mod)
... do stuff with data_joined
}
Sadly, the result causes that the shuffle inside the WHILE loop is causing a
jdbc call and that is very slow. It is not finding the data locally
How can I help myself?
Saif
