Hi All,
I am a newbie to spark and want to know if there is any performance
difference between map vs mapPartitions if I am doing strictly a per item
transformation?
For e.g.
reversedWords = words.map(w => w.reverse());
vs.
reversedWords = words.mapPartitions(pwordsIterator => {
List<String> pWordList = new ArrayList();
while(pwordsIterator.hasNext()) {
pWordList.add(pwordsIterator.next().reverse());
}
return pWordList
}
Is the instantiation of "pWordList" for mapPartitions justified for any
performance gain?
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/map-vs-mapPartitions-tp27490.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
