Hi All, I am a newbie to spark and want to know if there is any performance difference between map vs mapPartitions if I am doing strictly a per item transformation?
For e.g. reversedWords = words.map(w => w.reverse()); vs. reversedWords = words.mapPartitions(pwordsIterator => { List<String> pWordList = new ArrayList(); while(pwordsIterator.hasNext()) { pWordList.add(pwordsIterator.next().reverse()); } return pWordList } Is the instantiation of "pWordList" for mapPartitions justified for any performance gain? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/map-vs-mapPartitions-tp27490.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org