Thanks rxin, I still have a doubt about collect(). Word's number before reduceByKey() is about 200 million, and after reduceByKey() it decreases to 18 million. Memory for driver is initialized 15GB, then I print out "runtime.freeMemory()" before reduceByKey(), it indicates 13GB free memory. I think the memeory is enough for 18 million words. In additional, it's okay to broadcast 2GB to other worders, but it's difficult to collect 18 million words from worker to driver. Thanks a lot!
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/something-about-rdd-collect-tp16451p16458.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
