One last time to be sure I got it right, the executing sequence here goes like this?:
val usersMap = contacts.collectAsMap() #The contacts RDD is collected by the executors and sent to the driver, the executors delete the rdd contacts.map(v => (v._1, (usersMap(v._1), v._2))).collect() #The userMap object is sent again to the executors to run the code, and with the collect(), the result is sent again back to the driver 2015-02-26 11:57 GMT+01:00 Sean Owen <so...@cloudera.com>: > Yes, in that code, usersMap has been serialized to every executor. > I thought you were referring to accessing the copy in the driver. > > On Thu, Feb 26, 2015 at 10:47 AM, Guillermo Ortiz <konstt2...@gmail.com> > wrote: >> Isn't it "contacts.map(v => (v._1, (usersMap(v._1), v._2))).collect()" >> executed in the executors? why is it executed in the driver? >> contacts are not a local object, right? --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
