:40 AM
To: user@spark.apache.org
Subject: Re: The function of ClosureCleaner.clean
I am not sure specifically about specific purpose of this function but Spark
needs to remove elements from the closure that may be included by default but
not really needed so as to serialize it & send it to execu
I am not sure specifically about specific purpose of this function but
Spark needs to remove elements from the closure that may be included by
default but not really needed so as to serialize it & send it to executors
to operate on RDD. For example a function in Map function of RDD may
reference o
Hi, All
Before sc.runJob invokes dagScheduler.runJob, the func performed
on the rdd is "cleaned" by ClosureCleaner.clearn.
Why spark has to do this? What's the purpose?
