All,
Is a Function object in Spark reused on a given executor, or is sent and
deserialized with each new task?
On my project, we have functions that incur a very large setup cost, but then
could be called many times. Currently, I am using object deserialization to
run this intensive setup,
Unless its a broadcast variable, a new copy will be deserialized for every
task.
On Wed, Oct 14, 2015 at 10:18 AM, Starch, Michael D (398M) <
michael.d.sta...@jpl.nasa.gov> wrote:
> All,
>
> Is a Function object in Spark reused on a given executor, or is sent and
> deserialized with each new
